About the role
Responsibilities:
- Design, develop, and maintain ETL pipelines for data ingestion, transformation, and processing.
- Work with PySpark and Python to process large datasets efficiently.
- Optimise Redshift queries and manage data warehouse performance.
- Ensure data quality and integrity in the pipeline.
- Collaborate with data scientists, analysts, and software engineers to provide clean and reliable data.
- Automate workflows and monitor job execution for failures.
- Implement best practices in data engineering, performance tuning, and cost optimisation.
Requirements:
- 2+ years of experience in ETL pipeline development using Python and SQL.
- Hands-on experience with PySpark for big data processing.
- Strong knowledge of AWS Redshift, including query optimisation and performance tuning.
- Proficiency in SQL for complex queries, transformations, and aggregations.
- Good understanding of data modelling and best practices in data architecture.
- Strong problem-solving and debugging skills.
- Experience with workflow orchestration tools like Airflow is a plus.