About the role
Responsibilities:
- Design, develop, and optimise scalable ETL/ELT pipelines and data workflows using AWS services.
- Build and maintain enterprise data warehouses and data lakes on AWS (Redshift, S3 Glue).
- Develop batch and real-time data ingestion pipelines from APIs, Oracle databases, files, and other data sources.
- Design efficient data models (OLTP/OLAP, Star/Snowflake) and optimise query performance.
- Develop production-grade Python, PySpark, and advanced SQL for large-scale data processing.
- Implement workflow orchestration using Apache Airflow or AWS Step Functions.
- Ensure data quality, governance, security, lineage, and compliance across data platforms.
- Optimise cloud infrastructure for performance, scalability, reliability, and cost efficiency.
- Lead technical design discussions, mentor junior engineers, and drive engineering best practices.
- Support CI/CD, Infrastructure as Code (Terraform/CloudFormation), monitoring, and production deployments.
Requirements:
- Experience: 7-10 years in data engineering.
- Cloud: AWS (S3 Glue, Athena, Lambda, Step Functions, EMR/Glue Spark, Redshift, IAM, CloudWatch, CloudFormation/Terraform).
- Database: Oracle Database (PL/SQL, performance tuning, partitioning, and indexing).
- Programming: Python (PySpark, Pandas), advanced SQL.
- Orchestration: Apache Airflow / AWS MWAA, Step Functions.
- Data Engineering: ETL/ELT, Data Warehousing, Data Lakes, Data Modelling (Star/Snowflake), and Schema Design.
- Storage Formats: Parquet, Iceberg/Delta (preferred).
- Version Control and DevOps: Git, CI/CD.
- Security: IAM, encryption, and secrets manager.
Preferred:
- Experience with Spark (AWS Glue/EMR) for distributed data processing.
- Strong expertise in SQL optimisation, data quality frameworks, and pipeline monitoring.
- Experience leading technical initiatives, code reviews, and mentoring engineering teams.
- Excellent problem-solving and stakeholder communication skills.