About the role
We're seeking a Data Engineer with 3-4 years of experience to join our growing tech team. The ideal candidate will have hands-on experience in building and managing scalable data systemson AWS, with strong expertise in data pipelines, real-time streaming, and modern data frameworks in building and managing scalable data systems on AWS using Spark and modern data frameworks.
Responsibilities:
- Design, build, and maintain end-to-end data pipelines for ingestion, transformation, and delivery of high-volume data.
- Develop Spark-based ETL/ELT workflows for both batch and real-time streaming data.
- Core Skills: Spark, AWS (S3 Glue, Redshift, Kinesis), Kafka, Airflow, Python, SQL.
- Integrate data from multiple internal and external systems using Kafka, Kinesis, or other streaming frameworks.
- Build and manage data models, warehouses, and lakehouses using AWS services such as S3 Glue, Redshift, Athena, etc.
- Implement data quality checks, validation rules, and monitoring to ensure reliability and consistency.
- Collaborate with data analysts and scientists to provide clean, structured datasets optimised for analytics and ML.
- Work with orchestration tools (Airflow, MWAA, Step Functions, etc. ) for automated workflow scheduling.
- Continuously optimise data pipelines for cost, scalability, and performance.
Requirements:
- Strong programming skills in Python for data manipulation and automation.
- Hands-on expertise in Apache Spark (PySpark or Spark SQL) for large-scale data processing.
- Deep understanding of the AWS data ecosystem: S3 Glue, Lambda, Redshift, Athena, EMR, Kinesis, IAM.
- Experience with real-time streaming platforms such as Kafka, Kinesis, or Flink.
- Strong command of SQL and data modelling (star schema, dimensional modelling, partitioning).
- Proficiency with data orchestration and workflow management tools (Airflow, Step Functions, etc. ).
- Familiarity with Git, CI/CD, and modern development best practices.
- Experience working in Linux/Unix environments and handling large datasets efficiently.
Good to Have:
- Exposure to data lakehouse technologies (Delta Lake, Iceberg, Hudi).
- Understanding of data governance, cataloguing, and lineage tools (Glue Data Catalogue, Amundsen, DataHub).
- Familiarity with containerization and deployment (Docker, ECS, EKS).
- Basic understanding of AI/ML.
- Bachelor's degree in Computer Science, IT, Engineering, or related technical field.
- 3-4 years of experience in data engineering, big data, or analytics infrastructure.