About the role
Responsibilities:
- Design and build scalable data pipelines and ETL/ELT workflows.
- Develop data solutions using Python, SQL and PySpark/Spark.
- Design and implement AWS-based data architectures.
- Work with large-scale data across Data Lakes, Data Warehouses and Lakehouse platforms.
- Build and optimise batch and real-time data pipelines.
- Implement data quality, monitoring and pipeline reliability practices.
- Work with AI/ML teams to build data platforms supporting AI and analytics use cases.
- Contribute to technical architecture and mentor junior engineers.
- Collaborate with global clients and cross-functional teams.
Requirements:
- 8-12 years of experience in Data Engineering / Big Data Engineering.
- Strong AWS experience - Mandatory.
- Strong Cloud experience - Mandatory.
- Strong hands-on Python and SQL.
- Experience with PySpark / Spark.
- Strong ETL/ELT and data pipeline development experience.
- Experience with AWS services such as S3 Glue, EMR, Redshift, Athena, Lambda, Kinesis.
- Experience with Snowflake / Databricks / Redshift or similar data platforms.
- Strong understanding of data modelling and distributed systems.
- Experience with CI/CD and DevOps practices.
- Strong communication skills and ability to work with global stakeholders.
- Comfortable working in the 4:00 PM - 1:00 AM IST shift.
- Good to have: Palantir Foundry, Kafka, Airflow, Docker/Kubernetes, Terraform, MLOps/AI data pipelines.