About the role
Required Skills
· Design and develop ETL pipelines using PySpark, AWS EMR, and AWS Glue for large-scale data processing.
· Design and develop Lambda triggers
· Develop data ingestion pipelines
· Optimize Spark jobs for performance, scalability, and cost efficiency in cloud environments.
· Manage data transformations, cleansing, and enrichment to ensure high data quality.
· Monitor, troubleshoot, and enhance ETL workflows for reliability and efficiency.
Must Have
Hands-on experience in Python, Pyspark, EMR (or Glue or Lambda), IAM, ECS, S3
Good To Have
Athena, Redshift, Cloud formation
Qualifications: BACHELOR OF ENGINEERING