Python, Pyspark, AWS

Location: Hyderabad, India

Experience: 5–8 years

Employment Type: Full-time / Permanent

We are looking for a skilled Senior Data Engineer with strong experience in Python, PySpark, and AWS Cloud Services. The ideal candidate will design, build, and optimize large-scale data processing pipelines and ETL workflows, enabling reliable and high-performance data analytics solutions on the cloud.

• Design, develop, and maintain data pipelines using PySpark and Python.
• Build ETL processes to extract, transform, and load data from various structured and unstructured sources.
• Work with large-scale distributed data processing systems using Apache Spark.
• Develop and deploy scalable data solutions on AWS (EMR, Glue, S3, Lambda, Redshift, Athena, etc.).
• Implement data quality, validation, and governance frameworks to ensure reliable data delivery.
• Collaborate with data scientists, analysts, and other engineering teams to design end-to-end data flows.
• Optimize data workflows for performance and cost efficiency in a cloud environment.
• Maintain and monitor data infrastructure, resolving performance bottlenecks and production issues.

• 5+ years of experience in data engineering or backend development.
• Strong proficiency in Python and PySpark for large-scale data processing.
• Experience working with AWS cloud services — especially EMR, Glue, S3, Athena, Lambda, Redshift, and Step Functions.
• Hands-on experience with data ingestion frameworks and workflow orchestration tools (Airflow, Step Functions, etc.).
• Good understanding of SQL and NoSQL databases (PostgreSQL, Redshift, DynamoDB, MongoDB).
• Familiarity with data lake and data warehouse architectures.
• Experience with version control (Git) and CI/CD pipelines.
• Strong analytical, debugging, and problem-solving skills.

• Experience with Delta Lake, Databricks, or Snowflake.
• Knowledge of Kafka or Kinesis for real-time data streaming.
• Exposure to Terraform / CloudFormation for infrastructure as code.
• Understanding of data security and governance best practices.