About the role
We are looking for a hands-on Data Engineer with strong experience in big data engineering and real-time streaming systems. You will work closely with Data Platform, Analytics, and Machine Learning teams to design and build scalable, reliable, and low-latency data pipelines that enable real-time analytics and decision-making.
Responsibilities:
- Design, develop, deploy, and own real-time streaming applications using Apache Flink.
- Build and maintain Spark Structured Streaming pipelines for high-volume, real-time data processing.
- Tune and troubleshoot Flink and Spark workloads in production, including: Checkpointing, State management, Backpressure handling, Resource and performance optimization.
- Develop and optimize batch and streaming ETL/ELT pipelines using PySpark and Python.
- Write efficient and optimized SQL for data transformation, validation, and downstream analytics consumption.
- Build and deploy cloud-native data pipelines on AWS, leveraging services such as EMR, S3 AWS Glue, and Lambda.
- Design scalable solutions capable of meeting defined latency, throughput, reliability, and data accuracy SLAs.
- Implement monitoring, schema validation, error handling, and data quality checks to ensure pipeline reliability.
- Apply strong programming, data structures, and algorithms fundamentals to build performant, maintainable, production-grade systems.
- Work effectively across a complex ecosystem involving technology teams, vendors, business stakeholders, and other engineering teams.
- Contribute to Agile projects spanning multiple teams, business units, and organizations.
Requirements:
- 3-5 years of professional experience in Data Engineering / Big Data Engineering.
- Strong hands-on experience with Apache FlinkMandatory.
- Experience building and supporting production-grade real-time streaming applications.
- Strong experience with Apache Spark / Spark Structured Streaming.
- Hands-on expertise in PySpark and Python.
- Strong understanding of streaming architectures, distributed data processing, and large-scale data systems.
- Strong working knowledge of SQL and data transformation.
- Experience building data pipelines on AWS, particularly with: EMR, S3 AWS Glue, Lambda.
- Experience with performance tuning, debugging, monitoring, and optimization of production data pipelines.
- Ability to work across multiple stakeholders, teams, and technology ecosystems.
Good to Have:
- Programming experience in Java or Scala.
- Strong understanding of Data Structures and Algorithms (DSA).
- Experience working in Agile environments across multiple organizations or business units.
- Key Skills: Apache Flink, Spark Structured Streaming, Apache Spark, PySpark, Python, SQL, AWS, EMR, S3 AWS Glue, Lambda, Real-Time Streaming, ETL/ELT, distributed systems, and big data.