About the role
We are seeking an experienced Senior Data Engineer with strong expertise in PySpark and big data technologies to design, develop, and maintain scalable data pipelines in production environments. The ideal candidate should have hands-on experience processing large datasets, building reusable ingestion frameworks, and automating complex data workflows.
Responsibilities:
- Design, build, and maintain robust Apache Spark pipelines in production environments.
- Develop scalable and reusable data ingestion frameworks integrating data from multiple heterogeneous source systems.
- Process and transform large datasets (~500 GB+) while ensuring data quality, reliability, and performance.
- Optimise Spark jobs using a deep understanding of Spark architecture, including DAG execution, partitioning, shuffles, caching, executors, and cluster resource management.
- Work extensively with HDFS and various file formats to perform data cleansing, normalisation, transformation, and generation of curated datasets.
- Develop and maintain Python, PySpark, and Shell scripts to automate data processing workflows and eliminate manual interventions.
- Write efficient Oracle SQL queries for data extraction, transformation, validation, and reporting.
- Troubleshoot and optimise existing data pipelines to improve scalability and operational efficiency.
- Collaborate with cross-functional teams to understand business requirements and deliver high-quality data solutions.
Requirements:
- Strong hands-on experience with Apache Spark (PySpark) in production environments.
- Expertise in designing and scaling data ingestion frameworks across multiple source systems.
- Deep understanding of Spark internals: Driver/Executor architecture, DAG execution, partitioning, shuffles, caching, and resource management.
- Experience handling and transforming large datasets (500GB+).
- Strong proficiency in Python, PySpark, and Shell scripting.
- Advanced knowledge of Oracle SQL.
- Hands-on experience working with HDFS and different file formats.
- Strong understanding of data cleansing, normalisation, and transformation techniques.
- Ability to automate manual data processing activities through Python-based solutions.
- Excellent analytical, problem-solving, and debugging skills.