About the role
Responsibilities:
- Design and develop scalable batch and near-real-time ETL/ELT pipelines using Snowflake (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming).
- Build structured streaming pipelines using Kafka and Spark Structured Streaming.
- Design dimensional data models (Fact/Dimension, SCD Type 2).
- Orchestrate pipelines using Databricks Workflows / Apache Airflow.
- Integrate CI/CD pipelines using Jenkins, Git, Bitbucket/GitHub for automated deployment across DEV/UAT/PROD.
Requirements:
- Hands-on experience in Delta Lake optimisation and incremental processing strategies.
- Good to have: Apache Spark (Core, Structured Streaming), PySpark, Databricks (AWS/Azure), Advanced SQL DevOps & CI/CD Jenkins Git/GitHub/Bitbucket Programming Python SQL Cloud & Infrastructure AWS (S3 EMR, EC2 IAM, CloudWatch) Databricks Runtime & Cluster Management- Preferred Streaming & Integration Apache Kafka Snowflake integration (preferred) Airflow (preferred).
- Experience in financial services, payouts, or enterprise data platforms.
- Hands-on experience in Delta Lake optimisation and incremental processing strategies.
- Experience with Snowflake data warehousing.
- Strong understanding of distributed computing principles.