About the role
Responsibilities:
- Design and develop scalable batch and near-real-time ETL/ELT pipelines using Snowflake (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming).
- Build structured streaming pipelines using Kafka and Spark Structured Streaming.
- Design dimensional data models (Fact/Dimension, SCD Type 2). Orchestrate pipelines using Databricks Workflows / Apache Airflow. Integrate CI/CD pipelines using Jenkins, Git, Bitbucket/GitHub for automated deployment across DEV/UAT/PROD.
Requirements:
- Apache Spark (Core, Structured Streaming) PySpark Databricks (AWS/Azure) Advanced SQL DevOps & CI/CD Jenkins Git/GitHub/Bitbucket Programming Python SQL Cloud and Infrastructure AWS (S3 EMR, EC2 IAM, CloudWatch) Databricks Runtime & Cluster Management.
- Preferred Streaming & Integration Apache Kafka Snowflake integration (preferred) Airflow (preferred).
Preferred Qualification:
- Experience in financial services, payouts, or enterprise data platforms.
- Hands-on experience in Delta Lake optimisation and incremental processing strategies.
- Experience with Snowflake data warehousing.
- Strong understanding of distributed computing principles.