About the role
Responsibilities:
- AI-Augmented Pipeline Architecture: Design and build self-healing, schema-validated data pipelines on GCP (BigQuery, Dataflow, Pub/Sub) that power real-time analytics. Pipelines must be idempotent, observable, and auto-recover from failures without human intervention.
- Streaming and Batch Processing at Scale: Implement and optimise large-scale processing workflows using Apache Spark, Flink, or Beam. Own both real-time Kafka consumer pipelines and scheduled batch jobs, ensuring sub-hour SLAs for data freshness across all critical business domains.
- Schema Contracts and Data Integrity: Implement and enforce schema registry and data contract standards across all producers and consumers: target 100% data quality scores, no silent failures, and no downstream surprises.
- ML Pipeline Integration: Build and maintain feature stores and data pipelines that feed production ML models (recommender systems, demand forecasting, pricing). Collaborate with data science to ensure training-serving skew is eliminated.
- Data Governance and Cost Engineering: Champion data cataloguing, lineage tracking, and access-control policies. Monitor BigQuery slot usage and pipeline costs, and implement optimisations that reduce cloud spend by measurable percentages each quarter.
Requirements:
- DE2 3-6 years; end-to-end ownership of production pipelines at scale.
- Ability to ship data products independently with clear quality and latency guarantees.
Tech Stack:
- Languages: Python, Scala, Java, SQL.
- Processing: Apache Spark, Apache Beam / Dataflow, Apache Flink, Kafka, Databricks.
- Cloud and Storage: GCP (BigQuery, Dataflow, Pub/Sub, DataProc, and GCS) and AWS (S3 and EMR).
- Orchestration: Apache Airflow (Cloud Composer).
- Data Quality: Great Expectations, dbt.
- Infrastructure: Kubernetes, Docker, Terraform, Databricks.
- AI Productivity: Cursor, Claude.