About the role
Looking for a hands-on Lead Data Engineer to build scalable data pipelines and data products using Data Mesh principles. This is a core engineering role involving end-to-end system design, development, and optimisation.
Responsibilities:
- Build and maintain batch and real-time (streaming) data pipelines.
- Develop domain-driven data products and ensure data quality and reliability.
- Implement data contracts, governance, and observability.
- Optimise query performance and cost.
- Work closely with business and domain teams.
Requirements:
- 7+ years in Data Engineering.
- Strong SQL and Python.
- Experience with data modelling (star schema, normalisation).
- Hands-on with Apache Airflow (or similar).
- Experience with large-scale data processing.
Streaming / Real-Time (Important):
- Hands-on experience with real-time streaming pipelines.
- Tools like Apache Kafka, Spark Streaming, or similar.
- Understanding of event-driven architecture and Kappa architecture.
Tech Stack (Flexible):
- Any Data Warehouse (Snowflake / BigQuery / Redshift, etc. )
- Spark / PySpark.
- dbt / Dataform or similar.
- Object storage (S3 or similar).
- Hands-on experience with AWS (S3 Redshift, EMR, Glue or similar services).
Good to Have:
- Data Mesh / Data Product experience.
- Strong ownership and independent working style.
- Experience working with analysts/stakeholders.