About the role
Responsibilities:
- Own the complete medallion architecture Bronze, Silver, Gold pipeline: land raw events (object storage), clean, conform and dedupe them (Silver), and model curated aggregates (Gold).
- Design the multi-tenant data model: data partitioned and isolated per tenant and per facility, with row-level tenancy and ensure that isolation holds across every layer.
- Author and operate Airflow DAGs for both incremental ingestion and historical backfill.
- Build and support the streaming ingestion path: API polling, Kafka consumer Bronze and pair with the junior engineer who owns it.
Requirements:
- 6+ years in data engineering, with senior-level ownership of production pipelines.
- Strong Python for data work.
- Hands-on Airflow with DAG authoring, dependencies, and backfills.
- Solid PostgreSQL: schema/data modelling and query performance.
- Experience with medallion / layered ELT patterns (raw, cleaned, curated).
- Comfortable with S3 / object storage and partitioning strategies.
- Working knowledge of Kafka consumers.
- Hands-on stream processing with Apache Flink (or similar) event-time semantics, windowing, and stateful streaming for real-time pipelines.
- Very good at problem-solving.
- Tech stack: Python, Apache Airflow, Apache Flink, Apache Kafka, PostgreSQL, S3 / object storage, data lake/lakehouse (Delta Lake, Apache Iceberg), cloud data warehouse (Snowflake / BigQuery / Redshift), medallion / ELT architecture.
Nice to have:
- Multi-tenant data modelling and tenant-isolation experience.
- dbt or similar transformation tooling.
- Awareness of data governance / GDPR readiness.
- Experience mentoring junior engineers.