About the role
Responsibilities:
- Own the complete medallion architecture Bronze, Silver, Gold pipeline: land raw events (object storage), clean, conform and dedupe them (Silver), and model curated aggregates (Gold).
- Design the multi-tenant data model: data partitioned and isolated per tenant and per facility, with row-level tenancy and ensure that isolation holds across every layer.
- Author and operate Airflow DAGs for both incremental ingestion and historical backfill.
- Build and support the streaming ingestion path: API polling, Kafka consumer Bronze and pair with the junior engineer who owns it.
- Build real-time / stream processing on top of the ingestion path: low-latency transformations, event-time windowing and aggregations over Kafka streams to power near-real-time analytics.
- Establish data-quality and reconciliation practices, including Bronze/Silver/Gold row-count and integrity checks.
- Make pragmatic design decisions and document the patterns clearly for the team.
Requirements:
- 3+ years in data engineering, with senior-level ownership of production pipelines.
- Strong Python for data work.
- Hands-on Airflow with DAG authoring, dependencies, backfills.
- Solid PostgreSQL: schema/data modelling and query performance.
- Experience with medallion / layered ELT patterns (raw, cleaned, curated).
- Comfortable with S3 / object storage and partitioning strategies.
- Working knowledge of Kafka consumers.
- Hands-on stream processing with Apache Flink (or similar) event-time semantics, windowing, and stateful streaming for real-time pipelines.
- Experience with a data lake/lakehouse (e. g., Delta Lake, Apache Iceberg) or a cloud data warehouse (e. g., Snowflake, BigQuery, Redshift).
- A real data-quality and reconciliation mindset: you validate; you don't assume.
- Very good at problem-solving.