About the role
We are looking for a Senior Data Engineer to own, extend, and harden the multimodal data infrastructure that powers our AI training data supply chain. You will work directly with the Head of Data, building and maintaining production-grade pipelines that move real-world audio, video, and egocentric capture data from ingestion through AWS-scale storage into GPU-backed technical validation and finally to delivery-ready datasets for frontier AI lab customers.
This role sits at the intersection of data engineering and ML infrastructure; you will own the full pipeline: ingestion, lake management, quality validation, and compute handoff.
Requirements:
- 4+ years in a data engineering role with end-to-end pipeline ownership.
- Strong Python async patterns, subprocess management, API clients, and data processing at scale.
- Hands-on AWS Athena, Glue, S3 DynamoDB, and Lake Formation; production-grade, not just familiarity.
- Apache Hudi or Delta Lake schema evolution, partition strategies, and upsert patterns.
- SQL proficiency: able to write and optimize complex analytical queries.
- Experience with Airflow or an equivalent workflow orchestrator.
- Demonstrated experience with large-scale media data pipelines: audio/video format conversion, metadata extraction, chunking, and egocentric or multimodal datasets (hard requirement, not a plus).
- Understanding of how data format and I/O design affect downstream GPU compute workloads (WebDataset, sharded Parquet, tfrecord, or equivalent).