About the role
Requirements:
- 4+ years in a data engineering role with end-to-end pipeline ownership.
- Strong Python async patterns, subprocess management, API clients, and data processing at scale.
- Hands-on AWS Athena, Glue, S3, DynamoDB, and Lake Formation; production-grade, not just familiarity.
- Apache Hudi or Delta Lake schema evolution, partition strategies, and upsert patterns.
- SQL proficiency: able to write and optimize complex analytical queries.
- Experience with Airflow or an equivalent workflow orchestrator.
- Demonstrated experience with large-scale media data pipelines, audio/video format conversion, metadata extraction, and chunking, e. g., egocentric or multimodal datasets (hard requirement, not a plus).
- Understanding of how data format and I/O design affect downstream GPU compute workloads (WebDataset, sharded Parquet, tfrecord, or equivalent).
Nice to Have:
- Direct experience designing data handoff for GPU clusters (AWS Batch GPU instances, Ray, or SLURM).
- Familiarity with ML training data formats and dataset standards used by AI labs (Hugging Face datasets, WebDataset, dataset cards).
- Experience with rclone, large-scale file transfer, or cloud-to-cloud sync pipelines.
- Exposure to data lineage or provenance tooling (OpenLineage, DataHub, or custom metadata schemas).