About the role
- DataPipeline & Orchestration: Design, build, and run batch and scheduleddatapipelines (Airflow, Dagster, Prefect) and lead orchestrator upgrades.
- DataIngestion & Scraping: Own and extenddatacollectors and engines, including web/API ingestion, large-scale web scraping (Playwright, Selenium, Scrapy), parsing, normalization, and file merging.
- Distributed Processing: Build scalable distributed processing architectures (Spark on EMR,Databricks,Dataproc) and analytical query layers (Athena, Snowflake, BigQuery), including cross-cloud ingestion pipelines.
- Caching & Entity Matching: Improve entity matching and caching services (Redis, MemoryDB, DynamoDB) to optimize accuracy, latency, and operational cost.
- DataAccess & APIs: Develop and maintain high-performance Pythondataaccess layers and APIs for downstream ML and product teams.
- DataQuality & Observability: Owndataquality initiatives, including validation frameworks, retention policies, backfills, monitoring, and OpenTelemetry-based observability.
- Cross-Functional Collaboration: Partner withdatascience, forecasting, and platform teams to ship features safely behind feature flags (LaunchDarkly, Unleash).
- Engineering Leadership: Raiseengineering standards across the team through rigorous code reviews, technical design documents, and mentoring.
- Experience: 6+ years of software ordataengineering experience, including 3+ years of experience building and maintaining productiondatapipelines.
- Python Mastery: Strong Python development skills with a commitment to writing clean, tested code (using pytest, nox, or tox).
- Orchestration Expertise: Deep experience with workflow orchestration (Airflow DAG design, operators, scheduling, and failure handling) or modern alternatives like Dagster and Prefect.
- DataEcosystems: Hands-on experience with distributed compute (Spark) and clouddataecosystems (AWS EMR/Athena/S3,Databricks, Snowflake, GCP).
- Storage & Modeling: Solid SQL anddata-modeling skills, alongside practical experience with NoSQLdatabases, caching layers, and message queues (MongoDB, Redis, Kafka, RabbitMQ).
- Infrastructure Familiarity: Comfort deploying and managing containerized workloads on Kubernetes using Docker and Helm.
- Production Ownership: A proven track record of owning production pipelines, including participating in on-call rotations, incident response, and root-cause analysis.
- Communication: Clear written technical communication and a history of documenting architectural decisions.
- Scraping Infrastructure: Experience managing large-scale webdatacollection and distributed crawler infrastructure.
- DataScience / ML Concepts: Familiarity with record linkage, entity resolution, fuzzy matching, and NLP preprocessing (spaCy, NLTK).
- Multi-Cloud Pipelines: Hands-on experience building multi-clouddatapipelines (e.g., bridging GCP/BigQuery and AWS).
- ML Ops: Experience designingdatapipelines that directly feed machine learning or forecasting systems.