About the role
We are looking for a Lead Data Engineer who can own the design and development of scalable data platforms and pipelines while continuing to remain hands-on technically. The ideal candidate should have experience working in a product/technology company, building data engineering solutions at scale, and taking ownership of architecture, technical decisions, production reliability, and mentoring engineers.
Responsibilities:
- Own the architecture and end-to-end design of scalable batch and real-time data pipelines.
- Design and develop high-performance data platforms using Python, SQL, and Spark/PySpark.
- Build and optimize ETL/ELT pipelines, data ingestion frameworks, and transformation workflows.
- Work with Kafka / Spark Streaming / other streaming technologies for near-real-time data processing.
- Design data platforms using cloud technologies such as AWS / GCP / Azure.
- Work with technologies such as Airflow, Databricks, Snowflake, BigQuery, Delta Lake/Iceberg, or equivalent.
- Define data models, lakehouse/data warehouse architecture, partitioning, performance, and cost-optimization strategies.
- Own production data pipelines, including monitoring, data quality, observability, troubleshooting, and reliability.
- Participate in architecture/design reviews and make technical decisions for scalability and maintainability.
- Mentor and guide 3-8 engineers, conduct code/design reviews, and establish engineering best practices.
- Work closely with product, engineering, analytics, and other business stakeholders to translate requirements into scalable technical solutions.
- Drive migration, modernization, and optimization of existing data platforms where required.
Requirements:
- Experience: 7.5-10 years.
- Industry preference: Product/SaaS/Technology companies.
Must-Have Skills:
- Core: Python + SQL + Spark/PySpark.
- Data Engineering: ETL/ELT, Data Pipelines, Data Modelling, Data Warehousing/Lakehouse.
- Distributed Processing: Spark/PySpark + preferably Kafka/Streaming.
- Cloud: Strong experience in at least one of AWS, GCP, or Azure.
- Orchestration: Airflow / Cloud Composer / ADF / equivalent.
- Architecture: Data platform architecture, scalability, performance optimization, data quality.
- Leadership: Technical leadership, mentoring, architecture/design reviews, and stakeholder management.