About the role
We are looking for a hands-on Data Engineer with strong expertise in SQL, PySpark, Apache Spark, and Databricks, along with awareness of GenAI and AI-assisted coding tools. The candidate should be capable of building scalable data pipelines, troubleshooting performance issues, and optimizing data processing for both performance and cost.
Responsibilities:
- Design, develop, and maintain scalable data pipelines using SQL, PySpark, Spark, and Databricks.
- Build reliable batch/incremental data-processing solutions and data products.
- Troubleshoot and optimize Spark workloads for performance, scalability, and cost efficiency.
- Work with Delta Lake and modern data-platform architecture.
- Use AI coding agents to accelerate development, debugging, testing, and documentation.
- Collaborate with application teams, consultants, and managers.
Requirements:
- Strong problem-solving and analytical skills.
- Hands-on, engineering-oriented mindset.
- Ability to understand why a technical approach is selected, not just implement it.
- Clear and structured communication.
- Comfortable working independently and learning new technologies.
- Expert SQL and PySpark development.
- Strong understanding of Apache Spark internals: distributed processing, partitioning, shuffle, data skew, and execution model.
- Strong Databricks experience with Delta Lake, optimization, compute management, and cost optimization.
- Strong data-engineering fundamentals: ETL/ELT, data modelling, incremental processing, and data quality.
- Experience troubleshooting and optimizing large-scale data workloads.
GenAI and AI-Assisted Development:
- LLMs, RAG, embeddings, agents, and agentic AI.
- AI-assisted coding tools such as Claude Code, Codex, Cursor, Antigravity, or similar.
- Ability to review, validate, and improve AI-generated code.
Good to Have:
- DevOps knowledge: Git, CI/CD, Docker, and deployment practices.
- Exposure to NoSQL databases and their use cases.
- Data Engineer/Analytics Certificate in any of the clouds (AWS, Azure, or GCP).