About the role
Responsibilities:
- Design, develop, and maintain scalable data architecture and data engineering solutions on Azure and Databricks.
- Build robust batch and streaming data pipelines using Apache Spark, PySpark, Delta Lake, SQL, and related technologies.
- Develop and optimize complex data processing frameworks for structured, semi-structured, and high-volume datasets.
- Implement enterprise data lakehouse solutions aligned to business, analytics, and AI/ML requirements.
- Ensure data pipelines are reliable, performant, reusable, and built with strong engineering standards.
- Collaborate with business teams, data architects, analysts, data scientists, and cloud engineers to translate requirements into technical solutions.
- Support AI and ML initiatives by creating curated, governed, and feature-ready datasets.
- Work on performance tuning, troubleshooting, and optimization of Databricks workloads and Spark jobs.
- Establish and maintain data quality, lineage, security, governance, and compliance controls.
- Drive best practices in CI/CD, code quality, testing, release management, and operational support for data solutions.
- Contribute to platform improvements, architecture reviews, and technology evaluation activities
- Deliver high-quality, scalable, and secure data solutions aligned to business objectives.
- Enable data-driven and AI-ready capabilities through well-engineered data platforms.
- Contribute to Tata Digital's long-term data strategy and platform maturity.
Requirements:
- Demonstrate strong technical leadership in the Azure and Databricks ecosystems.
- Bachelor's or master's degree in computer science, information technology, engineering, or a related field.
- 6-8 years of experience in data engineering, data platform engineering, or related roles.
- Strong hands-on experience with Azure cloud services and Azure Databricks.
- Deep expertise in Apache Spark / PySpark, Delta Lake, SQL, and distributed data processing.
- Strong understanding of data architecture, data modeling, and modern lakehouse patterns.
- Experience building and supporting complex ETL/ELT pipelines in enterprise environments.
- Working exposure to AI and ML data pipelines, feature preparation, or data support for AI-based solutions.
- Experience with batch and streaming data processing.
- Solid understanding of data governance, security, access control, and data quality practices.
- Experience with orchestration tools such as Azure Data Factory, Databricks Workflows, Airflow, or similar.
- Strong debugging, analytical thinking, and problem-solving skills.
Competencies:
- Strong technical ownership and accountability.
- Ability to work on complex, high-impact data problems.
- Collaborative mindset with cross-functional stakeholders.
- Strong communication and documentation skills.
- Adaptability to evolving data and AI technologies.