About the role
Responsibilities:
- Architect, design, and deliver large-scale cloud-native data platforms primarily on AWS, with exposure to Azure and GCP, using Databricks and distributed processing frameworks to build highly scalable and resilient systems.
- Work hands-on across the technology stack - including Python, PySpark, Apache Spark, Databricks, AWS services, event-driven architectures, and SQL/NoSQL databases - to solve complex engineering challenges and maintain platform excellence.
- Lead organisation-wide technical initiatives focused on performance optimisation, scalability, reliability, security, governance, and cost efficiency.
- Review and influence critical architectural decisions across teams while establishing engineering best practices and architectural governance standards.
- Partner closely with global engineering, product management, architecture, and business stakeholders to align technical solutions with strategic business objectives.
- Own the end-to-end software development lifecycle, including requirements gathering, solution design, development, deployment, observability, and documentation.
- Mentor and guide senior and junior engineers, fostering a culture of innovation, accountability, collaboration, and technical excellence.
Requirements:
- B. E/B. Tech/M. Tech/MCA in Computer Science, Information Technology or a related field.
- 10-13 years of strong software engineering experience, with significant expertise in large-scale data engineering and distributed systems architecture.
- Solid expertise in Data Warehousing, Data Lakes, Delta Lake architecture, and modern big data ecosystem designs.
- Deep hands-on expertise in Databricks, Python, PySpark and Apache Spark, with proven experience building high-performance distributed data processing solutions and managing massive-scale datasets for analytics and business intelligence workloads.
- Strong experience with AWS services such as S3 Glue, Redshift, EMR, Athena, Lambda, and EventBridge for building scalable and reliable cloud-native data platforms.
- Experience with real-time and near real-time streaming technologies including Kafka, AWS Kinesis, SQS, and RabbitMQ.
- Strong understanding of both relational and NoSQL databases, including PostgreSQL, SQL Server, Aurora, RDS, MongoDB, and DynamoDB.
- Hands-on experience with Infrastructure as Code (IaC) tools such as Terraform or Ansible.
- Strong understanding of CI/CD and DevOps practices using tools such as Jenkins, Github/GitLab, Bitbucket, GoCD and automated deployment pipelines.
- Proven ability to influence technical direction, drive multi-functional initiatives, and lead complex engineering programs at scale.
- Exposure to Generative AI technologies including LLMs, RAG architectures, and Agentic AI systems, with experience designing and deploying AI-driven solutions.
Nice to have:
- AWS and Databricks certifications and experience with Azure / Google Cloud Platform (GCP)
- Working knowledge of TypeScript, Node. JS, and UX design and development.
- Experience building data platforms in privacy-safe or cleanroom environments.