About the role
Looking for a candidate with at least 8 years of extensive experience in Data Engineering
Proven record of successfully delivering software with a broad mix of languages technologies and platforms
Proven experience in Big Data Engineering
Experience working with Agile at Scale Lean and Continuous Delivery approaches such as Continuous Integration TestDriven Development and Infrastructure as Code
Technical Skills Required
Must have skills
Strong handson experience with Databricks musthave
Expertise in Apache Spark PySparkScala
Strong proficiency in SQL and data modelling
Strong handson experience in Power BI
Experience with Delta Lake and Lakehouse architecture
Handson experience with cloud platforms AWS preferred
Experience with orchestration tools
Knowledge of streaming frameworks Kafka
Proficiency in Python mandatory
Familiarity with Scala
Experience with Git CICD pipelines
Exposure to data cataloguing tools Unity Catalog
Proficient understanding of distributed computing principles
Good exposure to AWS Big data platform and services such as S3 Glue EMR DMS Athena RDS Redshift Kinesis etc
Management of EMR cluster with all included services
Strong working knowledge with RDBMS such as PostgreSQL Oracle SQL Server
Experience with writing complex SQL queries in PostgreSQL
Experience working with a source control system such as Azure DevOps GitHub
Nice to have skills
Data warehousing concepts
Experience supporting ML pipelines in Databricks
Familiarity with MLflow experiment tracking model registry
Familiarity with Databricks Mosaic AI Vector search embeddings pipelines
Understanding of Generative AI concepts LLMs embeddings RAG
Experience with infrastructure as code Terraform CloudFormation
Prior experience in Debezium or another CDC tool
Experience in building Workow orchestrators using Jenkins
Soft skills
Strong analytical and problemsolving abilities
Excellent communication and stakeholder management skills
Ability to work in a fastpaced agile environment
Ownership mindset and attention to detail