About the role
We are looking for an experienced Databricks Developer with strong hands-on expertise in building scalable data pipelines and solutions on Azure Databricks. The ideal candidate should have experience in data engineering, data transformation, and cloud-based analytics development.
Responsibilities:
- Develop, optimise, and maintain data pipelines using Azure Databricks.
- Work with PySpark/Spark SQL for large-scale data processing.
- Build ETL/ELT workflows integrating various data sources.
- Implement best practices for data quality, performance tuning, and cost optimisation.
- Collaborate with data architects, business stakeholders, and analytics teams to deliver end-to-end solutions.
- Deploy and monitor Databricks workflows and jobs in production environments.
- Manage Delta Lake, notebooks, and data versioning.
- Work with Azure services such as ADLS, ADF, Azure Synapse, etc. (preferred).
Requirements:
- Strong hands-on experience with Databricks and Azure Databricks.
- Expertise in PySpark, Spark SQL, and distributed data processing.
- Good understanding of Delta Lake, data lakehouse architecture.
- Experience with Azure ecosystem: Azure Data Factory, ADLS, Azure DevOps (added advantage).
- Solid understanding of ETL/ELT concepts and data modelling.
- Ability to troubleshoot performance issues and optimise code.
- Strong analytical and problem-solving skills.