About the role
We are looking for a skilled Databricks Engineer to design, develop, and optimise scalable data pipelines and data processing solutions. The ideal candidate should have strong experience with Databricks, Apache Spark, Python, SQL, and cloud-based data platforms, with the ability to transform complex datasets into reliable, analytics-ready data.
Responsibilities:
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.
- Build and optimise data processing workflows using PySpark and SQL.
- Work with large datasets to support analytics, reporting, and data-driven applications.
- Develop and manage data lakehouse solutions using Delta Lake and Lakehouse architecture.
- Implement data ingestion, transformation, validation, and quality checks.
- Optimise Spark jobs and Databricks workloads for performance and cost efficiency.
- Integrate data from multiple sources, including databases, APIs, and cloud storage.
- Work with cloud platforms such as Azure, AWS, or Google Cloud.
- Implement workflow orchestration, monitoring, error handling, and troubleshooting.
- Collaborate with data engineers, analysts, and software teams to deliver reliable data solutions.
- Follow best practices for data security, governance, access control, and documentation.
Requirements:
- 3-5 years of experience in Data Engineering or a related role.
- Hands-on experience with Databricks and Apache Spark.
- Strong programming skills in Python and PySpark.
- Strong SQL skills, including query optimisation.
- Experience building and maintaining ETL/ELT data pipelines.
- Knowledge of Delta Lake, data lakehouse architecture, and distributed data processing.
- Experience with at least one cloud platform: Azure, AWS, or GCP.
- Understanding of data modelling, data warehousing, and data quality management.
- Experience with Git, CI/CD, and workflow orchestration tools.
Good to Have:
- Experience working in the utility, energy, power, water, or gas industry.
- Familiarity with utility data, IoT, time-series data, or operational technology systems.
- Experience integrating data from multiple enterprise systems and APIs.
- Knowledge of streaming technologies such as Kafka or Spark Structured Streaming.
- Experience with Unity Catalogue, data governance, and access management.
- Familiarity with BI and visualisation tools such as Power BI or Tableau.