About the role
The client currently operates data science workloads in GCP and is migrating these workloads to Azure Databricks. Input data will continue to originate from GCP, data science processing will execute in Azure Databricks, and outputs will be written back to GCP. The MLOps engineer will play a critical role in model migration, workflow automation, CI/CD implementation, model lifecycle management, and cross-cloud data movement initiatives.
Responsibilities:
- Support migration of existing ML models from GCP to Azure Databricks.
- Analyse, replicate, and optimise existing data science model architectures.
- Build and maintain CI/CD pipelines using GitHub Actions.
- Implement and manage MLflow for model tracking, versioning, and lifecycle management.
- Develop scalable data & ML pipelines using Databricks and PySpark.
- Collaborate with data engineering teams on GCP-to-Azure Databricks data movement strategies.
- Build pipelines to move model outputs from Azure Databricks back to GCP.
- Provide architectural guidance and workflow optimisation recommendations.
- Improve testing coverage, monitoring, observability, and performance tuning.
- Drive cost optimisation and operational efficiency initiatives.
Requirements:
- Strong Azure Databricks expertise is mandatory.
- Hands-on experience with PySpark, MLflow, and GitHub Actions is required.
- Candidates must have exposure to both Azure and GCP ecosystems.
- Experience in ML model migration, operationalisation, and optimisation is highly preferred.
- Strong troubleshooting, performance tuning, and cost optimisation capabilities are essential.
Mandatory Technical Skills:
- Strong hands-on expertise in Azure Databricks with a deep understanding of Databricks internals.
- Advanced PySpark development, optimisation, and execution plan analysis.
- Experience building CI/CD pipelines using GitHub Actions.
- Strong experience with MLflow for model tracking, deployment, and lifecycle management.
- Knowledge of Terraform and Databricks infrastructure automation.
- Experience integrating workflows across GCP and Azure cloud platforms.
- Strong debugging, troubleshooting, and performance optimisation capabilities.
- Experience operationalising machine learning models in production environments.
- Cost optimisation mindset for cloud and data processing workloads.
Good to Have:
- Experience with cross-cloud data movement architectures.
- Understanding of data science model structures and workflows.
- Exposure to model monitoring, observability, and alerting frameworks.
- Experience collaborating closely with data science teams.