About the role
- Set up and maintain Databricks for ML: workspaces, clusters, instance pools, cluster policies, Repos, and runtime versions.
- Provide notebook and job templates, shared libraries, and pinned environments so experiments can be repeated.
- Build CI/CD to promote notebooks, jobs, workflows, and models across development, staging, and production environments using Git and GitHub Actions, Jenkins, or Azure DevOps, with Databricks Asset Bundles or the Jobs and Repos APIs.
- Track, package, register, and deploy models with MLflow on Databricks, including batch scoring and model serving jobs.
- Build Databricks Workflows for training, retraining, evaluation, and inference, with task dependencies, retries, and alerts.
- Support classification, forecasting as Python and Spark jobs on Databricks.
- Tune cluster size, autoscaling, Spark settings, and Delta Lake reads and writes so jobs stay reliable and cost-efficient.
- Convert Data Scientist notebooks into production pipelines with clear inputs, outputs, and fixed dependencies.
- Apply Databricks controls: Unity Catalog for data and model access, secret scopes, service principals, and model versioning.
- Fix failed jobs, slow Spark runs, library conflicts, and environment differences across workspaces.
- Write standards and runbooks for workspace layout, cluster policies, job templates, and promotion from dev to production.
- Bachelors or Master’s degree in Computer Science, Engineering, or a related field.
- 6–12 years in ML Engineering, MLOps, or platform engineering, including hands-on ML work on Databricks.
- Databricks experience with workspaces, clusters and policies, Jobs and Workflows, Repos, Delta Lake, and Spark (batch and streaming).
- MLflow on Databricks for experiment tracking, model packaging, and the model registry.
- Strong Python, plus working knowledge of PySpark or Spark SQL.
- CI/CD for Databricks with Git and GitHub Actions, Jenkins, or Azure DevOps.
- Experience deploying ML models on Databricks, Azure, AWS services.
- Clear debugging skills and the ability to work with data science and platform teams.
- Databricks Asset Bundles, Terraform, or workspace-as-code.
- Unity Catalog, cluster policies, and service principals for job runs.
- Azure ML services such as Azure Machine Learning and Amazon SageMaker for training, pipelines, or model deployment.
- Healthcare or pharmaceutical domain experience.
- Agile or Scrum.
- Python packaging on Databricks (pip or conda, and runtime libraries).
- SQL and PySpark for reading, joining, and transforming data on Databricks.
- Simple pipelines that ingest data, clean it, and store it in Delta Lake.
- Data quality checks and schema handling.
- Familiarity with healthcare or pharma data, such as EHR, clinical, or patient and drug datasets.
- Docker for packaging jobs and model services.
- Kubernetes and KServe for deploying and scaling models.
- Helm, Kubeflow, or Seldon for model deployment.
- MLflow, GitHub Actions, and Terraform for tracking, CI/CD, and environment setup.
- Model performance monitoring, data drift detection, alerting, and rollback for production models.