About the role
Responsibilities:
- Design, build, and maintain scalable data pipelines and Delta Lake architectures using Databricks on AWS.
- Develop and enhance data lakehouse/data warehouse solutions supporting front office trading and commodity analytics.
- Build production-grade ETL/ELT pipelines using Python, PySpark, and SQL.
- Work with data scientists, ML engineers, quants, and business stakeholders to prepare ML-ready datasets and support ML/AI pipelines.
- Perform Databricks administration, including cluster management, jobs, permissions, workflows, and platform resources.
- Implement and maintain CI/CD pipelines, automated testing, deployment automation, and monitoring for data engineering workflows.
- Implement MLOps practices using Databricks and MLflow, including model tracking, deployment, and monitoring.
- Provision and manage data infrastructure using Terraform / Infrastructure as Code (IaC).
- Integrate AWS services such as S3, Lambda, Glue, API Gateway, IAM, EC2, and EKS with Databricks.
- Participate in data modeling, architecture, code reviews, and cross-functional solutioning.
- Ensure data governance, security, access control, performance, reliability, and data quality standards.
- Optimize Databricks workloads, Spark jobs, clusters, and cloud infrastructure for performance and scalability.
- Identify opportunities to improve Databricks and AWS data platform capabilities.
- Support streaming and real-time data pipelines using Kafka or Spark Structured Streaming where required.
- Work closely with Front Office Trading and quantitative and analytics teams to deliver reliable data solutions.
- Take end-to-end ownership of data engineering solutions from design and development through deployment and production support.