About the role
- Design and implement scalable batch and streaming data pipelines using Azure Databricks, Delta Live Tables, and Apache Spark.
- Build and maintain ETL/ELT workflows orchestrated through Azure Data Factory and Databricks Workflows.
- Develop reusable and modular pipeline components following software engineering best practices.
- Architect and manage Lakehouse solutions using Delta Lake across Bronze, Silver, and Gold layers.
- Design and enforce data models, schemas, and governance policies using Unity Catalog.
- Optimize storage, partitioning, and query performance for large-scale datasets on ADLS Gen2.
- Manage Databricks clusters, compute policies, and job scheduling.
- Implement Infrastructure as Code (IaC) using Terraform or ARM templates.
- Integrate Databricks with Azure services including Synapse, Event Hubs, Key Vault, and Azure DevOps.
Requirements
- Strong proficiency inPySpark, Python, and SQLfor Big Data processing.
- Hands-on experience withDelta Lake, Delta Live Tables (DLT), and Medallion Architecture.
- Strong experience with Azure Data Services including:
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Azure Event Hubs
- Experience withDatabricks Unity Catalogfor data governance and access control.
- Experience implementing CI/CD pipelines usingAzure DevOpsorGitHub Actionsfor Databricks deployments.
- Strong understanding of distributed computing concepts, Spark optimization, partitioning, and performance tuning.
- Experience with streaming data processing usingStructured Streaming, Kafka, or Event Hubs.