About the role
Role Overview
We are looking for a Senior Databricks & Governance Engineer with 5+ years of Data Engineering experience to develop scalable data pipelines, implement enterprise data governance, and prepare unstructured data for Generative AI applications in a modern Lakehouse environment.
Key Responsibilities
-
Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
-
Orchestrate data workflows using Azure Data Factory (ADF).
-
Extract and process unstructured data from PDFs, images, logs, and text files.
-
Prepare data for RAG, LLM applications, embeddings, and vector search.
-
Implement data access controls, security, and lineage using Unity Catalog.
-
Integrate Databricks metadata with Collibra using REST APIs.
-
Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture.
-
Develop dimensional and Lakehouse data models.
-
Optimize performance using Z-Ordering, Liquid Clustering, and Delta Lake optimization techniques.
-
Ensure data quality, governance, metadata consistency, and pipeline reliability.
Mandatory Skills:
-
5+ years of Data Engineering experience
-
Strong hands-on experience with Azure Databricks
-
PySpark, Python, SQL, and Delta Lake
-
Azure Data Factory (ADF)
-
Unity Catalog — governance, access controls, and lineage
-
Collibra REST API integrations — metadata extraction and synchronization
-
GenAI / LLM data preparation, including unstructured data processing
-
RAG, embeddings, and vector search
-
Lakehouse and Medallion Architecture
-
Dimensional data modeling and performance optimization