About the role
Exp : 5-8 years exp
Azure Databricks: Strong hands-on experience with Databricks Lakehouse Architecture, Delta Lake, Unity Catalog, Databricks Apps, Workflows/Jobs, and Databricks Asset Bundles (DAB).
Real-Time Data Ingestion: Experience with Azure Event Hubs, Spark Structured Streaming, Auto Loader, Databricks Zerobus Ingest, and Delta Change Data Feed (CDF).
Python & SQL: Strong proficiency in Python, PySpark, and SQL for data engineering, transformation, optimization, and automation.
Configuration-Driven Framework: Experience designing reusable, modular, metadata-driven Python frameworks with configurable business rules, execution engines, and orchestration layers.
Lakebase (PostgreSQL): Hands-on experience with Lakebase for operational data storage, relational data modeling, SQL development, and Lakebase-to-Delta synchronization.
Databricks Lakehouse: Experience implementing Bronze, Silver, and Gold architectures, Delta tables, data transformations, and scalable analytical pipelines.
Data Pipeline Engineering: Ability to design batch and streaming pipelines with error handling, retries, replay, reconciliation, and failure recovery mechanisms.
Data Quality & Governance: Experience implementing validation frameworks, data quality checks, quarantine handling, audit logging, and Unity Catalog governance.
Data Modeling & Historical Processing: Strong understanding of relational data modeling, source-to-target mappings, CDC, SCD Type 2, and historical data synchronization.
Performance Optimization: Ability to optimize Spark workloads, SQL queries, Delta tables, job execution, and data processing costs.