Delivery Module Lead

JOB DESCRIPTION - SENIOR DATABRICKS ENGINEER

Azure Databricks | Spark & PySpark | Delta Lake | Lakeflow | Unity Catalog

Experience

8-10 years

Role Level

Senior Databricks Engineer

Primary Platform

Microsoft Azure and Azure Databricks

Domain

Insurance and Benefits Management

Seeking a hands-on Senior Databricks Engineer with 8-10 years of data engineering experience and strong expertise in Azure Databricks. The role will design, build, test, optimize and operationalize scalable batch and streaming pipelines, Delta Lake data products and governed Lakehouse solutions for analytics, BI and AI workloads, supported by Insurance and Benefits Management domain experience.

• Develop and optimize ETL/ELT pipelines using Azure Databricks, Spark, PySpark, Spark SQL and Python.
• Build Bronze, Silver and Gold data layers using Delta Lake, schema enforcement, incremental processing and reusable engineering patterns.
• Implement ingestion from APIs, databases, files, cloud storage and streaming sources using Auto Loader, Lakeflow and Azure orchestration services.
• Develop batch and near-real-time pipelines using Spark Structured Streaming, checkpoints, watermarking and resilient recovery patterns.
• Create curated tables, views and data products for Databricks SQL, Power BI, Genie, analytics and downstream AI/ML consumption.
• Apply Unity Catalog controls, lineage, data classification, access policies, masking and employer-level data isolation.
• Implement data-quality rules, validation, reconciliation, error handling, audit controls and operational observability.
• Optimize Spark jobs, Delta tables, queries, partitioning, file sizes, caching, Photon workloads, compute utilization and cloud cost.
• Build automated tests and CI/CD pipelines for notebooks, jobs, Lakeflow pipelines and data assets using Git and Azure DevOps.
• Participate in design and code reviews, troubleshoot production issues, maintain technical documentation and mentor junior engineers.

Microsoft Azure; Azure Databricks; Apache Spark; PySpark; Spark SQL; Python; advanced SQL; Delta Lake; Lakehouse and Bronze/Silver/Gold architecture; Auto Loader; Lakeflow Connect, pipelines and jobs; Spark Structured Streaming; Databricks Workflows; Databricks SQL and Photon; Unity Catalog; ADLS Gen2; Azure Data Factory or equivalent orchestration; REST APIs; JSON, CSV, Parquet and Avro; data modelling; ETL/ELT; data quality and reconciliation; Git; Azure DevOps or GitHub Actions; CI/CD; Terraform awareness; monitoring, logging, ing, security, performance tuning and cost optimization.

• 8-10 years of experience in data engineering, big-data development, ETL/ELT, data warehousing or cloud analytics.
• At least 4 years of strong hands-on experience developing production solutions on Azure Databricks or an equivalent Databricks environment.
• Proven experience building and supporting scalable Spark and Delta Lake pipelines for structured and semi-structured data.
• Experience with Unity Catalog, data lineage, access controls, data-quality frameworks and production deployment practices.
• Experience working with data architects, modellers, business analysts, QA, DevOps, security and BI teams in Agile delivery.
• Bachelor’s degree in Computer Science, Information Technology, Data Engineering or a related discipline; Databricks or Azure certification is preferred.

• Insurance or employee-benefits experience involving employer, client, plan, member, enrollment, carrier, claims or renewal data is preferred.
• Experience handling privacy-sensitive and regulated information with traceability, auditability and role-based access controls.
• Familiarity with EDI, HRIS, benefits-administration, CRM or document-derived data is advantageous.

Hands-on engineering depth, structured problem-solving, production ownership, data-quality discipline, performance tuning, collaboration, mentoring and the ability to balance scalability, reliability, governance, maintainability and cost.