Project Lead - BIDW

JOB DESCRIPTION - LEAD BI / DW / ETL ENGINEER

Azure Databricks | Data Warehousing | ETL/ELT | Power BI | Lakehouse

Experience

10-12 years

Role Level

Lead Engineer / Technical Lead

Primary Focus

BI, DW, ETL and Databricks

Domain

Insurance and Benefits Management

Seeking a hands-on Lead Engineer with 10-12 years of experience across Business Intelligence, Data Warehousing and ETL/ELT, including strong Azure Databricks expertise. The role will lead delivery of governed Lakehouse and warehouse solutions, scalable data pipelines, dimensional and semantic models, certified metrics and Power BI reporting for Insurance and Benefits Management use cases.

• Lead end-to-end design and delivery of BI, enterprise data warehouse, data-mart and Azure Databricks Lakehouse solutions.
• Design Bronze, Silver and Gold layers using Delta Lake, dimensional models, star/snowflake schemas, facts, dimensions and slowly changing dimensions.
• Develop and optimize ETL/ELT pipelines using Azure Databricks, Spark, PySpark, Spark SQL, Python, SQL and Azure Data Factory.
• Implement batch, incremental and streaming ingestion from databases, APIs, files, HRIS, benefits platforms, CRM and external feeds.
• Deliver curated datasets, semantic models, certified KPIs and reporting structures for Power BI, Databricks SQL and Genie.
• Define source-to-target mappings, transformation rules, data-quality checks, reconciliation, lineage and audit controls.
• Apply Unity Catalog governance, RBAC/ABAC, masking, access policies, employer isolation and secure data-sharing patterns.
• Optimize Spark workloads, Delta tables, SQL queries, data models, Power BI datasets, refresh performance, compute and storage cost.
• Establish engineering standards, reusable frameworks, automated testing, CI/CD, release controls, monitoring and production runbooks.
• Lead technical reviews, estimates, sprint delivery, defect resolution, stakeholder communication and mentoring of distributed engineering teams.

Azure Databricks; Apache Spark; PySpark; Spark SQL; Python; advanced SQL and T-SQL; Delta Lake; medallion architecture; data warehousing, ODS and data marts; dimensional modelling; star and snowflake schemas; SCD Types 1 and 2; ETL/ELT; Azure Data Factory; Auto Loader and Lakeflow; Databricks Workflows and SQL; Photon; Unity Catalog; ADLS Gen2; Power BI data modelling, DAX, Power Query, semantic models, gateways and refresh; SSIS, Informatica or equivalent ETL exposure; data profiling, quality, reconciliation and lineage; REST APIs; Git; Azure DevOps; CI/CD; Terraform awareness; monitoring, security and performance optimization.

• 10-12 years of experience in BI, data warehousing, ETL/ELT, data engineering or enterprise analytics, including technical leadership.
• Proven experience delivering production BI and DW platforms using Azure Databricks, Delta Lake and modern cloud-data patterns.
• Strong hands-on expertise in Spark/PySpark, SQL, dimensional modelling, ETL frameworks and Power BI integration.
• Experience leading data-engineering workstreams, conducting design and code reviews and mentoring engineers.
• Experience collaborating with architects, modellers, business users, QA, DevOps, security and reporting teams in Agile delivery.
• Bachelor’s degree in Computer Science, Information Technology, Data Engineering or a related discipline; Azure, Databricks or Power BI certification is preferred.

• Insurance or employee-benefits experience across employer, client, plan, member, enrollment, carrier, premium, claims and renewal data is preferred.
• Experience delivering governed metrics, operational reports, analytical dashboards and role-specific views for sensitive data.
• Familiarity with HRIS, EDI, benefits-administration, CRM, policy or document-derived data is advantageous.

Hands-on technical leadership, BI and data-warehouse design, ETL delivery ownership, analytical problem-solving, data-quality discipline, stakeholder communication, mentoring and the ability to balance business value, scalability, governance, performance and cost.