Senior Data Engineer

Senior Data Engineer (Platform Engineering focus)

The Opportunity

The Senior Platform Engineer designs, builds, automates, and operates the shared data and AI platform capabilities used by engineering, analytics, governance, and AI teams.

This senior hands-on role establishes reusable platform patterns across Databricks, Microsoft Azure, Microsoft Fabric, CI/CD, observability, security, and cost management. The role also provides technical leadership for platform reliability, operational maturity, and developer enablement.

What you’ll do…

Platform Architecture & Engineering

• Design and implement secure, scalable platform services for Databricks, Azure Data Lake, Microsoft Fabric, and supporting cloud services.
• Create reusable environment patterns for development, test, and production workloads.
• Implement workspace, network, identity, storage, compute, catalog, policy, and secrets-management configurations.
• Partner with Architecture and Security to translate enterprise standards into practical platform controls and reference implementations.

Automation, DevOps & Developer Experience

• Develop infrastructure-as-code modules, deployment pipelines, automated configuration, and policy-as-code controls.
• Establish CI/CD patterns for data pipelines, notebooks, libraries, semantic assets, and AI workloads.
• Build self-service templates and paved-road capabilities that improve delivery consistency and reduce manual provisioning.
• Maintain source-control, release, artifact, environment-promotion, and rollback practices.

Reliability, Observability & Cost Management

• Define and implement monitoring, alerting, logging, capacity, performance, availability, and recovery standards.
• Lead troubleshooting and root-cause analysis for complex platform incidents and recurring reliability issues.
• Implement operational dashboards, service-level indicators, runbooks, and on-call support patterns.
• Analyze platform usage and cost drivers; implement guardrails, tagging, budgets, and optimization recommendations.

Technical Leadership & Governance

• Lead platform design reviews and provide technical guidance to Platform Engineers and delivery teams.
• Develop engineering standards, operating procedures, support models, and platform documentation.
• Embed data governance, lineage, classification, access control, and audit requirements into platform capabilities.
• Evaluate platform features and emerging technologies, recommending pragmatic adoption paths aligned with business outcomes.

Who you are?

• 8+ years of experience in cloud platform engineering, DevOps, site reliability engineering, data platform engineering, or related roles.
• Deep hands-on experience with Databricks and Microsoft Azure services, including networking, identity, storage, compute, and security.
• Strong experience with infrastructure as code such as Terraform or Bicep and CI/CD platforms such as Azure DevOps or GitHub Actions.
• Experience designing production-grade observability, incident response, backup, recovery, and operational support capabilities.
• Strong scripting and automation skills using Python, PowerShell, Bash, or comparable languages.
• Working knowledge of lakehouse, data engineering, analytics, and AI workload patterns.
• Proven ability to lead complex technical initiatives, mentor engineers, and influence standards without direct authority.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field; equivalent practical experience will be considered.

Preferred Qualifications

• Experience with Unity Catalog, Microsoft Purview, Microsoft Fabric, MLflow, or AI platform services.
• Experience with policy-as-code, FinOps, container platforms, private networking, and enterprise identity integration.
• Cloud or Databricks certifications.
• Experience supporting globally distributed, business-critical data platforms.

#LI-VK