Databricks - Solution Architect

Role

We are hiring a Databricks Architect with deep expertise across the Databricks Lakehouse Platform to own the organization's data architecture — current state and target state — and provide technical oversight of the implementation delivered, signing off on its design and quality.

Key Responsibilities

• Own and maintain the current and target-state Databricks Lakehouse architecture: Delta Lake structures, Unity Catalog metastore/catalog design, Workflows/job orchestration, Delta Live Tables pipelines, and cluster/workspace topology.
• Define and govern Unity Catalog structure — catalogs, schemas, access-control models (RBAC/ABAC, row/column-level security), data lineage, and audit logging.
• Review pipeline and notebook architecture (DLT, Workflows/Jobs, orchestration patterns) for scalability, reliability, and maintainability.
• Set and enforce cluster policies, sizing, and autoscaling standards; monitor DBU consumption, Photon/serverless usage, and job efficiency to control platform cost.
• Define the platform's security architecture: encryption, network isolation (VNet/VPC peering, private link), IAM integration, and compliance alignment.
• Establish CI/CD and IaC standards for Databricks — Asset Bundles, Repos/Git integration.
• Design and evolve the data governance and metadata strategy (lineage, cataloging, quality checks) across the Lakehouse.
• Guide integration patterns with upstream/downstream systems — source ingestion, and consumption layers for BI, ML, and APIs.
• Lead performance tuning and troubleshooting for Spark jobs, query optimization, and workload management.
• Maintain architecture documentation, design decision records, and the platform roadmap.
• Review and sign off on deliverables from the implementation vendor to confirm they meet these architecture, security, and performance standards.

Required Skills and Experience

• 10-15 years of overall IT experience, with deep hands-on and architecture-level expertise across the Databricks Lakehouse Platform: Delta Lake, Unity Catalog, Delta Live Tables, Workflows, DBSQL, Photon, and MLflow.
• Strong command of at least one hyperscaler running Databricks (Azure, AWS, or GCP).
• Hands-on depth in Unity Catalog governance: catalog/schema/table design, access-control models, data lineage, and audit logging.
• Strong grasp of Spark internals, performance tuning, and cluster/job optimization.
• Working knowledge of Databricks CI/CD and IaC tooling — Asset Bundles, Repos/Git integration.
• Solid data engineering fundamentals: ETL/ELT pipeline design, data modeling, and orchestration.
• Experience with DBU-based cost governance/FinOps for Databricks.
• Excellent communication skills to work with cross-functional stakeholders and external delivery partners.
• Preferred: Databricks Certified Data Engineer Professional or Databricks Certified Solutions Architect; a cloud architecture certification (Azure/AWS).