About the role
- Lead the design, development, and delivery of enterprise-scale data solutions on the Databricks Lakehouse Platform.
- Provide technical leadership while remaining hands-on with development, troubleshooting, and performance optimization.
- Architect and implement ETL/ELT pipelines using PySpark, Spark SQL, and Databricks.
- Design and build batch and real-time/streaming data pipelines using Structured Streaming, Kafka, or Event Hubs.
- Implement Delta Lake and Medallion Architecture (Bronze, Silver, Gold) data platforms.
- Drive Spark performance tuning, query optimization, cluster sizing, and cost optimization initiatives.
- Lead code reviews, establish development standards, and mentor Data Engineers.
- Integrate data from APIs, cloud storage, databases, and enterprise applications.
- Implement CI/CD, DevOps, monitoring, and data governance best practices.
Collaborate with Architects, Product Owners, and Business stakeholders to define scalable data solutions