About the role
Responsibilities:
- Lead and contribute to the development of a self-service data platform for reporting and analytics.
- Design and build data solutions using Databricks, SQL, Python, Spark, and Delta Lake within the Azure ecosystem (Blob Storage, Data Factory, Event Hubs).
- Implement best practices for ETL/ELT processes, focusing on data quality management, partitioning, maintainability, and reusability.
- Collaborate with engineers, product managers, and business leaders to ensure seamless integration of the data platform with systems such as Tableau, Power BI, APIs, and other applications.
- Establish CI/CD pipelines and use infrastructure-as-code tools for deployment and monitoring (Git, Terraform, Azure DevOps/GitHub Actions/Jenkins, Azure Monitor/Datadog).
- Ensure adherence to organizational compliance standards and the Code of Conduct as defined in LogixGarden.
Requirements:
- 8+ years of data engineering experience.
- 3+ years in a senior or principal engineer role, with proven experience leading projects.
- Strong hands-on experience with Databricks and related integrations (Delta Live Tables, Auto Loader, Databricks Workflows / Apache Airflow, Unity Catalogue).
- Expertise in PySpark and/or Scala.
- Strong DataFrame programming experience (Spark, Pandas, NumPy).
- Proven ability to design scalable, secure, and maintainable data pipelines.
- Excellent communication and leadership skills, with a passion for mentoring.
- Solid understanding of data security and governance frameworks, with exposure to HIPAA or SOC 2 compliance preferred.
Preferred Skills:
- AWS cloud experience.
- AWS Databricks implementation expertise.
- Experience designing and implementing data security and governance frameworks.