About the role
Roles and Responsibilities:
Architecting and designing analytics platform solution for clients
Identifying data sources and defining data pipeline, segmentation, and performance engineering for
scale
Managing team of Data Analyst/ Data Engineers assigning work and reviewing quality of
deliverables.
Strong understanding of data modeling, data warehousing concepts and various analytical data models
Familiarity with business domains such as healthcare
Migration experience highly preferred
Azure Cloud experience highly preferred
Perform research with new technologies and third-party vendor technologies as required and
prototyping Design, develop and implement analytics based on the discussions with the team.
Programming skills in generic analytic platforms using Python/ Java/ Scala
Responsibilities:
Design, develop, and implement data processing pipelines using Databricks.
Administer and maintain Databricks clusters, ensuring optimal performance, reliability, and security.
Collaborate with cross-functional teams to deploy, configure, and manage Databricks environments.
Implement and enforce security best practices for Databricks workspaces, including access controls
and data encryption.
Monitor system health, troubleshoot issues, and perform routine maintenance tasks.
Work closely with data engineers and data scientists to understand workload requirements and
optimize cluster configurations.
Collaborate with IT teams to integrate Databricks with other enterprise systems.
Keep abreast of Databricks updates, patches, and new features, and implement them as needed.
Provide technical support and mentorship to junior administrators.
Develop and maintain ETL processes, ensuring data accuracy and consistency.
Work with business stakeholders to understand analytics requirements and translate them into
Databricks workflows.
Implement and maintain version control for Databricks notebooks and code.
Troubleshoot and debug Databricks jobs and provide solutions to performance issues.
Stay updated on Databricks best practices and incorporate them into development processes.
Qualifications:
Bachelor's degree in computer science, Information Technology, or a related field.
Proven experience as a Databricks Developer or similar role.
Proficient in Apache Spark, Databricks notebooks, and related technologies.
Strong programming skills in languages like Python, Scala, or SQL.
Experience with data modeling, ETL processes, and data warehousing concepts.
Familiarity with cloud platforms and integration with Databricks.
Excellent problem-solving, analytical, and communication skills.