About the role
The individual is highly skilled and experienced Databricks expert to design and development of endtoend data products This job will require expertise in developing and building data products in databricks using pysparkThe candidate will have a strong background in data engineering a deep understanding of cloud data stacks and a proven track record of delivering robust and scalable data solutions
Skills
Lead the design development and deployment of data products on the Databricks platform
Implementing maintaining and optimizing complex data pipelines to process large amounts of structured unstructured datasets
Proficient in Databricks Lakehouse and Apache Spark Framework Highly skilled in PythonPySpark and SQL
Experienced in developing data solutions on cloud platformspreferably AWS
Establish and advocate for best practices in data engineering principles and standards including Code quality data validation governance and security controls
Closely collaborate and engage with business stackholdersProduct owners to prepare and coordinate data sharingconduct workshops ensure data products developed should meet the business needs and provide actionable insights
Mentor and guide junior data engineers providing technical leadership and expertise
Comfortable in Agile Processes
Problem solving skills and hands on attitude and strong interpersonal and communication skills
Tools
Extensive experience with Databricks solutions PySpark Spark SQL and cloud data platforms preferably AWS Cloud
Strong fundamental knowledge of ETL ProcessesDatawarehousing DataLakes and Data Modeling
Proficiency in working with AWS or Azure cloud platforms including services like S3 Lambda Glue Azure Data Lake Azure SQL Data Warehouse etc
Handson experience with version control systems GitBitbucket and CICD pipelines
Excellent problemsolving skills and ability to troubleshoot complex data issues
Experience working with Agile methodologies Scrum Kanban
Strong communication and collaboration skills with the ability to work effectively in a team environment
Should Have
Databricks Pyspark Python Data Management ETL Design
Basic knowledge on Alteryx tools
Atlassian Tools JIRA Confluence
Agile Scrum methodology
Soft Skills
Process Oriented
Excellent verbalwritten communication skills including communicating technical issues to nontechnical audiences and presentation skills
Experience with team management