About the role
Responsibilities:
- Lead and focus on scaling the data infrastructure, streamlining the data delivery, and designing and implementing AI automation to replace manual operations.
- Engage closely with a wide array of stakeholders, including executive, product, data, and design teams, providing high-level support for data infrastructure challenges, advising on technical data issues, and implementing self-serve solutions.
- Develop sophisticated AI-based analytical tools that integrate with the data pipeline to provide deep insights into key business metrics, including customer growth and operational efficiency.
- Architect and manage extensive, sophisticated data sets to meet complex business needs and requirements.
- Engage closely with a wide array of stakeholders, including executive, product, data, and design teams, providing high-level support for data infrastructure challenges and advising on technical data issues.
- Make a meaningful impact on the lives of our community members.
- Collaborate and mentor other senior engineers while providing thoughtful guidance using code, design, and architecture reviews.
- Contribute to defining technical direction, planning the roadmap, escalating issues, and synthesising feedback to ensure team success.
- Estimate and manage team project timelines and risks.
- Participate in hiring and onboarding for new team members.
- Lead cross-team engineering initiatives.
- Constantly lEarnIng about new technologies and industry standards in data engineering.
Requirements:
- 4+ years of experience in designing, building, and maintaining data infrastructure, and the ability to lead complex projects.
- Bachelor's, Master's, or PhD degree in computer science, computer engineering, or a related technical discipline, or equivalent industry experience.
- Deep knowledge of Kafka, Proficiency in programming languages like Python and Scala, and a good understanding of ML engineering.
- Strong knowledge of distributed computing frameworks, such as Apache Spark, with cloud platforms such as AWS, Azure, or GCP.
- Working experience with Databricks would be nice to have.
- Experience using AI-assisted development tools (e. g., GitHub Copilot, Cursor, ChatGPT, or similar tools) as part of your software development workflow.
- Deep understanding of database design, SQL, and NoSQL databases.
- Experience in managing large datasets and optimising database performance.
- Proficiency in Git and Terraform, and experience in deploying continuous integration and continuous deployment (CI/CD) practices.
- Experience in managing event-driven systems, particularly with Kafka in cloud environments.
- Expertise in developing and implementing data governance frameworks, policies, and procedures to ensure data quality, compliance, and effective data management practices.
- Deep understanding of data security principles, including encryption, decryption, and secure data storage and transfer protocols.