About the role
As the Lead, Data Engineering, you will play a pivotal role in designing and prototyping cutting-edge solutions that empower our operations with timely and relevant data. You'll explore new tools and build cost-effective solutions using the powerful Snowflake and AWS Data Platform. In this position, you will also lead and inspire our cross-vendor Data Engineering team, driving key initiatives that shape the future of our organisation.
Responsibilities:
- Develop, customise, and manage integration tools, databases, warehouses, and analytical systems.
- Develop data pipelines from internal and external data sources to support new product launches and drive data quality across data products.
- Build automation and monitoring frameworks that capture metrics and operational KPIs for data pipeline quality and performance.
- Assist in implementation of best practices around systems integration, security, performance and data management.
- Partner with the business, IT, and technical subject matter experts to develop robust solutions.
- Implement DevOps, DataOps and Agile methodologies to improve KPIs like cycle times, consistency, and quality.
- Monitor the overall performance and stability of the system; adjust and adapt automated pipeline as data, models, and/or requirements change.
- Test the reliability and performance of data engineering pipelines and support the testing team with data validation activities.
- Partner with various Snowflake Engineering subject matter experts, including vendor partners, project managers and business teams, to scope and build customer-facing content, modules, tools and proof of concepts.
- Research and cultivate state-of-the-art data engineering methodologies to develop into a Snowflake technology stack developer.
Requirements:
- Bachelor's degree in engineering, computer science, or related field.
- 3 - 5 years of total work experience, with at least 4 years of experience in Python /Snowflake Data Engineering tool stack.
- At least one key data engineering professional certification (e. g., SnowPro Associate and Core).
- Experience with data management tools such as Airflow, Airbyte and DBT.
- Good scripting, data modelling, and programming skills.
- Understanding of cloud architecture principles and best practices.
- Experience in the design and build of end-to-end solutions that meet business requirements and adhere to scalability, reliability, and security standards.
- Familiarity with version control systems such as GitHub, GitHub Actions and DevOps practices for CI/CD pipelines.
Desired Qualifications:
- Subject matter expert for Snowflake Data, Analytics & AI with experience Snowflake architecture design is preferred.
- Good understanding of migration and modernisation strategies and approaches is preferred.
- Experience with Python, EKS Implementations, Spark, Kafka, and/or Flask is preferred.