About the role
Responsibilities:
- Design, develop, and maintain robust data pipelines using Airflow, DBT, and Python within a BigQuery environment.
- Implement and maintain data integrations with various sources and targets using tools like Airbyte, Elementary, and Metaflow, with a focus on optimizing data storage and retrieval in BigQuery.
- Write advanced SQL queries and scripts for data manipulation and retrieval in BigQuery.
- Develop and optimize data models and schemas in Preset and other visualization tools, tailored for performance in BigQuery.
- Utilize Pyspark for big data processing tasks, ensuring compatibility with BigQuery's infrastructure.
- Work with APIs, Docker, and GCP services to enhance and automate data workflows, particularly focusing on integration with BigQuery.
- Apply best practices in continuous integration and continuous deployment (CI/CD) environments using testing and pipeline validation techniques.
- Employ asynchronous architectures, queues, caching strategies, and distributed computing to enhance system performance.
- Write custom code in Python and utilize Jinja within DBT for dynamic pipeline generation, specifically designed for efficiency in BigQuery.
- Adopt design patterns and engineering best practices to ensure code scalability and maintainability.
Key outcomes:
- Develop a Unified Data Foundation: Spearheaded the creation of a single source of truth that centralizes our data assets, ensuring consistency and reliability across the board.
- Drive Impact Across Teams: Your work will directly influence the success of our business analysts, data scientists, and machine learning engineers by providing them with the crucial data they need to excel.
- Optimize Business Logic: Design and manage data tables that support and enhance business logic functions, delivering tailored data solutions that drive strategic decision-making.