About the role
The core responsibilities for the job include the following:
Data Warehousing and BigQuery Development:
- Design and implement data warehouse solutions using Google BigQuery.
- Develop, optimise, and maintain complex SQL queries and data models.
- Create efficient partitioning, clustering, and indexing strategies.
Data Pipeline Development:
- Build scalable data pipelines using GCP services (Dataflow, Dataproc, Cloud Composer).
- Perform data ingestion from various sources (APIs, databases, cloud storage).
- Ensure data transformation, cleansing, and validation.
Performance Optimisation:
- Optimise BigQuery queries for cost and performance.
- Monitor query execution and tune workloads.
- Implement best practices for storage and compute efficiency.
Integration and ETL/ELT:
- Implement ETL/ELT workflows using BigQuery, Cloud Storage, and Pub/Sub.
- Integrate data from multiple cloud and on-prem systems.
- Automate workflows and scheduling.
Collaboration and Reporting:
- Work closely with data analysts, data scientists, and business teams.
- Support reporting tools like Looker, Data Studio (Looker Studio), and Tableau.
- Translate business requirements into scalable data solutions.
Requirements:
- 3-5 years of experience in data engineering / BigQuery development.
- Strong expertise in SQL (mandatory).
- Hands-on experience with Google BigQuery.
- Experience with data modelling (star/snowflake schemas).
- GCP Tools and Services: BigQuery, Cloud Storage, Dataflow / Dataproc, Pub/Sub, Cloud Composer (Airflow).
Programming Skill:
- Proficiency in Python / SQL scripting.
- Experience with Spark (PySpark) is a plus.