About the role
The candidate will have responsibilities across the following functions:
Data Warehousing and BigQuery Development:
- Design and implement data warehouse solutions using Google BigQuery.
- Develop, optimise, and maintain complex SQL queries and data models.
- Create efficient partitioning, clustering, and indexing strategies.
Data Pipeline Development:
- Build scalable data pipelines using GCP services (Dataflow, Dataproc, Cloud Composer).
- Perform data ingestion from various sources (APIs, databases, cloud storage).
- Ensure data transformation, cleansing, and validation.
Performance Optimisation:
- Optimise BigQuery queries for cost and performance.
- Monitor query execution and tune workloads.
- Implement best practices for storage and compute efficiency.
Integration and ETL/ELT:
- Implement ETL/ELT workflows using BigQuery, Cloud Storage, and Pub/Sub.
- Integrate data from multiple cloud and on-prem systems.
- Automate workflows and scheduling.
Collaboration and Reporting:
- Work closely with data analysts, data scientists, and business teams.
- Support reporting tools like Looker, Data Studio (Looker Studio), and Tableau.
- Translate business requirements into scalable data solutions.
The core requirements for the job include the following:
Core Technical Skills:
- 3-5 years of experience in data engineering / BigQuery development.
- Strong expertise in SQL (mandatory).
- Hands-on experience with Google BigQuery.
- Experience with data modelling (star/snowflake schemas).
GCP Tools and Services:
- BigQuery
- Cloud Storage
- Dataflow / Dataproc
- Pub/Sub
- Cloud Composer (Airflow)
Programming Skills:
- Proficiency in Python / SQL scripting.
- Experience with Spark (PySpark) is a plus.