About the role
We are looking for a highly skilled GCP Data Engineer with strong experience in building scalable data pipelines and working on Google Cloud Platform services. The ideal candidate will have hands-on expertise in Dataproc, Dataflow, BigQuery, Cloud Functions, and strong SQL skills to handle large-scale data processing and transformation.
Responsibilities:
- Design, develop, and maintain end-to-end data pipelines on GCP.
- Build and manage batch and streaming data pipelines using Dataflow (Apache Beam).
- Develop and optimise Dataproc-based Spark/Hadoop workflows.
- Perform data transformation, cleansing, and aggregation using SQL and BigQuery.
- Create and manage serverless workflows using Cloud Functions.
- Work with Cloud Composer (Airflow) for pipeline orchestration.
- Handle data ingestion from multiple sources (APIs, databases, files, streams).
- Ensure data quality, validation, and reconciliation across systems.
- Optimise performance of pipelines and queries for large-scale datasets.
- Collaborate with cross-functional teams (Data Scientists, Analysts, Business teams).
Requirements:
- Strong experience in GCP Data Engineering.
- Hands-on with: Google Cloud Dataproc, Google Cloud Dataflow (Apache Beam), BigQuery, Cloud Functions.
- Strong proficiency in SQL (complex queries, performance tuning).
- Experience working with large datasets and distributed data processing systems.
- Good understanding of ETL/ELT pipelines.
- Experience with batch and streaming data processing.
Preferred Skills:
- Experience with Cloud Composer (Airflow).
- Knowledge of Pub/Sub for streaming ingestion.
- Exposure to data lake/data warehouse architecture.
- Experience with Python / PySpark.
- Understanding of CI/CD pipelines and DevOps practices.
- Data governance, security, and compliance knowledge.