About the role
Job Summary
We are looking for an experienced GCP Data Engineer with strong expertise in Python, Advanced SQL, BigQuery, Dataflow, Apache Beam, and Apache Iceberg. The candidate should have hands-on experience designing and developing scalable data pipelines on Google Cloud Platform, along with a good understanding of SSIS and enterprise CI/CD processes/tools.
Key Responsibilities
- Design, develop, and maintain scalable data engineering pipelines on GCP.
- Build batch and streaming data pipelines using Google Cloud Dataflow and Apache Beam.
- Develop high-quality data processing and automation solutions using Python.
- Write and optimize complex SQL queries for large-scale datasets.
- Design and optimize analytical data models and workloads in BigQuery.
- Implement data lake and open-table solutions using Apache Iceberg.
- Perform data ingestion, transformation, validation, and integration across multiple data sources.
- Apply BigQuery optimization techniques including partitioning, clustering, query optimization, and cost management.
- Troubleshoot pipeline failures, data-quality issues, and performance bottlenecks.
- Work with existing SSIS packages and understand legacy ETL/integration workflows where required.
- Implement automated build, testing, deployment, and release processes using internal WF CI/CD tools.
- Collaborate with architects, developers, analysts, and business stakeholders to translate requirements into scalable data solutions.
- Follow enterprise standards for data security, governance, logging, monitoring, and production support.
Required Skills
- Python: Strong hands-on development experience, including data processing and automation.
- Advanced SQL: Excellent knowledge of complex queries, joins, CTEs, window functions, data transformations, and performance optimization.
- GCP Data Engineering: Advanced hands-on experience designing and implementing cloud-based data solutions.
- BigQuery: Strong experience with data modeling, large-scale querying, partitioning, clustering, and performance optimization.
- Dataflow: Hands-on experience building and supporting batch and streaming pipelines.
- Apache Beam: Strong understanding of Beam programming concepts and pipeline development using Python.
- Apache Iceberg: Experience with Iceberg table formats, schema evolution, partitioning, and large-scale data lake architecture.
- SSIS: Good understanding of SSIS-based ETL workflows and packages.
- CI/CD: Experience with enterprise CI/CD processes for build, deployment, and release automation.
- Strong troubleshooting, analytical, and problem-solving skills.
Preferred Qualifications
- 5+ years of overall Data Engineering experience.
- 3+ years of hands-on GCP Data Engineering experience preferred.
- Experience working with high-volume enterprise datasets and production-grade data pipelines.
- Knowledge of data warehousing, data lakes, ETL/ELT patterns, and modern cloud data architectures.
- Experience working in Agile/Scrum delivery environments.
Primary Skill
GCP Data Engineering
Secondary Skills
Python | Advanced SQL | BigQuery | Dataflow | Apache Beam | Apache Iceberg | SSIS | CI/CD