About the role
Job Summary:
We are seeking a highly skilled GCP Data Engineer with strong expertise in Python, PySpark, and the Google Cloud Platform (GCP) ecosystem, particularly BigQuery and Dataproc. The ideal candidate will play a critical role in designing, building, and optimizing large-scale data processing pipelines and data solutions on GCP.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines using PySpark and Python on GCP.
- Work extensively with BigQuery for data warehousing, querying, and analytics.
- Leverage Dataproc to process large datasets and manage Spark/Hadoop jobs.
- Build ETL/ELT workflows that efficiently ingest, transform, and store data from various sources.
- Optimize performance and cost-efficiency of data pipelines and queries.
- Collaborate with data scientists, analysts, and other engineering teams to deliver data solutions.
- Implement best practices in data engineering, including CI/CD, version control, and monitoring.
- Ensure data quality, consistency, and security across all data systems.
Required Skills and Qualifications:
- 3+ years of experience in data engineering or a similar role.
- Proficiency in Python and experience with PySpark for distributed data processing.
- Strong hands-on experience with Google Cloud Platform (GCP) services, especially:
- BigQuery
- Dataproc
- Cloud Storage, Pub/Sub, Cloud Functions (nice to have)
- Solid understanding of data modeling, data warehousing, and data lake architectures.
- Experience building and managing ETL/ELT pipelines in a cloud environment.
- Familiarity with DevOps practices, CI/CD, Git, and infrastructure-as-code (Terraform is a plus).
- Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment.
Preferred Qualifications:
- GCP Professional Data Engineer certification.
- Experience with additional GCP tools such as Cloud Composer, Dataflow, or Looker.
- Knowledge of data governance, security, and compliance standards.