About the role
Responsibilities:
- Design, build, and maintain scalable batch and streaming data pipelines on GCP.
- Ingest data from multiple sources, including traditional databases and API-based integrations, into our cloud data platform.
- Develop data and feature engineering workflows that support analytics, BI, and machine learning use cases.
- Orchestrate and schedule pipelines with tools such as Cloud Composer/Airflow and Vertex AI Pipelines.
- Contribute to data architecture design, choosing the right storage and compute options for each use case.
- Optimize pipeline and cluster performance for reliability, scalability, and cost.
- Package and deploy data workloads with Docker and CI/CD practices.
- Work with data scientists, analysts, and business teams to understand how data is used and ensure it is accurate, available, and fit for purpose.
Requirements:
- Strong programming skills in Python, SQL, and PySpark.
- Hands-on experience with at least one cloud platform (GCP preferred).
- Experience working with Docker.
- Experience managing environments and dependencies (e. g., venv, pip).
- Experience with orchestrators such as Vertex AI Pipelines or Airflow.
- Solid knowledge of data engineering and feature engineering techniques.
- Proficiency in at least one of Apache Spark, Apache Beam, or Apache Flink.
- Understanding of streaming concepts such as windowing, late-arriving data, and triggers.
Should have:
- Hands-on experience with distributed computing.
- Working experience in data architecture design.
- Knowledge of storage and compute options and when to choose each.
- Good understanding of cluster and pipeline optimization strategies.
- Experience using GCP tools to build end-to-end data pipelines for different scenarios, including ingestion from traditional databases and API-based sources.
- Experience with CI/CD pipelines and deployment methodologies.
- A business mindset: understanding the data and how it will be used for BI and analytics.
- Good to have: Hands-on experience with Kubernetes.
Tools and Technologies (GCP):
- Storage: Cloud SQL, Cloud Storage, Cloud Bigtable, BigQuery.
- Ingestion: Pub/Sub, Cloud Functions, Google Kubernetes Engine (GKE).
- Scheduling and Orchestration: Cloud Composer, Airflow, and Vertex AI Pipelines.
- Processing: Cloud Dataproc, Cloud Dataflow, Apache Spark.
- DevOps: Docker, CI/CD (e. g., GitHub/GitLab Actions).