About the role
Roles & Responsibilities
Owns the end-to-end tHands-on implementer responsible for building and testing individual pipelines: Datastream CDC/OTL configuration, Cloud Composer DAGs, PySpark-to-BigQuery SQL conversions, materialized view creation, and report connector reconfiguration. Works under a Tech Lead within a pod, executes the implementation plan, and produces evidence (logs, recon outputs) for QA sign-off.echnical design for the ClickHouse-to-BigQuery migration: medallion architecture (Bronze/Silver/Gold), Datastream/CDC topology, schema-drift handling strategy, and the Technical Design Document (TDD). Acts as the single point of technical authority during the 8-week engagement, signs off on architecture decisions, and resolves cross-pod design conflicts (Postgres, MongoDB, PySpark, reporting).
Primary Skillset
• Configuring and operating Google Cloud Datastream streams (one-time load + CDC) for PostgreSQL (Cloud SQL) and/or MongoDB Atlas sources into BigQuery• Writing and maintaining Cloud Composer (Airflow) DAGs for scheduled/micro-batch ingestion, using parameterised/templated patterns• Converting PySpark batch jobs to BigQuery SQL — using BQ-native Spark-to-SQL conversion utilities for standard jobs, and manually re-implementing complex UDFs/window functions where required• Writing BigQuery SQL for Bronze/Silver/Gold transformations: cleansing, deduplication (via BQ Data Engineering Agent or SQL), and materialized view definitions• Implementing Dataflow (Apache Beam) jobs to flatten nested MongoDB documents into BigQuery tables, handling document schema variability (SAFE/COALESCE patterns)• Configuring BigQuery native features: column-level PII masking policies, scheduled exports to GCS (including Iceberg format), and basic Cloud Monitoring/Audit Logging setup
Secondary Skillset
• Reconfiguring BI tool connectors (e.g., Metabase JDBC connector switch from ClickHouse to BigQuery) and validating report output• Writing automated reconciliation scripts (row-count, checksum comparisons between source and BigQuery)• Exposure to ClickHouse (reading schemas/data for migration purposes — not requiring ClickHouse admin expertise)• Shell scripting / Python for automation glue (recon scripts, batch reconfiguration helpers)• Git-based version control and CI basics for pipeline code• Awareness of data security/compliance concepts (PII handling, audit logging) sufficient to implement to spec