About the role
As our Senior Data Engineer, you will be a key driver of our data engineering strategy, focusing on the design, implementation, and maintenance of our data infrastructure on Google Cloud Platform (GCP). You will leverage your expertise in GCP services (Data Engineering Pipeline, BigQuery, DataFlow etc.) and related technologies to build and enhance our data pipelines and platforms, ensuring they are scalable, reliable, and adaptable to our evolving needs. Responsibilities Your responsibilities will include: • Platform Migration and Modernization: Lead setting up of data in GCP, ensuring data integrity and designing for scalability and performance. • Pipeline Development and Enhancement: Design, develop, and maintain robust and scalable data pipelines using GCP services (BigQuery, DataFlow, Batch processing etc.). This includes creating reusable components, implementing best practices, and ensuring the pipelines' maintainability and extensibility. Experience with automated data lineage is essential. • Data Governance and Security: Implement and maintain data governance and security best practices within the GCP environment, ensuring compliance with relevant regulations. • CI/CD Integration: Integrate data pipelines into our continuous integration and continuous delivery (CI/CD) pipeline. • Performance Optimization: Continuously monitor and optimize the performance of data pipelines and platforms to ensure fast and reliable data processing. • Troubleshooting and Debugging: Identify, troubleshoot, and resolve issues related to data pipelines and platforms, providing timely solutions to ensure continuous data flow. • Reporting and Analysis: Generate reports on data pipeline performance and data quality, analyzing results to identify trends and areas for improvement.
Skills Required:
Google Cloud Platform - Biq Query, Data Flow, Dataproc, Data Fusion, TERRAFORM, Tekton,Cloud SQL, AIRFLOW, POSTGRES, Airflow PySpark, Python, API
Skills Preferred:
Big Query,, AIRFLOW
Experience Required:
Engineer 3 Exp: Prac. In 2 coding lang. or adv. Prac. in 1 lang. 6+ years in IT; 4+ years in development
Experience Preferred:
Experience in GCP, Big-Query, Astronomer and exp in any ETL technology
Education Required:
Bachelor's Degree