About the role
- Design, develop, and maintain scalable data pipelines using technologies such as Apache Hadoop, Spark, and Kafka.
- Implement ETL (Extract, Transform, Load) processes to ingest, clean, and process large volumes of data from various sources.
- Optimize data processing workflows to ensure high performance and efficiency.
- Should have strong knowledge on SQL/OLAP databases like Big Query, or similar like Hive.
- Should have strong knowledge on Object Storage like Google Storage, or similar like S3, HDFS.
- Should have strong knowledge on Messaging services like Pub/Sub, or similar like SNS, Kafka maybe RabbitMQ.
- Hands on Experience in Iaas/PaaS service provisioning and automation tools like Terraform.
- Good to have knowledge on Cloud functions, Cloud build/Jenkins, Cloud Networking