About the role
We are looking for an experienced Data Engineering Lead who can join us as part of the core team and build Zluri with us.
Responsibilities:
- Build, maintain, and optimize the data pipeline architecture at Zluri.
- Develop ETL pipelines based on the requirements following best practices, specifications, and standards.
- Work with project teams to provide input and assistance in the design, development, and support of ETL and database components following the standard software development lifecycle.
- Work with the team to ensure that ETL and database components conform to global standards via thorough code reviews.
- Mentor junior programmers in the team and plan their skill development.
- Determine the technical needs for the team to work on an assigned project or third-party tools.
- Documenting the data requirements, data format, and tool dependencies.
- Flexibility and ability to adapt to new technological changes as per project need.
Requirements:
- 5+ years of experience in solving multi-dimensional data problems.
- Knowledge of multiple cloud storage options like S3 Redshift, or BigQuery is a must.
- Should be well versed with MySQL and NoSQL.
- Knowledge of Pandas and Spark is a must. Knowledge of working with DBT is a plus.
- Knowledge of Python, SQL, and shell script is a must, and Javascript (Node.js ) is good to have.
- Knowledge of streaming services like AWS Streams/Kafka/Google Pub/Sub is good to have.
- Knowledge of queue systems like RabbitMQ and SQS and building pipelines using the same is very much needed.
- Knowledge of Docker and Kubernetes is good to have.
- Familiarity with multiple cloud providers (AWS, Azure, and GCP) is good to have.
- Should be familiar with strategies for real-time analytics and batch processing.
- Master's or bachelor's degree in computer science or the equivalent in experience.