About the role
Responsibilities:
- Responsible for leading a team of talented data engineers responsible for designing, building, and maintaining scalable data pipelines and infrastructure.
- Work closely with cross-functional teams to ensure client data systems meet the highest standards of quality and performance.
- Lead, mentor, and develop a team of data engineers, fostering a collaborative and inclusive team environment.
- Conduct regular performance reviews, provide feedback, and set goals for team members.
- Identify and address skill gaps, and provide opportunities for professional development.
- Plan, execute, and deliver data engineering projects on time and within scope.
- Coordinate with stakeholders to gather requirements, set priorities, and define project timelines.
- Ensure projects align with overall business objectives and data strategy.
- Oversee the design, development, and maintenance of data pipelines, ETL processes, and data warehouse.
- Ensure data quality, integrity, and security across all data engineering projects.
- Identify opportunities for process improvements and drive initiatives to enhance the efficiency and effectiveness of data operations.
Requirements:
- Has a strong conceptual understanding of Data Warehousing and, ETL, and data.
- Governance and Security, Cloud Computing, and Batch and Real-Time data processing.
- Ability to build/drive reusable frameworks that can drive efficiency of the overall data system.
- Has executed and lead multiple projects, including streaming, batch, large data pipelines, etc.
- Manages conversations with the client stakeholders to understand the Internal Use Only requirement and translate it into technical outcomes.
Required Tech Stack:
- Strong experience with Databricks, Spark, and cloud platforms (Azure, AWS, GCP).
- Architect and deploy cloud-based data solutions (Azure, AWS, GCP).
- Define CI/CD strategies for data pipelines using Terraform, Azure DevOps, or GitHub Actions.
- Implement data cataloguing, lineage tracking, and access control (Unity Catalogue, Collibra, Alation).
- Ensure compliance with GDPR, CCPA, and industry-specific data security policies.
- Develop strategies for distributed computing, parallel processing, and caching mechanisms.
Required Non-Tech Stack:
- Partner with data architects, product managers, and business leaders to define data requirements and align engineering efforts with business objectives.
- Define data engineering standards and playbooks to streamline development.
- Oversee end-to-end project execution, from scoping to delivery.
- Stay updated with emerging trends in data engineering, AI, and analytics to continuously improve architectures.
- Evaluate and recommend new data tools, frameworks, and best practices.
- Ability to translate complex technical concepts into business-friendly language.
- Excellent communication, leadership, and stakeholder management.
Good to Have Tech Stack:
- Experience with machine learning and advanced analytics technologies.
- Familiarity with data visualisation tools and techniques.
- Knowledge of data security and privacy practices.
- Understanding of data governance and compliance frameworks.
- Experience with containerization and orchestration technologies (e. g., Docker, Kubernetes).
- Experience with graph databases and graph processing frameworks.
- Experience with data virtualisation and data federation techniques.
- Proficiency in data profiling and data quality management.