About the role
We are looking for an experienced ETL / Data Engineer with strong hands-on expertise in Informatica, Hadoop, Apache Spark, and Snowflake. The candidate will be responsible for designing, developing, and maintaining scalable data pipelines and ETL processes, integrating data from multiple sources into enterprise data platforms. The ideal candidate should have strong SQL and data warehousing knowledge, experience working with large datasets, and the ability to troubleshoot and optimise data pipelines.
Responsibilities:
- Design, develop, and maintain robust ETL/ELT data pipelines using Informatica and related technologies.
- Develop and optimise data processing jobs using Apache Spark / PySpark.
- Work with Hadoop ecosystem technologies such as HDFS, Hive, and related components.
- Develop data ingestion and transformation pipelines into Snowflake.
- Write complex SQL queries, stored procedures, and data transformation logic.
- Perform data extraction from multiple sources, including relational databases, flat files, APIs, and cloud platforms.
- Implement data cleansing, validation, transformation, and reconciliation processes.
- Develop scalable and reusable ETL workflows following data engineering best practices.
- Monitor production data pipelines and troubleshoot failures, performance issues, and data quality problems.
- Optimise Spark jobs, SQL queries, ETL workflows, and Snowflake workloads for performance and cost.
- Work closely with business analysts, data architects, developers, and other stakeholders to understand data requirements.
- Participate in data migration, modernisation, and platform transformation initiatives.
- Ensure data security, quality, governance, and reliability across data pipelines.
- Create and maintain technical documentation for ETL processes, mappings, workflows, and data pipelines.