About the role
The senior data engineer will build and maintain data lake and data warehousing solutions for a B2B multi-tenant SaaS solution. The position requires a demonstrated ability to design processes from end to end and a strong technical understanding of modern data pipelines and strategies. In-depth knowledge of multiple data sources, data design patterns, ETL transformation, SQL and performance tuning are required.
Responsibilities:
- In order of significance, list the essential functions and responsibilities of the position.
- Participate in the entire data pipeline lifecycle, focusing on building data lakes, data warehouses and related processes.
- Lead code and design reviews with junior and peer developers.
- Write clean code to develop data pipelines and necessary infrastructure as code.
- Collaborate with cross-functional teams to understand requirements and effectively design appropriate solutions.
- Monitor and troubleshoot performance, scalability, and security issues, making improvements as needed.
- Participate in the evaluation of new technologies and tools that can improve the performance of our system.
- Participate in the data governance process: Contribute to the design and documentation of our data lake platform. Proficient in test automation solutions for data and able to optimise performance.
- Gather technical and design requirements: Communicate effectively with technical and non-technical team members to ensure alignment and successful execution of projects and continuous improvement. Participate in an on-call rotation.
Requirements:
- List applicable knowledge, skills, experience, certificates, licenses, and other criteria required or preferred for this position. (e. g., Management experience required.
- Experience in the development of modern data architecture, analytics, data governance, AI/ML, or related areas.
- Experience with AWS big data technologies, including EMR, Glue, S3 EKS, Lambda, Athena, RDS, MKS/Kafka/Kinesis.
- In-depth knowledge of the entire data engineering process (design, development, deployment).
- Exposure to relational and NoSQL databases such as PostgreSQL, Cassandra, DynamoDB, MongoDB, etc., and data modelling principles.
- Experience with DevOps, including CI/CD pipelines, containerised deployment/Kubernetes, and infrastructure-as-code/AWS CloudFormation/Terraform.
- Proficient in SQL (SQL Server, Postgres) and Python.
- Identify opportunities for process improvement as well as efficiency in solutions.
- Teamwork skills with a problem-solving attitude and a willingness to take a variety of approaches.
- Strong experience with legacy data warehouses and ETL pipelines.
- Demonstrated ability to mentor junior team members.
- Excellent analytical and time management skills, with a proven ability to deliver value independently.
- Strong written and verbal communication skills, with demonstrated experience providing technical input to technical and non-technical stakeholders.
- Ability to participate in on-call rotation.
Preferred:
- Experience with multi-tenant, B2B2C SaaS offerings.
- FinTech industry experience.
- Understanding of MPP Data Warehouse (Redshift, Greenplum).