About the role
Responsibilities:
- Design and develop data processing solutions using various ETL tools for data ingestion, cleansing, transformation, and job orchestration in the Azure cloud environment.
- Implement robust data security with role-based access control, data masking, and encryption for sensitive data.
- Monitor and optimise data storage and data processing performance using monitoring tools, along with implementing archive strategies and query tuning.
Requirements:
- Hands-on experience with Synapse Pipelines, Spark Pools, Synapse SQL Pools, and SQL Managed Instance
- 4-6 years of experience in a data engineering role, with at least 3-5 years of hands-on experience in Azure cloud technologies.
- Strong expertise in Azure Synapse Data Pipelines (knowledge of ADF is also valuable).
- Proficiency in PySpark, Spark SQL, and T-SQL.
- Familiarity with Azure Functions App (coded in Python) is a plus.
- Deep understanding of cloud technologies and the ability to deliver high-quality, large-scale projects on time.
- Excellent problem-solving skills and a passion for data engineering in the cloud.