About the role
We are seeking a highly skilled Azure Data Engineer with experience in designing, developing, and maintaining modern cloud-based data platforms. The ideal candidate should have strong expertise in Azure Data Factory, Azure Databricks, Spark, SQL, Python, data warehousing, and DevOps practices. The role requires hands-on experience in building scalable data pipelines, implementing Lakehouse architecture, and delivering enterprise-grade data solutions.
Responsibilities:
- Design and develop scalable data ingestion, transformation, and orchestration solutions on Azure.
- Build and maintain batch and streaming data pipelines using Azure Databricks and Azure Data Factory.
- Implement Lakehouse architecture using Delta Lake and Azure Data Lake Storage Gen2
- Develop high-performance data transformation logic using Spark DataFrame APIs and Spark SQL.
- Optimize data processing workloads through caching, persistence, partitioning, and performance tuning techniques.
- Collaborate with architects, analysts, and business stakeholders to deliver data solutions.
- Implement CI/CD and source control best practices using Git and branching strategies.
- Support production deployments, troubleshooting, and operational excellence.
Requirements:
- Strong problem-solving and analytical skills.
- Ability to perform coding assessments in Spark SQL, SQL, and Python.
- Experience working in Agile delivery environments.
- Excellent communication and stakeholder management skills.
- Ability to mentor junior engineers and drive technical excellence.
Mandatory Technical Skills:
- Azure Data Factory (ADF): Orchestration, ingestion from on-premises and cloud sources, triggers, linked services, and Integration Runtime (IR).
- Azure Databricks: Data ingestion from ADLS Gen2 Delta Lake, CRUD operations, time travel, optimization, merge operations, streaming, and Auto Loader.
- Apache Spark: Data transformation using DataFrame APIs, Spark SQL, caching, persistence, and performance optimization.
- Data Warehousing: Strong understanding of Slowly Changing Dimensions (SCD) and data loading strategies.
- SQL: Advanced SQL development and coding expertise. (Window Function/CTE)
- Python: Data handling using Pandas, object-oriented programming concepts, classes, and objects.
- DevOps: Git usage, source code management, and branching strategy awareness.
Good to Have Skills:
- ADF Data Flows.
- Dimensional Modelling (Facts and Dimensions).
- Real-time analytics and event-driven architecture.
- Exposure to Azure Synapse Analytics and related Azure data services.