About the role
- Design, develop, and maintain scalable data ingestion and transformation pipelines using Azure Synapse Pipelines, Spark, SQL, and Azure Data Factory.
- Implement incremental data processing using watermarking, Change Data Capture (CDC), Change Tracking, or timestamp-based mechanisms.
- Develop and optimize Dedicated SQL Pool and Serverless SQL Pool solutions for large-scale data warehousing.
- Perform DWU right-sizing based on workload patterns to balance performance and cost.
- Identify and implement cost optimization opportunities, including workload scheduling, pause/resume automation, storage optimization, and query tuning.
- Optimize SQL queries, partitioning strategies, indexing, and data distribution (Hash, Round Robin, Replicated tables) to improve performance.
- Design and maintain dimensional data models (Star/Snowflake schemas) for reporting and analytics.
- Monitor pipeline execution, troubleshoot failures, and ensure high availability and reliability.
- Implement CI/CD pipelines using Azure DevOps for deployment automation.
- Collaborate with business stakeholders, architects, and BI teams to deliver scalable analytics solutions.
- Ensure data quality, governance, security, and compliance across the data platform.
- Document technical designs, operational procedures, and best practices.