About the role
Role: Ensures data accuracy, consistency, and integrity across data pipelines and ETL processes.
Skills:
i. Proficiency in analyzing data from different sources and validating transformations.
ii. Strong understanding of data structures and relational databases.
iii. Ability to write complex SQL queries for-
· Data Comparison (Source vs. target).
· Aggregation and integrity checks.
· Data reconciliation during migrations.
iv. Familiarity with ETL processes to validate data pipelines.
v. Experience with tools Azure Data Factory, Azure Synapse.
vi. Knowledge of data profiling, data validation, and data cleansing methods.
vii. Familiar with automation in data validation, potentially using tools like QuerySurge.
viii. Understanding of data quality metrics and governance frameworks to ensure compliance during migrations.
ix. Ability to validate Data Lineage, and business rules.
x. Experience working with cross -functional teams, including developers, DBA's and BA, to ensure data validation success.
xi. Proficient in documenting data validation plans, test cases, and results for audits and reviews.
Ideal Candidate Profile:
- Strong experience in data validation, ETL testing, and test data management.
- Proficient in Python, Azure Data Factory, and Synapse (or comparable tools like Informatica).
- Background in digital transformation, particularly cloud migrations.
- Ability to work independently, take ownership, and thrive in a fast-paced setting.
Key Responsibilities & Requirements:
- Data Flow & Validation:
- Understand how data flows from one system to another.
- Experience in data validation and ensuring data integrity across systems.
- Strong understanding of ETL processes, particularly from a data validation perspective.
- Ability to analyze ETL flows and improve existing processes.
- Test-Driven Development (TDD):
- Experience with TDD methodologies is required.
- Test Data Management & Mocking:
- Candidates should understand test data management best practices.
- Ability to mock up datasets for testing purposes.
- Hiring manager noted that previous candidates struggled with this concept.