About the role
- Perform source-to-target reconciliation for event/transaction-style datasets across ingestion and curated layers.
- Validate completeness, timeliness, and consistency of daily/near-real-time ingestion pipelines.
- Identify and investigate data anomalies (missing records, spikes/drops, duplicates, out-of-range values, schema drift).
- Validate correctness of KPIs and metrics used in dashboards and reports (e.g., volumes, rates, turnaround times, error trends).
- Ensure referential integrity across key entities (e.g., events/encounters, units/resources, locations, agencies/sites).
- Partner with engineering teams to debug discrepancies, confirm root causes, and validate fixes.
- Build and maintain automated data validation frameworks using Python (PyTest/Unittest) and data quality libraries/tools.
- Develop reusable validation assets:
- reconciliation templates
- data quality rules (nulls, ranges, uniqueness, referential integrity)
- regression suites for critical pipelines
- Integrate validation into CI/CD and orchestration workflows (e.g., validation gates per pipeline run).
- Produce validation artifacts (test reports, defect summaries, data quality scorecards) for ongoing monitoring.
- Use AI tools (LLMs, Qodo, etc.) to accelerate:
- validation query generation
- reconciliation logic
- test case creation (including edge-case discovery)
- Apply AI-assisted techniques for anomaly detection ideas, boundary scenarios, and coverage expansion.
- Validate all AI-generated outputs against actual data and documented business rules before adoption.
Required Qualifications
- Strong SQL skills for reconciliation and validation (joins, aggregation, window functions, profiling).
- Strong hands-on Python experience for test automation and data validation (not just scripting).
- Experience in ETL/ELT QA, data testing, analytics validation, or data quality engineering.
- Understanding of event-driven / transaction-based systems and common pipeline failure patterns (late-arriving data, corrections, duplicates).
- Experience validating large datasets and reporting layers with high reliability expectations.
Preferred Qualifications
- Experience with data quality tools such as:
- Great Expectations (preferred)
- custom Python validation frameworks
- dbt tests / similar validation frameworks
- Familiarity with cloud data platforms and modern data stacks (any of):
- Snowflake / BigQuery / Redshift
- Databricks / Spark
- Airflow / ADF / Prefect
- Experience with sensitive data handling practices (privacy/security considerations).
- Exposure to interoperability/integration patterns (APIs, HL7/FHIR, EDI, batch feeds)—as applicable to the environment.
What Success Looks Like
- Automated validation coverage for critical pipelines (freshness, completeness, reconciliation).
- Early detection of anomalies and measurable reduction in downstream reporting issues.
- Repeatable, documented validation rules aligned to business definitions and metrics.
- Efficient collaboration with engineering teams to shorten incident triage and fix verification cycles.