ETL Testing With Python

Department: Data Engineering / Quality Assurance

Reports To: QA Lead / Data Engineering Manager

We are looking for a detail-oriented and analytical ETL Test Engineer with strong Python skills to join our Data Engineering team. The ideal candidate will be responsible for validating ETL pipelines, ensuring data accuracy, completeness, and quality across various stages of the data lifecycle.

• Develop and execute automated and manual test cases for ETL processes and data pipelines.
• Validate data transformations, data integrity, and data quality in staging, warehouse, and reporting layers.
• Write Python scripts for data validation and automated testing of ETL jobs.
• Work closely with data engineers to understand business logic and data flows.
• Build test harnesses and data validation frameworks using Python or PySpark.
• Perform data profiling and exploratory analysis to understand data anomalies.
• Identify, log, and track defects and work closely with development teams for resolution.
• Maintain test data and environments to support various test scenarios.
• Participate in Agile/Scrum processes including sprint planning and retrospectives.

• Bachelor’s degree in Computer Science, Information Systems, or related field.
• 3+ years of experience in ETL/data testing with strong understanding of data warehousing concepts.
• Proficient in Python for scripting and automation.
• Experience in testing tools/frameworks like pytest, unittest, or custom Python-based frameworks.
• Strong SQL skills; ability to write complex queries for data validation.
• Hands-on experience with ETL tools (e.g., Informatica, SSIS, Talend, Airflow, or custom pipelines).
• Familiarity with data formats like JSON, Parquet, Avro, and CSV.
• Good understanding of database systems (e.g., PostgreSQL, MySQL, Snowflake, Redshift, BigQuery).
• Experience with version control systems like Git.

• Experience with PySpark, Spark SQL, or big data platforms.
• Exposure to CI/CD pipelines and automated testing tools (e.g., Jenkins).
• Knowledge of cloud platforms such as AWS, GCP, or Azure (especially services like S3, Glue, BigQuery, etc.).
• Experience in data observability or data quality tools like Great Expectations or Deequ.
• Familiarity with BDD frameworks like Behave or Cucumber.

• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• Ability to work independently and manage multiple priorities in a fast-paced environment.