QA Test Engineer ETL

Role Summary

The QA/Test Engineer is responsible for building and executing comprehensive test strategies for all ingestion and ETL pipelines. This role ensures that every pipeline meets the 98–99% data quality pass rate required by the SOW before production promotion, and that automated tests are embedded in the CI/CD process.

Key Responsibilities

• Design and implement a reusable test framework for unit, integration, and end-to-end testing of data ingestion and ETL pipelines.
• Write and maintain unit tests for individual transformation logic, connector behavior, and schema validation.
• Develop integration tests that validate end-to-end data flow from source ingestion through raw → curated → consumption layers.
• Build automated test suites that execute as part of the CI/CD pipeline, gating production deployments.
• Validate data quality check behavior: confirm that DQ rules (completeness, schema conformance, record counts, freshness) correctly trigger fail/alert actions.
• Track and report test coverage, defect rates, and pipeline validation results for each milestone.
• Collaborate with Data/Cloud Engineers to define test cases, edge cases, and acceptance criteria for each data source.
• Support performance and load testing for streaming and high-volume batch sources where representative test data is available.
• Produce test artifacts, test plans, and validation reports as part of milestone documentation.

Required Skills & Qualifications

• 8-10 years of experience in QA/test engineering with a focus on data pipelines, ETL processes, or data platforms.
• Strong proficiency in Python for test automation and data validation scripting.
• Experience with data testing frameworks (e.g., Great Expectations, dbt tests, pytest, or custom frameworks).
• Hands-on experience testing data pipelines on AWS (Glue, Spark, S3, Athena, Redshift).
• Understanding of data quality dimensions: completeness, accuracy, consistency, freshness, schema conformance.
• Experience integrating automated tests into CI/CD pipelines (GitHub Actions, CodePipeline, Jenkins).
• Ability to write SQL queries for data validation and reconciliation.
• Familiarity with test data management strategies for sensitive or regulated data.
• Strong documentation skills for test plans, test cases, and validation reports.
• Experience working in Agile/Scrum teams with 2-week sprint cycles.

Preferred Skills

• Experience with performance/load testing for data pipelines.
• Familiarity with contract testing for APIs and schema registries.
• Knowledge of data observability tools (Monte Carlo, Datafold, or similar).