Senior Data Engineer
nextSource · United States · 2 wk ago
RemoteRemoteInformation Technology$70–$90/hrOther
About the role
We are seeking a highly skilled Data Quality Engineer to help define and implement end-to-end testing strategies for a modern AWS-based data platform. This role is ideal for someone who enjoys working across complex data ecosystems, building automation, validating data pipelines, and ensuring that high-volume data exchanges are accurate, reliable, and scalable.
Responsibilities
- Define end-to-end testing scope based on solution architecture, technical documentation, and project requirements
- Design and implement comprehensive testing strategies aligned with enterprise QA standards
- Develop, maintain, and enhance test scripts and automation frameworks for AWS data platforms
- Validate data pipelines from ingestion through transformation and reporting
- Build automated testing solutions for:
- End-to-end data validation
- Data ingestion accuracy
- Transformation logic integrity
- Data pipeline reliability
- Data reconciliation and anomaly detection
- Perform testing across key technologies, including:
- AWS Redshift
- AWS DMS
- AWS Glue
- Pyspark Deequ
- EventBridge
- Data lakes
- Python-based data pipelines
- Apache Airflow
- dbt
- Conduct test coverage analysis across data engineering workflows
- Prepare, manage, and validate test data
- Review solution architecture, data models, design documents, and technical documentation
- Partner with Data Engineering, BI, DevOps, Product, and QA teams to identify risks, evaluate testing impacts, and ensure high-quality delivery
Requirements
- Proven experience in data engineering testing, data QA, ETL validation, or data platform quality engineering
- Strong hands-on experience with AWS data services, including Redshift, Glue, and DMS
- Proficiency with Python for data validation, test automation, and scripting
- Experience testing orchestrated workflows using Apache Airflow
- Hands-on experience validating dbt models and transformation logic
- Familiarity with AWS CDK or infrastructure validation concepts
- Experience with data quality frameworks such as PySpark Deequ or similar tools
- Strong understanding of:
- Data warehousing concepts
- ETL and ELT pipelines
- Data validation techniques
- Schemavalidation
- Data reconciliation
- Anomaly detection
- Pipeline reliability testing
Nice to have
- Experience designing enterprise-level test strategies for modern data platforms
- Knowledge of CI/CD pipelines for data engineering and automated testing
- Experience working in Agile or Scrum environments
- Familiarity with data observability frameworks
- Experience with BI testing, especially within AWS QuickSight or similar reporting platforms
Benefits
- Healthcare Benefits including Medical, Prescription, Vision & Dental Coverage | United Healthcare
- Voluntary STD & LTD | New York Life
- 401(k) Retirement Planning | Fidelity
- Discount Program
Pay
Pay Rate Range: $70-$90 per hr (Depending on Experience)