Senior Data Engineer - Quality

About the role

The Senior Data Engineer - Quality is a hands-on technical role responsible for designing and building robust, scalable, end-to-end testing frameworks for modern data pipelines. This role focuses on ensuring data quality across ingestion (APIs, SQL Server, flat files) and transformation using Medallion Architecture (Bronze, Silver, Gold).

Responsibilities

• Design and implement scalable end-to-end testing frameworks for data pipelines
• Validate ingestion from APIs, SQL Server, and flat files
• Ensure data quality across Medallion architecture layers (Bronze, Silver, Gold)
• Build automated checks for schema validation, data integrity, and transformations
• Create reusable validation patterns using Great Expectations or similar frameworks
• Leverage Claude Code and prompt engineering to accelerate test generation and standardization
• Create reusable AI-driven testing assets (.md skills, templates) and workflows for AI-assisted coding and testing
• Integrate testing into CI/CD pipelines (Azure DevOps, GitHub Actions)
• Collaborate with data engineers, Product, and DevOps to define data quality standards and acceptance criteria
• Maintain data reliability, observability, and test coverage
• Investigate data quality issues and drive root-cause resolution

Requirements

• Required Education and/or Experience:
• Bachelor's degree in Computer Science, Information Technology, or related field.
• A minimum of 5 years of experience in data quality engineering, data testing, or data platform engineering.
• Strong experience building automated testing frameworks for data pipelines.
• Hands-on experience with data validation tools (Great Expectations, DQX - Databricks Data Quality Framework, dbt tests).
• Proficiency in Python and strong SQL skills.
• Experience with AI-assisted development tools (e.g., Claude Code, Copilot).
• Strong understanding of prompt engineering and reusable AI workflows.
• Experience validating data ingestion from APIs, relational databases, and flat files.
• Deep understanding of data transformation validation and Medallion Architecture.
• Experience integrating testing into CI/CD pipelines (Azure DevOps, GitHub Actions, etc.).
• Familiarity with data observability and monitoring practices.
• Preferred Experience:
• With Azure Databricks and Apache Spark (PySpark).
• Familiarity with Delta Lake and Unity Catalog.
• Experience building reusable AI prompt libraries or skills (.md or similar formats).
• Hands-on experience using Claude Code for automation or framework generation.
• Knowledge of data contracts and schema evolution strategies.
• Experience with test data management and synthetic data generation.
• Familiarity with infrastructure as code (Terraform, Bicep).
• Experience with streaming data pipelines (Kafka, Event Hubs).

Qualifications

• Strong organizational skills.
• Strong attention to detail.
• Good judgment.
• Strong interpersonal communication skills.
• Strong analytical and problem-solving skills.
• Able to work harmoniously and effectively with others.
• Able to preserve confidentiality and exercise discretion.
• Able to work under pressure.
• Able to manage multiple projects with competing deadlines and priorities.

Skills

• Strong organizational skills.
• Strong attention to detail.
• Good judgment.
• Strong interpersonal communication skills.
• Strong analytical and problem-solving skills.
• Able to work harmoniously and effectively with others.
• Able to preserve confidentiality and exercise discretion.
• Able to work under pressure.
• Able to manage multiple projects with competing deadlines and priorities.

Benefits

Our compensation package includes a target salary range of $100,000 - $150,000 for candidates located in Illinois. Salaries vary by location and are based on numerous factors, including, but not limited to, the relevant market, skills, experience, and education of the selected candidate. Our benefits program includes bonus eligibility and a comprehensive benefits package. For more information, visit Sidley.com/Benefits.