Senior Databricks Data Engineer, Data Testing & Validation

About the role

We partner with global enterprises to design and build large-scale data platforms that power the products and operations at the center of their business. The platform replaces an existing integration system that today connects the client's engineering tools and produces the reports their teams rely on. The rule is simple: the data in Databricks has to match what that system holds and what its reports say. It has to match on day one, and it has to keep matching every time new data arrives. You will write the tests that prove it. You are a data engineer, not a manual tester. Your code checks that every table reconciles to the legacy system or its reports, that incremental loads add exactly what changed and nothing twice, and that duplicates never make it through. When a pipeline breaks one of those rules, your tests catch it before the client does.

Responsibilities

• Build the automated test framework for the platform's pipelines, in PySpark and SQL, and run it on every deployment throughout all environments.
• Write reconciliation tests that compare Databricks tables to the legacy system and its reports, with clear output on where they differ and why.
• Test incremental loading: change data capture, appends, and batch refreshes. Prove that reruns are idempotent, that merges and upserts update the right rows, and that late or re-delivered files do not double-load.
• Test for duplicates on composite keys and on the relationships that link records across source systems.
• Write unit tests for transformation logic so that keying, joins, and business rules are verified in isolation before they run against real data.
• Define data quality expectations in the pipelines themselves, so bad data is quarantined or fails the run instead of landing silently.
• Test how pipelines behave when source files change shape: new columns, renamed columns, missing columns, and mixed types.
• Wire the tests into Databricks Workflows and the CI/CD pipeline so they gate promotion between environments.
• Test Unity Catalog access controls, row filters, column masks, and classification tags to confirm that export-controlled data is only visible to authorized users.
• Plan and run user acceptance testing sessions with client stakeholders, capture feedback, and track defects to resolution.
• Document test results and acceptance evidence for each deliverable, and report quality status in weekly reviews.

Requirements

• 5+ years in data engineering, including building test and validation frameworks for production pipelines, not only running tests written by others.
• Strong PySpark and SQL, used to write reconciliation and assertion logic at scale.
• Hands-on experience reconciling a new platform against a legacy system or its reports, and explaining the variances.
• Deep understanding of incremental load correctness: CDC and append semantics, idempotency, MERGE behavior, watermarks, and late-arriving data.
• Experience finding and resolving duplicates on composite keys across multiple sources.
• Experience with a Python test framework such as pytest, and with data testing tools such as chispa, Great Expectations, or pipeline expectations in Delta Live Tables or Lakeflow.
• Experience running tests from Databricks Workflows and CI/CD across multiple environments.
• Organized, detail-oriented, and comfortable holding a delivery team and a client to a quality bar on a tight timeline.
• Clear written communication - test evidence and variance reports are the core deliverables.
• Ability to complete the client's background check and onboarding and to work on client-furnished equipment.
• US citizenship.
• Fluent English, both written and spoken.

Skills

• Hands-on experience with Databricks, Unity Catalog, or Delta Lake.
• Python or PySpark for automating data validation.
• Testing Unity Catalog access controls, row filters, and column masks.
• Prior work in a defense, aerospace, or FedRAMP environment, or with CUI or ITAR-controlled data.
• Familiarity with PLM or MBSE tools such as Teamcenter or Cameo, or