Senior Databricks Engineer / Ingestion Lead

About the role

We partner with global enterprises to design and build large-scale data platforms that power the products and operations at the center of their business. This engagement is a Databricks lakehouse foundation for a US defense manufacturer, and we are looking for a Senior Databricks Engineer to own the ingestion layer. The sources include change data capture out of a PLM system, JDBC ingestion against an ERP where session-level initialization parameters decide whether row-level security hands you the right rows or quietly the wrong ones, and bill-of-materials reconciliation across three systems of record that each describe the same part differently, requiring a decision on which representation is correct. This is a hands‑on senior role with real ownership: with only two people on the engagement, ingestion is yours end‑to‑end — you make the pattern calls per source and sit in client discovery sessions rather than receiving tickets, working alongside the engagement architect who owns platform direction and governance. The engagement begins with a discovery and design phase, where you go deep on each source system and design the right ingestion pattern for it, and then moves into the build phase, executing what you designed.

Responsibilities

• Run source discovery across each system in scope, documenting data structures, change patterns, extraction constraints, data quality, and current-state integration topology.
• Assess extraction options per source, including CDC feasibility from PLM, JDBC access to Oracle E-Business Suite under its row-level security model, SQL Server change patterns, and file-based ingestion from S3.
• Design the ingestion pattern for each source (Lakeflow Connect, custom JDBC, Auto Loader, or federation) with the tradeoffs written down and defensible.
• Model bill-of-materials data across PLM, ERP, and MBSE sources, keyed on part number plus revision, including how conflicting representations of the same part are reconciled into the bronze and silver layers.
• Build the ingestion layer: Lakeflow Connect pipelines including CDC from Teamcenter PLM, custom JDBC connectors against Oracle E-Business Suite with session-level initialization parameter handling, Auto Loader for incremental S3 ingestion, and SQL Server ingestion patterns.
• Handle schema evolution deliberately, with automatic handling of new columns, rescued data columns for unexpected shapes, and controlled failure on renamed columns rather than silent data loss.
• Operate nightly batch patterns with idempotent reruns and backfill support, quarantining bad source data rather than dropping it.
• Apply strong engineering practices including testing, CI/CD, and version control, and write expectations and quality checks into pipelines so failures are attributable to a source and a rule.
• Document for handoff, producing each source's ingestion contract and