Senior Databricks Data Engineer, Spreadsheet & Flat-File Sources

About the role

We partner with global enterprises to design and build large-scale data platforms that power the products and operations at the center of their business. Not all of that data lives in enterprise systems. Manufacturing and supply chain teams maintain parts, supplier, and program data in Excel workbooks and flat-file repositories. These are authoritative sources, but they have no schema, no change log, and no owner-enforced structure. Each workbook is a small, undocumented system of its own. You will bring those sources onto the platform. You will inventory them, build ingestion that turns inconsistent workbooks and files into governed Delta tables, and make the result join cleanly to parts and revisions from the engineering systems, so the digital thread includes the data that today lives only in spreadsheets.

Location: This is a fully remote position. This position supports a U.S. Government contract that requires all personnel to possess U.S. Citizenship. This role is remote within the United States only. Candidates must reside in the United States while assigned to our client and be physically located within the United States when performing project work or accessing project systems or data. Availability to work overlapping Pacific, Central, or Eastern U.S. time zones.

Responsibilities

• Inventory the Excel workbooks and file exports that business and engineering teams rely on, and document their structure, owners, and downstream use.
• Build Auto Loader pipelines to ingest workbooks and flat files landed in S3, with explicit schemas and typed Bronze and Silver tables.
• Parse multi-sheet, inconsistently structured workbooks at scale: shifting header rows, merged cells, embedded totals, mixed types in a column, and layouts that differ between versions of the same file.
• Configure schema evolution in Databricks declarative pipelines, including new column handling, rescued data columns, and controlled failure on renamed columns.
• Implement data quality expectations that catch the failure modes these sources produce, such as duplicate rows, blank keys, and silently changed formats.
• Conform ingested data to the platform's part number plus revision keying so it joins to BOM and engineering data from other sources.
• Ensure lineage is captured for every pipeline, orchestrate with Databricks Workflows, and document each source for handoff.
• Work with the business owners of each workbook to confirm meaning, business rules, and what "correct" looks like.
• Translate spreadsheet logic, such as