Sr. Databricks Data Engineer

About the role

Halvik Corp is a leading WOB business delivering Digital Services, Advanced Analytics, AI/ML, Cyber Security, and cutting‑edge technology to U.S. government agencies. The Senior Databricks Data Engineer / Technical Lead will support the U.S. Department of Transportation’s data modernization effort by designing, building, migrating, and operating secure, scalable data pipelines in a Databricks Lakehouse.

Responsibilities

• Strategic Execution: Lead design and implementation of reusable ingestion and transformation patterns for legacy and external data sources, including migration to the AWS/OneDOT environment.
• Operational Oversight: Build and operate Bronze, Silver, and Gold pipelines; manage batch and incremental loads, orchestration, dependencies, retries/recovery, monitoring, alerting, and production support.
• Collaboration: Partner with Technical Project Manager, Data Architects, business/data SMEs, BI developers, Cloud/DevSecOps, security, and governance teams to translate requirements into implementable data solutions.
• Technical Performance: Develop and optimize Python, PySpark, and SQL workloads using Databricks, Delta Lake, Unity Catalog, Auto Loader/Lakeflow, and Databricks Jobs/Workflows with a focus on reliability, performance, and cost.
• Data Quality & Governance: Implement source‑to‑target reconciliation, data validation, schema evolution, lineage, access controls, documentation, and reusable engineering standards for production pipelines.

Minimum Requirements

• Education: Bachelor’s degree in Computer Science, Data Engineering, Information Technology, Engineering, or a related field (equivalent experience may be considered).
• Experience: Minimum 7 years of professional data engineering experience, with significant hands‑on work on Databricks/Spark and production data pipelines.
• Technical Proficiency: Advanced Python, PySpark, and SQL skills; strong experience with Databricks, Delta Lake, AWS/S3, batch/file ingestion (CSV/JSON), REST APIs, Git/CI‑CD, and data pipeline orchestration.
• Compliance: Must meet applicable DOT contract security and suitability requirements and demonstrate secure handling of PII and other sensitive data, including access controls and secrets management.
• Certifications: Relevant Databricks and/or AWS data engineering certification preferred; equivalent demonstrated hands‑on experience will be considered.

Preferred Expertise

• Proven success modernizing legacy data platforms (e.g., Sybase, SAP IQ) into Databricks, including source‑to‑target reconciliation, cutover, and production stabilization.
• Experience with Unity Catalog, metadata/configuration‑driven ingestion, Auto Loader/Lakeflow, Databricks Jobs/Workflows, lineage/cataloging tools, and scalable multi‑source ingestion frameworks.
• Federal/DOT delivery experience; familiarity with Agile/Kanban, production O&M, cloud cost optimization, and data governance.
• Exposure to streaming and AI/ML data pipelines is a plus.

Benefits

• Company‑supported medical, dental, vision, life, short‑term disability (STD), and long‑term disability (LTD) insurance.
• 11 federal holidays and paid time off (PTO).
• Performance‑based incentives for individual and team achievements.
• 401(k) plan with company matching.
• Flexible Spending Accounts for commuter, medical, and dependent care expenses.
• Tuition assistance.
• Charitable contribution matching.