Databricks Architect and Engineering Lead

About AbarVa AbarVa is building Nexus, an AI-native enterprise execution platform that helps organizations turn strategic decisions into governed action, measurable outcomes, and sustained business value. We are building Nexus alongside complex enterprise Data & AI transformations, particularly in regulated industries. The Opportunity We are looking for an experienced Databricks Architect and Engineering Lead to lead the architecture and engineering execution of large-scale data modernization and migration programs. This is a hands-on architecture and engineering leadership role. We are looking for someone who has spent significant time personally building modern data platforms, has deep Databricks expertise, and can lead engineering teams through complex enterprise migrations. This is not a strategy-only or architecture-only position. The successful candidate must be able to design, build, review code, troubleshoot complex engineering issues, establish engineering standards, and lead teams through production delivery. Required Experience 10+ years of hands-on data engineering experience5+ years of hands-on Databricks experienceCurrent Databricks certification is mandatorySignificant experience as a Databricks Architect, Data Engineering Lead, Principal Data Engineer or equivalentDeep hands-on expertise with Databricks, Apache Spark, PySpark and SQLStrong experience with Delta Lake, Unity Catalog, Databricks Workflows and modern Databricks ingestion and transformation patternsDemonstrated experience designing and implementing enterprise-scale Bronze, Silver and Gold architecturesProven experience leading large-scale migrations from legacy data platforms to DatabricksExperience assessing existing data workloads, ETL pipelines, data models, reports, KPIs and downstream dependenciesStrong experience designing metadata-driven ingestion and transformation frameworksStrong knowledge of data quality, reconciliation, lineage, observability, security and governanceStrong cloud platform experience; AWS experience is highly desirableProven ability to lead and mentor teams of data engineers while remaining technically hands-onStrong communication skills and experience working directly with senior client and technology stakeholders What Youll Do Own the architecture and engineering execution of large-scale Databricks modernization and migration programsAssess complex legacy data environments and develop practical modernization strategiesDetermine which workloads should be migrated, refactored, redesigned, consolidated or retiredDesign scalable enterprise Bronze / Silver / Gold architecturesLead migration of legacy data warehouses, ETL platforms, analytics environments and reporting ecosystems to DatabricksArchitect reusable, metadata-driven ingestion, transformation and orchestration frameworksLead enterprise data modeling and development of reusable data productsEstablish engineering standards for performance, security, governance, lineage, observability and data qualityLead reconciliation and parallel-run activities to prove Databricks outputs against trusted legacy platformsDefine migration waves, cutover plans and production stabilization approachesConduct architecture, design and code reviewsTroubleshoot complex engineering, scalability and performance issuesLead and mentor distributed engineering teamsPartner directly with business and technology leaders to translate business outcomes into scalable data solutionsIntroduce modern AI-assisted engineering approaches to improve development productivity while maintaining appropriate governance and human reviewHelp develop reusable AbarVa engineering methods, accelerators and intellectual property Large-Scale Migration Experience Candidates should have personally led significant migrations involving platforms such as: NetezzaTeradataOracle / ExadataSQL ServerHadoop ecosystemsTraditional enterprise data warehousesLegacy ETL and integration platformsExternally hosted analytics platformsEnterprise reporting and BI environments Candidates Should Be Able To Clearly Explain How They Assessed the existing data landscapeIdentified workload and reporting dependenciesRationalized legacy pipelines and reportsDesigned the target Databricks architectureSequenced migration wavesReconciled legacy and Databricks resultsManaged parallel operationsExecuted production cutoverStabilized and optimized the new environment Architecture & Leadership Expectations This role requires someone who can lead from the front technically. You Will Be Expected To Own critical architecture decisionsLead engineering teams through complex delivery programsEstablish architecture and engineering standardsMentor architects and data engineersConduct detailed design and code reviewsMake difficult architecture and engineering trade-offsWork directly with senior client technology and business leadersTranslate complex technical decisions into clear business implicationsTake accountability from architecture through successful production implementation We are not looking for someone who only manages engineers or produces architecture diagrams. You must be comfortable getting deeply hands-on when required. Industry Experience Experience in healthcare, financial services, insurance or another highly regulated industry is a significant advantage. Healthcare Experience Is Particularly Valuable, Including Exposure To Health-plan / payer dataClaimsMembership, enrollment and eligibilityMedicare and MedicaidClinical and provider dataEpic / Epic ClarityHealthcare financial and operational analyticsPHI and other regulated data environments Deep healthcare expertise is not mandatory, but experience designing platforms where security, privacy, governance, auditability and data quality are critical is strongly preferred. Work Authorization Candidates must be legally authorized to work in the United States on a W-2 basis without current or future visa sponsorship. AbarVa is not able to provide employment visa sponsorship for this role. What Makes This Role Different You will not simply create an architecture and hand it to another team. You will be expected to lead discovery, architecture, engineering, migration, reconciliation, cutover and production delivery. We are looking for a Databricks architect, hands-on engineer and engineering leader who has successfully delivered large-scale modernization programs and can build and lead high-performing teams.