Databricks SME (Remote)

Responsibilities

• Implement and manage linked services, mount points, and catalog configuration within Databricks and VA’s VHA Data Lake.
• Develop and optimize PySpark (Python/Scala) jobs, notebooks, and workflows in Databricks.
• Build ETL/ELT pipelines using Azure Databricks Lakehouse features.
• Design and implement Spark-based data processing for analytics workloads.
• Optimize Databricks Spark jobs for performance, cost, and scalability (partitioning, caching, tuning, etc.).
• Collaborate with data scientists, analysts, infrastructure specialists and business SME to deliver production-ready data solutions.
• Experience with Delta Lake and Synapse.
• Ensure data quality, governance, and security best practices.
• Act as subject matter experts to the workgroups from start to finish in their migration to the cloud.
• Help workgroups set up their resources and migrate their workloads.
• Support automated data warehouse access provisioning for eligible users, ensuring seamless integration with VA Cloud Data Warehouse (CDW) systems.
• Provide white‑glove customer success services, including onboarding sessions, office hours, rapid async support, and creation of reusable templates, guides, and best‑practice documentation.
• Collaborate with Customer Success, Business Analysts, and Platform Engineering teams to refine intake processes, improve user satisfaction, and standardize workspace deployment.
• Develop migration guidance and self‑service resources to accelerate platform adoption and reduce user friction when transitioning from legacy data systems.
• Troubleshoot user issues related to data access, workspace configuration, pipelines, cataloging, or permissions, escalating to engineering teams when needed.

Qualifications

• Bachelor’s degree in Information Technology or a related field. (or commensurate experience)
• 12+ years of experience in business analysis, project management, or a similar role.
• Strong experience with Databricks and Azure data services, including workspace administration, catalog creation, linked services, storage mounts, and access configuration.
• Proficiency in data engineering fundamentals such as ETL/ELT pipeline development, data modeling, and managing structured and unstructured datasets.
• Able to automate provisioning workflows and infrastructure using scripting or infrastructure‑as‑code tools (Python, PowerShell, Bash, Terraform, ARM, or Bicep).
• Strong troubleshooting and problem‑solving abilities for diagnosing platform, data access, and pipeline issues and collaborating across teams for resolution.
• Effective communication skills with the ability to document technical processes, create reusable templates, and support users through onboarding, office hours, and white‑glove assistance.