Databricks Architect

Salary Range: $110,000–$140,000 a year

Requirements

• 10+ years of hands-on data engineering experience, with at least 3 years focused on the Databricks/Spark ecosystem
• Deep, hands-on expertise with the Databricks Lakehouse Platform, including Delta Lake, Structured Streaming, Lakeflow Declarative Pipelines (formerly Delta Live Tables), Databricks SQL, and cluster/serverless configuration and optimization
• Hands-on experience with Unity Catalog Business Semantics, including Metric Views (measures, dimensions, materialization), Domains, and Pages/Glossary to define governed, reusable KPIs once and serve them consistently across SQL, BI, and AI agents
• Working knowledge/exposure of AI/BI Genie (Genie Spaces, Genie Ontology, trusted assets), Genie Code for agentic pipeline/SQL development, Databricks One for business-user consumption, and Mosaic AI for building and serving AI/ML models and agents
• Expert-level proficiency in Python and PySpark; advanced SQL skills
• Strong understanding of data modeling principles, including dimensional modeling (Kimball), data warehousing concepts, and ETL/ELT design patterns
• Proven experience working with a major cloud provider (Azure, AWS, or GCP), particularly with data storage (e.g., S3) and related services
• Experience with software engineering best practices, including version control (Git), code reviews, testing, and CI/CD
• Databricks certification

Responsibilities

• Design, code, and deploy robust and scalable batch and streaming data pipelines using PySpark, Spark SQL, and Lakeflow (Lakeflow Connect for ingestion and Lakeflow Declarative Pipelines) to ingest data from sources such as Point-of-Sale (POS), e-commerce platforms, loyalty systems, and marketing clouds
• Implement complex data transformations and business logic within the Medallion architecture (Bronze, Silver, Gold layers); build and optimize the final "Gold" dimension tables that serve as the single source of truth
• Define governed business KPIs on top of the Gold layer using Unity Catalog Metric Views (business semantics) so metrics are computed once and reused consistently across BI, SQL, and Genie
• Implement data quality frameworks and cleansing routines to ensure the accuracy and trustworthiness of Customer 360 data
• Proactively monitor, debug, and tune Databricks jobs and Spark clusters for performance and cost-efficiency; implement best practices for partitioning, caching, and data layout in Delta Lake
• Work with DevOps teams to manage Databricks environments, clusters, and job deployments using tools like Terraform and AWS DevOps/GitHub Actions; champion and implement CI/CD best practices for data pipelines
• Implement data governance features within Databricks Unity Catalog, including data lineage tracking, fine-grained access controls (row/column-level security and ABAC policies), tags, and data masking to ensure compliance and security across BI and AI/agent workloads
• Partner closely with Functional Consultants, Data Scientists, and Analytics Engineers to understand their data requirements and deliver well-structured, consumption-ready datasets
• Build and govern Unity Catalog Business Semantics, authoring Metric Views (measures, dimensions, joins, synonyms, materialization), Domains, Pages, and Glossary terms, and certifying trusted assets so every dashboard, SQL query, notebook, and AI agent works from the same governed definitions
• Configure and curate AI/BI Genie Spaces and the Genie Ontology (instructions, trusted assets, example queries, synonyms) to deliver accurate natural-language, conversational analytics for business users, and enable consumption through AI/BI Dashboards and Databricks One

Benefits

• Discretionary annual incentive
• Comprehensive medical coverage (medical & health, dental & vision)
• Disability planning & insurance, pet insurance plans
• Maternal & parental leaves
• Auto & home insurance, identity theft protection
• Commuter benefits & certification & training reimbursement
• Vacation, time off, sick leave & holidays
• Legal assistance, 401K plan, performance bonus, college fund, student loan refinancing