AWS Databricks Architect

About the role

We are looking for an experienced AWS Databricks Architect to design, implement, and optimize scalable Lakehouse solutions across AWS as well as Azure cloud platforms. The ideal candidate will provide technical leadership in data architecture, platform engineering, data governance, performance optimization, and enterprise-grade Databricks implementations.

Responsibilities

• Design and architect end-to-end Databricks Lakehouse solutions across AWS and Azure environments.
• Define scalable data architecture patterns for ingestion, transformation, storage, analytics, and reporting workloads.
• Lead implementation of Delta Lake medallion architecture, Databricks SQL, Unity Catalog, and workflow orchestration.
• Optimize Apache Spark jobs, clusters, notebooks, pipelines, and workloads for performance, reliability, and cost efficiency.
• Implement security, access control, secrets management, data governance, and compliance best practices across cloud platforms.
• Collaborate with data engineering, BI, data science, DevOps, and business teams to deliver high-quality data solutions.
• Guide CI/CD, infrastructure-as-code, monitoring, and production deployment practices for Databricks workloads.
• Provide technical leadership, architecture reviews, troubleshooting support, and mentoring to project teams.

Requirements

• 12+ years of experience in data engineering, data architecture, or big data platform implementation in AWS cloud platform, with knowledge of Azure.
• Strong hands-on experience with Databricks platform architecture, workspace setup, cluster configuration, jobs, notebooks, and workflow orchestration.
• Expertise in AWS services such as S3, IAM, VPC, Glue, Lambda, CloudWatch, and related data platform services.
• Expertise in Azure services such as ADLS, Azure Data Factory, Azure Synapse, Key Vault, Entra ID, and Azure Monitor.
• Strong knowledge of Apache Spark, PySpark, SQL, Delta Lake, and Lakehouse architecture concepts.
• Experience with data security, governance, access controls, Unity Catalog, encryption, and cloud networking concepts.
• Hands-on experience with performance tuning, cost optimization, debugging, and production support of Databricks workloads.
• Good understanding of CI/CD, Git, Terraform or equivalent infrastructure-as-code tools, and DevOps practices.

Benefits

• Comprehensive medical plan covering medical, dental, and vision.
• Short-term and long-term disability coverage.
• 401(k) plan with company match.
• Life insurance.
• Vacation time, sick leave, and paid holidays.
• Paid paternity and maternity leave.

Pay

The range displayed reflects the minimum and maximum salary target for the position across all US locations. Within the range, individual pay is determined by work location, job level, and additional factors including job-related skills, experience, and relevant education or training. Other forms of compensation may include an annual performance-based bonus, sales incentive pay, or other forms of bonus or variable compensation.