Jobs · Information Technology · California

Azure Databricks Infrastructure Engineer

Tredence Inc. · San Jose, CA · 1 mo ago
HybridInformation TechnologyFull-time
We need to output HTML fragment with sections: headings

for section headings like About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. Use
  • for any list of duties/requirements/benefits, and

    for prose. Do NOT repeat the job title as a heading (the page already shows it) and do NOT put Job ID / requisition numbers as headings — fold minor metadata into a short

    or drop it. Preserve ALL substantive information; do NOT invent or summarize away detail. Remove cookie/navigation/legal/EEO/equal-opportunity/scam-warning boilerplate, duplicated lines, and application instructions. IMPORTANT: include a section ONLY when the posting has real, specific content for it — never emit a heading followed by filler/placeholder text (e.g. "commensurate with experience", "not specified", "N/A", "varies", "TBD") and never create Pay/Schedule/Benefits sections unless the posting actually states them. Prefer a few substantive sections over many thin ones. Output ONLY the HTML fragment — no /, no markdown code fences. We need to parse the posting. Content: Role description Azure Databricks Infrastructure Engineer Location - Remote Position Summary We are seeking an experienced Azure Databricks Infrastructure Engineer to execute migrations of Azure Databricks environments between Azure tenants. The successful candidate will have deep expertise in Databricks, Azure cloud services, identity and access management, networking, data engineering, and platform modernization. This role is responsible for executing a migration strategy that minimizes risk, downtime, and business disruption. The ideal candidate will combine strong technical capabilities with consulting skills and be comfortable interacting with architects, platform teams, security organizations, and business stakeholders. Key Responsibilities Migration Planning Analyze tenant-specific dependencies, networking requirements, DNS, Private Endpoints, and identity integrations. Define migration waves and cutover strategies. Databricks Migration Execution Migrate Databricks workspaces between Azure tenants. Rebuild or migrate: · Workspaces · Compute policies · Job schedules · Repositories · Workflows · Secret scopes · Access controls Migrate notebooks, libraries, packages, and configuration artifacts. Validate Unity Catalog and data governance configurations. Implement Infrastructure as Code (IaC) and automation where applicable. Security and Identity Migrate and validate: Microsoft Entra ID integrations Managed identities Service principals RBAC assignments Databricks groups and permissions Coordinate Key Vault migration and secret rotation activities. Ensure compliance with the Principle of Least Privilege (PLP). Support security reviews and governance approvals. Data Migration Assess and migrate associated datasets and storage integrations. Validate Delta Lake tables, metadata, and schemas. Develop rollback and recovery procedures. Automation and DevOps Develop Infrastructure-as-Code solutions using: Terraform Bicep ARM Templates Automate migration and validation activities wherever possible. Required Technical Skills Azure Azure Databricks Microsoft Entra ID Azure Virtual Networks (VNets) Private Endpoints Azure Key Vault Azure Storage Accounts Azure Data Lake Storage Gen2 (ADLS Gen2) Azure Monitor Log Analytics Azure Policy Azure RBAC Databricks Workspace Administration Unity Catalog Delta Lake Databricks Repos MLflow Cluster Management Cluster Policies Secret Scopes Databricks REST APIs Automation & DevOps Terraform Azure DevOps GitHub Actions PowerShell Azure CLI Databricks CLI Required Experience 7+ years of Azure cloud experience. 5+ years of Azure Databricks administration and engineering experience. Demonstrated experience migrating Azure resources across tenants. Experience with large-scale data platform migrations. Experience designing secure cloud landing zones. Experience implementing Infrastructure as Code (IaC). Experience working in regulated enterprise environments. Preferred Qualifications Azure Solutions Architect Expert certification. Databricks Administrator Certification. Databricks Solution Architect Certification. Experience with Unity Catalog migrations. Experience with mergers, acquisitions, divestitures, or tenant-to-tenant transformations. Deliverables The resource should be capable of independently producing: Migration strategy and wave plan. Detailed migration runbooks. Cutover plans. Rollback plans. Operational handover documentation. Ideal Candidate Profile A senior-level architect or lead engineer who has previously executed Azure Databricks tenant-to-tenant migrations, understands both platform and data engineering concerns, can engage with executive stakeholders, and can lead migration activities from assessment through production cutover with minimal supervision. We need to produce sections: maybe "About the role" (position summary), "Responsibilities", "Requirements" (maybe combine required technical skills and required experience?), "Qualifications" (preferred qualifications), "Skills" (maybe technical skills?), "Benefits" (none), "Pay" (none), "Schedule" (none). We should only include sections that have real specific content. So we can include About the role (position summary), Responsibilities, Requirements (maybe combine required technical skills and required experience), Qualifications (preferred qualifications), Skills (maybe technical skills list). However we need to avoid duplication: The posting has "Required Technical Skills" and "Required Experience". Could combine into a "Requirements" section with sub-lists: Technical Skills and Experience. Or we could have separate sections: "Technical Skills" and "Experience". But the instruction says use headings like About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. So we should map content to those headings. Thus:

    About the role

    Position summary...

    Responsibilities

    • ... list each responsibility bullet.

    Requirements

    maybe include required technical skills and required experience as sub-lists? Could do
    • Technical Skills: ...
    • Experience: ...
    but that might be less granular. Better to have separate subheadings? Not allowed? Only those headings allowed. Could embed paragraphs.

    Qualifications

    for preferred qualifications.

    Skills

    maybe we could list technical skills again? But that duplicates. Perhaps we can put technical skills under Skills, and experience under Requirements. Let's decide: - About the role: position summary paragraph. - Responsibilities: list of responsibilities (migration planning, execution, security, data migration, automation/devops). - Requirements: required experience (7+ years Azure, 5+ years Databricks, etc.) - Qualifications: preferred qualifications (certifications, etc.) - Skills: required technical skills list. Thus we avoid duplication. Now we need to extract details precisely. About the role: "We are seeking an experienced Azure Databricks Infrastructure Engineer to execute migrations of Azure Databricks environments between Azure tenants. The successful candidate will have deep expertise in Databricks, Azure cloud services, identity and access management, networking, data engineering, and platform modernization. This role is responsible for executing a migration strategy that minimizes risk, downtime, and business disruption. The ideal candidate will combine strong technical capabilities with consulting skills and be comfortable interacting with architects, platform teams, security organizations, and business stakeholders." That's the position summary. Responsibilities: The posting has headings: Migration Planning, Databricks Migration Execution, Security and Identity, Data Migration, Automation and DevOps. Under each there are bullet points. We need to list each as
  • . Let's extract: Migration Planning: - Analyze tenant-specific dependencies, networking requirements, DNS, Private Endpoints, and identity integrations. - Define migration waves and cutover strategies. Databricks Migration Execution: - Migrate Databricks workspaces between Azure tenants. - Rebuild or migrate: * Workspaces * Compute policies * Job schedules * Repositories * Workflows * Secret scopes * Access controls - Migrate notebooks, libraries, packages, and configuration artifacts. - Validate Unity Catalog and data governance configurations. - Implement Infrastructure as Code (IaC) and automation where applicable. Security and Identity: - Migrate and validate: * Microsoft Entra ID integrations * Managed identities * Service principals * RBAC assignments * Databricks groups and permissions - Coordinate Key Vault migration and secret rotation activities. - Ensure compliance with the Principle of Least Privilege (PLP). - Support security reviews and governance approvals. Data Migration: - Assess and migrate associated datasets and storage integrations. - Validate Delta Lake tables, metadata, and schemas. - Develop rollback and recovery procedures. Automation and DevOps: - Develop Infrastructure-as-Code solutions using: * Terraform * Bicep * ARM Templates - Automate migration and validation activities wherever possible. We need to preserve all details. We'll flatten into
  • items, maybe nested lists? The instruction says use
    • for any list of duties/requirements/benefits. It doesn't forbid nested lists, but we can keep simple: each bullet as
    • . For sub-bullets we could also make separate
    • items. Thus we will produce a
        with many
      • items. Requirements: Required Experience: - 7+ years of Azure cloud experience. - 5+ years of Azure Databricks administration and engineering experience. - Demonstrated experience migrating Azure resources across tenants. - Experience with large-scale data platform migrations. - Experience designing secure cloud landing zones. - Experience implementing Infrastructure as Code (IaC). - Experience working in regulated enterprise environments. Qualifications: Preferred Qualifications: - Azure Solutions Architect Expert certification. - Databricks Administrator Certification. - Databricks Solution Architect Certification. - Experience with Unity Catalog migrations. - Experience with mergers, acquisitions, divestitures, or tenant-to-tenant transformations. Skills: Required Technical Skills list: - Azure - Azure Databricks - Microsoft Entra ID - Azure Virtual Networks (VNets) - Private Endpoints - Azure Key Vault - Azure Storage Accounts - Azure Data Lake Storage Gen2 (ADLS Gen2) - Azure Monitor Log Analytics - Azure Policy - Azure RBAC - Databricks Workspace Administration - Unity Catalog - Delta Lake - Databricks Repos - MLflow - Cluster Management - Cluster Policies - Secret Scopes - Databricks REST APIs - Automation & DevOps - Terraform - Azure DevOps - GitHub Actions - PowerShell - Azure CLI - Databricks CLI We need to preserve all. We'll list each as
      • . For sub-items like Automation & DevOps we could list each. Now we need to ensure we don't include filler sections like Benefits, Pay, Schedule (none). So we omit those. We also need to include location? The posting says Location - Remote. Should we include that? It's minor metadata; we could fold into a short

        under About the role maybe. The instruction: do NOT put Job ID / requisition numbers as headings — fold minor metadata into a short

        or drop it. Location is minor metadata; we can include a short

        maybe after the position summary. But we already have About the role section; we could add a sentence: "Location: Remote." Or we could add a

        Location: Remote.

        inside About the role. We must not repeat the job title as a heading. Already we are not. Now produce HTML fragment:

        About the role

        We are seeking an experienced Azure Databricks Infrastructure Engineer to execute migrations of Azure Databricks environments between Azure tenants. The successful candidate will have deep expertise in Databricks, Azure cloud services, identity and access management, networking, data engineering, and platform modernization. This role is responsible for executing a migration strategy that minimizes risk, downtime, and business disruption. The ideal candidate will combine strong technical capabilities with consulting skills and be comfortable interacting with architects, platform teams, security organizations, and business stakeholders.

        Location: Remote.

        Responsibilities

        • Analyze tenant-specific dependencies, networking requirements, DNS, Private Endpoints, and identity integrations.
        • Define migration waves and cutover strategies.
        • Migrate Databricks workspaces

Similar jobs