Lead Databricks Engineer

Location: Columbus

Job Type: C2C

Primary Skills: Databricks, Terraform, Databricks Unity Catalog

About the role

We are seeking a DiTA/Predictive Attrition developer responsible for implementation, modification and maintenance. The role involves working closely with data engineers, architects, and business stakeholders to ensure data integration, transformation, and loading processes within the cloud environment.

Responsibilities

• Implement, modify, and maintain data pipelines and integration processes.
• Collaborate with data engineers, architects, and business stakeholders.
• Ensure data transformation and loading within the cloud environment.

Technical Requirements

• Databricks, Genie One, Unity Catalog
• SQL Warehouse and compute setup (DABS pipeline)
• CI/CD and MLOps principles
• AWS MLOps tools: SageMaker, ECS, Lambda, SSM Parameter Store, EventBridge
• Infrastructure as Code with Terraform
• Python programming
• Generative AI platforms: OpenAI, Claude, GCP LLM models
• Observability and monitoring dashboards

Qualifications

• 7–10 years of relevant experience.