Senior AI Ops Engineer

About the role

The KLA Services team headquartered in Milpitas, CA is seeking a Senior AI Ops Engineer to join our team. This role will be pivotal in architecting and delivering the automation layer that enables fast, reproducible, and scalable model development.

Responsibilities

• Implement and operate experiment tracking, lineage, and reproducibility standards (datasets, code, configs, artifacts, metrics) using MLflow/W&B or equivalents.
• Build CI/CD for ML: tests (unit/integration), packaging, reproducibility checks, policy gates, automated deployment and rollback strategies.
• Design workflow orchestration for large-scale ML jobs (scheduled runs, triggered retrains, parameter sweeps, gated releases) using tools such as Airflow/Kubeflow/Argo or equivalents.
• Architect, build, and own automated pipelines for model training, fine-tuning (e.g., PEFT/LoRA), evaluation, and promotion across environments (dev → staging → production).
• Enable and optimize distributed GPU training (throughput, reliability, and cost), including checkpointing, mixed precision, fault tolerance, and spot/preemptible handling where applicable.
• Develop evaluation harnesses and automated benchmark suites (quality, safety, latency, and cost) with clear, repeatable reporting to compare runs and releases.

Qualifications

• Strong proficiency in Python and experience building robust automation frameworks and production-grade services for ML workloads.
• Hands-on experience with experiment tracking and model lifecycle tooling (e.g., MLflow, Weights & Biases) and reproducible ML workflows.
• Practical experience fine-tuning modern deep learning models (e.g., Transformers) and familiarity with parameter-efficient approaches (LoRA/PEFT).
• Familiarity with RLHF concepts and pipelines (preference data, reward models, policy optimization) and how to operationalize human-in-the-loop workflows.
• Experience with containerization (Docker), orchestration (Kubernetes), and operating GPU workloads reliably at scale.
• Experience with CI/CD, version control (Git), and Infrastructure-as-Code (Terraform/Bicep or equivalent).
• Excellent problem-solving skills across distributed systems (training jobs, pipelines, compute infrastructure) and strong communication to partner with research and engineering teams.
• Prior experience in a similar industry and/or operating ML platforms with stringent IP/security requirements is a plus.
• Bachelor’s degree in Computer Science, Software Engineering, or related field.
• 5+ years of experience in MLOps/Platform Engineering/DevOps/ML Engineering (or demonstrated equivalent impact), including owning production systems and leading cross-team initiatives.

Base Pay Range

$111,900.00 - $190,200.00 Annually

Benefits

• Medical, dental, vision, life, and other voluntary benefits
• 401(K) including company matching
• Employee stock purchase program (ESPP)
• Student debt assistance
• Tuition reimbursement program
• Development and career growth opportunities and programs
• Financial planning benefits
• Wellness benefits including an employee assistance program (EAP)
• Paid time off and paid company holidays
• Family care and bonding leave