Principal Engineer - Perf and Benchmarking

About the role

We're looking for a Principal Engineer to lead CoreWeave's Benchmarking & Performance team. Your responsibilities include defining the multi-year benchmarking strategy, leading MLPerf submissions, designing and maintaining benchmarking services, and collaborating with key partners.

Responsibilities

• Define the multi-year benchmarking strategy and roadmap; prioritize models/workloads and hardware tiers.
• Build, lead, and mentor a high-performing team of performance engineers and data analysts.
• Establish governance for claims: documented methodologies, versioning, reproducibility, and audit trails.
• Lead end-to-end MLPerf Inference and Training submissions: workload selection, cluster planning, runbooks, audits, and result publication.
• Coordinate optimization tracks with NVIDIA (CUDA, cuDNN, TensorRT/TensorRT-LLM, Triton, NCCL) to hit competitive results; drive upstream fixes where needed.
• Design a Kubernetes-native, repeatable benchmarking service that exercises CoreWeave stacks across SUNK, Kueue, and Kubeflow pipelines.
• Measure and report p50/p95/p99 latency, jitter, tokens/s, time-to-first-token, cold-start/warm-start, and cost-per-token/request across models, precisions, batch sizes, and GPU types.
• Maintain a corpus of representative scenarios and data sets; automate comparisons across software releases and hardware generations.
• Build CI/CD pipelines and K8s controllers/operators to schedule benchmarks at scale; integrate with observability stacks and results warehouses.
• Implement supply-chain integrity for benchmark artifacts (SBOMs, Cosign signatures).
• Partner with NVIDIA, key ISVs, and OSS projects to co-develop optimizations and upstream improvements.
• Support Sales/SEs with authoritative numbers for RFPs and competitive evaluations; brief analysts and press with rigorous, defensible data.

Requirements

10+ years building distributed systems or HPC/cloud services, with deep expertise on large-scale ML training or similar high-performance workloads.

Proven track record of architecting or building planet-scale data systems (e.g., telemetry platforms, observability stacks, cloud data warehouses, large-scale OLAP engines).

Deep understanding of GPU performance (CUDA, NCCL, RDMA, NVLink/PCIe, memory bandwidth), model-server stacks (Triton, vLLM, TensorRT-LLM, TorchServe), and distributed training frameworks (PyTorch FSDP/DeepSpeed/Megatron-LM).

Proficient with Kubernetes and ML control planes; familiarity with SUNK, Kueue, and Kubeflow in production environments.

Excellent communicator able to interface with executives, customers, auditors, and OSS communities.

Nice to have:

• Experience with time-series databases, log-structured merge trees (LSM), or custom storage engine development.
• Experience running MLPerf submissions (Inference and/or Training) or equivalent audited benchmarks at scale.
• Contributions to MLPerf, Triton, vLLM, PyTorch, KServe, or similar OSS projects.
• Experience benchmarking multi-region fleets and large clusters (thousands of GPUs).
• Publications/talks on ML performance, latency engineering, or large-scale benchmarking methodology.

Qualifications

Master's degree in Computer Science, Electrical Engineering, or related field.

Hands-on experience with large-scale data systems and distributed systems.

Strong problem-solving and analytical skills.

Ability to work independently and as part of a team.

Skills

Experience with large-scale ML training or similar high-performance workloads.

Expertise in GPU performance (CUDA, NCCL, RDMA, NVLink/PCIe, memory bandwidth).

Knowledge of model-server stacks (Triton, vLLM, TensorRT-LLM, TorchServe).

Familiarity with distributed training frameworks (PyTorch FSDP/DeepSpeed/Megatron-LM).

Proficiency with Kubernetes and ML control planes.

Familiarity with SUNK, Kueue, and Kubeflow in production environments.

Excellent communication skills.

Experience with time-series databases, log-structured merge trees (LSM), or custom storage engine development.

Experience running MLPerf submissions (Inference and/or Training) or equivalent audited benchmarks at scale.

Contributions to MLPerf, Triton, vLLM, PyTorch, KServe, or similar OSS projects.

Experience benchmarking multi-region fleets and large clusters (thousands of GPUs).

Publications/talks on ML performance, latency engineering, or large-scale benchmarking methodology.

Benefits

The base salary range for this role is $206,000 to $333,000. The starting salary will be determined based on job-related knowledge, skills, experience, and market location.

In addition to base salary, our total rewards package includes a discretionary bonus, equity awards, and a comprehensive benefits program (all based on eligibility).

These benefits include:

• Medical, dental, and vision insurance - 100% paid for by CoreWeave
• Company-paid Life Insurance
• Voluntary supplemental life insurance
• Short and long-term disability insurance
• Flexible Spending Account (FSA)
• Health Savings Account (HSA)
• Tuition Reimbursement
• Ability to participate in Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health
• Family-Forming support provided by Carrot
• Paid Parental Leave
• Flexible, full-service childcare support with Kinside
• 401(k) with a generous employer match
• Flexible PTO
• Catered lunch each day in our office and data center locations
• A casual work environment
• A work culture focused on innovative disruption

What We Offer

CoreWeave offers a competitive salary, a variety of benefits, and a casual work environment. These include:

• Medical, dental, and vision insurance
• Company-paid Life Insurance
• Voluntary supplemental life insurance
• Short and long-term disability insurance
• Flexible Spending Account (FSA)
• Health Savings Account (HSA)
• Tuition Reimbursement
• Ability to participate in Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health
• Family-Forming support provided by Carrot
• Paid Parental Leave
• Flexible, full-service childcare support with Kinside
• 401(k) with a generous employer match
• Flexible PTO
• Catered lunch each day in our office and data center locations
• A casual work environment
• A work culture focused on innovative disruption

Export Control Compliance

This position requires access to export controlled information. To conform to U.S. Government export regulations applicable to that information, applicant must either be (A) a U.S. person, (B) eligible to access the export controlled information without a required export authorization, or (C) eligible and reasonably likely to obtain the required export authorization from the applicable U.S. government agency. CoreWeave may, for legitimate business reasons, decline to pursue any export licensing process.