Software Engineer, Inference Infrastructure
Tesla · Palo Alto, CA · 1 mo ago
On-siteEngineering$140k–$300k/yrFull-time
What You'll Do
- Own & scale the AI inference cluster — the physical and software platform that runs AI workloads on custom AI hardware across thousands of boards
- Build & improve job scheduling, hardware onboarding, and cluster self-healing systems that keep the fleet running at 95%+ uptime
- Design & implement inference pipelines that unify evals, sims, rollouts, and visualizations across AI and Optimus teams
- Build developer tooling that makes compiler-produced artifacts easy to run, validate, and debug on real hardware at scale
- Contribute to flashing, inventory management, and fleet management infrastructure for different hardware generations
- Work closely with Compiler, AI, and Optimus teams to understand their bottlenecks and build infrastructure that removes them
What You'll Bring
- Strong backend engineering fundamentals — distributed systems, job orchestration, reliability, and scale
- Experience with hardware accelerator infrastructure — TPUs, custom AI chips, or similar; strong understanding of what it means to manage a large fleet of accelerators and keep them healthy/utilized
- Familiarity with cluster orchestration — Kubernetes, SLURM, or similar bare metal & containerized environments
- Proficiency in Python; familiarity with PyTorch, Go or C++ is a plus
- Experience with low-level systems concepts — networking, file systems, process management
- Strong ownership mindset — comfortable navigating ambiguous problems, diving into unfamiliar codebases, and driving things to completion without hand-holding
- Experience building CI/CD pipelines for hardware-in-the-loop validation, expertise in fleet management or device provisioning at scale, and familiarity with gRPC, distributed task queues, or high-throughput data pipelines are nice-to-haves
- Exposure to MLIR or compiler toolchains is a nice-to-have (helpful for working with compiler-produced artifacts and understanding the compilation pipeline)
Benefits
- Competitive pay
- Medical plans
- Family-building, fertility, adoption and surrogacy benefits
- Dental (including orthodontic coverage) and vision plans
- Company Paid (Health Savings Accounts) HSA Contribution
- 401(k) with employer match
- Employee Stock Purchase Plans
- Other financial benefits
- Basic Life, AD&D Short-term and long-term disability insurance
- Employee Assistance Program
- Sick and Vacation time
- Paid Holidays
- Commuter benefits
- Employee discounts and perks program
Expected Compensation
$140,000 - $300,000/annual salary + cash and stock awards + benefits