Senior Solutions Architect, First Time Deployment Validation - NVIS

About the role

The First Time Deployment Team owns first-time execution of NVIDIA's latest products and systems; gathering install and bring-up evidence, operationalizing the validation process, documenting blockers and finding solutions to launch AI Factories at scale. Our results are spread across NVIDIA so we can succeed at scale.

Responsibilities

• Set up, adjust, and verify AI factory environments across multi-GPU and multi-node Linux clusters.
• Ensure configurations align with guidelines for NCCL, collectives, and distributed training frameworks.
• Own the execution of key AI/LLM benchmarks, including setup, orchestration, result collection, and analysis.
• Investigate and resolve issues when training jobs or benchmarks fail, hang, or underperform.
• Build and improve observability for AI factories (metrics, logs, traces, dashboards) to understand workload behavior and system health.
• Develop automation (Python, Shell) for running benchmarks, collecting results, and performing regression checks.
• Examine communication patterns and NCCL usage for AI/LLM workloads, concentrating on collectives such as AllReduce and AllToAll.
• Recommend changes to job configuration, parallelism strategies, and cluster settings to improve throughput, latency, and scaling efficiency.
• Work closely with hardware, software, networking, datacenter, and product teams to prepare AI factories for customer use.
• Contribute to documentation, guidelines, and readiness collateral that support internal collaborators and customer-facing teams.

Requirements

• Bachelor’s degree or equivalent experience in Computer Science, Mathematics, Engineering, Physics, or related field.
• More than 6+ years of experience managing Linux-based systems in HPC, distributed systems, or extensive AI/ML settings.
• Hands-on experience running AI/ML workloads on multi-GPU and/or multi-node clusters, with practical knowledge of NCCL.
• Solid grasp of collective communication patterns, particularly AllReduce and AllToAll, and how they are applied in contemporary ML/LLM training.
• Familiarity with LLM training and/or inference workflows using frameworks such as PyTorch or TensorFlow.
• Proficiency with Python and Shell/Bash for scripting, automation, and tooling.
• Experience with benchmarking (crafting, executing, and interpreting performance benchmarks).
• Comfortable working with observability data (metrics, logs, dashboards) to troubleshoot and optimize complex distributed workloads.
• Strong communication skills and the ability to work effectively with cross-functional teams.

Qualifications

• Experience with AI factory or large-scale AI infrastructure build, deployment, or operations.
• Background in HPC performance engineering, SRE, or systems performance analysis for GPU-accelerated environments.
• Familiarity with observability stacks (e.g., metrics/monitoring, logging, tracing systems) used for large distributed systems.
• Experience building automation and CI-style pipelines for running and validating benchmarks at scale.
• Demonstrated desire to use AI to solve practical problems, improve workflows, and guide data-driven decisions.

Skills

• Experience with AI factory or large-scale AI infrastructure build, deployment, or operations.
• Background in HPC performance engineering, SRE, or systems performance analysis for GPU-accelerated environments.
• Familiarity with observability stacks (e.g., metrics/monitoring, logging, tracing systems) used for large distributed systems.
• Experience building automation and CI-style pipelines for running and validating benchmarks at scale.
• Demonstrated desire to use AI to solve practical problems, improve workflows, and guide data-driven decisions.

Benefits

• Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.
• The base salary range is 148,000 USD - 235,750 USD.
• You will also be eligible for equity and benefits.

Pay

• Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.
• The base salary range is 148,000 USD - 235,750 USD.

Schedule

• Full time