Principal Engineer, AI Infrastructure (R4941)

Platform Ownership

Define and operate the core AI and data platform across training, simulation, data management, evaluation, and deployment.

Compute Strategy and Infrastructure

Own where and how workloads run across on-premise, cloud, and hybrid environments. Drive capacity planning, utilization, and cost-per-compute decisions, including support for classified and air-gapped systems.

Training and Simulation Systems

• Build infrastructure for distributed training (supervised learning, RL/MARL, foundation models) and large-scale, multi-fidelity simulation.
• Ensure training and simulation systems operate together without bottlenecks.

Data Platform

• Ingest and manage multi-modal sensor data (EO, IR, radar, EW, IMU).
• Establish dataset versioning, data lineage, feature storage, data cataloging, and classification-aware storage and access controls.

MLOps, Evaluation, and Model Lifecycle

• Establish a consistent workflow for experiment tracking, model registry, artifact provenance, and automated validation.
• Implement evaluation and V&V gates so models meet defined standards before deployment.

Deployment and Operational Feedback

• Own the pipeline from training to deployment, including model optimization (e.g., distillation, quantization, pruning), deployment to edge systems, monitoring, drift detection, and retraining triggers.

Customer AI Infrastructure

• Define how AI infrastructure is deployed in customer environments across on-premise, cloud, hybrid, and sovereign settings.
• Establish a consistent approach that avoids one-off solutions while adapting to operational constraints.

Platform Standardization

Define common tools, interfaces, and workflows across teams. Reduce duplication while maintaining flexibility where needed.

Cross-Team Partnership

Work directly with Hivemind and other autonomy teams to ensure the platform supports real workloads and evolves with program needs.

Key Outcomes

• Faster iteration from idea to trained model to evaluated result
• High utilization of compute resources with clear visibility into usage and cost
• Sustained high utilization of GPU resources under production workloads
• Consistent end-to-end lifecycle: development, evaluation, deployment, monitoring, and retraining
• Repeatable data loop: telemetry, scenario extraction, retraining, and redeployment
• Reliable deployment of optimized models to edge systems
• Broad platform adoption across autonomy programs
• Repeatable approach for deploying AI infrastructure in customer environments

Required Qualifications

• Experience building and operating ML infrastructure at scale (100+ GPU clusters, distributed systems)
• Experience defining compute strategy, including on-premise vs cloud tradeoffs, capacity planning, and cost management
• Strong understanding of ML workloads, including foundation models, RL/MARL, simulation-based training, and fine-tuning
• Ability to debug and resolve system issues when needed

Preferred Qualifications

• Experience in defense or classified environments (e.g., air-gapped systems, SCIFs)
• Experience with simulation-heavy ML systems (robotics, autonomy, or similar domains)
• Experience deploying and optimizing models for edge hardware
• Familiarity with HPC systems (schedulers, parallel storage, high-speed networking)