AI Research Computing Infrastructure Engineer

Key Roles/Responsibilities

• Design and implement next-generation high-performance computing (HPC) environments that leverage container-driven workflows for GPU-accelerated research.
• Build and maintain container orchestration systems for batch and distributed workloads.
• Integrate containerized job workflows with existing HPC schedulers and storage systems.
• Develop and maintain job templates for batch GPU training and multi-node distributed computing.
• Automate deployment, configuration, and scaling through infrastructure-as-code and CI/CD practices.
• Monitor, benchmark, and optimize system performance, reliability, and resource utilization.
• Collaborate with researchers to containerize and optimize legacy workflows for scalable execution.
• Lead evaluation of emerging tools (e.g., Prefect, Ray, Airflow, Dagster) for workflow orchestration and distributed computing.
• Contribute to the development of tools and bridges between orchestration frameworks and traditional HPC environments.

Basic Qualifications

• Possession of Bachelor’s degree from an accredited college/university according to the Council for Higher Education Accreditation (CHEA) or four (4) years relevant experience in lieu of degree. Foreign degrees must be evaluated for U.S. equivalency.
• In addition to the education requirement, a minimum of eight (8) years of related experience.
• Strong Linux systems engineering and administration experience.
• Hands-on experience with container orchestration tools such as Kubernetes, Nomad, Run:AI, etc.
• Hands-on experience with scripting/programming skills (Python, Bash, or Go) for automation, monitoring, and job orchestration.
• Experience with infrastructure-as-code / automation tooling (Terraform, Ansible, Packer, or equivalent).
• Familiarity with system performance analysis, monitoring, and tuning.
• Comfortable with small-team environments and taking end-to-end ownership of compute infrastructure.
• Ability to obtain and maintain a security clearance.

PREFERRED QUALIFICATIONS

• Experience with multi-node distributed ML frameworks (PyTorch DDP, Ray, Horovod, TensorFlow, etc.).
• Familiarity with pipeline orchestration tools (Prefect, Airflow, Dagster, Kubeflow).
• Understanding of resource management and scheduling concepts (queues, allocations, GPU device plugins, gang scheduling, multi-node coordination).
• Understanding of storage integration with high-performance clusters (POSIX + object storage, VAST or similar).
• Familiarity with cloud GPU environments (AWS, GCP, Azure) and hybrid workflows.
• Familiarity with workflow orchestration/pipeline tools (Argo, Kubeflow, Ray, MLFlow).
• Good communication and documentation skills, the ability to make complex infrastructure understandable to researchers and other engineers.