Senior HPC Engineer

About the role

Millennium’s Infrastructure organization designs, engineers, and operates a robust global computing platform supporting WorldQuant’s quantitative research. We are seeking a Senior HPC Engineer to join our team in a senior, hands-on role building and evolving large-scale, high-throughput HPC and GPU platforms that underpin AI- and machine-learning-driven research.

Responsibilities

• Design, build, and operate large-scale, high-throughput HPC and GPU clusters (for example, tens of thousands of CPU cores and hundreds of GPUs) supporting AI and machine-learning workloads.
• Collaborate with other HPC engineers and subject-matter experts to co-design system architectures, review designs, and share knowledge.
• Partner with storage specialists to architect and maintain high-performance, low-latency storage solutions, including parallel or scale-out file systems.
• Work closely with researchers, data scientists, and engineers to understand computational needs and translate them into effective, scalable system designs.
• Monitor, analyze, and optimize performance across compute, scheduling, networking, and storage layers.
• Build and maintain automation and infrastructure-as-code for provisioning, configuration, monitoring, and lifecycle management, with an emphasis on repeatability and simplicity.
• Participate in design reviews, operational discussions, and post-incident reviews with a focus on learning, collaboration, and system improvement rather than blame.
• Explore alternative approaches to scheduling, data layout, cluster architectures, and GPU utilization through small experiments or prototypes, using data to guide decisions.
• Produce clear documentation, diagrams, and reusable tooling that enable others to operate, debug, and extend the platform.
• Stay current with advancements in HPC, GPU computing, networking, and storage, and help assess where new technologies can add real value.

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related technical field; a Master’s or PhD is a plus.
• Typically 7+ years of hands-on experience designing, building, and operating HPC or large-scale compute environments.
• Deep, practical experience with at least one major HPC scheduler (such as Slurm), including using it to operate large-scale or high-throughput clusters in production.
• Hands-on experience with GPU-accelerated computing, including NVIDIA GPUs and associated software ecosystems.
• Strong Linux systems engineering skills and comfort working close to the operating system, drivers, and hardware.
• Experience designing or operating high-performance storage systems, including parallel or scale-out file systems.
• Curious, evidence-driven problem solving, including experimenting with different approaches and using data to inform decisions.
• A collaborative working style that values listening, respectful discussion, and incorporating different perspectives — whether you are more quiet and reflective or more vocal in group settings.
• Clear written and verbal communication skills, and an ability to explain complex ideas in a way that works for different audiences.
• A strong sense of ownership for outcomes, paired with openness to feedback, learning, and evolving systems over time.

Qualifications

• Experience with Kubernetes, Run:ai, or other workload orchestration platforms alongside traditional HPC schedulers.
• Familiarity with Lustre, GPFS / Spectrum Scale, or similar high-performance storage technologies.
• Exposure to cloud-based HPC environments (e.g., GCP or other major cloud providers).
• Experience supporting quantitative research, finance, or other demanding compute-intensive workloads.
• Interest in applying AI or ML techniques to infrastructure (for example, optimization, anomaly detection, or predictive analysis).

Skills

• Strong understanding of HPC architecture, including parallel programming models, MPI, OpenMP, and CUDA.
• Experience with cloud-native tools and services, such as Docker, Kubernetes, and AWS/Azure/GCP services.
• Knowledge of distributed systems, including fault tolerance, load balancing, and data replication.
• Experience with scripting languages like Python, Bash, or Perl for automating tasks and analyzing system performance.
• Ability to troubleshoot and resolve complex technical issues independently or as part of a team.

Benefits

The estimated base salary range for this position is $175,000 to $250,000, which is specific to New York and may change in the future. Millennium pays a total compensation package which includes a base salary, discretionary performance bonus, and a comprehensive benefits package.

Pay

The estimated base salary range for this position is $175,000 to $250,000, which is specific to New York and may change in the future.

Schedule

Full-time, Monday through Friday, with occasional evening and weekend work required for project deadlines.