AI Storage Solutions Expert
Bitdeer is a world-leading technology company for AI and Bitcoin mining infrastructure, committed to providing comprehensive Bitcoin mining solutions and building AI computational infrastructure. Headquartered in Singapore, Bitdeer has deployed data centers across the United States, Norway, Bhutan, and Ethiopia.
About the role
You own the IO layer that trains the models—and the signals we need to predict storage faults before a checkpoint stalls a $50M training run. Bitdeer is building an AI-operated GPU cloud. Storage is where AI workloads either fly or fall over: a slow parallel read can starve a 1,000-GPU job; a stalled checkpoint can waste a full training epoch. In this role, you deploy and operate the high-performance storage layer for AI training and inference across NeoCloud's US data centers, and you feed the AIOps substrate with the signals it needs to catch storage regressions before they page a customer.
Responsibilities
- Deploy and operate parallel/distributed storage systems: WEKA, VAST Data, Ceph, DDN/Lustre.
- Design storage architectures optimized for AI workload patterns—checkpoint I/O bursts, sequential dataset reads, KV cache for inference.
- Implement multi-tenant storage isolation with per-tenant QoS, quotas, and access controls; configure and optimize GPU Direct Storage for direct GPU-to-storage data paths.
- Deploy and manage storage networking (NFS over RDMA, NVMe-oF, high-speed storage fabrics) and Nvidia CMX for cluster-wide storage orchestration.
- Diagnose and tune storage performance: IOPS, throughput, latency profiling with fio, IOR, mdtest; own the runbook for common failure modes.
- Plan storage capacity aligned with GPU cluster growth and customer workload projections; manage firmware, data migration, and DR procedures.
- Instrument storage telemetry—IO tail latency, checkpoint durations, NVMe SMART, filesystem health, RDMA counters—into the metrics/logs/traces store the platform team runs.
- Partner with the platform team to define the storage-fault predictor: which signals, which labels (from your incidents), which false-positive tolerances.
- Convert every novel incident into an automation: SOPs become runbook-as-code, runbook-as-code becomes an agent-executable remediation.
What Success Looks Like
- In Year 1: Observability and a baseline predictor for the top 3 storage-fault classes on our fabric.
- Storage-incident MTTR measurably lower than at hire.
- The Nvidia GB200-class clusters we build out ship on your storage design.
Requirements
- 5+ years in enterprise or HPC storage operations, with at least 2 years supporting AI/ML workloads.
- Hands-on deployment and operations experience with at least two of: WEKA, VAST Data, Ceph, DDN/Lustre.
- Strong understanding of AI training I/O patterns: checkpoint frequency, dataset loading, shuffle buffers.
- Experience with high-performance storage networking (NFS over RDMA, NVMe-oF).
- Knowledge of GPU Direct Storage and RDMA-based data transfer.
- Proficiency in storage performance benchmarking and tuning (fio, IOR, mdtest).
- Experience implementing multi-tenant storage with isolation and QoS.
- Strong Linux systems knowledge (kernel tuning, filesystem internals, block device management).
- Instinct for turning ops toil into ML signal—you've either shipped an anomaly detector for storage/IO telemetry or you can articulate the labels and features you'd need to.
- Runbook-as-code mindset—every SOP you write should be executable by a machine within a quarter.