Member of Technical Staff - Research Engineer

About the role

Black Forest Labs is a frontier research lab developing generative models for image and video creation, including Latent Diffusion, Stable Diffusion, and FLUX. The Research Engineer will improve the performance, reliability, and numerical stability of large-scale training systems by optimizing GPU workloads, profiling distributed training, developing low-precision and kernel-level improvements, and collaborating with researchers.

Responsibilities

• Improve the performance, reliability, and numerical stability of production training runs for large multimodal generative models
• Profile full training steps across model code, attention, kernels, data loading, encoders, communication, optimizer steps, checkpointing, and memory pressure
• Implement and validate GPU-level optimizations: fused kernels, attention paths, low-precision matmuls, quantization kernels, CUDA/Triton/CuTe/CUTLASS experiments, and no-compile alternatives where they make sense
• Push lower-precision training forward, including FP8 / MXFP8 / FP4-style paths, weight and activation quantization, accumulation choices, convergence risk, and quality tradeoffs against baseline training runs
• Work with researchers to translate architecture changes into efficient training implementations, and help distinguish real model-quality progress from changes that only look good in a microbenchmark
• Debug distributed training failures: NaNs, loss spikes, silent numerical drift, memory leaks, stragglers, bad nodes, NCCL issues, and throughput cliffs
• Build benchmarking and profiling harnesses that make performance claims trustworthy across hardware, shapes, sequence lengths, and training configurations
• Help the training team move quickly when an urgent bottleneck appears, while turning repeated failures into better abstractions and tools