Principal Systems Software Engineer
Position Overview
We are looking for a Principal Engineer to architect, build, and own the end-to-end data pipeline that drives our high-throughput diagnostic instrument platform.
Key Responsibilities
Own the architecture of the complete data path from image acquisition to final processed output
Design pipeline stages with clear interfaces, flow control, and backpressure mechanisms
Ensure the pipeline sustains continuous high-throughput operation across extended instrument runs
Define data formats, handoff protocols, and buffering strategies between pipeline stages
Architect for graceful degradation — the system must handle transient failures without data loss or pipeline stalls
Establish performance budgets and SLAs for each pipeline stage and monitor adherence
Image Acquisition & On-Instrument Processing
Develop and optimize real-time image acquisition from high-speed sensors on the instrument
Implement low-latency, high-bandwidth data capture with minimal frame loss
Design on-instrument preprocessing stages that reduce data volume before offload
Manage memory and storage constraints within the instrument compute environment
Ensure deterministic, repeatable performance under sustained acquisition loads
GPU-Accelerated Signal & Image Processing
Develop and maintain GPU compute pipelines using CUDA for signal and image processing
Implement DSP algorithms including frequency-domain analysis, deconvolution, filtering, and detection
Manage host-to-GPU data transfers and ensure efficient use of GPU resources
Profile GPU workloads to identify issues and validate performance headroom
Balance numerical accuracy against throughput requirements
Job Orchestration & Distributed Processing
Design and implement job queuing, scheduling, and orchestration across instrument, local HPC, and cloud compute
Build robust work distribution that maximizes resource utilization across heterogeneous compute
Implement backpressure handling so upstream stages throttle gracefully when downstream is saturated
Design comprehensive error handling, retry logic, and dead-letter strategies for failed jobs
Implement priority scheduling to balance real-time instrument processing with batch reprocessing
Monitor queue depths, processing latencies, and resource utilization with actionable alerting
Linux Systems & Performance
Configure and tune Linux systems for reliable, high-throughput operation across instrument and HPC nodes
Tune kernel parameters (scheduler, NUMA, IRQs, huge pages) as needed for stable pipeline performance
Understand and manage DMA paths, PCIe topology, and device-to-memory data movement
Profile and diagnose system-level issues using perf, ftrace, eBPF, and similar tools
Ensure system configurations are reproducible and documented across instrument and HPC environments
HPC Compute Platform & Algorithm Infrastructure
Co-design the HPC compute platform architecture with DevOps — define computational requirements, job flow, and data access patterns while DevOps provisions and manages the infrastructure
Define how algorithms are deployed, versioned, and rolled into production on the HPC platform — support safe side-by-side execution of new and existing algorithm versions
Design compute allocation strategies that balance real-time instrument processing, batch algorithm development/validation, and historical data reprocessing
Design the data handoff between instrument-side processing and HPC/cloud compute — formats, staging, transfer protocols
Specify when and how workloads should burst from local HPC to cloud (AWS) based on pipeline load and priority
Optimize data movement across high-speed networks (RDMA, InfiniBand, high-speed Ethernet) between instrument, HPC, and storage
Design for scalability — the architecture must accommodate increasing instrument throughput, additional instruments, and growing algorithm complexity
Reliability & Observability
Instrument every pipeline stage with metrics, logging, and tracing
Build real-time dashboards showing pipeline health, throughput, latency, and queue state
Design automated recovery mechanisms for common failure modes
Implement data integrity checks and validation at pipeline stage boundaries
Support root-cause analysis and post-mortem investigation for pipeline incidents
Establish runbooks and operational procedures for pipeline operations
Qualifications
Education: PhD – 10 yrs, MS – 14 yrs and BS – 17 yrs of experience in Computer Science, Electrical Engineering, or related field.
Experience & Technical Leadership: 15+ years of professional software engineering experience in performance-critical systems, track record of architecting and delivering complex, multi-stage data processing pipelines, demonstrated technical leadership — ability to drive architecture decisions and mentor engineers, experience operating systems at industrial-grade reliability and throughput requirements, systems programming & GPU computing, expert-level C/C++ and systems programming on Linux, solid experience with CUDA programming and GPU pipeline development (required), strong understanding of computer architecture: CPU caches, NUMA, memory hierarchies, PCIe, DMA, experience with Python for tooling, orchestration, and pipeline glue, experience with performance profiling and diagnostics tools (perf, ftrace, Nsight, or similar), pipeline & orchestration experience designing multi-stage data pipelines with flow control, buffering, and backpressure management, strong understanding of error handling, retry strategies, and fault recovery in performance-critical systems, experience with job scheduling and work distribution across heterogeneous compute resources, familiarity with workflow orchestration frameworks (Airflow, Celery, custom solutions, or similar) is a plus, signal processing & algorithms, practical experience implementing DSP or image processing algorithms in production systems, familiarity with frequency-domain analysis, filtering, and detection algorithms, ability to reason about numerical accuracy and throughput tradeoffs, data movement, storage & networking, experience optimizing data transfer across high-speed networks (RDMA, InfiniBand, high-speed Ethernet), understanding of shared storage architectures, tiered storage strategies, and high-throughput data staging, experience defining compute platform requirements and collaborating effectively with infrastructure teams, familiarity with algorithm deployment and versioning in production computing environments