Principal Systems Software Engineer

Position Overview

We are looking for a Principal Engineer to architect, build, and own the end-to-end data pipeline that drives our high-throughput diagnostic instrument platform.

Key Responsibilities

• Own the architecture of the complete data path from image acquisition to final processed output

• Design pipeline stages with clear interfaces, flow control, and backpressure mechanisms

• Ensure the pipeline sustains continuous high-throughput operation across extended instrument runs

• Define data formats, handoff protocols, and buffering strategies between pipeline stages

• Architect for graceful degradation — the system must handle transient failures without data loss or pipeline stalls

• Establish performance budgets and SLAs for each pipeline stage and monitor adherence

Image Acquisition & On-Instrument Processing

• Develop and optimize real-time image acquisition from high-speed sensors on the instrument

• Implement low-latency, high-bandwidth data capture with minimal frame loss

• Design on-instrument preprocessing stages that reduce data volume before offload

• Manage memory and storage constraints within the instrument compute environment

• Ensure deterministic, repeatable performance under sustained acquisition loads

GPU-Accelerated Signal & Image Processing

• Develop and maintain GPU compute pipelines using CUDA for signal and image processing

• Implement DSP algorithms including frequency-domain analysis, deconvolution, filtering, and detection

• Manage host-to-GPU data transfers and ensure efficient use of GPU resources

• Profile GPU workloads to identify issues and validate performance headroom

• Balance numerical accuracy against throughput requirements

Job Orchestration & Distributed Processing

• Design and implement job queuing, scheduling, and orchestration across instrument, local HPC, and cloud compute

• Build robust work distribution that maximizes resource utilization across heterogeneous compute

• Implement backpressure handling so upstream stages throttle gracefully when downstream is saturated

• Design comprehensive error handling, retry logic, and dead-letter strategies for failed jobs

• Implement priority scheduling to balance real-time instrument processing with batch reprocessing

• Monitor queue depths, processing latencies, and resource utilization with actionable alerting

Linux Systems & Performance

• Configure and tune Linux systems for reliable, high-throughput operation across instrument and HPC nodes

• Tune kernel parameters (scheduler, NUMA, IRQs, huge pages) as needed for stable pipeline performance

• Understand and manage DMA paths, PCIe topology, and device-to-memory data movement

• Profile and diagnose system-level issues using perf, ftrace, eBPF, and similar tools

• Ensure system configurations are reproducible and documented across instrument and HPC environments

HPC Compute Platform & Algorithm Infrastructure

• Co-design the HPC compute platform architecture with DevOps — define computational requirements, job flow, and data access patterns while DevOps provisions and manages the infrastructure

• Define how algorithms are deployed, versioned, and rolled into production on the HPC platform — support safe side-by-side execution of new and existing algorithm versions

• Design compute allocation strategies that balance real-time instrument processing, batch algorithm development/validation, and historical data reprocessing

• Design the data handoff between instrument-side processing and HPC/cloud compute — formats, staging, transfer protocols

• Specify when and how workloads should burst from local HPC to cloud (AWS) based on pipeline load and priority

• Optimize data movement across high-speed networks (RDMA, InfiniBand, high-speed Ethernet) between instrument, HPC, and storage

• Design for scalability — the architecture must accommodate increasing instrument throughput, additional instruments, and growing algorithm complexity

Reliability & Observability

• Instrument every pipeline stage with metrics, logging, and tracing

• Build real-time dashboards showing pipeline health, throughput, latency, and queue state

• Design automated recovery mechanisms for common failure modes

• Implement data integrity checks and validation at pipeline stage boundaries

• Support root-cause analysis and post-mortem investigation for pipeline incidents

• Establish runbooks and operational procedures for pipeline operations

Qualifications

• Education: PhD – 10 yrs, MS – 14 yrs and BS – 17 yrs of experience in Computer Science, Electrical Engineering, or related field.

• Experience & Technical Leadership: 15+ years of professional software engineering experience in performance-critical systems, track record of architecting and delivering complex, multi-stage data processing pipelines, demonstrated technical leadership — ability to drive architecture decisions and mentor engineers, experience operating systems at industrial-grade reliability and throughput requirements, systems programming & GPU computing, expert-level C/C++ and systems programming on Linux, solid experience with CUDA programming and GPU pipeline development (required), strong understanding of computer architecture: CPU caches, NUMA, memory hierarchies, PCIe, DMA, experience with Python for tooling, orchestration, and pipeline glue, experience with performance profiling and diagnostics tools (perf, ftrace, Nsight, or similar), pipeline & orchestration experience designing multi-stage data pipelines with flow control, buffering, and backpressure management, strong understanding of error handling, retry strategies, and fault recovery in performance-critical systems, experience with job scheduling and work distribution across heterogeneous compute resources, familiarity with workflow orchestration frameworks (Airflow, Celery, custom solutions, or similar) is a plus, signal processing & algorithms, practical experience implementing DSP or image processing algorithms in production systems, familiarity with frequency-domain analysis, filtering, and detection algorithms, ability to reason about numerical accuracy and throughput tradeoffs, data movement, storage & networking, experience optimizing data transfer across high-speed networks (RDMA, InfiniBand, high-speed Ethernet), understanding of shared storage architectures, tiered storage strategies, and high-throughput data staging, experience defining compute platform requirements and collaborating effectively with infrastructure teams, familiarity with algorithm deployment and versioning in production computing environments