Data/Full Stack Engineer, Data Storage & Ingestion Consultant

Responsibilities

• Architect ingest & storage: Choose and implement an on-prem hardware and data pipeline design or a cloud/S3 alternative with explicit cost and performance tradeoffs at multi-petabyte scale.
• Set up a sustained-write ingest path ≥1 GB/s with adequate burst headroom (camera/frame-to-disk), including networking considerations, cooling, and throttling safeguards.
• Incorporate on-the-fly compression/downsampling options and quantify CPU budget vs. write-speed tradeoffs; document when/where to compress to control $/PB.
• Integrate with acquisition workflows ensuring image data and metadata are compatible with downstream stitching/flat-field correction pipelines.
• Enable downstream compute: Expose the data to segmentation/analysis stacks (local GPU nodes or cloud).

Requirements

• 5+ years designing and deploying high-throughput storage or HPC pipelines (≥1 GB/s sustained ingest) in production.
• Deep hands-on with: NVMe RAID/striping, ZFS/MDRAID/erasure coding, PCIe topology, NUMA pinning, Linux performance tuning, and NIC offload features.
• Proven delivery of multi-GB/s ingest systems and petabyte-scale storage in production (life-sciences, vision, HPC, or media).
• Experience building tiered storage systems (NVMe → HDD/object) and validating real-world throughput under sustained load.
• Practical S3/object-storage know-how (AWS S3 and/or on-prem S3-compatible systems) with lifecycle, versioning, and cost controls.
• Data integrity & reliability: snapshots, scrubs, replication, erasure coding, and backup/DR for PB-scale systems.
• Networking: 25/40/100 GbE (SFP+/SFP28), RDMA/ RoCE/iWARP familiarity; switch config and path tuning.
• Ability to spec and rack hardware: selecting chassis/backplanes, RAID/HBA cards, NICs, and cooling strategies to prevent NVMe throttling under sustained writes.

Skills

• Experience with microscopy or scientific imaging ingest at frame-to-disk speeds, including Micro-Manager-based pipelines and raw-to-containerized format conversions.
• Experience with life science imaging data a plus.

Outcomes (what Success Looks Like)

• Within 2 weeks: Implement an immediate data-handling strategy that reliably ingests our initial data streams.
• Within 2 weeks: Deliver a documented medium-term data architecture covering storage, networking, ingest, and durability.
• Within 1 month: Operationalize the medium-term pipeline in production (ingest → buffer → long-term store → compute access).
• Ongoing: Maintain ≥95% uptime for the end-to-end data-handling pipeline after setup.

Pay

Contract, $100-300/hour

Schedule

Immediate start