Staff Engineer

About the role

We are looking for a hands-on Staff Software Engineer to help design and build our Cluster management platform. DDN – Infinia engineers come from diverse backgrounds and are considered among the highest tier of technology game changers globally. They have created, architected, and pioneered numerous advancements in data-enabling technologies, delivering groundbreaking ideas that have shaped and transformed the storage industry.

Requirements

• 8+ years of backend development experience, including deep proficiency in Go for building high-performance, low-overhead system components.
• Production-Scale Observability Expertise: Hands-on experience implementing and operating telemetry pipelines for software-defined clustered, distributed, or cloud-native solutions.
• Deep Mastery of the Metrics Stack: Strong technical knowledge of Prometheus (operators, alerting rules, scraping mechanics) and the VictoriaMetrics stack for long-term, high-cardinality storage.
• Telemetry Industry Standards: Practical experience with the OpenTelemetry (OTel) ecosystem, including custom OTel collector configurations, instrumentation SDKs, and data processing.
• Systems-Level Troubleshooting: A solid understanding of Linux networking, filesystems, and how clustered storage applications behave under heavy I/O workloads.
• Collaborative Mindset: Proven ability to work effectively across geographically distributed teams, driving technical clarity through code reviews and clear documentation.

Responsibilities

• Execute the Telemetry Architecture: Take on core projects within the observability domain, ensuring seamless integration between our proprietary Go infrastructure and open-source tools.
• Optimize the Observability Stack: Help design and refine how OpenTelemetry, Prometheus, and VictoriaMetrics handle the massive metrics volume generated by our storage cluster.
• Drive Code Excellence: Act as a key technical contributor to our Software Defined Storage control plane, writing clean, performant Go code and providing rigorous code reviews.
• Full Lifecycle Engineering: Participate actively within the Scrum model—from initial design and coding to automated testing, usability reviews, and release.
• Document and Standardize: Ensure our telemetry frameworks are well-documented, making it easy for other engineering teams to instrument their components properly.
• Global Support Rotation: Contribute to our global team on-call rotation, leveraging your own observability tools to provide high-level technical support for our distributed footprint.

Qualifications

• Built and Maintained Telemetry Pipelines: A proven track record of developing or extending proprietary Go components to efficiently ingest, process, and forward massive streams of metrics, logs, and traces.
• Optimized Resource Consumption: Experience managing the CPU and memory footprint of monitoring agents to ensure they do not compete with core storage data paths.
• Strong Testing & Regression Habits: Dedication to writing robust unit and integration tests to ensure telemetry components remain stable during live cluster upgrades.
• Independent Feature Delivery: A proven ability to take ownership of complex technical initiatives and independently make progress in a fast-paced environment.
• Concept Visualization: Ability to turn abstract cluster state data into logical, well-structured telemetry frameworks that bring visibility to complex system scenarios.