Staff Software Engineer - Forecast Engine

About the role

Join us to put AI to work for people. We operate like a small startup, moving quickly, delivering early, keeping process light, and keeping momentum.

Responsibilities

• Design and develop scalable, maintainable, and reusable software components with a strong emphasis on performance, determinism, and reliability.
• Collaborate with product managers and FinOps partners to translate planning and budgeting requirements into well-architected solutions, owning features from design through delivery.
• Build intuitive and extensible interfaces for forecast consumption (Lightdash models, alert payloads, and APIs) ensuring flexibility for finance and capacity-planning use cases.
• Contribute to the design and implementation of new Forecast Engine capabilities while enhancing existing simulation, validation, and publish paths.
• Integrate automated testing into development workflows to ensure consistent quality across releases, including determinism (byte-identical output) and forecast-accuracy regression checks.
• Develop comprehensive test strategies covering functional, regression, integration, and accuracy aspects (period-over-period identity, backtest grading against real actuals).
• Foster a culture of continuous learning and improvement by sharing best practices in engineering and quality.
• Promote a culture of engineering craftsmanship, knowledge-sharing, and thoughtful quality practices across the team.
• Own the architecture of the Forecast Engine and the automation layer around it: scheduled runs, variance/budget tracking, and alerting.
• Lead technical decision-making on forecast cadence, reconciliation against actuals, alert routing, and the contract between the simulation core and downstream consumers.
• Establish best practices for forecast automation: idempotent scheduled runs, deterministic reproducibility, fail-loud data contracts, and no silent fallbacks.
• Define how forecast signals (variance, budget breach, capacity headroom, migration drift) are computed, thresholded, and surfaced.
• Drive innovation in forecasting and planning automation, including the responsible use of AI/ML tooling to accelerate development and analysis.
• Build the automation that runs the Forecast Engine on a schedule via Argo Workflows, with retries, alerting on failure, and run-to-run reproducibility.
• Implement variance and budget tracking: reconcile each forecast against plan and against the latest actuals, compute deltas at the grains that matter (provider, region, pod, workload), and persist a queryable variance history.
• Implement alerting that fires on budget breach, forecast drift, capacity thresholds, and pipeline health, routed to Splunk and the team's notification channels.
• Integrate with planning systems so plan/budget targets flow into the engine and forecast outputs flow back out to the planning surface.
• Drive the Future Capacity Reservation (FCR) handoff: translate the forecast of fleet growth and migration timing into reservation recommendations (how much capacity, which providers/regions/pods, and by when), aligned to hyperscaler procurement lead-time windows and reconciled with Cloud Operations so the same capacity is never reserved twice.
• Create and extend the Rust simulation core (period loop, growth, migration, routing, packing, sizing, validation) and its streaming Trino read and Iceberg publish paths.
• Create and maintain the Lightdash forecast and variance marts (standard dbt models on the published tables) that finance and capacity partners consume.
• Design the forecast data contract (the upstream view the engine reads) so data-quality problems halt loudly and are fixed at the source, never papered over downstream.
• Implement scheduled, observable forecast runs with full run lineage: inputs, seed, config, output location, and metrics for every run.
• Establish observability and monitoring for the Forecast Engine: run success rates, forecast latency, memory ceilings, accuracy drift, and alert-delivery health, emitted to Splunk and the observability stack.
• Establish an automation foundation that scales from a handful of scheduled scenarios to a broad, multi-scenario forecasting program.
• Create and promote scheduled, parameterized forecast scenarios with opinionated structure: pinned config, deterministic seeds, validated inputs, and published outputs.
• Establish guardrails: input data contracts, resource/memory ceilings, and loud halts that surface real problems instead of producing wrong-but-quiet numbers.
• Collaborate closely with FinOps analysts and capacity planners to rapidly iterate on variance definitions, alert thresholds, and the signals that matter, without over-engineering.
• Prioritize forecast reliability, accuracy tracking, and clear alerting over feature breadth.
• Use modern AI development tools (e.g., Claude Code, Cursor, GitHub Copilot) to accelerate development, testing, and analysis, and help the team adopt effective, well-validated AI-assisted practices.
• Collaborate with DevOps and platform teams on scheduling infrastructure, CI/CD pipelines, and Splunk/observability integration.
• Partner with FinOps Tools team members working on Trino, dbt, Lightdash, and Iceberg to ensure seamless integrations.
• Partner with finance and capacity-planning stakeholders to ensure forecasts, variance, and alerts map to how they actually plan and budget.

Qualifications

• Experience in leveraging or critically thinking about how to integrate AI into work processes, decision-making, or problem-solving.
• Strong skills in a systems or backend language (Rust, Go, Java, C++, or similar) and in Python for data tooling, automation, and analysis.
• Proven track record building automated, scheduled data or forecasting pipelines that run reliably in production.
• Demonstrated ability to deliver at high velocity: shipping production-quality software fast, in tight iteration loops, without sacrificing reliability.
• Proven track record of greenfield development and building from scratch in environments with evolving requirements.
• Hands-on experience building variance/anomaly detection, budget or SLA tracking, or alerting systems at scale.
• Experience integrating with observability and logging platforms (Splunk, Datadog, Prometheus/Grafana, or similar).
• Experience with workflow orchestration systems (Argo, Airflow, or similar) and with the modern data stack.
• Strong knowledge of data structures, algorithms, object-oriented and data-oriented design, design patterns, and performance optimization.
• Familiarity with automated testing frameworks and integrating tests into CI/CD pipelines.
• Understanding of software quality principles including reliability, determinism, observability, and production readiness.
• Ability to troubleshoot complex systems and optimize performance and memory across the stack.
• Experience validating data correctness: reconciling pipeline outputs against ground-truth actuals and catching silent regressions.
• Comfort with development tools such as IDEs, debuggers, profilers, source control, and Unix-based systems.
• Full professional proficiency in English.
• Forecasting & simulation: time-series or simulation-based forecasting, scenario modeling, and reconciliation of forecasts against actuals.
• Variance & alerting: budget vs. actual tracking, anomaly/threshold detection, alert routing, and noise control (deduplication, suppression, severity).
• Observability: Splunk (search, dashboards, alerts) and metrics/logging integration for pipeline and forecast health.
• Orchestration: Argo Workflows or similar: scheduled runs, retries, idempotency, failure alerting.
• Modern data stack: Trino, dbt, Iceberg, Lightdash, or similar lakehouse architecture.

Skills

• Experience in leveraging or critically thinking about how to integrate AI into work processes, decision-making, or problem-solving.
• Strong skills in a systems or backend language (Rust, Go, Java, C++, or similar) and in Python for data tooling, automation, and analysis.
• Proven track record building automated, scheduled data or forecasting pipelines that run reliably in production.
• Demonstrated ability to deliver at high velocity: shipping production-quality software fast, in tight iteration loops, without sacrificing reliability.
• Proven track record of greenfield development and building from scratch in environments with evolving requirements.
• Hands-on experience building variance/anomaly detection, budget or SLA tracking, or alerting systems at scale.
• Experience integrating with observability and logging platforms (Splunk, Datadog, Prometheus/Grafana, or similar).
• Experience with workflow orchestration systems (Argo, Airflow, or similar) and with the modern data stack.
• Strong knowledge of data structures, algorithms, object-oriented and data-oriented design, design patterns, and performance optimization.
• Familiarity with automated testing frameworks and integrating tests into CI/CD pipelines.
• Understanding of software quality principles including reliability, determinism, observability, and production readiness.
• Ability to troubleshoot complex systems and optimize performance and memory across the stack.
• Experience validating data correctness: reconciling pipeline outputs against ground-truth actuals and catching silent regressions.
• Comfort with development tools such as IDEs, debuggers, profilers, source control, and Unix-based systems.
• Full professional proficiency in English.
• Forecasting & simulation: time-series or simulation-based forecasting, scenario modeling, and reconciliation of forecasts against actuals.
• Variance & alerting: budget vs. actual tracking, anomaly/threshold detection, alert routing, and noise control (deduplication, suppression, severity).
• Observability: Splunk (search, dashboards, alerts) and metrics/logging integration for pipeline and forecast health.
• Orchestration: Argo Workflows or similar: scheduled runs, retries, idempotency, failure alerting.
• Modern data stack: Trino, dbt, Iceberg, Lightdash, or similar lakehouse architecture.

Benefits

We offer a competitive benefits package including:

• Health insurance
• Retirement savings plan
• Flexible work arrangements
• Professional development opportunities
• Employee recognition programs

Pay

Competitive salary based on experience and qualifications.

Schedule

Remote work option available.