Data Scientist

About the role

The Data Scientist will play a crucial role in evaluating and improving a suite of locally-hosted language models. The objective is to ensure these models provide high-quality responses and operate reliably under various conditions. This role requires expertise in designing and implementing evaluation frameworks, testing for adversarial conditions, and developing predictive models that can forecast operational trends.

Responsibilities

• Evaluation & Response Quality: Design and own evaluation harnesses for LLM and agentic outputs; build and calibrate LLM-as-judge pipelines; validate judges against human labels and control for bias and variance.
• Adversarial & Robustness Testing: Red-team the system with prompt injection, jailbreaks, tool-misuse, and edge-case discovery; design chaos and stress tests to probe model and agent reliability under degraded or hostile conditions.
• Retrieval & Agentic Trajectory Analysis: Evaluate retrieval quality over the document corpus; analyze multi-step agent trajectories; assess intent classification and routing quality.
• Behavioral Regression & Drift: Build standing evaluation that catches quality and behavioral regressions; monitor output distribution and quality drift in production; recommend and validate fixes through levers available with local models.
• Predictive & Trend Modeling: Build, ship, and own production models that forecast and surface trends from operational telemetry; wire predictive signals into the LLM and agentic layer so forecasts and trends feed reasoning, advisories, and operator-facing recommendations.
• Domain & Value Analytics: Apply AIOps/NOC analysis where it's the product; quantify the economics of the system; connect it to customer-facing value metrics like MTTR and operator-hours.
• Method & Innovation: Use LLM-assisted workflows to scale the work itself; draft analyses, generate synthetic evaluation cases, and bootstrap labeled data for human refinement.

Requirements

• Bachelor's or Master's in Data Science, Computer Science, Statistics, Mathematics, or a related field or equivalent experience.
• 3+ years in data science, ML, or applied quantitative analysis.
• Strong applied statistics, with the judgment to design sound experiments and significance tests on noisy, non-deterministic outputs.
• Experience building, deploying, and monitoring predictive or time-series models in production.
• Demonstrated work evaluating, analyzing, or improving LLM or NLP systems.
• Proficiency in Python.
• Strong SQL and comfort querying large analytical datasets.
• Fluency with foundation models and hands-on experience with the modern LLM evaluation and tooling layer.
• Ability to build analysis and visualization in code.

Qualifications

• Experience getting strong results out of small or self-hosted/local models under compute, memory, or latency constraints.
• Experience with retrieval-augmented systems and retrieval evaluation at scale.
• Experience with agentic frameworks and tool-use/orchestration analysis, including human-in-the-loop and replayable-state patterns.
• Familiarity with red-teaming or adversarial robustness for LLMs.
• Domain background in IT operations — AIOps, NOC, ITSM, observability, or anomaly detection on logs and telemetry.
• Experience with large-scale analytical and big-data stores.
• Cloud experience for data science and ML workloads.
• Exposure to enterprise security and compliance constraints in a delivery context.