Staff Software Engineer — Search Platform, Ingestion & Indexing

About the role

The Advanced Content Engineering (ACE) team is seeking a Staff Software Engineer to lead the design, implementation, and operational health of the document ingestion pipeline and search index management systems.

Responsibilities

• Design, develop, and own the end-to-end document ingestion pipeline, including fault tolerance, version ordering, and at-least-once delivery guarantees.
• Own the platform's Protobuf-based document schema and schema registry integration, establishing schema governance standards and ensuring reliable serialization.
• Lead the migration of ingestion infrastructure from OpenSearch to Vespa, including design of Vespa document processors, custom Kafka feeders, and application package architecture.
• Define, build, test, deploy, scale, and operate custom models integrated into the ingestion pipeline, including inference serving, latency SLO management, and scaling.
• Build and operate the model promotion pipeline, ensuring the platform team can operate model updates independently without depending on the research team for production changes.
• Develop and maintain integration contracts between custom models and downstream pipeline components, ensuring search pipeline consumers are not broken by changes upstream.
• Instrument model serving for production observability, enabling the team to detect regressions or model drift without requiring the fine-tuning team's involvement.
• Own the search engine layer end-to-end, design and operate Vespa (and OpenSearch during transition) index configurations, ranking profiles, schema definitions, and application package lifecycle management.
• Implement and maintain zero-downtime index management, including shadow indexing, blue/green index promotion, and rolling reindex workflows.
• Design and implement resilient fault tolerance mechanisms, such as dead-letter queues, retry strategies with exponential backoff, circuit breakers, and consumer lag monitoring.
• Design and implement agentic search infrastructure, including explicit latency budgets per retrieval hop, chunking and result compression strategies optimized for token economy in context windows, and index boundary definitions that give agents clean, predictable tool contracts.
• Instrument the query and retrieval stack for online analytics, including real-time query latency and throughput monitoring, query log collection for session analysis, and the infrastructure to support A/B and interleaved ranking experiments in production.
• Partner with TR Labs and research scientists to ensure that new search components can be evaluated in isolation, with automated offline evaluation on every build and a clear path from evaluation results to production promotion decisions.
• Treat delivery friction as the enemy, identifying and removing obstacles that slow the team's ability to ship ingestion and indexing changes to production safely and frequently.
• Collaborate closely with TR Labs and research scientists to integrate new chunking strategies, embedding models, and enrichment techniques into the pipeline in a safe, well-instrumented, and ethically responsible way.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8+ years of software engineering experience, with demonstrated progression to staff-level or equivalent technical leadership — including ownership of a functional area and leadership of significant cross-functional projects.
• Deep expertise in distributed stream processing: designing, building, and operating high-throughput, fault-tolerant event-driven pipelines using Kafka or equivalent technologies at production scale.
• Production experience with Vespa, OpenSearch, or Elasticsearch — including schema design, ranking profile configuration, and end-to-end application lifecycle management.
• Mastery of Python with strategic awareness of language and framework selection; strong software engineering fundamentals including test strategy, performance architecture, and system design.
• Proficiency with AWS cloud services used in data pipeline and search infrastructure (MSK, ECS, Lambda, DynamoDB, Step Functions, CloudWatch), with infrastructure-as-code experience (Terraform or AWS CDK).
• Demonstrated ability to take full operational responsibility end-to-end — defining SLOs, building observability, running on-call, and driving systematic improvements from incident retrospectives — with a track record of shipping to production frequently and removing delivery friction proactively.
• Comfort and fluency with AI-assisted development tools; you use them to move faster and produce higher-quality work, not as a novelty.
• Track record of establishing architectural principles, cross-system design patterns, and documentation standards that improve the broader team’s engineering quality.