Staff Software Engineer — Search Platform, Ingestion & Indexing

About the role

The Advanced Content Engineering (ACE) team is seeking a Staff Software Engineer to lead the design, implementation, and operational health of the document ingestion pipeline and search index management systems.

Responsibilities

• Design, develop, and own the end-to-end document ingestion pipeline, including fault tolerance, version ordering, and at-least-once delivery guarantees.
• Own the platform's Protobuf-based document schema and schema registry integration, establishing schema governance standards and ensuring reliable serialization.
• Lead the migration of ingestion infrastructure from OpenSearch to Vespa, including design of Vespa document processors, custom Kafka feeders, and application package architecture.
• Define and maintain integration contracts between custom models and downstream pipeline components, ensuring the platform team can operate model updates independently.
• Instrument model serving for production observability, enabling the team to detect regressions or model drift without requiring the fine-tuning team's involvement.
• Build and operate zero-downtime index management, including shadow indexing, blue/green index promotion, and rolling reindex workflows.
• Develop the Component Registry and Index Registry, focusing on correctness, observability, and safe concurrent modification.
• Implement and maintain the full-reindex and incremental-update orchestration logic, including change detection, document tracking, Kafka topic management, and DynamoDB-backed state management.
• Design and implement agentic search infrastructure, including explicit latency budgets per retrieval hop, chunking and result compression strategies, and index boundary definitions.
• Instrument the query and retrieval stack for online analytics, enabling the team to monitor and analyze real-time query latency and throughput, collect query logs for session analysis, and support A/B and interleaved ranking experiments in production.
• Partner with TR Labs and research scientists to ensure new search components can be evaluated in isolation, with automated offline evaluation on every build and a clear path from evaluation results to production promotion decisions.
• Treat delivery friction as the enemy, identifying and removing obstacles that slow the team's ability to ship ingestion and indexing changes to production safely and frequently.
• Collaborate closely with TR Labs and research scientists to integrate new chunking strategies, embedding models, and enrichment techniques into the pipeline in a safe, well-instrumented, and ethically responsible way.
• Deliver effective presentations on complex technical concepts to both technical and non-technical stakeholders, developing strategic plans for technology implementation that align with business objectives.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8+ years of software engineering experience, with demonstrated progression to staff-level or equivalent technical leadership.
• Deep expertise in distributed stream processing, including design, building, and operating high-throughput, fault-tolerant event-driven pipelines using Kafka or equivalent technologies at production scale.
• Production experience with Vespa, OpenSearch, or Elasticsearch, including schema design, ranking profile configuration, and end-to-end application lifecycle management.
• Mastery of Python with strategic awareness of language and framework selection; strong software engineering fundamentals including test strategy, performance architecture, and system design.
• Proficiency with AWS cloud services used in data pipeline and search infrastructure (MSK, ECS, Lambda, DynamoDB, Step Functions, CloudWatch), with infrastructure-as-code experience (Terraform or AWS CDK).
• Experience operationalizing ML models in production, including inference serving, model promotion pipelines, canary rollouts, and production observability for model quality signals.
• Familiarity with agentic retrieval patterns, multi-hop retrieval, latency budget management across retrieval hops, context window optimization, and stateful session design.
• Experience with online search analytics, including instrumenting systems for query performance monitoring, A/B or interleaved ranking experiments, and query log analysis to surface relevance gaps.
• Experience with embedding pipelines, vector indexing, and hybrid (dense + sparse) retrieval architectures in a production context.
• Familiarity with Protobuf schema design and schema registry governance patterns (Confluent Schema Registry or equivalent).
• Experience building self-service or multi-tenant platform infrastructure.