Staff Software Engineer — Search Platform, Ingestion & Indexing

About the role

The Advanced Content Engineering (ACE) team is seeking a Staff Software Engineer to lead the design, implementation, and operational health of the document ingestion pipeline and search index management systems.

Responsibilities

• Design, develop, and own the end-to-end document ingestion pipeline — a Kafka-based stream processing architecture that moves documents through parsing, chunking, enrichment (entity extraction, embedding generation, metadata enrichment), and indexing stages — including all fault tolerance, version ordering, and at-least-once delivery guarantees.
• Define, build, test, deploy, scale, and operate pluggable, configurable pipeline components (parsers, chunkers, enrichers, indexers) that client teams can assemble into custom topologies via the platform's self-service APIs, while maintaining reliable, observable, and performant execution.
• Own the platform's Protobuf-based document schema and schema registry integration — establishing schema governance standards, enforcing backward-compatible evolution, and ensuring reliable serialization across all pipeline stages.
• Lead the migration of ingestion infrastructure from OpenSearch to Vespa, including design of Vespa document processors, custom Kafka feeders, and application package architecture — resolving complex technical challenges that have little or no precedent within the team.
• Customize and operationalize models integrated into the ingestion pipeline — re-ranking models, embedding models, and enrichment components — including inference serving behind a stable API surface, latency SLO management, hardware and runtime configuration (batching, quantization), and scaling.
• Build and operate the model promotion pipeline: the CI/CD workflow that moves a model artifact from the fine-tuning team through staging to production, including versioning, canary rollouts, and rollback mechanisms — ensuring the platform team can operate model updates independently without depending on the research team for production changes.
• Define and maintain integration contracts between custom models and downstream pipeline components — governing input/output schemas, compatibility requirements, and the governance process for model updates that ensures search pipeline consumers are not broken by changes upstream.
• Instrument model serving for production observability: latency distributions, throughput, error rates, and quality signals such as re-ranking score distributions — enabling the team to detect regressions or model drift without requiring the fine-tuning team’s involvement.
• Own the search engine layer end-to-end: design and operate Vespa (and OpenSearch during transition) index configurations, ranking profiles, schema definitions, and application package lifecycle management — applying architectural principles that scale to the platform's long-term content and tenancy goals.
• Implement and maintain the Component Registry and Index Registry — the platform's catalog of reusable processing components and active index configurations — with a focus on correctness, observability, and safe concurrent modification.
• Develop the full-reindex and incremental-update orchestration logic, including change detection, document tracking, Kafka topic management, and DynamoDB-backed state management.
• Design and implement resilient fault tolerance mechanisms — dead-letter queues, retry strategies with exponential backoff, circuit breakers, consumer lag monitoring — that make the pipeline robust to downstream failures and transient errors.
• Drive system-level performance architecture: profiling ingestion throughput and indexing latency, identifying bottlenecks, and implementing optimizations that meet platform SLOs under peak load.
• Treat delivery friction as the enemy: identify and remove obstacles that slow the team's ability to ship ingestion and indexing changes to production safely and frequently — improving CI/CD pipelines, deployment automation, and local development workflows as a standing priority.
• Instrument pipeline components with distributed tracing, structured logging, and rich metrics — establishing documentation standards and knowledge management practices so that the team and platform consumers can understand system behavior at all times.
• Partner with TR Labs and research scientists to ensure that new search components can be evaluated in isolation — with automated offline evaluation on every build and a clear path from evaluation results to production promotion decisions.
• Deliver effective presentations on complex technical concepts to both technical and non-technical stakeholders; develop strategic plans for technology implementation that align with business objectives.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8+ years of software engineering experience, with demonstrated progression to staff-level or equivalent technical leadership — including ownership of a functional area and leadership of significant cross-functional projects.
• Deep expertise in distributed stream processing: designing, building, and operating high-throughput, fault-tolerant event-driven pipelines using Kafka or equivalent technologies at production scale.
• Production experience with Vespa, OpenSearch, or Elasticsearch — including schema design, ranking profile configuration, and end-to-end application lifecycle management.
• Mastery of Python with strategic awareness of language and framework selection; strong software engineering fundamentals including test strategy, performance architecture, and system design.
• Proficiency with AWS cloud services used in data pipeline and search infrastructure (MSK, ECS, Lambda, DynamoDB, Step Functions, CloudWatch), with infrastructure-as-code experience (Terraform or AWS CDK).
• Demonstrated ability to take full operational responsibility end-to-end — defining SLOs, building observability, running on-call, and driving systematic improvements from incident retrospectives — with a track record of shipping to production frequently and removing delivery friction proactively.
• Comfort and fluency with AI-assisted development tools; you use them to move faster and produce higher-quality work, not as a novelty.
• Track record of establishing architectural principles, cross-system design patterns, and documentation standards that improve the broader team’s engineering quality.