Staff Software Engineer — Search Platform, Ingestion & Indexing

About the role

The Advanced Content Engineering (ACE) team is seeking a Staff Software Engineer to lead the design, implementation, and operational health of the document ingestion pipeline and search index management systems. This role involves owning the design, development, and operation of the document ingestion pipeline, including its fault tolerance, version ordering, and at-least-once delivery guarantees.

Responsibilities

• Design and implement pluggable, configurable pipeline components (parsers, chunkers, enrichers, indexers) that client teams can assemble into custom topologies via the platform's self-service APIs, while maintaining reliable, observable, and performant execution.
• Own the platform's Protobuf-based document schema and schema registry integration, establishing schema governance standards, enforcing backward-compatible evolution, and ensuring reliable serialization across all pipeline stages.
• Lead the migration of ingestion infrastructure from OpenSearch to Vespa, including design of Vespa document processors, custom Kafka feeders, and application package architecture, resolving complex technical challenges that have little or no precedent within the team.
• Customize and operationalize custom models integrated into the ingestion pipeline, including inference serving behind a stable API surface, latency SLO management, hardware and runtime configuration (batching, quantization), and scaling.
• Define and maintain integration contracts between custom models and downstream pipeline components, governing input/output schemas, compatibility requirements, and the governance process for model updates that ensures search pipeline consumers are not broken by changes upstream.
• Instrument model serving for production observability, including latency distributions, throughput, error rates, and quality signals like re-ranking score distributions, enabling the team to detect regressions or model drift without requiring the fine-tuning team's involvement.
• Own the search engine layer end-to-end, design and operate Vespa (and OpenSearch during transition) index configurations, ranking profiles, schema definitions, and application package lifecycle management, applying architectural principles that scale to the platform's long-term content and tenancy goals.
• Implement and maintain zero-downtime index management, including shadow indexing, blue/green index promotion, and rolling reindex workflows that keep the platform available during major infrastructure changes.
• Develop the full-reindex and incremental-update orchestration logic, including change detection, document tracking, Kafka topic management, and DynamoDB-backed state management.
• Design and implement resilient fault tolerance mechanisms, such as dead-letter queues, retry strategies with exponential backoff, circuit breakers, consumer lag monitoring, and drive system-level performance architecture, including profiling ingestion throughput and indexing latency, identifying bottlenecks, and implementing optimizations that meet platform SLOs under peak load.
• Treat delivery friction as the enemy, identifying and removing obstacles that slow the team's ability to ship ingestion and indexing changes to production safely and frequently, improving CI/CD pipelines, deployment automation, and local development workflows as a standing priority.
• Instrument pipeline components with distributed tracing, structured logging, and rich metrics, establishing documentation standards and knowledge management practices so that the team and platform consumers can understand system behavior at all times.
• Deliver effective presentations on complex technical concepts to both technical and non-technical stakeholders, developing strategic plans for technology implementation that align with business objectives.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8+ years of software engineering experience, with demonstrated progression to staff-level or equivalent technical leadership, including ownership of a functional area and leadership of significant cross-functional projects.
• Deep expertise in distributed stream processing, designing, building, and operating high-throughput, fault-tolerant event-driven pipelines using Kafka or equivalent technologies at production scale.
• Production experience with Vespa, OpenSearch, or Elasticsearch, including schema design, ranking profile configuration, and end-to-end application lifecycle management.
• Mastery of Python with strategic awareness of language and framework selection; strong software engineering fundamentals including test strategy, performance architecture, and system design.
• Proficiency with AWS cloud services used in data pipeline and search infrastructure (MSK, ECS, Lambda, DynamoDB, Step Functions, CloudWatch), with infrastructure-as-code experience (Terraform or AWS CDK).
• Demonstrated ability to take full operational responsibility end-to-end, defining SLOs, building observability, running on-call, and driving systematic improvements from incident retrospectives, with a track record of shipping to production frequently and removing delivery friction proactively.
• Comfort and fluency with AI-assisted development tools, using them to move faster and produce higher-quality work, not as a novelty.
• Track record of establishing architectural principles, cross-system design patterns, and documentation standards that improve the broader team's engineering quality.