Vectorization Expert

Responsibilities

• Lead the design and development of scalable vectorization pipelines to convert text, PDFs, and multimodal data into high‑quality embeddings using models like BGE, Ada, or Cohere.
• Manage and optimize vector databases (Pinecone, Weaviate, Milvus, pgvector), including building efficient indexing strategies (HNSW, DiskANN).
• Develop advanced chunking and parsing techniques to maintain context and improve retrieval accuracy.
• Build and fine‑tune hybrid search workflows combining dense vector search with sparse keyword methods (e.g., BM25).
• Create evaluation frameworks and gold-standard datasets to measure retrieval performance (Hit Rate, MRR, Faithfulness).
• Implement re-ranking pipelines using cross‑encoders to refine results before sending them to LLMs.
• Work closely with engineering and product teams to ensure scalable, high‑performance retrieval syste