Vector Database

Cosine similarity: measures the angle between two vectors regardless of magnitude. Returns 1.0 for identical direction (most similar), 0 for orthogonal (unrelated), -1 for opposite.

Complete RAG pipeline: embed documents → store in Chroma → retrieve → generate with Claude

from anthropic import Anthropic
import chromadb
from chromadb.utils import embedding_functions

# ── 1. Set up Chroma vector database ──────────────────────────────────────
client = chromadb.Client()  # in-memory; use chromadb.PersistentClient() for disk

# Use OpenAI embeddings (or swap for sentence-transformers for free local embed)
embed_fn = embedding_functions.OpenAIEmbeddingFunction(
    api_key="YOUR_OPENAI_KEY",
    model_name="text-embedding-3-small"  # 1536-dim, $0.02 per 1M tokens
)

collection = client.create_collection("lumichats_docs", embedding_function=embed_fn)

# ── 2. Ingest documents ────────────────────────────────────────────────────
documents = [
    "LumiChats charges ₹69 per active day. You only pay on days you use it.",
    "LumiChats Study Mode locks all AI answers to specific pages of your uploaded PDF.",
    "LumiChats supports 40+ models including Claude Sonnet 4.6, GPT-5.4, and Gemini 3 Pro.",
    "LumiChats Agent Mode enables multi-step autonomous task execution using frontier models.",
]
collection.add(
    documents=documents,
    ids=[f"doc_{i}" for i in range(len(documents))]
)

# ── 3. Retrieve relevant context for a query ──────────────────────────────
query = "How much does LumiChats cost?"
results = collection.query(query_texts=[query], n_results=2)
context = "\n".join(results["documents"][0])

# ── 4. Generate answer with Claude using retrieved context ─────────────────
anthropic = Anthropic()
response = anthropic.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=512,
    system=f"""Answer questions using only the provided context.
Context:
{context}

If the answer isn't in the context, say so explicitly.""",
    messages=[{"role": "user", "content": query}]
)
print(response.content[0].text)
# → "LumiChats charges ₹69 per active day — you only pay on days you actually use it."

1. What is approximate nearest neighbour (ANN) search and why is it used instead of exact nearest neighbour? (Answer: Exact nearest neighbour in 1536-dimensional embeddings space requires comparing the query vector against every stored vector — O(n·d) time and impractical for millions of vectors. ANN algorithms (HNSW, IVF, LSH) sacrifice a small amount of accuracy (miss a few true nearest neighbours) for massive speed improvements — typically 100–1000× faster. Production vector databases use ANN: Pinecone, Weaviate, Qdrant, and pgvector all use HNSW as their primary index.)
2. What is HNSW (Hierarchical Navigable Small World) and why is it the dominant ANN algorithm? (Answer: HNSW builds a multi-layer graph where vectors are connected to their nearest neighbours. The top layers are sparse graphs with long-range connections (for fast approximate traversal). Bottom layers are dense with precise neighbourhood connections. Search: enter at the top layer, greedily navigate toward the query, descend to more detailed layers. Insert/search are both O(log n) — unlike tree structures which degrade in high dimensions. HNSW dominates because it achieves 95%+ recall at 100–1000× speedup over brute force.)
3. In a RAG system, when would you use hybrid search (vector + keyword) instead of pure vector search? (Answer: Hybrid search combines dense (embedding) and sparse (BM25/TF-IDF) retrieval. Use hybrid when: (1) Queries include exact terms that must be matched (product IDs, proper nouns, technical terms). Pure vector search may retrieve semantically similar but wrong product. (2) Domain vocabulary is specialised — embeddings may not capture domain-specific term similarity. (3) Users mix broad conceptual queries with specific searches. Reciprocal Rank Fusion (RRF) combines the two ranking lists. Weaviate and Qdrant both support hybrid search natively.)
4. What is the embedding dimensionality trade-off for vector databases? (Answer: Higher dimensions (e.g., 3072 for text-embedding-3-large): more nuanced semantic representation, higher search quality. Costs: more storage per vector (3072 × 4 bytes = 12KB vs 384 × 4 bytes = 1.5KB for small embeddings), slower indexing and search, higher memory usage. Lower dimensions (e.g., Matryoshka embeddings can be truncated to 256 dims): ~12× storage reduction, ~4× faster search, small accuracy loss. Matryoshka Representation Learning (MRL) trains models so early dimensions capture the most important information — enabling dimension selection at retrieval time.)
5. What is the 'semantic gap' problem in vector search and how does query rewriting address it? (Answer: Semantic gap: user queries are often short, keyword-like, and expressed differently than the documents they target. A query 'Python list comprehension' may not retrieve a document titled 'Compact syntax for creating lists in Python' even if they are semantically equivalent — embedding similarity depends on training distribution. Query rewriting: use an LLM to expand or rephrase the query into multiple forms. HyDE (Hypothetical Document Embeddings): generate a hypothetical answer to the query and embed THAT — the answer embedding is closer in space to the actual answer document.)