Embeddings

Generating a real embedding with OpenAI text-embedding-3-small

from openai import OpenAI
import numpy as np

client = OpenAI()  # uses OPENAI_API_KEY env variable

def embed(text: str) -> np.ndarray:
    response = client.embeddings.create(
        model="text-embedding-3-small",  # 1536 dimensions
        input=text
    )
    return np.array(response.data[0].embedding)

king   = embed("king")
queen  = embed("queen")
banana = embed("banana")

print(f"Embedding shape: {king.shape}")    # (1536,)
print(f"First 5 values:  {king[:5]}")
# e.g. [ 0.021, -0.083,  0.045, -0.012,  0.067, ...]
# 1536 numbers — each individually meaningless,
# but together they encode the word's meaning

Cosine similarity: ranges from −1 (opposite meaning) to 1 (identical meaning). 0 = unrelated.

Computing cosine similarity between embeddings

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """Cosine similarity between two vectors."""
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# Using the embeddings from the previous example:
print(f"king  ↔ queen:  {cosine_similarity(king, queen):.4f}")   # ~0.87
print(f"king  ↔ banana: {cosine_similarity(king, banana):.4f}")  # ~0.21
print(f"king  ↔ king:   {cosine_similarity(king, king):.4f}")    # 1.0000

# For bulk comparisons, normalize first (cosine = dot product of unit vectors):
def normalize(v: np.ndarray) -> np.ndarray:
    return v / np.linalg.norm(v)

king_n, queen_n = normalize(king), normalize(queen)
# Now: similarity = np.dot(king_n, queen_n)

The gender relationship is encoded as a consistent geometric direction in embedding space.

Demonstrating word arithmetic with embeddings

# Semantic arithmetic in embedding space
man    = embed("man")
woman  = embed("woman")

# Compute the "analogy" vector
target = king - man + woman

# Find which word's embedding is closest to the result
candidates = {"king": king, "queen": queen, "woman": woman, "banana": banana}

similarities = {
    word: cosine_similarity(target, vec)
    for word, vec in candidates.items()
}

best = max(similarities, key=similarities.get)
print("Nearest to (king - man + woman):", best)  # queen ✓

# Output: queen (similarity ~0.89)
# This works because the gender direction (man→woman)
# is consistent across semantic spaces

Minimal semantic search with embeddings + cosine similarity

import numpy as np
from openai import OpenAI

client = OpenAI()

def embed_batch(texts: list[str]) -> np.ndarray:
    """Embed multiple texts in one API call (efficient)."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    return np.array([r.embedding for r in response.data])


# --- 1. Index your knowledge base ---
documents = [
    "Photosynthesis converts sunlight into glucose in plant cells.",
    "The mitochondria is the powerhouse of the cell.",
    "DNA replication occurs during the S phase of the cell cycle.",
    "The Eiffel Tower was built in 1889 in Paris, France.",
]

doc_embeddings = embed_batch(documents)   # shape: (4, 1536)

# Normalize for fast cosine similarity via dot product
norms = np.linalg.norm(doc_embeddings, axis=1, keepdims=True)
doc_embeddings_norm = doc_embeddings / norms


# --- 2. Query ---
def search(query: str, top_k: int = 3):
    query_emb = embed_batch([query])[0]                  # (1536,)
    query_emb = query_emb / np.linalg.norm(query_emb)   # normalize

    # Cosine similarity = dot product when both are normalized
    scores = doc_embeddings_norm @ query_emb             # (4,)

    top_indices = np.argsort(scores)[::-1][:top_k]
    return [(documents[i], float(scores[i])) for i in top_indices]


results = search("how do plants make food?")
for doc, score in results:
    print(f"Score {score:.3f}: {doc}")

# Score 0.847: Photosynthesis converts sunlight into glucose in plant cells. ✓
# Score 0.432: The mitochondria is the powerhouse of the cell.
# Score 0.381: DNA replication occurs during the S phase of the cell cycle.

• Recommendation systems — embed user history and items; find items closest to the user's "taste vector"
• Duplicate detection — find near-identical documents in a corpus (cosine similarity > 0.97)
• Classification — train a lightweight classifier (logistic regression, SVM) on top of frozen embeddings — often beats fine-tuning for small datasets
• Clustering — K-Means or HDBSCAN over embeddings to discover semantic groups without labels
• Cross-lingual search — multilingual models embed English and Hindi into the same space; search Hindi docs with an English query
• Anomaly detection — inputs far from the distribution of "normal" embeddings may indicate unusual or adversarial inputs

1. What is the dot product between two unit vectors and why does cosine similarity use it? (Answer: Cosine similarity = (A·B)/(||A||·||B||). For unit vectors (||A||=||B||=1): cosine_similarity = A·B directly. Range: -1 (opposite directions) to +1 (same direction), 0 (orthogonal/unrelated). Used for embeddings because length is normalised — similarity reflects angular relationship (semantic closeness) not vector magnitude. In practice: normalise embeddings before cosine comparison; this also makes nearest neighbour search faster (dot product only).)
2. What is the difference between sentence embeddings and word embeddings? (Answer: Word embeddings (Word2Vec, GloVe): one vector per word in the vocabulary, context-independent. 'bank' has the same vector whether 'river bank' or 'bank account.' Sentence embeddings (SBERT, OpenAI text-embedding): one vector per sentence/passage, context-aware. The vector for 'I went to the bank to deposit money' reflects the financial sense. Sentence embeddings are generated by averaging token embeddings or using [CLS] token from a transformer — capturing the full contextual meaning of the input.)
3. What is semantic search vs keyword search and when is each appropriate? (Answer: Keyword search (BM25/TF-IDF): finds documents containing the exact query terms. Fast, interpretable, handles technical terms and product IDs exactly. Fails when query uses different vocabulary than documents. Semantic search (embedding similarity): finds semantically similar documents even with different vocabulary — 'affordable car' matches documents about 'budget vehicle' and 'cheap automobile.' Slower (requires embedding + ANN lookup). Use keyword for: exact product IDs, medical codes, legal citations. Use semantic for: user intent queries, cross-lingual retrieval, FAQ matching.)
4. What is the 'curse of dimensionality' problem for high-dimensional embeddings? (Answer: In high dimensions (768, 1536, 3072), the volume grows exponentially — almost all points become roughly equidistant from each other. Nearest neighbour search loses discriminative power: the difference between the closest and farthest neighbour becomes proportionally small. Practical effect: at very high dimensions, cosine similarities cluster around 0 for all pairs. Mitigations: dimensionality reduction (PCA to 256 dims), Matryoshka embeddings (encode most information in first 256 dims), and ANN algorithms (HNSW) that navigate the manifold structure rather than brute-force comparing all distances.)
5. What is fine-tuning embedding models and when is it necessary? (Answer: Pretrained embeddings (text-embedding-3-large, BAAI/bge): trained on general web text. May not capture domain-specific similarity. Fine-tuning: train on (query, positive_document, negative_document) triplets from your domain using contrastive loss. When necessary: (1) Highly specialised vocabulary (legal, medical, chemical). (2) Custom similarity notion (you want 'similar' to mean something specific). (3) When out-of-box retrieval quality is below 70% accuracy. Fine-tuned embedding models often improve RAG retrieval by 10–20 percentage points on domain-specific tasks.)