Real-life analogy: The city map

Imagine every word is a point on a map of concepts. Words that mean similar things are placed in the same neighborhood: cat, kitten, dog, puppy are all near each other in the 'animals' district. Paris, London, Berlin are in the 'European capitals' district. The distance and direction between points encode relationships: the vector from 'man' to 'woman' is the same as the vector from 'king' to 'queen'. Word embeddings are exactly this map — learnt automatically from billions of words of text.

Word2Vec — learning from context

Word2Vec (Mikolov et al., Google 2013) trains a shallow neural network on one of two tasks: CBOW (Continuous Bag of Words) predicts the center word from its context window. Skip-gram predicts context words from the center word. Neither task is the real goal — the weights of the hidden layer are the word embeddings, learnt as a by-product.

\text{Skip-gram objective: } \max \sum_{t=1}^{T} \sum_{-c \leq j \leq c, j \neq 0} \log P(w_{t+j} \mid w_t)

from gensim.models import Word2Vec

# Tokenised corpus: list of sentences (lists of words)
sentences = [
    ["the", "king", "rules", "the", "kingdom"],
    ["the", "queen", "is", "the", "ruler"],
    ["man", "works", "at", "the", "office"],
    ["woman", "works", "at", "the", "office"],
    ["paris", "is", "the", "capital", "of", "france"],
    ["berlin", "is", "the", "capital", "of", "germany"],
]

model = Word2Vec(
    sentences,
    vector_size=50,   # embedding dimensions (typically 100-300)
    window=3,         # context window size
    min_count=1,      # ignore words with freq < min_count
    sg=1,             # 1 = Skip-gram, 0 = CBOW
    epochs=100,
)

# Semantic similarity
print(model.wv.most_similar("king", topn=3))
# [('queen', 0.97), ('ruler', 0.92), ('kingdom', 0.88)]

# Word analogy: king - man + woman = queen
result = model.wv.most_similar(positive=["king", "woman"], negative=["man"])
print(result[0])   # ('queen', 0.96)

# Cosine similarity
print(model.wv.similarity("paris", "berlin"))   # ~0.91 (both capitals)
print(model.wv.similarity("paris", "dog"))       # ~0.12 (unrelated)

GloVe and FastText

GloVe (Global Vectors, Stanford 2014) takes a different approach: instead of a prediction task, it directly factorises the word co-occurrence matrix of the entire corpus. GloVe embeddings encode global corpus statistics — not just local context windows — making them particularly good for syntactic relationships.

FastText (Facebook 2016) extends Word2Vec by representing each word as a bag of character n-grams. 'apple' = {ap, app, ppl, ple, le, }. This handles morphologically rich languages (Turkish, Finnish), rare words, and misspellings gracefully — even words not seen in training can be represented by averaging their character n-gram embeddings.

Static vs contextual embeddings: Word2Vec, GloVe, and FastText are static: the word "bank" has one embedding regardless of whether it means river bank or financial bank. BERT and GPT produce contextual embeddings: the same word gets different vectors depending on its sentence context. For most modern NLP tasks, contextual embeddings (BERT, GPT) significantly outperform static ones.

Negative sampling — the trick that made Word2Vec trainable

The naive Skip-gram objective requires a softmax over the entire vocabulary for every training pair — with a 100,000-word vocabulary, that is 100,000 dot products per update. Negative sampling replaces this with a series of tiny binary classifications: for each true (center, context) pair, sample k random "negative" words (k = 5–20) and train the model to score the true pair high and the fake pairs low. Cost drops from |V| to k+1 computations — a ~5,000× speedup that made training on billions of words practical on 2013 hardware.

• Negatives are drawn from the unigram distribution raised to the ¾ power — a hand-tuned compromise that samples frequent words often, but not as often as their raw frequency (which would waste updates on 'the' and 'of').
• Frequent-word subsampling: very common words are randomly dropped from training contexts, which both speeds training and measurably improves embedding quality for rare words.
• Deep connection: Levy & Goldberg (2014) proved Skip-gram with negative sampling implicitly factorizes the PMI (pointwise mutual information) matrix of word co-occurrences — Word2Vec and GloVe are mathematically closer cousins than they appear.

Bias, and whether static embeddings still matter in 2026

Word embeddings learn whatever the corpus contains — including its stereotypes. The landmark finding (Bolukbasi et al., 2016): in Word2Vec trained on Google News, the analogy man : computer programmer :: woman : ? returned homemaker. Gender, ethnic, and religious associations are measurably encoded in embedding geometry (the WEAT test quantifies this), and debiasing by projecting out a "gender direction" helps less than hoped — bias reappears through indirect associations. This line of research is a core reason model cards and bias evaluations are now standard practice.

Still worth learning in 2026? Yes — three reasons: First, production: fastText remains a workhorse for language identification and high-throughput classification where a transformer is 1000× too slow, and modern static models distilled from sentence transformers (e.g., Model2Vec-style approaches) revive the idea for cheap retrieval at massive scale. Second, concepts: negative sampling, cosine similarity, and the vector-arithmetic view of meaning are the direct ancestors of every RAG pipeline and vector database today. Third, interviews and exams still love it — CBOW vs Skip-gram and the analogy test are perennial questions.

Practice questions

1. What is the dimensionality problem with one-hot vectors that Word2Vec solves? (Answer: One-hot vectors have dimension |V| (10k-100k+) and are orthogonal — all words are equidistant. Word2Vec uses 100-300 dimensions and encodes semantic similarity via cosine distance.)
2. In Skip-gram, given the sentence "the cat sat on the mat" with window=2 and center word "sat", what are the training pairs? (Answer: (sat, cat), (sat, the), (sat, on), (sat, the) — all words within distance 2.)
3. Why does FastText outperform Word2Vec on rare words? (Answer: FastText represents words via character n-grams. Rare words share n-grams with common words, so their embeddings inherit some meaning even with few training examples.)
4. What does the analogy "Paris - France + Germany = ?" test in word embeddings? (Answer: Berlin. Tests that "capital of" relationships are encoded as consistent vectors. Result = model.wv.most_similar(positive=["Paris", "Germany"], negative=["France"]).)
5. GloVe is called "Global" because: (Answer: It factors the global word co-occurrence matrix of the entire corpus, rather than only looking at local context windows like Word2Vec.)