BLEU — evaluating translation quality

\text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right) \quad BP = \begin{cases} 1 & c > r \\ e^{1-r/c} & c \leq r \end{cases}

from nltk.translate.bleu_score import corpus_bleu, sentence_bleu, SmoothingFunction
from rouge_score import rouge_scorer
import numpy as np

# ── BLEU for machine translation ──
# References: one or more human translations (list of lists)
# Hypotheses: machine translations (list of token lists)
references = [
    [['the', 'cat', 'is', 'on', 'the', 'mat']],      # 1 reference for sent 1
    [['the', 'dog', 'barked', 'at', 'the', 'mailman']],  # 1 reference for sent 2
]
hypotheses = [
    ['the', 'cat', 'is', 'on', 'mat'],           # Missing one 'the'
    ['the', 'dog', 'barked', 'at', 'a', 'man'],  # 'the mailman' → 'a man'
]

# Corpus-level BLEU (standard for MT evaluation)
bleu_score = corpus_bleu(references, hypotheses)
print(f"Corpus BLEU: {bleu_score:.4f}")

# Sentence-level BLEU with smoothing
smooth = SmoothingFunction().method1
for ref, hyp in zip(references, hypotheses):
    score = sentence_bleu(ref, hyp, smoothing_function=smooth)
    print(f"Sentence BLEU: {score:.4f} | Hyp: {' '.join(hyp)}")

# Individual n-gram precision breakdown
from nltk.translate.bleu_score import modified_precision
from fractions import Fraction
ref = [['the', 'cat', 'is', 'on', 'the', 'mat']]
hyp =  ['the', 'cat', 'is', 'on', 'mat']
for n in range(1, 5):
    prec = modified_precision(ref, hyp, n=n)
    print(f"  P_{n}: {float(prec):.3f}")
# P_1: 0.800 (4 of 5 unigrams match)
# P_2: 0.750 (3 of 4 bigrams match)
# P_3: 0.667 ...
# P_4: 0.500 ...

# ── ROUGE for summarization ──
scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)

reference_summary = """The transformer architecture revolutionised NLP by replacing
recurrent networks with self-attention mechanisms for parallel processing."""

hypothesis1 = """Transformers changed NLP by using self-attention instead of
recurrent networks, enabling faster parallel training."""

hypothesis2 = """Scientists invented a new cooking technique in 2019."""

for hyp_name, hyp in [("Good", hypothesis1), ("Bad", hypothesis2)]:
    scores = scorer.score(reference_summary, hyp)
    print(f"
{hyp_name} summary ROUGE:")
    for metric, score in scores.items():
        print(f"  {metric}: P={score.precision:.3f}, R={score.recall:.3f}, F1={score.fmeasure:.3f}")
# ROUGE-1: measures unigram overlap
# ROUGE-2: measures bigram overlap
# ROUGE-L: measures longest common subsequence

# ── BERTScore: semantic similarity beyond exact match ──
try:
    from bert_score import score as bert_score
    # BERTScore uses contextual BERT embeddings to compare semantic similarity
    # Better than BLEU/ROUGE for paraphrases (same meaning, different words)
    cands = ["The cat sat on the mat"]
    refs  = ["A feline rested on the rug"]   # Paraphrase — different words, same meaning
    P, R, F1 = bert_score(cands, refs, lang='en', verbose=False)
    print(f"
BERTScore F1 for paraphrase: {F1.mean():.3f}")  # ~0.88 (high!)
    # BLEU would give 0 (no word overlap). BERTScore captures semantic equivalence.
except ImportError:
    print("pip install bert-score")

ROUGE — summarization evaluation

Limitations of BLEU and ROUGE: BLEU and ROUGE measure surface overlap — not semantic quality. A paraphrase ("quick brown fox" → "fast auburn fox") scores BLEU=0 despite being a perfect translation. A repetitive summary ("The cat sat on the mat. The cat sat on the mat.") can score high ROUGE despite being useless. Modern evaluation increasingly uses human ratings, BERTScore, or LLM-as-judge approaches (GPT-4 evaluating model outputs).

Perplexity — evaluating language models

PPL(W) = P(w_1 w_2 \ldots w_N)^{-1/N} = \exp\left(-\frac{1}{N}\sum_{i=1}^N \log P(w_i \mid w_1, \ldots, w_{i-1})\right)

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch, math

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model     = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

def perplexity(text: str, stride: int = 512) -> float:
    """Compute perplexity with stride trick for long texts."""
    encodings = tokenizer(text, return_tensors='pt')
    max_length = model.config.n_positions   # 1024 for GPT-2
    seq_len    = encodings.input_ids.size(1)
    nlls, n_tokens = [], 0

    for begin in range(0, seq_len, stride):
        end       = min(begin + max_length, seq_len)
        target_len = end - begin
        input_ids  = encodings.input_ids[:, begin:end]
        target_ids = input_ids.clone()
        target_ids[:, :-target_len] = -100   # Ignore context tokens in loss

        with torch.no_grad():
            nll = model(input_ids, labels=target_ids).loss
        nlls.append(nll * target_len)
        n_tokens += target_len

    return math.exp(torch.stack(nlls).sum() / n_tokens)

texts = {
    'Simple English': "The cat sat on the mat. The dog ran in the park.",
    'Scientific': "Transformer architectures utilize multi-head self-attention mechanisms.",
    'Random': "Purple elephant dances quantum philosophy seventeen banana.",
    'Code': "def fibonacci(n): return n if n < 2 else fibonacci(n-1) + fibonacci(n-2)",
}
for label, text in texts.items():
    ppl = perplexity(text)
    print(f"PPL={ppl:6.1f}: {label}")
# Simple English: low PPL (well-represented in training)
# Scientific: medium PPL (technical vocabulary)
# Random: very high PPL (semantically incoherent)
# Code: medium-high PPL (GPT-2 saw little code)

BLEU vs ROUGE vs perplexity — which metric should you use?

The three metrics are not competitors — they answer different questions, and using the wrong one is the most common evaluation mistake in NLP projects. The deciding question is simple: do you have a reference text, and does your task reward precision or recall?

The intuition behind the precision/recall split: a translation that invents content is wrong, so BLEU penalizes unmatched output n-grams (precision). A summary that omits the main point is wrong, so ROUGE rewards covering the reference's n-grams (recall). This is why reporting BLEU on a summarization task — or ROUGE on translation — quietly measures the wrong failure mode.

The perplexity comparison trap: Perplexity is only comparable across models that share a tokenizer and test set. A model with a 200k vocabulary will show lower perplexity than one with a 50k vocabulary on identical text — it makes fewer, larger predictions — with no difference in quality. Never compare the published perplexity of, say, a Llama model against a GPT model and conclude anything. Perplexity also says nothing about instruction-following or factuality: a model can have excellent perplexity and still be useless as an assistant.

What practitioners actually do in 2026: BLEU and ROUGE survive as cheap regression checks and for comparability with published baselines — not as quality verdicts. Real evaluation stacks combine them with BERTScore (or its trained cousin COMET, now the standard for translation quality) and LLM-as-judge scoring on a task-specific rubric, then validate the judge against a few hundred human ratings. Rule of thumb: use n-gram metrics to detect regressions, use embedding or judge metrics to make decisions.

Practice questions

1. Machine translation BLEU = 0.65. Is this good? (Answer: Yes — human-level translation achieves BLEU ≈ 0.60-0.70 on standard benchmarks. BLEU > 0.6 is considered near-human quality. Commercial MT systems (Google Translate) typically achieve BLEU 0.50-0.65 on standard WMT benchmarks.)
2. Your team reports ROUGE-1 for a translation system and BLEU for a summarizer. What is wrong? (Answer: The metrics are swapped. Translation has essentially one correct output, so precision matters — BLEU penalizes invented content. Summarization is judged on coverage of the source's key points, so recall matters — ROUGE rewards capturing reference n-grams. Swapping them measures the wrong failure mode: a rambling translation and an incomplete summary would both score deceptively well.)
3. Model A (50k vocab) has PPL=12; Model B (200k vocab) has PPL=8 on the same text. Is B better? (Answer: Unknown — the comparison is invalid. Perplexity depends on the tokenizer: a larger vocabulary means fewer tokens per sentence and each prediction covers more text, mechanically lowering perplexity. Perplexity is only comparable across models sharing a tokenizer and test set. Compare them on downstream task benchmarks instead.)
4. Why does BLEU use a brevity penalty? (Answer: Without it, a model could achieve perfect precision by outputting just one highly probable word (e.g., "the") that appears in all references. P_1 = 1.0 but the output is useless. Brevity penalty multiplies by e^(1-r/c) when the hypothesis c is shorter than reference r — penalizing short translations.)
5. ROUGE-1 = 0.9, ROUGE-2 = 0.3 for a summary. What does this suggest? (Answer: High ROUGE-1 means the summary covers key individual words from the reference. Low ROUGE-2 means few consecutive word pairs match — the summary may have rearranged key terms without preserving phrase structure. The summary covers the right vocabulary but possibly in a different order or with different context.)
6. A language model has PPL=10 on the training set and PPL=200 on the test set. What is happening? (Answer: Severe overfitting — the model has memorized the training distribution and generates that text easily (PPL=10) but fails to generalize to new text (PPL=200). A good model should have similar PPL on train and test. Train PPL should be slightly lower than test PPL.)
7. BERTScore is better than BLEU for paraphrase evaluation. Why? (Answer: BLEU counts exact word overlap — "quick fox" and "fast fox" share only one word, giving low BLEU. BERTScore encodes both texts with BERT and computes cosine similarity of contextual embeddings — semantically similar words (quick≈fast, fox≈fox) score high even without exact match. BERTScore is more aligned with human judgments of translation quality.)