BERT & Encoder-Only Models

BERT masked language modeling inference

from transformers import pipeline, AutoTokenizer, AutoModelForMaskedLM
import torch

# High-level: use fill-mask pipeline
mlm = pipeline("fill-mask", model="bert-base-uncased")
results = mlm("The [MASK] sat on the mat.")
for r in results[:3]:
    print(f"{r['token_str']:12s} ({r['score']:.3f})")
# cat        (0.812)
# dog        (0.043)
# man        (0.018)

# Lower-level: get BERT hidden states (for downstream use)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")

inputs = tokenizer("The quick brown fox", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs, output_hidden_states=True)

# Last hidden state: (batch=1, seq_len, hidden=768)
hidden_states = outputs.hidden_states[-1]
cls_embedding = hidden_states[:, 0, :]   # [CLS] token for classification tasks
token_embeddings = hidden_states[:, 1:-1, :]  # word tokens (no special tokens)

Fine-tuning BERT for text classification (sentiment analysis)

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset
import numpy as np

model_name = "distilbert-base-uncased"   # 66M params — fast, good quality
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=2  # positive / negative
)

dataset = load_dataset("imdb")
tokenized = dataset.map(
    lambda x: tokenizer(x["text"], truncation=True, max_length=512),
    batched=True
)

args = TrainingArguments(
    output_dir="./bert-imdb",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    learning_rate=2e-5,          # Small LR — pretrained weights are delicate
    eval_strategy="epoch",
    fp16=True,
)

trainer = Trainer(
    model=model, args=args,
    train_dataset=tokenized["train"].shuffle(seed=42).select(range(5000)),
    eval_dataset=tokenized["test"].select(range(1000)),
    compute_metrics=lambda p: {
        "accuracy": (np.argmax(p.predictions, axis=1) == p.label_ids).mean()
    }
)
trainer.train()
# Typical result: ~93% accuracy on IMDb after 3 epochs, ~10 minutes on a T4 GPU

SBERT sentence embeddings for semantic similarity and RAG

from sentence_transformers import SentenceTransformer, util
import torch

# Pretrained SBERT model fine-tuned specifically for semantic similarity
model = SentenceTransformer("all-MiniLM-L6-v2")  # 22M params, fast, great quality

# Encode sentences to fixed 384-dim vectors
sentences = [
    "What is the capital of France?",
    "Paris is the capital city of France.",
    "The Eiffel Tower is in Paris.",
    "Python is a programming language.",
]
embeddings = model.encode(sentences, convert_to_tensor=True)  # (4, 384)

# Semantic similarity (cosine)
sims = util.cos_sim(embeddings, embeddings)
print("Q vs Answer 1:", sims[0][1].item())   # → ~0.85 (high — semantically related)
print("Q vs Answer 2:", sims[0][2].item())   # → ~0.62 (medium)
print("Q vs Code:", sims[0][3].item())        # → ~0.10 (low — unrelated)

# For RAG: encode your document chunks once, store in vector DB
# At query time: encode query, find top-k by cosine similarity

1. What is masked language modeling (MLM) and how does it pre-train BERT? (Answer: MLM: randomly mask 15% of input tokens ([MASK]), train the model to predict the original token from context. Unlike causal LM (GPT), which only attends to previous tokens, MLM gives BERT bidirectional context — each token attends to all tokens. This enables BERT to learn deep bidirectional representations: predicting [MASK] in 'The [MASK] sat on the mat' uses both 'The' and 'on the mat'. The 15% masking: 80% replaced with [MASK], 10% random word, 10% unchanged — prevents the model from only learning [MASK] tokens.)
2. What is BERT's next sentence prediction (NSP) task and why was it later found to be unhelpful? (Answer: NSP: train BERT to classify whether sentence B follows sentence A in the original text (50% true pairs, 50% random). Intended to improve sentence-pair tasks (NLI, QA). RoBERTa (2019) ablated NSP and found removing it IMPROVED performance on most benchmarks. NSP was too easy — the model learned topic mismatch rather than real discourse understanding. Random sentence pairs often have different topics, making classification trivial. Modern BERT variants (RoBERTa, DeBERTa) drop NSP, training longer on MLM only.)
3. What is the difference between BERT-base and BERT-large, and when would you choose each? (Answer: BERT-base: 12 layers, 12 heads, 768 d_model, 110M parameters. BERT-large: 24 layers, 16 heads, 1024 d_model, 340M parameters. BERT-large scores ~2–4 points higher on GLUE benchmarks. BERT-base: preferred for production (2-3× faster inference, 3× less memory). BERT-large: preferred for research or when maximum accuracy matters and you have GPU budget. For most production NLP tasks (NER, classification, sentence similarity): BERT-base fine-tuned on domain data outperforms BERT-large on general data.)
4. What is the [CLS] token in BERT and how is it used for classification? (Answer: [CLS] (classification token): prepended to every input sequence. During pre-training, the hidden state of [CLS] at the final layer is used to predict NSP — so BERT is trained to aggregate sequence-level information into [CLS]. For downstream classification fine-tuning: take the [CLS] final hidden state (768-dim vector), add a linear classifier head on top, and fine-tune on labeled data. [CLS] acts as a sequence summary vector. Alternative: mean-pool all token embeddings — often performs similarly or better for sentence similarity tasks (Sentence-BERT).)
5. What is DeBERTa and what two innovations improved on BERT's attention? (Answer: DeBERTa (He et al., Microsoft 2020): (1) Disentangled attention: content and position embeddings are separate — attention between tokens computed using 4 terms: content-to-content, content-to-position, position-to-content, position-to-position. Better position-aware representations. (2) Enhanced mask decoder (EMD): uses absolute position information in the final decoding layers while using relative position in attention. DeBERTa-v3 (186M params) outperforms BERT-large (340M) and RoBERTa-large (355M) on GLUE/SuperGLUE while being smaller and faster.)