Knowledge Distillation

Hinton et al. (2015) distillation loss: a weighted sum of (1) standard cross-entropy with the ground truth, and (2) KL divergence between the teacher and student output distributions at temperature T. Temperature T > 1 softens the distributions, amplifying information in the non-maximum logits. α balances the two objectives.

Response distillation pipeline — the most practical form: fine-tune a small model on teacher outputs

from datasets import Dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer
import anthropic
import json

# ── Step 1: Generate teacher outputs ─────────────────────────────────────
client = anthropic.Anthropic()

def get_teacher_response(prompt: str) -> str:
    """Get a high-quality response from Claude Sonnet (the teacher)."""
    msg = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}]
    )
    return msg.content[0].text

# Your domain-specific prompts (e.g. customer support, medical Q&A, legal)
student_prompts = [
    "What are the side effects of ibuprofen?",
    "Explain the difference between a debit card and a credit card.",
    # ... thousands more domain prompts
]

distillation_data = []
for prompt in student_prompts:
    response = get_teacher_response(prompt)
    distillation_data.append({
        "text": f"<|user|>\n{prompt}\n<|assistant|>\n{response}"
    })

# ── Step 2: Fine-tune a small student model on teacher outputs ────────────
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

dataset = Dataset.from_list(distillation_data)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    args=TrainingArguments(
        output_dir="./phi3-distilled-domain",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        learning_rate=2e-5,
        # Use LoRA via peft_config for efficiency — see LoRA article
    ),
)
trainer.train()
# Result: A 3.8B model with domain expertise from a 100B+ teacher.

1. What is 'dark knowledge' and why does it improve student model training? (Answer: Dark knowledge: the probability distribution a teacher model assigns to all classes (soft labels). Example: for an image of a cat, the teacher might output [cat: 0.7, tiger: 0.2, dog: 0.1]. The non-cat probabilities encode the teacher's knowledge about similarity relationships — tiger is more cat-like than dog. Training on hard labels [cat: 1, tiger: 0, dog: 0] loses this similarity structure. Soft labels carry much more information per example, enabling the student to learn better representations with less data.)
2. What is temperature scaling in knowledge distillation and why is it important? (Answer: Temperature T in soft targets: p_i = exp(z_i/T) / Σ exp(z_j/T). High T flattens the distribution — makes the soft labels even softer, exposing more dark knowledge (tiny probabilities become more visible). Low T sharpens — approaches hard labels. Standard distillation uses T=3–5 for soft targets, T=1 for hard targets. The final loss combines soft loss (at T) + hard loss (at T=1): L = α × T² × KL(teacher_soft, student_soft) + (1-α) × CE(student_logits, hard_labels). T² rescales the soft gradient to match hard gradient magnitude.)
3. What is the difference between offline distillation, online distillation, and self-distillation? (Answer: Offline: train teacher fully first, then train student on teacher outputs — classic approach. Teacher is fixed throughout student training. Online (mutual learning): teacher and student train simultaneously, sharing knowledge with each other. No pretrained teacher needed — multiple students teach each other. Self-distillation: a model distils knowledge to itself — deeper layers teach shallower layers, or later training epochs teach earlier epochs. Born-Again Networks: retrain same architecture using soft labels from a trained copy, consistently outperforming the original.)
4. What is feature-based distillation vs response-based distillation? (Answer: Response-based: match only final outputs (logits/probabilities). Simplest but loses intermediate representation information. Feature-based (FitNets, CRD): match intermediate layer activations — student's hidden states should match teacher's hidden states at corresponding depths. Requires projection layers if student and teacher have different hidden dimensions. Feature-based distillation transfers more structural knowledge but is more complex to implement. Combined approaches (match both outputs and features) typically perform best.)
5. Why is knowledge distillation particularly effective for BERT compression? (Answer: BERT is heavily over-parameterised for most downstream tasks — much of its capacity is not needed. DistilBERT retains 97% of BERT's GLUE performance with 40% fewer parameters and 60% faster inference by distilling from BERT. BERT's multi-head attention naturally produces soft, information-rich targets. Task-specific distillation (after fine-tuning) is more effective than general distillation. Further: TinyBERT distils both intermediate representations and attention matrices — achieving 97% performance with 7.5× compression.)