Large Language Model (LLM)

Cross-entropy training loss: the model is penalized for assigning low probability to the correct next token. Minimizing this over trillions of tokens teaches the model language structure, facts, and reasoning.

Autoregressive generation: how LLMs produce text token by token

import torch
import torch.nn.functional as F

def generate(model, tokenizer, prompt: str, max_new_tokens=50, temperature=1.0):
    """Minimal autoregressive generation loop — this is what every LLM inference does."""
    input_ids = tokenizer.encode(prompt, return_tensors="pt")

    for _ in range(max_new_tokens):
        with torch.no_grad():
            logits = model(input_ids).logits   # shape: (1, seq_len, vocab_size)

        # Take logits for the LAST token position only
        next_token_logits = logits[:, -1, :] / temperature   # (1, vocab_size)

        # Convert to probabilities and sample
        probs = F.softmax(next_token_logits, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)  # (1, 1)

        # Append to sequence and continue
        input_ids = torch.cat([input_ids, next_token], dim=1)

        # Stop at end-of-sequence token
        if next_token.item() == tokenizer.eos_token_id:
            break

    return tokenizer.decode(input_ids[0], skip_special_tokens=True)

Kaplan scaling laws: loss decreases as a power law with model size and data size. αN ≈ 0.076, αD ≈ 0.095 from the original paper.

1. Pretraining: Self-supervised next-token prediction on a massive corpus (trillions of tokens). Trains for weeks to months on thousands of GPUs. Cost: $1M–$100M+. Produces a base model that can complete text but doesn't follow instructions or refuse harmful requests.
2. Post-training: (a) Supervised Fine-Tuning (SFT) on (instruction, response) demonstration pairs — teaches instruction-following. (b) RLHF or DPO on human preference data — teaches helpfulness, safety, and alignment. Total cost: $1K–$10M. Produces the conversational model you interact with.

Pretraining data mix — how training corpora are assembled (LLaMA 3 style)

# LLaMA 3 training data composition (approximate, based on Meta's paper)
TRAINING_MIX = {
    "general_web":      {"weight": 0.50, "source": "Common Crawl (cleaned)"},
    "code":             {"weight": 0.17, "source": "GitHub, Stack Overflow"},
    "books":            {"weight": 0.10, "source": "Books, long-form text"},
    "scientific":       {"weight": 0.08, "source": "arXiv, PubMed, academic papers"},
    "wikipedia":        {"weight": 0.05, "source": "Wikipedia (multilingual)"},
    "math":             {"weight": 0.05, "source": "Math Stack Exchange, proofs"},
    "multilingual":     {"weight": 0.05, "source": "CC news in 30+ languages"},
}

total_tokens = 15e12   # 15 trillion tokens

for name, info in TRAINING_MIX.items():
    tokens = total_tokens * info["weight"]
    print(f"{name:20s}: {tokens/1e12:.2f}T tokens  ({info['source']})")

1. What is an LLM's context window and what happens when content exceeds it? (Answer: Context window = the maximum number of tokens the model can process in one forward pass (input + output combined). Claude 3.5: 200K tokens. GPT-4o: 128K. LLaMA 3.1: 128K. When content exceeds the limit: API throws a context_length_exceeded error (you must chunk or summarise). Within the window: all tokens receive full attention — no information is lost. Common workaround: sliding window chunking with overlap, RAG (retrieve only relevant portions), or summarisation of earlier context.)
2. Why do LLMs sometimes confidently state false information (hallucinate)? (Answer: LLMs are trained to predict the next most probable token given context — not to verify factual accuracy. They have no internal fact-checking mechanism. Hallucination occurs when: (1) the fact was rare or absent in training data, (2) the model interpolates between related facts incorrectly, (3) the model confabulates plausible-sounding completions when uncertain. Mitigation: Retrieval-Augmented Generation (RAG) grounds responses in verified sources; Constitutional AI training reduces confident false statements.)
3. What is the difference between a base model and an instruction-tuned model? (Answer: Base model (pretrained): trained to predict next tokens on massive text corpora — continues text in the style of its training data. Chatting with a base model: it completes your message as if continuing a document, not responding helpfully. Instruction-tuned (chat model): fine-tuned on (instruction, response) pairs + RLHF — trained to be a helpful assistant that follows directions. All deployed products (ChatGPT, Claude, Gemini) are instruction-tuned. Base models are used for research and further fine-tuning.)
4. How do LLMs achieve few-shot learning from examples in the prompt without weight updates? (Answer: In-context learning: the model uses the examples in the prompt as implicit task specification. The transformer's attention mechanism can identify the pattern across examples and apply it to the new input. This works because during pretraining, the model saw countless documents that implicitly demonstrated tasks through examples. Few-shot ICL is thus a form of pattern matching using the model's pretrained world knowledge rather than gradient-based adaptation.)
5. What is the difference between LLM fine-tuning and prompting for a specific task, and when should you use each? (Answer: Prompting: craft instructions/examples in the prompt to guide the model. Zero cost, instant to change, works well for tasks the model already has capability for. Fine-tuning: train model weights on task-specific data. Higher upfront cost, produces consistent behaviour, better for tasks requiring style/format consistency, domain-specific knowledge, or reliably following complex instructions. Rule of thumb: try prompting first. If you hit consistent failure modes despite good prompting, and you have 100+ quality examples, fine-tune.)