Small Language Models (SLMs)

Run a 7B model locally with Ollama — the fastest way to get started with SLMs in 2026

# Install Ollama (macOS/Linux/Windows WSL)
curl -fsSL https://ollama.ai/install.sh | sh

# Download and run Llama 3.2 3B (fits in ~2GB RAM)
ollama run llama3.2:3b

# Or try Microsoft's Phi-3 Mini (3.8B, one of the best SLMs)
ollama run phi3:mini

# Or Mistral 7B — great general-purpose SLM
ollama run mistral:7b

# Once running, you can also use it via API (OpenAI-compatible!)
curl http://localhost:11434/api/generate \
  -d '{
    "model": "phi3:mini",
    "prompt": "Explain gradient descent in 2 sentences.",
    "stream": false
  }'

# Or use the Python client (works with the openai SDK):
# from openai import OpenAI
# client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# response = client.chat.completions.create(model="phi3:mini", messages=[...])

1. What is the key insight behind Microsoft's Phi series of small language models? (Answer: The Phi hypothesis: model performance is primarily limited by training data quality, not parameter count. Phi-1 (1.3B): trained on 'textbook quality' Python tutorials and coding exercises (7B tokens) — outperforms much larger models on coding benchmarks. Phi-2 (2.7B): extends to general reasoning with synthetic educational content. Phi-3-mini (3.8B): matches GPT-3.5 on many benchmarks. Key insight: a small model trained on 10B high-quality tokens can outperform a 10× larger model trained on 100B noisy web tokens.)
2. What are the practical advantages of SLMs (≤7B params) over frontier LLMs (70B+) for enterprise deployment? (Answer: (1) Cost: inference cost 10–100× lower. (2) Latency: faster response times on same hardware. (3) Local deployment: 7B at INT4 runs on a laptop — no API dependence. (4) Privacy: sensitive data never leaves the organisation. (5) Customisation: fine-tuning a 7B model is accessible on one A100; fine-tuning a 70B model requires a cluster. (6) Reliability: no API outages or rate limits. Trade-offs: lower capability on complex reasoning; less general knowledge.)
3. What tasks are SLMs surprisingly competitive on compared to much larger models? (Answer: (1) Narrow domain Q&A after fine-tuning: a 3B model fine-tuned on medical literature often outperforms GPT-4 on specific medical subspecialty questions. (2) Structured extraction (NER, classification): task-specific fine-tuned SLMs match frontier performance. (3) Code completion within a specific codebase: fine-tuned on the company's code. (4) Simple chat and FAQ: for well-defined, limited-scope conversations. (5) Embedding and semantic search: small embedding models often match large ones for retrieval tasks.)
4. What is model distillation and how is it used to create SLMs? (Answer: Knowledge distillation: a large 'teacher' model generates high-quality outputs on a large dataset; a small 'student' model is trained on these outputs. The student learns from the teacher's soft probability distributions (dark knowledge) rather than hard labels — capturing the teacher's uncertainty and generalisation. Microsoft's Phi-3 and Google's Gemma distilled from larger proprietary models. Distillation typically achieves 80–90% of the teacher's performance with 10× fewer parameters.)
5. What is speculative decoding's relationship to SLMs? (Answer: SLMs serve as draft models in speculative decoding: the fast SLM generates k tokens speculatively; the slower large model verifies them in one forward pass. The SLM should be in the same model family as the large target model (same tokenizer). Llama 3.2 1B serves as draft for Llama 3.1 70B — the 1B model runs ~20× faster and proposes tokens that the 70B accepts ~70% of the time, achieving ~2.5× end-to-end speedup. SLMs are thus infrastructure components for efficient large model deployment.)