Chain-of-Thought (CoT) Reasoning

All three CoT approaches using the OpenAI API — from simplest to most powerful

from openai import OpenAI

client = OpenAI()
MODEL = "gpt-4o"

def ask(messages):
    return client.chat.completions.create(
        model=MODEL, messages=messages, temperature=0
    ).choices[0].message.content

PROBLEM = "If a train travels 120 km in 1.5 hours, then stops for 30 minutes, then travels another 90 km in 1 hour, what is its average speed for the entire journey including the stop?"

# ─── Approach 1: Standard (no CoT) ───────────────────────────────────────────
standard = ask([{"role": "user", "content": PROBLEM}])
# Often gives wrong answer: jumps to (120+90)/(1.5+1) = 84 km/h, forgetting the stop

# ─── Approach 2: Zero-shot CoT ───────────────────────────────────────────────
zero_shot_cot = ask([{"role": "user", "content": PROBLEM + "\n\nLet's think step by step."}])
# Model breaks down: total distance=210km, total time=1.5+0.5+1=3h, avg=70 km/h ✓

# ─── Approach 3: Few-shot CoT ─────────────────────────────────────────────────
few_shot_system = """Solve math problems by thinking step by step.
Show each calculation on its own line.
Clearly label: Total distance, Total time, Final answer."""

few_shot_messages = [
    {"role": "system",    "content": few_shot_system},
    {"role": "user",      "content": "A car goes 60 km in 1 hour, stops 15 min, goes 45 km in 45 min. Avg speed?"},
    {"role": "assistant", "content": "Total distance: 60 + 45 = 105 km\nTotal time: 1 + 0.25 + 0.75 = 2 hours\nAverage speed: 105 / 2 = 52.5 km/h"},
    {"role": "user",      "content": PROBLEM},
]
few_shot_cot = ask(few_shot_messages)
# Most reliable: follows demonstrated reasoning structure exactly

# ─── Approach 4: Self-consistency (best accuracy) ────────────────────────────
import re
from collections import Counter

def self_consistent_answer(problem: str, n_samples: int = 10) -> str:
    """Generate n independent CoT solutions and take majority vote."""
    answers = []
    for _ in range(n_samples):
        resp = client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": problem + "\n\nThink step by step, then give the final numeric answer."}],
            temperature=0.7,   # diversity needed for voting to help
        ).choices[0].message.content
        # Extract the last number mentioned as the answer
        nums = re.findall(r'\d+\.?\d*', resp)
        if nums: answers.append(nums[-1])

    if answers:
        most_common = Counter(answers).most_common(1)[0][0]
        return most_common
    return "No consensus"

answer = self_consistent_answer(PROBLEM, n_samples=15)
print(f"Self-consistent answer: {answer} km/h")  # → 70

1. Computation allocation: Each generated token is a full forward pass through the model. Generating 100 reasoning tokens means 100× more "compute" applied to the problem before giving an answer — essentially a soft form of multi-step computation that the model's fixed layers can't perform in a single pass.
2. External working memory: LLMs have no internal state beyond the context window. Writing intermediate results to the context externalizes memory. Without CoT, intermediate values computed in early layers are lost before the answer layer is reached. With CoT, those values persist as tokens in context.
3. Knowledge pathway activation: Reasoning through a problem step-by-step activates different, more relevant knowledge paths than jumping directly to an answer. The intermediate tokens serve as attention anchors that pull in more precise knowledge from the model's weights.

1. What empirical finding by Wei et al. (2022) established chain-of-thought as a major prompting technique? (Answer: Wei et al. (Google, 2022) showed that appending 'Let's think step by step' or providing multi-step reasoning examples dramatically improved performance on arithmetic, commonsense, and symbolic reasoning benchmarks — but ONLY for models above ~100B parameters. For smaller models, CoT hurt performance. This scale threshold finding was critical: it meant CoT is an emergent capability of large models, not a general prompting technique. The paper showed 40–60% accuracy improvements on GSM8K with CoT vs direct answering.)
2. What is the difference between zero-shot CoT and few-shot CoT prompting? (Answer: Zero-shot CoT: simply append 'Let's think step by step.' to the prompt — no examples provided. Few-shot CoT: provide 3–8 examples of (question, reasoning chain, answer) before the target question. Few-shot CoT outperforms zero-shot CoT on complex reasoning tasks because the examples demonstrate the expected reasoning format and depth. Zero-shot CoT is simpler (no example curation) and often sufficient for well-defined problems. Few-shot CoT is preferred for novel reasoning patterns where the model needs to see the expected structure.)
3. What is self-consistency decoding and how does it improve CoT performance? (Answer: Self-consistency (Wang et al. 2022): sample k reasoning chains independently (temperature > 0), execute each to get k answers, take majority vote. The diversity of reasoning paths reduces reliance on any single chain that may contain errors. Key insight: multiple correct paths lead to correct answers; multiple incorrect paths rarely agree on the same wrong answer. GSM8K improvement: CoT+self-consistency (k=40): 88% vs CoT alone: 57%. Trade-off: k× more inference compute and API cost.)
4. What is the 'unfaithful reasoning' problem in chain-of-thought? (Answer: CoT reasoning chains may not reflect the model's actual internal computation. Lanham et al. (2023): models sometimes give incorrect CoT but correct final answers (unused reasoning), and correct CoT but incorrect answers (reasoning not actually guiding the output). Faithfulness of CoT is debated: the reasoning might be post-hoc rationalization of an answer computed through other mechanisms. This matters for safety: if a model's stated reasoning is unfaithful, we cannot use it to understand or verify model behavior.)
5. When does CoT hurt performance compared to direct answering? (Answer: CoT hurts for: (1) Simple factual questions — 'What is the capital of France?' Adding 'Let me reason...' wastes tokens and can introduce errors. (2) Tasks that are pattern-matched from training data — models can answer faster and more accurately without reasoning steps. (3) Small models (<10B) — they lack the capacity to reason effectively in CoT; forced reasoning introduces errors. Rule: use CoT for tasks requiring multi-step computation or reasoning. Skip CoT for simple retrieval, classification, or pattern matching.)

Practical CoT templates for common task types — copy and adapt

# Template 1: Mathematical problem solving
MATH_COT = """Solve the following problem. Show all work.

Format your response exactly as:
## Setting up the problem
[identify knowns, unknowns, relevant formulas]

## Step-by-step solution
[numbered steps, show each calculation]

## Verification
[check your answer makes sense]

## Final answer
[state the answer clearly]

Problem: {problem}"""

# Template 2: Code debugging
DEBUG_COT = """Debug the following code. Think through it systematically.

## What the code is supposed to do
[brief description of intent]

## Reading through the code
[trace execution, note what each section does]

## Identifying the bug
[exactly where and what is wrong, and why]

## The fix
[corrected code with explanation]

## Testing the fix
[confirm it works on the given test case]

Code:
```{language}
{code}
```
Error: {error}"""

# Template 3: Multi-source analysis
ANALYSIS_COT = """Analyze the following question. Be systematic.

## What is being asked
[restate the core question in your own words]

## Key considerations
[factors that affect the answer, tradeoffs]

## Evidence and reasoning
[for each consideration, what does the evidence say?]

## Counter-arguments
[what would someone who disagrees say?]

## Conclusion
[balanced assessment with clear reasoning]

Question: {question}"""