Chain-of-thought and self-consistency: reasoning prompts that work

Asking a language model to think step by step is the single most impactful prompting trick ever discovered. Two papers from 2022 turned it from a folk technique into a science: chain-of-thought (CoT) prompting showed that asking for intermediate reasoning steps unlocks multi-step problem solving, and self-consistency showed that sampling many such chains and letting them vote pushes accuracy much further.

But "think step by step" is not free, and it's not always a win. This tutorial covers how both techniques work, what the research measured, when step-by-step reasoning helps, when it hurts, and how to put sampling into practice in your own apps.

How chain-of-thought prompting works#

In the standard setup, you give a model a question and it answers directly. In chain-of-thought prompting, you provide a few worked examples where each answer includes its reasoning steps before the final answer. The model learns the pattern and produces its own intermediate steps before answering.

Wei et al. (2022, NeurIPS) demonstrated the effect on arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains were striking: prompting a 540B-parameter language model with just eight chain-of-thought exemplars achieved state-of-the-art accuracy on the GSM8K benchmark of math word problems — surpassing even finetuned GPT-3 with a verifier. The reasoning ability, the authors argued, emerges naturally in sufficiently large models; the prompt just elicits it.

Then Kojima et al. (2022, NeurIPS) showed you don't even need examples. Appending a single instruction — "Let's think step by step" — before each answer turned large instruction-tuned models into decent zero-shot reasoners. With the same single prompt template, zero-shot CoT lifted MultiArith accuracy from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% on text-davinci-002, with similar magnitudes of improvement on PaLM-540B. Few-shot CoT remains stronger when you have good examples, but zero-shot CoT is the practical default when you don't.

The intuition is simple: breaking a problem into explicit steps lets the model "use its own output as scratch paper." Each intermediate step becomes context the model can check and build on, instead of forcing the whole computation into a single leap.

When step-by-step reasoning helps#

CoT shines on tasks that decompose into multiple dependent steps — the kind of thing you'd call "showing your work" in school:

• Math word problems and arithmetic — the classic wins (GSM8K, SVAMP, AQuA).
• Symbolic and logical reasoning — tracking shuffled objects, date understanding, last-letter concatenation.
• Commonsense reasoning that needs chaining — strategy questions where the answer follows from two or three facts.
• Code generation with planning — outlining the approach before writing the code.

A useful rule of thumb: if the task is one a capable human would solve with pen and paper rather than at a glance, CoT will probably help. The benefit is also scale-dependent — it helps large models much more than small ones. The ability to produce useful reasoning chains appears to be an emergent property that shows up once models are large enough.

When it hurts#

CoT is not a universal upgrade. There are well-documented cases where it backfires:

• Small models can get worse. The Wei et al. paper noted that forcing CoT on smaller models can reduce accuracy — they lack the capacity for coherent intermediate steps and just add confident-sounding noise.
• Simple factual recall and single-step tasks. Asking "What is the capital of France?" with a full reasoning trace buys nothing and costs extra tokens on every query — typically tens to hundreds of extra output tokens per call, which adds latency and API cost at scale.
• Overconfidence in the trace. The written reasoning looks authoritative even when it's wrong. Each step can still contain an error, and a plausible-sounding chain can make a wrong answer harder to distrust. Turpin et al. (2023) showed that a model's stated chain of thought doesn't always reflect its actual decision process — biasing features in the prompt can shift predictions substantially while the reasoning text never mentions them. Never treat a CoT trace as a faithful audit log.
• Native reasoning models. Modern reasoning models (OpenAI's o-series, DeepSeek-R1, Anthropic's extended thinking) already reason internally at inference time. Adding your own "think step by step" scaffolding can override the model's better plan, duplicate its internal work, or just waste the reasoning budget. With these models, state the goal, inputs, and output format clearly and let them manage the reasoning.

The practical takeaway: use CoT where you have measured that it helps — multi-step reasoning on capable models — and default to a direct answer everywhere else.

Self-consistency: sample, then vote#

A single chain of thought still follows one path, and a single path can wander. Wang et al. (2022, published at ICLR 2023) replaced naive greedy decoding with a simple ensemble strategy called self-consistency:

1. Sample a diverse set of reasoning paths using temperature or nucleus sampling, instead of decoding one greedy chain.
2. Marginalize — discard the reasoning text and take a majority vote over the final answers.

The intuition: a hard problem usually has many valid lines of reasoning that converge on the one correct answer, while incorrect reasoning scatters across many different wrong answers. Correct thinking converges; errors don't. Agreement is evidence.

The gains were large: on top of chain-of-thought prompting, self-consistency added +17.9% on GSM8K, +11.0% on SVAMP, +12.2% on AQuA, +6.4% on StrategyQA, and +3.9% on ARC-challenge. With PaLM-540B, that meant GSM8K accuracy moving from 56.5% to 74.4% — a gain achieved with no training, no new model, no extra prompting craft, just more inference compute. The technique is also robust: the authors report it improves accuracy across different language models and model scales.

This paper quietly founded the test-time-compute era. "Spend more compute at inference to get better answers" is the same thesis that later drove reasoning models, majority-voting pipelines, and tree-of-thought search.

Putting it into practice#

Self-consistency is easy to implement in an application. Here's the recipe:

1. Craft the prompt once. Use your best CoT prompt — zero-shot ("Let's think step by step") or few-shot with worked examples — and require a clearly parseable final answer (e.g., "Final answer: 42").
2. Sample N paths in parallel. Use temperature around 0.7 and issue N independent calls (or one batch). Start with 5–10 samples; gains roughly plateau after a few dozen, so there's no need for the paper's maximum of 40.
3. Extract and vote. Parse each final answer and take the majority. Ties can be broken with an extra sample or by falling back to the greedy path.
4. Spend the budget where it counts. Only self-consistency the hard items: run a cheap direct pass first, and escalate to sampled voting when the model's confidence is low or the stakes are high.

Watch the failure modes: self-consistency assumes the correct answer is the modal answer. If the model has a systematic misconception, every path can converge on the same wrong answer — voting amplifies a shared bias, not just shared reasoning. Keep a held-out evaluation set so you know the vote is actually helping on your task, not just on the paper's benchmarks.

Takeaway#

Chain-of-thought turned "show your work" into the most important prompting technique in AI, and self-consistency turned it into a decoding strategy: sample several chains, ignore the prose, vote on the answers. The pattern to remember is convergent answers are evidence — when independently sampled reasoning paths keep landing on the same result, trust it more.

Use it deliberately. Reserve CoT and self-consistency for multi-step reasoning where you've measured the payoff, skip them for simple tasks, stay skeptical of the written traces, and remember that with modern reasoning models the sampling usually happens inside the model itself. The skill isn't knowing the trick — it's knowing when the trick earns its cost.