When you ask a modern model for the weather in Boston and it "just knows" to call get_current_weather(location="Boston"), no magic is involved. The model is still doing what it always did — predicting the next token — except its training taught it a special dialect: emit a structured request instead of prose, wait for the world to answer, then continue. This article unpacks the machinery behind that trick: the training regimes that teach it, the parsing layers that make it reliable, and the design split that determines who actually runs the code.

The core trick: a model that writes requests, not answers#

Every function-calling system, regardless of provider, follows the same loop:

  1. You send the model a request plus tool definitions (name, description, and parameters in JSON Schema).
  2. The model decides whether a tool would help, and if so emits a structured block naming the tool and its arguments.
  3. Your code executes the function and sends the result back.
  4. The model reads the result and produces a final answer (or issues another call).

The crucial point: the model never executes anything. It only generates text that describes a call. Everything around that text — training, schema constraints, execution — is engineered plumbing. Understanding function calling means understanding three layers: how the model learns the dialect, how the dialect is parsed and enforced, and how the conversation loop is orchestrated.

Layer 1: Teaching the dialect — from prompting to fine-tuning#

The earliest approaches treated tool use as a prompting problem. ReAct (2022) showed that interleaving reasoning traces with actions — "Thought: I need the weather. Action: search_weather(Boston)" — in just a handful of examples let models alternate between thinking and acting, improving interactive decision-making on benchmark tasks. OpenAI's WebGPT (December 2021) fine-tuned GPT-3 with imitation learning and human feedback to operate a text-based browser, showing that trained tool use could produce answers humans preferred over human-written ones in head-to-head comparisons.

The conceptual leap came from Toolformer (Schick et al., Meta AI Research, February 2023; arXiv 2302.04761), which removed human labeling from the equation entirely:

  • Sample: the model proposes candidate API calls at various positions in ordinary text.
  • Execute: every candidate call is actually run, and the result is spliced into the sequence.
  • Filter: a call survives only if the result reduces the model's prediction loss on the following tokens — in plain terms, the tool result genuinely made the future text easier to predict.
  • Fine-tune: the model is trained on the filtered corpus, learning not just how to format calls, but when they're worth making.

The tools were humble — a calculator, a question-answering system, two search engines, a translation system, a calendar — but the result was a 6.7B-parameter GPT-J model that learned tool use from what the paper's abstract calls "nothing more than a handful of demonstrations for each API," and became competitive with models an order of magnitude larger on tasks where tools help. The core insight persists in everything that followed: you don't hardcode when to call; you let utility (measured in prediction loss) select the training data, and the model internalizes the judgment.

Commercialization followed fast. On June 13, 2023, OpenAI launched function calling for gpt-4-0613 and gpt-3.5-turbo-0613 — the first major commercial implementation. Per OpenAI's announcement, these models were fine-tuned to both detect when a function needs to be called and respond with JSON that adheres to the function signature. Developers describe functions via JSON Schema in a functions (later tools) parameter of the chat completions endpoint; the model then either answers directly or outputs a JSON object with a function name and arguments. Your application executes it and returns the output for a final response. The rest of the industry converged on this pattern: Anthropic added tool use to the Claude API, and Google, Mistral, Cohere, and open-source families (Llama 3.1, Mistral v0.3) shipped their own variants within roughly a year.

Takeaway for practitioners: prompting alone is a demo; reliable function calling is a fine-tuned capability. The Toolformer recipe — generate candidates, execute them, keep only the ones that reduce loss — remains the most scalable way to build training data, because it needs no human annotation.

Layer 2: Making the output parseable — schemas and constrained decoding#

A model that intends to call a function is useless if its output can't be parsed into one. This is the parsing layer, and it has two parts.

JSON Schema as the universal interface. Every major provider converged on JSON Schema (or a compatible subset) to describe parameters: name, description, type, properties, required fields. The same schema does double duty: it goes into the model prompt so the model knows the contract, and it validates the output on your side before execution. Tool descriptions matter enormously here — a crisp description is the model's primary signal for tool selection, while the parameter schema governs argument extraction. Writing good tool definitions (explicit descriptions, typed enums instead of free strings, clear required fields) is arguably the highest-leverage tuning you can do for function-calling quality.

Constrained decoding. Even fine-tuned models occasionally emit malformed JSON, and a single bad token invalidates a call. The fix works at the token level: given the schema, the serving stack builds a state machine over the JSON grammar and, at each generation step, masks out (assigns near-zero probability to) tokens that would violate the schema at that position. The model still chooses freely among the valid tokens — so creativity in argument values is preserved — but the output is structurally guaranteed. This is the difference between "JSON mode" (guarantee the output is JSON) and function calling (guarantee the JSON also names a real tool with valid arguments, and that the model decided a call was warranted).

Combined with special role tags in the conversation format — messages marked as tool calls and tool results, distinct from user/assistant turns — this turns an autoregressive text engine into something that reliably emits machine-readable requests.

Layer 3: Orchestrating the loop — who executes, and how calls compose#

The last layer is the conversation loop, and it's where providers diverge most.

  • Client-side execution (OpenAI, Gemini default, most open-source setups): the model returns the call; your code runs it, handles errors, and feeds the result back as a new message. Maximum control, but every call is a network round trip and your code owns retries, timeouts, and secrets.
  • Server-side execution (Anthropic's server tools): for certain built-in tools (like web search), Anthropic's infrastructure runs the tool and injects results directly — the round trip disappears. The tradeoff is a smaller, provider-curated toolset.
  • Automatic execution (Gemini's Python SDK): a middle ground where the SDK converts Python functions to declarations, executes calls automatically, and handles the response cycling for you.

Calls also compose in three patterns worth recognizing: parallel invocation (independent calls issued together — supported explicitly by most modern APIs), sequential chaining (one call's output becomes another's input, requiring multiple loop iterations), and conditional calling (later calls depend on earlier results). The model's training data must cover all three; a model fine-tuned only on single calls will underperform on multi-step tasks. This is exactly what agentic benchmarks probe: Berkeley's Function-Calling Leaderboard, for instance, evaluates AST accuracy (did the call match the function signature's structure?), tool selection accuracy, and hallucination rate (did it invent a tool that doesn't exist?) — with later versions extending to multi-turn and agentic scenarios.

What the research frontier looks like now#

The field has moved from "can models call tools?" to "how do we make tool use cheap, fast, and safe?"

  • Efficiency at the edge: small specialized models (e.g., Google's ~270M-parameter FunctionGemma for on-device calling) and parallel-decoding research aim to emit structured calls in fewer tokens with lower tail latency — critical when a single malformed token kills a call.
  • Protocol standardization: Anthropic's Model Context Protocol (announced November 2024) standardizes the layer above function calling — how applications expose tools and context to models — addressing the fragmentation where every provider's message format differs even though all use JSON Schema underneath.
  • Simulated and synthetic trajectories: following the Toolformer spirit, labs now train on a mix of real tool interactions (e.g., via MCP servers) and LLM-generated synthetic trajectories over diverse APIs, plus preference tuning to make multi-turn tool use concise rather than verbose.

The practical takeaway#

Function calling works because three mechanisms reinforce each other:

  1. Training (Toolformer-style filtered trajectories, supervised fine-tuning, RL) teaches the model when a tool helps and how to format the request.
  2. Parsing (JSON Schema definitions, constrained decoding at the token level) makes the request machine-readable by construction.
  3. Orchestration (the call → execute → result → respond loop, with parallel/chained/conditional composition) turns individual calls into agents.

If your tool-using system is unreliable, diagnose it in that order. Most production failures live in layer 2 — and the fix is better schemas, not a bigger model.