A 2-bit model that calls your tools: hands-on with Cactus Needle 3

Your smart-home hub has no internet connection, 256 MB of RAM, and a user who just said "goodnight." Somewhere in that sentence are three actions — kill the lights, drop the thermostat, lock the door — and exactly zero room for a language model to improvise. This is the job Cactus Needle 3 was built for: a 121-million-parameter model quantized to about two bits per weight that turns plain-language commands into tool calls, structured data, and embeddings, entirely on-device. In the next twenty minutes you will install it, wire up real tools, watch it execute a multi-step routine, and see what happens when it tries to invent an argument it was never given.

Why this is blowing up now#

Needle sits at ≈12,700 stars and 866 forks (checked September 27, 2026) on GitHub's weekly trending charts, up hundreds of stars in days — the strongest signal this week for a model you can actually download rather than just read about. The repo (cactus-compute/needle, Apache-2.0, maintained by Cactus Compute) has 319 commits behind it, so this is not a weekend demo; it is a maintained runtime with platform engines for Linux, macOS, Windows, Android, iOS, watchOS, and even WASM.

The reason for the spike is architectural, not marketing. General-purpose small models are trained to chat and then retrofitted for tool use; Needle 3 is a Laddered Simple Attention Network trained from the start for three narrow jobs — tool calls, structured extraction, embeddings — and it trades away general chat capacity to win there. The project's own benchmark chart reports it beating models ten times its size on mobile tool-call exact-match accuracy and matching models two to three times bigger on extraction field F1. Those are the vendor's numbers from their own suite, not mine — I did not re-run their benchmarks — but the claim I could test, that a ~35 MB download runs useful tool-calling on a CPU with under 100 MB of RAM, checked out exactly as described.

What you'll need#

• Python 3.9 or newer (I used 3.12 on Linux; macOS and Windows are supported). The package's only hard dependency is huggingface_hub.
• ~40 MB of disk and no GPU. The engine is a ~1.3 MB native library and the full 20-layer weights are a ~35 MB .cact file (the project describes 8–29 MB binaries for shallower depth slices). My runs peaked at 98.6 MB of RAM — measured, not estimated.
• No accounts, no API keys, no network at inference time. Everything below runs offline once the one-time download finishes. Cost: $0.

Step 1: Install and fetch the model#

pip install "cactus-needle==3.0.4"

The pin is deliberate, not superstition. Version 3.0.5 was current when I wrote this, but it points its engine auto-download at a build (3.0.2) that is not published on the project's Hugging Face repo — the fetch 404s. Version 3.0.4 points at engine 3.0.1, which exists, so the first-run download works. If you are reading this later and 3.0.5+ is fixed upstream, unpin; until then, ==3.0.4 is the version whose install path I verified end to end.

On first use, the package fetches two files once and caches them under ~/.cache/cactus-needle/: the platform engine and needle3.cact, the weights. One privacy note from the README, worth knowing before you run anything: telemetry is on by default in the binary. Turn it off with environment variables:

export NEEDLE_TELEMETRY=0
export DO_NOT_TRACK=1

Step 2: Your first tool call#

Needle's whole programming model fits in one decorator. The function's signature gives the argument types, the docstring becomes the tool description the model reads, and run() closes the loop — the model picks the tool, fills the arguments, your function executes, and the results come back:

import needle

@needle.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
out = agent.run("what's it like in Lagos right now?")
print(out["results"])
# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

This ran exactly as shown on a CPU-only Linux VM. The model extracted "Lagos" from the sentence, called get_weather, and run() executed it and returned the result. Total new concepts: one decorator, one constructor, one method.

Step 3: Read the response like an operator#

The full response object is where Needle shows it was designed for automation rather than chat. Here is the real, untrimmed response from the call above:

{
  "type": "respond",
  "success": true,
  "function_calls": [],
  "results": [{"city": "Lagos", "temp_c": 27, "sky": "clear"}],
  "reasoning": "User asked a question; respond with the concrete value from result.",
  "confidence": 0.6698,
  "prefill_tps": 8.8,
  "decode_tps": 4.5,
  "peak_ram_mb": 98.6
}

Four fields deserve your attention. confidence is a calibrated score from a learned head (0.67 here) — the project's docs describe routing on it: act above your threshold, ask to confirm in the middle, refuse below it. prefill_tps / decode_tps tell you the model did ~9 prompt tokens/sec and ~4.5 generated tokens/sec on a plain CPU; tool calls are short, so a full turn completes in seconds. peak_ram_mb is self-reported memory — under 100 MB, which is why this fits on hardware where even a 1B-parameter chat model does not.

One sharp edge I verified the hard way: turns accumulate. Every run() stays in the conversation, so after several queries an off-topic request once returned a previous turn's weather result instead of an empty list. The fix is in the API — call agent.reset() between independent queries, or construct with stateless=True. With a fresh agent, the off-topic behavior the README promises is exactly what you get:

agent = needle.Needle(tools=[get_weather])
out = agent.run("Write me a haiku about the ocean.")
print(out["function_calls"], out["results"], out["reasoning"])
# [] [] 'No tool available for creating or editing text.'

No tool covers poetry, so the model returns empty lists and says so, with confidence 1.0. For an automation runtime, a clean refusal beats a creative guess every time.

Step 4: A real end-to-end routine#

Now the scenario from the intro: three real tools, one sentence, all local.

import needle

@needle.tool
def set_light(room: str, state: str):
    "Turn a room light on or off. state must be 'on' or 'off'."
    return {"room": room, "light": state}

@needle.tool
def set_thermostat(temp_c: int):
    "Set the home thermostat temperature in Celsius."
    return {"thermostat_c": temp_c}

@needle.tool
def set_door_lock(locked: bool):
    "Lock (true) or unlock (false) the front door."
    return {"front_door": "locked" if locked else "unlocked"}

agent = needle.Needle(tools=[set_light, set_thermostat, set_door_lock])
out = agent.run(
    "Goodnight: turn off all the lights, "
    "set the thermostat to 19, and lock the front door."
)
print(out["results"])
# [{'room': 'all', 'light': 'off'},
#  {'thermostat_c': 19},
#  {'front_door': 'locked'}]

Three actions, one turn, confidence 0.75 — real output from my run. But look closely at the first result: the model collapsed "all the lights" into a single call with room="all" instead of one call per room. Your tool received it and executed it; whether "all" means anything is your code's decision. This is the single most important design lesson for Needle, and the project's own tool-design guide says the same thing: one tool per action, and handle group names inside the tool (fan out to every room yourself), because the model will happily hand you a collective noun. I kept the quirk in because it is the kind of thing that only shows up when you actually run the system.

Now the more important test — what happens when the model wants an argument the user never gave:

agent = needle.Needle(tools=[set_thermostat])
out = agent.run("set the bedroom thermostat")  # no temperature given
print(out["function_calls"], out["results"])
# [] []
print(out["validation"])
# {'ungrounded': ['set_thermostat.temp_c'], 'negation': False}

The model internally proposed set_thermostat(room="bedroom", temp_c=20) — inventing 20 degrees. Needle's strict grounding check caught it: the fabricated argument is flagged in validation.ungrounded, the call is moved to suppressed_calls, and nothing executes. For a model that drives physical devices, this is the feature that matters most: a missing argument produces a refusal, not a hallucinated thermostat setting. (The check targets numbers, dates, and other groundable values; keep strict=True, the default.)

Step 5: Structured extraction that always parses#

Same model, second job: declare a shape, hand over messy text, get typed fields back. A byte-level grammar compiled from your schema constrains every generated token, so the output parses by construction:

schema = {
    "name": "invoice",
    "description": "Extract invoice fields from text.",
    "parameters": {
        "type": "object",
        "properties": {
            "vendor":   {"type": "string", "description": "Vendor name"},
            "total":    {"type": "number", "description": "Total amount"},
            "currency": {"type": "string", "description": "Currency code"},
            "due_date": {"type": "string", "description": "Due date YYYY-MM-DD"},
        },
        "required": ["vendor", "total"],
    },
}

res = needle.extract(
    "Invoice from Acme Corp for $1,250.00 USD, due 2026-10-15. "
    "Thank you for your business.",
    schema,
)
print(res)
# {'vendor': 'Acme Corp', 'total': 1250.0,
#  'currency': 'USD', 'due_date': '2026-10-15'}

Every field correct, types correct (total came back a float, not the string "$1,250.00"). In strict mode — the default — values that contradict the input raise ExtractionValidationError instead of being returned silently, the same grounding philosophy as the tool-call suppression in Step 4. This is the boring, reliable core of on-device automation: receipts, forms, notifications, and voice commands turned into typed records with no cloud round-trip.

Step 6: Embeddings from the same model#

The third job needs no tools at all — the same weights return a vector per sentence, so a device can do local similarity matching and routing:

agent = needle.Needle(tools=[])
v = agent.embed("turn on the living room lights")
print(len(v))  # 3072

I measured cosine similarities across five sentence pairs. Paraphrases scored 0.94–0.96 ("turn on the living room lights" vs "switch the living room lights on": 0.9603; "set an alarm for 7am" vs "wake me up at seven in the morning": 0.9580), while unrelated pairs scored 0.89–0.92. The ordering is correct — similar above, unrelated below — but the margin is narrow, so treat these as a coarse local signal and calibrate your own threshold on your own data rather than trusting a magic number. For on-device intent routing ("does this utterance look like a lighting command?") it is usable; for fine-grained semantic search, a dedicated embedding model still wins.

When to use this vs the alternatives#

Reach for Needle when the device is the constraint — wearables, smart-home hubs, robots, in-car systems, microcontrollers — and the job is do the thing rather than talk about the thing. Its advantages are all verified above: grammar-constrained outputs, calibrated confidence for act/confirm/refuse routing, and a grounding validator that suppresses invented arguments instead of executing them. Reach for something else when you need open-ended conversation (this model trades that away by design), when you need the deepest reasoning on tangled plans (cloud models still lead), or when your command set is truly fixed (a parser is smaller and fully deterministic).

Two directions the project documents that I did not run hands-on: fine-tuning (pip install "cactus-needle[train]", then needle finetune data.jsonl --epochs 10 --out adapter.safetensors and needle build --lora adapter.safetensors --layers 8 --out tuned.cact to export a shallower slice) and deployment (needle build --platform linux-arm64 --layers 8 --out ./pi fetches a sub-1 MB engine plus weights for the target). Both are documented in the repo's guides; I left them out of the verified path because training needs a JAX stack I did not install for this tutorial.

The takeaway#

The surprise of Needle 3 is not that a 121M-parameter model can call tools — it is how seriously it takes not calling them. Empty lists for off-topic requests, suppressed calls for invented arguments, calibrated confidence on every turn: the whole design reads like it was written by people who have watched a model confidently mis-set a thermostat. At ~35 MB of weights and under 100 MB of RAM on a CPU, it puts that discipline on hardware where the cloud never reaches. Install the pinned version, decorate your functions, reset between independent queries — and handle room="all" in your own code.