Run a 763-billion-parameter model from your SSD: hands-on with InferSSD

InferSSD (wici-ai/inferssd, Apache-2.0, 206 GitHub stars) streams the experts of a 763-billion-parameter mixture-of-experts model from SSD instead of holding them in RAM or VRAM. This guide installs it on Apple Silicon or Linux with NVIDIA CUDA, starts DeepSeek-V4.1-Flash in terminal chat, serves it behind a browser UI and an OpenAI-compatible API, and states the 470 GiB download and the slow decode plainly.

• Who: InferSSD, by wici-ai on GitHub. A C++ runtime built on llama.cpp and ggml that indexes GGUF files and streams mixture-of-experts weights from SSD through a bounded RAM cache.
• What: Runs models larger than installed RAM or VRAM. Only the routed experts each token needs are read from SSD; dense layers run on Metal (macOS) or CUDA (Linux), and routed experts run on the CPU on macOS or on a CUDA tier on Linux.
• When: Repository created 22 October 2026. First public release v0.1.0 on 30 October 2026; v0.1.1 on 31 October 2026. At research time on 1 November 2026: 206 GitHub stars. Licence: Apache-2.0.
• Models: Exactly two tested GGUF checkpoints: DeepSeek-V4.1-Flash-1017, 763B total / 8–16B active, Q8_0; and Muse-Spark-Open, 816B total / 42B active, UD-Q4_K_XL. No smaller or test model is listed.
• Needs: Apple Silicon macOS with Metal, or Linux with an NVIDIA GPU and CUDA. Windows support is in progress. First run downloads about 470 GiB to ~/.local/share/inferssd/models; the benchmarked DeepSeek GGUF files take about 473 GiB. Building from source takes about 2 to 12 minutes.
• Speed: Project-reported, 4K prompt tier, 256 output tokens, same hardware vs llama.cpp: on an RTX 5090 rig, DeepSeek prefill 164.500 tokens/s (20.41×) and decode 7.270 tokens/s (8.39×); on a MacBook Pro with M5 Max and 48 GB unified memory, decode 2.874 tokens/s (3.29×). Decode is slow in absolute terms on every tested machine.

Why this matters now

A mixture-of-experts model is mostly weights that any single token will never use. DeepSeek-V4.1-Flash has 763 billion parameters in total, but only 8 to 16 billion are active for a given token. The usual local setup still wants the whole model in memory before it answers anything, which puts a model of this size out of reach of an ordinary workstation. InferSSD attacks the other side of the problem: keep the weights on the SSD, index them, and fetch only the experts the router selects, a few at a time, through a bounded cache in RAM.

The dense layers — the parts every token uses — run on the GPU: Metal on Apple Silicon, CUDA on Linux. On macOS the routed experts run on the CPU; on Linux with CUDA there is an added GPU cache tier. There is no memory setting to choose. The first launch measures the machine — CPUs, memory, GPUs and the drive holding the model, in one to three minutes — and the engine sizes its caches from what it finds. That measurement is the whole configuration story for the cli and serve commands below.

The project’s own benchmarks, measured on a 4K prompt tier with 256 output tokens and decode counted over tokens 129 to 256, report large prefill speedups over llama.cpp on the same hardware, and decode that is faster than llama.cpp but still slow in absolute terms:

Read those numbers as the project’s, on its test rigs, for one run each on the Mac — not as a promise for your machine. The honest headline is in the decode column: this makes a 763B model runnable on a workstation, not fast. If you need interactive speed, a smaller model held fully in memory will still beat it. Every command below is the project’s own, checked against the repository, its quick start and its benchmark notes on 1 November 2026.

What you’ll need

• A supported machine: an Apple Silicon Mac (Metal), or Linux on x86_64 or ARM64 with an NVIDIA GPU and the CUDA toolkit. Windows support is in progress — do not start there.
• Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools; Linux also needs the CUDA toolkit, liburing headers, and systemd with cgroup v2. Most containers, and WSL without systemd enabled, do not provide what the Linux launcher requires, and it checks before starting the server.
• A fast SSD with room to spare. The first run downloads about 470 GiB of model files; the benchmarked DeepSeek GGUF files take about 473 GiB. Plan for at least 500 GiB free on the drive that will hold the weights, and put the weights on your fastest SSD — the whole design is streaming from it.
• Time, not money. Building from source takes about 2 to 12 minutes depending on the machine, the first launch measures the machine for one to three minutes, and the download takes as long as your connection needs for 470 GiB. No API key is required by the steps below; model access uses the standard Hugging Face token store or HF_TOKEN if one is needed.
• No other model files. Only the two tested checkpoints in the key facts are accepted by the -hf commands below, so there is no small model to practise on first.

Step 1: Check the machine and the disk before you download anything

Confirm the platform and the free space first, because the download is the point of no return in this guide. On macOS or Linux, check that the home directory’s drive has the room:

df -h ~

If the weights will live on a different drive, check that drive instead, and note its path for Step 3 — the --model-dir flag chooses where the weights go. The test rigs in the project’s benchmarks all used 2 TB SSDs: a 2 TB internal drive on the MacBook Pro, 2 TB PCIe 4.0 ×4 drives on the RTX 5060 Ti and RTX 5090 rigs, and a 4 TB PCIe 5.0 ×4 drive on the DGX Spark. A slower or nearly full drive will not stop the install, but it is exactly what this design is sensitive to, since every expert fetch is a disk read. Decide now: Apple Silicon Mac, or Linux with NVIDIA CUDA. Anything else, including Windows, is outside what the project currently supports.

Step 2: Install InferSSD with the one-command installer

The installer is a source installer, not a prebuilt binary release: it compiles the engine for your machine’s CPU and GPU, which is why the build takes 2 to 12 minutes. Run it as the project documents it:

curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh

It checks for a C/C++ compiler, Git, and Python 3.9 or later with venv, and on macOS the Xcode Command Line Tools must be installed; on Linux it needs nvcc on PATH, under /usr/local/cuda, or through CUDACXX, plus liburing headers. It installs CMake, Ninja, Hugging Face support and the framework into a private virtual environment under ~/.local/share/inferssd/releases, builds the server and the machine probes, and installs the inferssd command at ~/.local/bin/inferssd. It does not edit shell profiles or install system packages. If that directory is not on your PATH, follow the export instruction the installer prints and add it to your shell profile.

Confirm the command is reachable before going further:

command -v inferssd

It should print a path ending in inferssd. If it prints nothing, the PATH step above is the fix — do not re-run the download commands until the command resolves. Rerunning the installer later is also how you upgrade: builds use a new release directory and publish the command only after build and schema checks, so a failed build keeps the previous command, and downloaded models are not touched.

Step 3: Start DeepSeek-V4.1-Flash in terminal chat

One command downloads the model, measures the machine if this is the first launch, and opens terminal chat:

inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

The first run downloads about 470 GiB of model files to ~/.local/share/inferssd/models and measures your machine. The -hf form fetches the repository metadata, pins the resolved commit, checks the complete shard set and disk space, then downloads just those GGUF files — it never downloads every quantization. Keep all shards together and do not edit them. To put the weights on a specific drive, add --model-dir:

inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF --model-dir /path/on/ssd

Replace /path/on/ssd with a real directory on your fastest SSD. There is no memory setting to choose: InferSSD uses the memory available and keeps its own margin for the host. Once chat opens, terminal chat keeps conversation history; /clear starts a new conversation. Ask it something that exposes the routing idea, for example:

Explain mixture-of-experts routing in two paragraphs.

Expect a slow first answer. Model loading is excluded from the benchmark timings quoted above, and decode on the tested hardware runs at roughly 3 to 7 tokens per second, so a two-paragraph answer takes its time. Stop with Ctrl+C, or /exit in chat. Each launch keeps its receipts under ~/.local/share/inferssd/state/runs.

Step 4: Serve it behind the browser UI and the OpenAI-compatible API

Terminal chat proves the model runs. The server form is what other tools can use:

inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

Open http://127.0.0.1:8080 in your browser for the web UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. The server listens on loopback port 8080; --port changes it. If the weights are already downloaded, this start is far quicker than Step 3 — the download and the machine measurement are done once and reused. Stop it the same way, with Ctrl+C. If the browser UI loads but answers crawl, that is the decode rate from the key facts, not a broken install: on the project’s own Mac rig, DeepSeek decodes at 2.874 tokens per second.

Step 5: One response, the Muse model, and weights you already have

Three variations cover the rest of the documented surface. First, a single response that stops the owned server afterwards — useful for a scripted smoke test:

inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF -p 'Explain MoE.' -n 256

Second, the other tested model. Muse selects its UD-Q4_K_XL quantization automatically:

inferssd cli -hf unsloth/Muse-Spark-Open-816B-A42B-GGUF

Muse is the larger model of the two — 816B total, 42B active — and its project-reported decode is slower still: 1.097 tokens per second on the Mac rig, 1.520 on the RTX 5090 rig. Start it only if you have the disk and the patience for it, and remember its GGUF is a separate download from the DeepSeek one.

Third, weights you already hold locally, including fully offline launches after installation. Point -m at the first shard of a complete local copy, with all shards together in one directory:

inferssd serve -m /path/to/first-shard.gguf

The -hf form accepts only the two README repositories and their tested quantizations, with explicit :Q8_0 and :UD-Q4_K_XL suffixes also accepted, and --revision pins a model revision. Do not point it at a different repository expecting it to work: the quick start says these two, and only these two, are accepted.

Step 6: See what it chose for your machine, then upgrade safely

To inspect every setting and its default, write them all to a file and copy only the lines you want to fix into your own settings file, as the configuration guide describes:

inferssd configuration write --output settings-defaults.ssd

Pass your own file with --settings settings.ssd. The launch choices that matter most here — context, machine profile, and re-measuring — are overridden with --context, --machine and --reprofile. The context defaults to 8192 tokens. Do not copy benchmark settings from the project’s notes as deployment defaults: the quick start warns that the README benchmark settings describe recorded runs, not universal defaults.

To upgrade, rerun the installer from Step 2 — v0.1.1, released 31 October 2026, fixes the Linux source install with recent CUDA toolkits and inferssd profile after a one-command install. To uninstall, stop active sessions, remove the inferssd command from the chosen bin directory, and remove the installation’s releases, current and state entries. Keep models to preserve the downloaded weights: deleting 470 GiB by accident is the one uninstall mistake that hurts.

What you built

A local server for a 763-billion-parameter model that no workstation here could hold in memory: installed from source, measured to its own machine, downloaded once to SSD, chat-tested in the terminal, and served on loopback behind a browser UI and an OpenAI-compatible API endpoint. Nothing in the setup sends prompts to a hosted provider — the model runs on your hardware, the weights sit in ~/.local/share/inferssd/models (or the directory you chose), and each launch leaves receipts under ~/.local/share/inferssd/state/runs. The trade is explicit and worth repeating: you bought runnability with disk space and patience, not speed.

Honest limitations

• Decode is slow. Project-reported decode for DeepSeek is 2.874 tokens/s on the M5 Max Mac rig and 7.270 tokens/s on the RTX 5090 rig; Muse is slower on both. Long answers take minutes. Prefill speedups over llama.cpp do not change that.
• Two models, no on-ramp. Only DeepSeek-V4.1-Flash-1017 (Q8_0) and Muse-Spark-Open (UD-Q4_K_XL) are tested, and there is no small model to verify the install with before committing about 470 GiB of disk and a long download.
• Benchmarks are the project’s own. One run each on the Mac, a 4K prompt tier, 256 output tokens, and comparisons of complete configurations — the methodology notes say the results do not isolate a single optimisation or support a pooled speedup. Treat them as evidence the approach works, not as your expected throughput.
• Platform limits are real. Windows support is in progress. Linux needs systemd with cgroup v2 and the memory controller delegated to the user manager; most containers and WSL without systemd enabled will not satisfy the launcher’s checks. macOS needs Apple Silicon and Metal.
• Young and source-only. First public release 30 October 2026, v0.1.1 the next day. There are no prebuilt packages yet; every install compiles from source in 2 to 12 minutes, and behaviour may change between releases. Rerun the installer to upgrade, and re-check the quick start when you do.
• Weights carry their own licences. InferSSD itself is Apache-2.0, but the model weights remain subject to their publishers’ licences, and the text-model GGUF omits the vision and MTP/DSpark parts of the official release. The optional v1 expert pack format records no checkpoint fingerprint, so keep each pack paired with its source checkpoint and rebuild it whenever the shards change.