Run a 321B AI model on your own GPU: a hands-on guide to Project Maya

Models this size usually need a datacenter. Project Maya — an MIT-licensed, open-source engine led by Peasant Smith (@mw00) with co-maintainer @needmorevram, about 900 GitHub stars in its first ten days — runs GLM-5.3-Flash (zai-org, MIT) on the hardware you already own: 321 billion parameters, about 18 billion active per token, up to a million tokens of context. The trick is an engine built for this one model: it keeps the experts your conversation uses most on the GPU, the next ones in RAM, and the rest on your NVMe SSD, moving them as you chat. It measures your GPUs, CPU, RAM and SSD and tunes itself to them. Nothing leaves your machine. This guide installs it, picks the right quant for your memory pool, and wires it into your tools through an OpenAI- and Anthropic-compatible API.

• Project Maya (mw00/project-maya) is MIT-licensed, led by Peasant Smith (@mw00) and co-maintained with @needmorevram; it reached about 900 GitHub stars in the ten days after its October 6, 2026 release.
• It runs one model: GLM-5.3-Flash by zai-org (MIT) — 321 billion parameters, about 18 billion active per token, up to 1 million tokens of context.
• Maya's own quants: Maya-S (default, 96.5 GB, 97.9% of the FP8 model's zero-shot accuracy); Maya-S24 (94.7 GB, for cards of 24 GB or less); Maya-M (116 GB, 97.9%, calibrated toward tool calls and front-end code); Maya-L (156.3 GB, 99.2%).
• Real measured speeds: 118 tokens/s decode on 4× RTX 4090, 81 on one RTX PRO 6000, 30+ on a single RTX 5090; the server handles up to 8 parallel requests.
• Hardware floor: one NVIDIA GPU (compute capability 7.0+, i.e. V100 / RTX 20 series and newer), 32 GB of RAM, a fast NVMe SSD (~100 GB for Maya-S), Linux x86-64, CUDA 12.x, Python 3.10+. Windows 10/11 and AMD are experimental.

Frontier-class open weights have quietly become a hardware problem, not a software one: the models are free and permissively licensed, but serving them usually means cloud GPUs and an API bill. Project Maya attacks exactly that gap. Its engine — grown from Strata and rewritten for GLM-5.3-Flash, with quantization formats and CPU kernels from ggml/llama.cpp — tiers the model’s mixture-of-experts layers across every memory tier you have: the experts your current conversation uses most stay in VRAM, the next ones in RAM, the rest on your NVMe SSD, swapped in as the chat moves. That is how a 321-billion-parameter model fits on a single gaming card.

You also get the full appliance around it: a chat UI with per-chat instructions, edit-and-regenerate and HTML previews, a live monitor that shows down to what fills each GPU’s memory, thinking levels (Off, Low, High — the default — or Max) in Settings or per request, image input, and two compatible APIs — OpenAI’s at http://127.0.0.1:8080/v1 and Anthropic’s at http://127.0.0.1:8080/v1/messages — with tool calls and up to eight requests answered together. Let’s build it.

What you’ll need

• GPU: one NVIDIA card, compute capability 7.0 or newer (V100, RTX 20 series and newer) — or up to 16 that share the model. More VRAM is faster. Pascal (Tesla P40/P100, GTX 10) is experimental, not validated yet.
• RAM: 32 GB runs it. More RAM keeps more experts close and is faster.
• Disk: a fast NVMe SSD with room for the model (~100 GB for Maya-S, ~160 GB for Maya-L) plus 1.1 GB for the picture encoder. Not a hard disk.
• System: Linux x86-64 (a CPU with AVX2 is best), the NVIDIA driver, CUDA 12.x (CUDA 13 works for RTX 20 and newer, not for the V100 or Pascal), g++, Python 3.10+. Windows 10/11 is experimental (run START-MAYA.bat, not WSL2); AMD (RX 7900 XT/XTX and friends, ROCm 7) is experimental and text-only for now.

The setup checks all of this, prints the exact command for anything missing, and installs nothing system-wide.

Step 1: Clone and run the guided setup

One command fetches the project; the setup script does the rest interactively. Answer a few questions — Enter takes the recommended choice — and the setup shows every download before it starts, builds everything once (20–40 minutes plus the download), and starts the model.

git clone https://github.com/mw00/project-maya.git && cd project-maya
./setup.sh

When it finishes, open http://127.0.0.1:8080. Ctrl+C stops the model, and running ./setup.sh again starts it right away. If the download stalls, just run ./maya.sh again — it continues where it stopped.

Step 2: Pick the quant for your memory pool

The setup asks which quant to download and recommends one for your PC. Maya’s own quants are made from Z.ai’s FP8 release with statistics from the FP8 model, and each keeps the model’s MTP block, which drafts tokens ahead. Accuracy figures are measured against the FP8 model’s zero-shot accuracy:

There is also Maya-M-Derisked (116 GB): Maya-M with a community weight edit that reduces blanket refusals — experimental, not measured yet. And community EXL3 quants (3.05 and 4.05 bpw) run on Maya via --model EXL3-3.05bpw, NVIDIA-only and experimental.

Step 3: Chat in the dashboard

The browser UI at http://127.0.0.1:8080 is more than a demo page. You get a chat with per-chat instructions, edit-and-regenerate and HTML previews (history is kept in your browser), a live Monitor of the model and your hardware down to what fills each GPU’s memory, a context size you can change in Settings, and the four thinking levels — Off, Low, High (the default) or Max — in Settings or per request. Attach a picture in the chat, or send image parts through either API.

Step 4: Point your apps and coding agents at it

This is where Maya earns its keep. Your apps and coding agents talk to it through an OpenAI-compatible API at http://127.0.0.1:8080/v1 and Anthropic’s at http://127.0.0.1:8080/v1/messages — tool calls included, streamed as the model writes them. The README’s own example is Claude Code:

# Route Claude Code through your local Maya instance
ANTHROPIC_BASE_URL=http://127.0.0.1:8080 claude

A long prompt’s reading progress streams too when a request asks for it ("return_progress": true), and a web page served from another address can call the API once you allow its address ("cors_origins" in the setup docs). Because up to eight requests are answered together, one PC can serve several agent sessions — or several people — at once.

Step 5: Calibrate it to your hardware

Maya measures your machine during setup, but the explicit calibration pass squeezes more out of it:

# Tune the engine to your exact hardware (10-15 minutes)
./maya.sh --calibrate

# Serve several requests at once (up to 8); kept in the config
./maya.sh --parallel 4

On one GPU, a token of each parallel request goes through the model together, so the dense weights are read once for all of them — measured at ×1.67 for two and ×2.02 for four on hardware that holds every expert in GPU memory. Two more levers from the docs:

• Long contexts: new setups keep each attention layer’s cache in RAM with a window of it in VRAM (--kv-streaming on), so VRAM holds more experts — a 1M-token context fits on one RTX 3090, with prompts 4–9× faster at 512K.
• Faster answers with a DFlash2 drafter (NVIDIA RTX 30 / A100 and newer): ./maya.sh --setup --draft dflash2 downloads a small drafter that drafts up to 7 tokens a round and gives the same answers — +19% on one RTX 3090 against the MTP block. A speed calibration measures it on your PC and keeps it only where it is faster. Note the license: the drafter is CC BY-NC-ND 4.0, non-commercial use only, and downloads only if you choose it.

And here is the honest benchmark table from the repo — real machines, real users, most measured with ./maya.sh --bench (a token is about ¾ of a word):

Step 6: Serve it on your LAN — carefully

Want the model from another device on your network? The README gives the exact incantation, with a warning in bold:

./maya.sh --setup --host 0.0.0.0 --api-key <secret>

Always set a key. Binding to 0.0.0.0 exposes an unauthenticated 321B-parameter API to your network otherwise — this is the one command in the guide where skipping a flag is a security hole, not a preference.

What you built

A private, self-hosted frontier-class model: Z.ai’s 321-billion-parameter GLM-5.3-Flash running on your own GPUs, with a browser chat and hardware monitor, OpenAI- and Anthropic-compatible APIs serving up to eight parallel requests, thinking levels and image input, and an engine calibrated to your exact hardware — with nothing leaving your machine.

Honest limitations

• It eats your RAM while it runs. The engine pins most of the free RAM (all but about 6 GB) for its expert tier, so other programs get little RAM while Maya is up. This is a dedicated-workstation workload, not a background tab.
• First install is heavy. A 20–40 minute build plus a ~100 GB model download onto a fast NVMe SSD — on a spinning disk, forget it.
• Speed is expert-bandwidth bound. A single consumer card gives 30+ tokens/s, not datacenter-class hundreds; it climbs steeply with VRAM (118 tokens/s on four RTX 4090s). If you need 1M context on one card, --kv-streaming on is what makes it fit.
• Windows and AMD are experimental, as are Pascal GPUs and the community EXL3 quants (NVIDIA-only).
• The optional DFlash2 drafter is non-commercial (CC BY-NC-ND 4.0) — skip it if the machine earns money.
• One-model engine. Maya is built for GLM-5.3-Flash specifically; new Z.ai releases need the maintainers’ new quants. And Maya-M-Derisked (reduced blanket refusals) is experimental and not benchmarked yet.