FreeToken runs a 753B model on one workstation GPU

FreeToken is a serving engine that runs very large Mixture-of-Experts models on ordinary hardware. It splits the work between GPU and CPU, measuring bandwidth as it goes and moving experts around to suit.

The headline claim is a 753B-parameter model, GLM-5.2, running on a single workstation GPU. That needs an RTX PRO 6000 and 512GB of host DDR5, and it returns 14.9 tokens per second.

Smaller setups do better on speed. An RTX 5090 serves Qwen3.6-35B at 77 to 83 tokens per second, and DeepSeek-V4-Flash at 22 to 25. Even an 8GB RTX 4060 laptop manages 39.3 tokens per second on a 35B model.

Time to first token is under 44 seconds in the worst case, which is slow to wait for but workable in a batch job.

Why it matters

MoE models only activate a fraction of their weights per token. GLM-5.2 uses 40B of its 753B at a time. Most serving engines do not exploit that well on small hardware, so the whole model has to fit in GPU memory. FreeToken places experts dynamically instead.

It is Apache 2.0 on GitHub, on PyPI as freetoken v0.1.2, and packaged as a desktop app. You need an NVIDIA GPU on driver r580 or later, on Linux or Windows.

Source: MarkTechPost