Hugging Face blog author Tarek Ziadé reports on an experiment to run Qwen3.8-Flash-Next, a mixture-of-experts (MoE) model whose expert weights total about 121 GB, on a 32 GB Mac. Experts make up more than 90 percent of such models' weights, so the author argues that a machine with modest memory can run a large MoE model if it streams in only the experts the router selects for each token.
The post benchmarks several inference approaches on Qwen3.6-35B-A3B using a single 24 GB A10G GPU. Strata, an engine that keeps frequently used experts in VRAM and moves the rest to RAM or disk, reached 82 to 88 tokens per second on Q8_0. llama.cpp with --fit on reached 63 to 64 tokens per second, and vLLM with FP8 and expert-only CPU offload reached 20.1. Transformers with Accelerate managed 0.16 tokens per second. The author attributes that result to Accelerate copying every expert of an offloaded layer to the GPU on every token, with no caching. Strata has also been reported at 13.5 tokens per second for Qwen3.8-Flash-Next on a 32 GB Mac M4.
The author's goal is to modify Transformers itself so that it streams only the routed experts through a bounded cache. The excerpt ends before the final results, so it does not show whether the Transformers work can match Strata or llama.cpp. The author says the Transformers version will probably remain too slow for daily coding work. For Australian businesses and professionals, this changes nothing in practice today. The approach is experimental and offers no production speed advantage over existing tools. Teams that handle personal information may eventually benefit from keeping inference on local hardware, since data stays off third-party servers, but that alone does not address obligations under the Privacy Act 1988. Developers with a 32 GB or larger machine could test llama.cpp's CPU expert offload at no cost, but there is no reason to change current workflows.
Key Points
Mixture-of-experts models keep most of their weights in expert banks, so streaming only the routed experts can run large models on limited memory.
In the author's benchmark on Qwen3.6-35B-A3B, Strata reached 82 to 88 tokens per second, ahead of llama.cpp and vLLM.
Transformers with Accelerate offloading managed about 0.16 tokens per second because it copies every expert of an offloaded layer on every token.
The author is modifying Transformers to stream routed experts through a bounded cache, but the excerpt does not include final results.
For most Australian businesses the work is experimental and changes nothing in current workflows.