// HACKER NEWS — CYBERSECURITY
Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses
How much GPU RAM do you actually need to run Qwen3.8 27B without sacrificing quality?
The full BF16 model weighs 55 GB, putting it beyond most consumer hardware.
Yet the 17 GB Q4_K_M matches the full model on a popular agentic coding benchmark, Terminal-Bench 2.1. It fits on a 24 GB card such as RTX 4090, still leaving room for about 64k tokens of context.
Compression eventually hits a cliff. At 1 bit, the model performs around random chance on GPQA Diamond, and longer reasoning makes it worse.
Qwen3.8 27B GGUF quantizations available from Unsloth on Hugging
Face. So much to choose from! I will check 8-bit Q8_0 (29 GB), 4-bit Q4_K_M (17 GB), 2-bit UD-Q2_K_XL (10.7 GB), and the smallest one possible, 1-bit UD-IQ1_S (6.2 GB).
Previously, I investigated the Qwen3.6 27B model, which was good at generating SVG pelicans even at 12GB, and maintained most of its knowledge up to 16GB.
At the same time, in Reddit threads, many complain that all quantizations, even the 8-bit ones, give worse results - with people asking why your local LLM feels dumber than it is. Are these complaints grounded?
Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks. Some noise might be irrelevant for solving tasks, as (say) a quantized model generates an answer of precisely the same quality, paraphrased a bit. In other cases, a single different token might be a logical error, or even abruptly end the output.
So, I focus on directly measuring results on popular benchmarks - GPQA Diamond, instruction-following IFBench, programming Terminal-Bench 2.1.
First, to replicate official results of the full model BF16, and then to see how quantization affects results.
I burned around $3,000 on Modal GPUs when I ran models with llama.cpp using a build from 16 August 2026 as earlier builds do not work for this model. I could have run it on my own laptop, in principle, but (unlike pelican-generation), these are time-consuming benchmarks.
Note that I use F16 KV-cache regardless of model quantization, weighing around 2.3 GB per 32k tokens.
I used Unsloth quantizations: v2 for the 2-, 4-, and 8-bit models, and v3 for the 1-bit models. Unsloth replaced the v2 files on 19 August 2026, so the exact files used for most tests are no longer available.