Measured by us · Nov 17, 2026

Quantizing Haru-4B, a full-duplex voice model: 10.5 GB to 4.8 GB

Result

Only the language backbone goes to 4-bit: 8.4 GB of BF16 weights become a 2.74 GB Q4_K_M GGUF, and the whole bundle drops from 10.54 GB to 4.84 GB. A T4 runs the backbone at 68.2 tok/s (15 ms a step, well inside the 133 ms frame); a 4-core EPYC manages 8.4 tok/s (119 ms), too slow for real-time duplex.

Model

Haru-4B

5.27B params, 7.5 Hz duplex

Compute

CPU + T4

4-core EPYC, Colab T4

Sizes

Exact bytes

from the published files

Metric

GB on disk

plus llama-cli tok/s

Size on disk (GB) · lower is better

024681012MeanFlow audio head, FP160.73VibeVoice audio codec, FP161.37Qwen3.5-4B backbone, GGUF Q4_…2.74Our split bundle (Q4_K_M + FP…4.84Upstream checkpoint, BF16 (al…10.54

Dot = measured, dashed line = best. Hollow dot = reference row (agreement ceiling or baseline), not a contender.

ModelSize on diskParams (B)Share of params %Bits per weighttok/s, 4-core EPYCtok/s, T4
MeanFlow audio head, FP16open0.730.366.916.02 - -
VibeVoice audio codec, FP16open1.370.6913.016.01 - -
Qwen3.5-4B backbone, GGUF Q4_K_Mopen2.744.2179.95.218.468.2
Our split bundle (Q4_K_M + FP16 audio)4.845.27100.07.35 - -
Upstream checkpoint, BF16 (all parts)baseline10.545.27100.016.00 - -

The hookLink to this section

Haru-4B (from higurashi-ai) is an open-weight spoken dialogue model that listens and speaks at the same time. It works in 133 ms frames, 7.5 per second, and ships as a 10.54 GB BF16 checkpoint with official runtimes for CUDA and Apple MLX only.

We wanted it smaller and closer to llama.cpp. Pointing convert_hf_to_gguf.py at it fails: the architecture is HaruForConditionalGeneration, and the audio codec and flow head are not transformer layers llama.cpp knows. So we took it apart.

The setup, in one pictureLink to this section

Haru-4B checkpoint (10.54 GB, BF16, 5.27B params)
   ▼  read tensor headers over HTTP range requests, no full download
   ▼  slice by prefix with memory-mapped safetensors
lm.*     Qwen3.5-4B backbone  4.21B ─► rename to model.*, tied embeddings
                                       convert_hf_to_gguf.py --no-mtp (BF16 GGUF)
                                       llama-quantize Q4_K_M ─► 2.74 GB
codec.*  VibeVoice codec      0.69B ─► keep FP16 ─► 1.37 GB
head.*   MeanFlow head        0.36B ─► keep FP16 ─► 0.73 GB
   ▼
llama-cli on the backbone: 4-core AMD EPYC, Tesla T4

What the numbers sayLink to this section

  • The backbone is where the bytes are. It holds 80% of the parameters, so it is the only part worth quantizing. Q4_K_M takes it to 2.74 GB, 5.2 bits per weight (llama.cpp keeps the 248k-token embedding table at higher precision).
  • The bundle is 54% smaller. 10.54 GB becomes 4.84 GB, and the audio parts are untouched, so the codec and head produce the same audio they did before.
  • A T4 has room to spare. 68.2 tok/s is about 15 ms per backbone step. One step per 133 ms frame leaves most of the budget for the codec and the flow head.
  • A small CPU does not. 8.4 tok/s on 4 EPYC cores is about 119 ms per step, leaving roughly 14 ms for everything else in the frame. Full-duplex would fall behind within seconds.

The dirty laundryLink to this section

  • We have not run full-duplex audio end to end on the quantized model. The speeds are text-generation runs on the backbone. The C++/ggml runtime that would feed audio embeddings into llama.cpp is a scaffold, not a working engine yet.
  • Single runs, no saved logs. Treat the tok/s numbers as indicative, not a benchmark with error bars.
  • No perplexity or audio-quality score. Keeping the codec and head in FP16 is a design choice: 4-bit audio decoders are known to add metallic artifacts, and we did not test that ourselves.
  • Memory bandwidth math is an estimate. Reading the 2.74 GB backbone 7.5 times a second needs about 21 GB/s; shared cloud CPUs often sit below that.

The recipeLink to this section

  1. Inspect before you download. A safetensors header scan over HTTP gives every tensor name, shape and dtype, so you can see which prefix holds the weight.
  2. Slice the backbone into a plain causal LM. Rename lm.* to model.*, drop the duplicate lm_head.weight and set tie_word_embeddings: true, or saving fails with "Some tensors share memory".
  3. Convert with --no-mtp, then llama-quantize ... Q4_K_M. The BF16 GGUF is the intermediate; Q4_K_M is the one to ship.
  4. Leave audio decoders in FP16. They are small (2.1 GB together) and quality-sensitive.
  5. Budget per frame, not per token. For a 7.5 Hz duplex model, one backbone step must fit well under 133 ms with room for the audio parts. A T4 fits; a 4-core CPU does not.

What's nextLink to this section

A working ggml runtime for the codec and the MeanFlow head, so the whole frame loop runs in one native process. Then a cascaded fallback (whisper.cpp + a small GGUF LLM + Piper) for CPU-only boxes, measured with the same frame budget.

Method & environment
  • Question: Can a full-duplex spoken dialogue model be shrunk enough to run on cheap hardware, and which parts can safely go to 4-bit?
  • Compute: 4-core AMD EPYC (Linode) + Tesla T4 (Colab)
  • Method: Sizes are the exact byte counts of the upstream checkpoint and of our published files (GB = 10^9 bytes). Parameter counts come from a safetensors header scan; bits per weight is bytes x 8 / params, which also checks the counts (the FP16 parts land on 16.0). Speeds are single llama-cli generation runs on the backbone alone (-t 4 on CPU, -ngl 99 on the T4); raw logs were not kept, and audio quality was not scored.
  • Numbers are imported directly from the run's result files; nothing is typed by hand.