Qwen3.8-Flash-Next

Also see the in-progress official-BF16 EXL3 campaign: exl3-350hq-375-2026-09-15 (3.50 HQ + 3.75 MTP6, plan + journey).

udq4kxl-cpu-moe-200k-2026-09-20 — UD-Q4_K_XL GGUF (104 GiB) CPU-MoE deployment: -ot expert placement tuning, 12/12/12 GPU split, 31.1 tok/s @ 204800/q8.

vllm-int4-autoround-3x3090-2026-09-20 — INT4-Mixed-AutoRound via patched vLLM PP3 podman deploy: 67.4 tok/s 1-stream / 73 aggregate @ 262144. New champion.

Qwen3.8-Flash-Next abliterated UD-IQ4_XS: quantization, deployment and template A/B

• Date: 2026-09-06 (single-day run)
• Host: laguna-s21 — 3x RTX 3090 (72 GB aggregate VRAM), 128 GB RAM, Ubuntu Linux
• Inference stack: llama.cpp flash-next fork at WORKSTATION/llama.cpp-flash-next (--repack, --op-offload, -Cr CPU pinning), plus the stock ggml-org build for the quantization pipeline
• Serving: qwen36-multi-proxy on :12434, backends on loopback
• Agent: hermes, default provider qwen-local

This experiment covers the complete pipeline for moltriver/Qwen3.8-Flash-Next-Uncensored-FP8 → Unsloth-style UD-IQ4_XS GGUF: torchao-FP8 conversion, live agentic importance-matrix collection, proxy family deployment as the hermes agent default, and an 8-cell benchmark (2 model variants × thinking on/off × chat template sharp/baked) on auto-graded cybersecurity tasks at a 16,384-token completion cap.

1. Quantization job (quant-jobs/fp8-unc-udiq4xs-6h)

2. Proxy deployment (clone of the UD-Q4 family pattern)

Serving parameters were cloned from the qwen38-unsloth-q4 (UD-Q4_K_XL) family: sharp chat-template hook, Q8_0 KV cache, 204,800-token context, batch 16,384 / ubatch 400, 14 pinned CPU threads (0-13), metrics enabled. Hardware placement follows the existing flash-next families because the model (88 GB) exceeds VRAM (72 GB):

• PLE table on CPU — per_layer_token_embd (29 GB of IQ4_NL) pinned via -ot '^per_layer_token_embd\.weight$=CPU'; all other tensors GPU-resident across 3 cards.
• The flash-next fork with --repack --op-offload is ~4x faster than the stock-build CPU-expert layout used during the quant job (86.7 vs 21.8 t/s single-stream generation).
• Family qwen38-flash-next-ablit-udiq4xs-200k, backend localhost, variants instruct / general / coding / thinking exposed as qwen38abq4-200k-* on :12434.

Root cause found during bring-up

Requests through the proxy hung indefinitely (clients saw HTTP 000/timeouts) while the backend itself was healthy. _backend_is_ready() in the proxy requires /v1/models to advertise exactly the family’s backend_model string. Our launch script used --alias qwen38abq4-200k while the family key is qwen38-flash-next-ablit-udiq4xs-200k → never “ready” → the GPU-acquisition wait (900 s) deadlocked on VRAM held by the proxy’s own backend. Fix: SERVED_MODEL_NAME must equal backend_model (the convention every other family follows). After the fix, a cold proxy launch from warm page cache answers its first request in ~17 s.

Standalone verification (before moving to the proxy)

A tmux-deployed standalone server on 0.0.0.0:12703 with a watchdog (hard deadline 1,500 s, 10 s progress trail, automatic self-test) was used to validate behaviour in isolation: ready in 21 s from warm page cache, self-test reply correct, 67.7 GB VRAM, 86.7 t/s single-stream non-thinking generation. See scripts/abq4_standalone.sh.

Known limitation: vision

The flash-next fork’s mtmd loader aborts (ggml_backend_buffer_set_usage in clip_model_loader::load_tensors) when loading this model’s F16 mmproj, so the proxy family runs text-only. Vision remains available through a stock-build server if required.

3. Hermes agent default

$WORKSTATION/.hermes/config.yaml: model.default: qwen38abq4-200k-thinking, provider qwen-local (proxy :12434); gateway restarted. End-to-end verified from the session store: user “Reply with the single word: pong” → assistant “pong” (1 m 32 s including agent boot).

4. Benchmark design

Variants (the dense UD-Q4_K_XL, flash-next UD-Q3_K_XL and FP8 vLLM weights had previously been deleted from disk, so the live comparison set is):

Cells: each variant × {instruct, thinking} × {sharp, baked} template = 8 cells, run sequentially on dedicated backends (:12703 / :12701) outside the proxy to avoid interference. Thinking mode is driven per request via chat_template_kwargs.enable_thinking; sampling follows the deployed presets (thinking: T=0.6/top_p 0.95; instruct: T=0.7/top_p 0.8).

Tasks (auto-graded, 10 points each, /30 per cell):

All cells: max_tokens 16,384, per-task timeout 1,500 s, one pass per task (n=1; task-level scores are deterministic checks, not human ratings). tok/s is estimated from streamed content + reasoning characters ÷ 3.6 chars/token; per-task estimates are aggregated by median.

“Sharp” = Qwen Sharp Chat Templates via --chat-template-file; “baked” = the GGUF’s embedded template (--jinja only).

5. Results

6. Findings

1. The sharp template is what makes thinking mode real on the abliterated model. With the baked template, enable_thinking is ignored: thinking cells score identically to non-thinking (18/30) and the CTF is never solved. With sharp, reasoning engages and separates properly into reasoning_content — 28/30 and the only CTF solve in the matrix.
2. An enabler, not a general booster: the same template on Q4_K_M does not unlock the CTF (18/30 either way). The quality win is the combination abliterated model + working thinking.
3. In thinking mode, abq4 also finalizes the practical tasks faster (T1+T2 = 38.3 s vs 54.8 s for q4km+sharp, and 60.4 s vs 365.8 s for the baked templates), despite lower raw tok/s (38 vs 51).
4. Template effect on q4km instruct: sharp halves total finalize time (158.8 s vs 308.3 s baked) and scored 18 vs 15.
5. Speed ceiling: standalone single-stream non-thinking generation reaches ~87 t/s (67.7 GB VRAM across 3x 3090, PLE on CPU, Q8_0 KV, 204,800-token context).
6. Caveats: n=1 per cell; T1/T2 are keyword/regex graded (ceiling effects possible); tok/s are character-based estimates, not tokenizer-exact. The 10-point quality gap between the top cell and the rest is driven entirely by the deterministic CTF task, which none of the seven other cells solved.

Deployed outcome: proxy family qwen38-flash-next-ablit-udiq4xs-200k with the sharp template hard-coded; hermes default = qwen38abq4-200k-thinking.

7. Contents and reproduction

Reproduce the A/B locally (backends run outside the proxy; both models must exist on disk):

## ablit UD-IQ4_XS cells (sharp + baked)
MMPROJ_PATH=none python3 scripts/bench_cyber.py
## q4km thinking cells
MMPROJ_PATH=none python3 scripts/bench_q4km_think.py   # companion driver, same harness

Host-side artifacts (not committed): model shards under WORKSTATION/UD-IQ4_XS, family entry in WORKSTATION/qwen36_model_families.json, hermes config at $WORKSTATION/.hermes/config.yaml.