| Model | Mode | Pass % (strict) | easy | hard | ultra | Median s | Max s | >120 s | Decode tok/s | Trunc | VRAM / memory |
|---|---|---|---|---|---|---|---|---|---|---|---|
| A Swift-1.5 27B IQ3_S (Win) | off | 66% (33/50) | 9/10 | 16/20 | 8/20 | 8.9 | 26.4 | 0 | 14.4 | 0 | 12070 MiB VRAM (218 free) |
| A Swift-1.5 27B IQ3_S (Win) | on | 94% (47/50) | 10/10 | 19/20 | 18/20 | 51.2 | 210.1 | 6 | 14.6 | 1 | 12070 MiB VRAM (218 free) |
| B Bonsai-2-27B-Abl PQ2_0 (.54) | off | 72% (36/50) | 10/10 | 14/20 | 12/20 | 4.8 | 15.5 | 0 | 33.6 | 0 | 10135 MiB VRAM (2153 free) |
| B Bonsai-2-27B-Abl PQ2_0 (.54) | on | 94% (47/50) | 10/10 | 17/20 | 20/20 | 26.6 | 68.2 | 0 | 33.6 | 0 | 10135 MiB VRAM (2153 free) |
| C Qwen3.8-27B GSQ-RCO IQ3_S (Win) | off | 68% (34/50) | 10/10 | 16/20 | 8/20 | 9.4 | 55.6 | 0 | 14.8 | 2 | 12070 MiB VRAM (218 free) |
| C Qwen3.8-27B GSQ-RCO IQ3_S (Win) | on | 90% (45/50) | 10/10 | 19/20 | 16/20 | 59.9 | 207.4 | 9 | 14.9 | 5 | 12070 MiB VRAM (218 free) |
| D Gemma4-12B Q4_K_M+MTP (.54) | off | 66% (33/50) | 9/10 | 16/20 | 8/20 | 2.4 | 6.3 | 0 | 62.0 | 0 | 11187 MiB VRAM (1101 free) |
| D Gemma4-12B Q4_K_M+MTP (.54) | on | 76% (38/50) | 8/10 | 16/20 | 14/20 | 8.0 | 40.6 | 0 | 51.6 | 1 | 11187 MiB VRAM (1101 free) |
| E Xing4.0-29B-A4B IQ3_XXS (.54) | off | 36% (18/50) | 9/10 | 7/20 | 2/20 | 7.3 | 39.3 | 0 | 21.1 | 1 | 10921 MiB VRAM (1367 free) |
| E Xing4.0-29B-A4B IQ3_XXS (.54) | on | 32% (16/50) | 9/10 | 6/20 | 1/20 | 69.0 | 157.1 | 17 | 20.7 | 13 | 10921 MiB VRAM (1367 free) |
| F MiMo-9B MLX 4bit (Mac) | off | 52% (26/50) | 10/10 | 14/20 | 2/20 | 6.6 | 18.8 | 0 | 18.7 | 0 | ~7.2-9.4 GB unified (Mac) |
| F MiMo-9B MLX 4bit (Mac) | on | 52% (26/50) | 9/10 | 14/20 | 3/20 | 6.7 | 176.2 | 3 | 20.6 | 3 | ~7.2-9.4 GB unified (Mac) |
| F MiMo-9B MLX 4bit (Mac) | on_forced | 64% (32/50) | 9/10 | 16/20 | 7/20 | 17.4 | 162.9 | 2 | 18.9 | 2 | ~7.2-9.4 GB unified (Mac) |
| G Hermes-4-14B-OBL EXL3 4.0 (.54) | off | 8% (4/50) | 8/10 | 13/20 | 4/20 | 25.5 | 32.1 | 0 | 32.8 | 46 | 10669 MiB VRAM (1619 free; Q6 cache, chunk 512) |
| G Hermes-4-14B-OBL EXL3 4.0 (.54) | on | 10% (5/50) | 8/10 | 10/20 | 5/20 | 94.4 | 106.5 | 0 | 13.1 | 40 | 10669 MiB VRAM (1619 free; Q6 cache, chunk 512) |
| H Bonsai-2-27B MLX 2bit (Mac) | off | 62% (31/50) | 10/10 | 13/20 | 8/20 | 14.9 | 50.4 | 0 | 9.9 | 0 | 9966 MB footprint (Mac) |
| H Bonsai-2-27B MLX 2bit (Mac) | on | 94% (47/50) | 10/10 | 18/20 | 19/20 | 87.0 | 260.2 | 16 | 9.8 | 0 | 9966 MB footprint (Mac) |
| I Bonsai-2-27B-Abl PQ2_0 (Mac) | off | 72% (36/50) | 10/10 | 14/20 | 12/20 | 12.0 | 51.3 | 0 | 11.6 | 0 | ~10.0 GB footprint (Mac) |
| I Bonsai-2-27B-Abl PQ2_0 (Mac) | on | 90% (45/50) | 10/10 | 16/20 | 19/20 | 81.7 | 246.7 | 17 | 11.0 | 0 | ~10.0 GB footprint (Mac) |
| J54 Bonsai-2-27B PQ2_0+MTP (.54) | off | 66% (33/50) | 10/10 | 14/20 | 9/20 | 3.8 | 9.7 | 0 | 53.0 | 0 | 10963 MiB VRAM (1325 free) |
| J54 Bonsai-2-27B PQ2_0+MTP (.54) | on | 96% (48/50) | 10/10 | 18/20 | 20/20 | 15.8 | 43.3 | 0 | 52.0 | 0 | 10963 MiB VRAM (1325 free) |
| JMac Bonsai-2-27B PQ2_0+MTP (Mac) | off | 66% (33/50) | 10/10 | 14/20 | 9/20 | 13.1 | 50.8 | 0 | 12.1 | 0 | ~10.0 GB footprint (Mac) |
| JMac Bonsai-2-27B PQ2_0+MTP (Mac) | on | 96% (48/50) | 10/10 | 19/20 | 19/20 | 67.1 | 172.6 | 10 | 11.3 | 0 | ~10.0 GB footprint (Mac) |
| L ThinkingCap-Qwen3.8-27B IQ3_S (Mac) | off | 68% (34/50) | 10/10 | 16/20 | 8/20 | 23.1 | 134.2 | 2 | 6.4 | 2 | 12.6 GB weights, footprint ~9.9 GB + paging (Mac) |
| L ThinkingCap-Qwen3.8-27B IQ3_S (Mac) | on | 92% (46/50) | 10/10 | 18/20 | 18/20 | 81.9 | 310.9 | 14 | 6.2 | 0 | 12.6 GB weights, footprint ~9.9 GB + paging (Mac) |
| M ZDTaichu5.0-9B MLX 8bit (Mac, text-only) | off | 46% (23/50) | 10/10 | 11/20 | 2/20 | 6.7 | 33.2 | 0 | 13.5 | 0 | 12 GB footprint peak (Mac) |
| M ZDTaichu5.0-9B MLX 8bit (Mac, text-only) | on | 82% (41/50) | 10/10 | 17/20 | 14/20 | 65.0 | 227.1 | 10 | 13.3 | 1 | 12 GB footprint peak (Mac) |
| K54 Bonsai-2 PQ2_0+DFlash2 (.54, draft on CPU) | off | 66% (33/50) | 10/10 | 14/20 | 9/20 | 8.5 | 31.5 | 0 | 14.8 | 0 | 11203 MiB VRAM (1085 free; draft on CPU) |
| K54 Bonsai-2 PQ2_0+DFlash2 (.54, draft on CPU) | on | 96% (48/50) | 10/10 | 19/20 | 19/20 | 52.0 | 243.7 | 7 | 14.1 | 1 | 11203 MiB VRAM (1085 free; draft on CPU) |
| KMac Bonsai-2 PQ2_0+DFlash2 (Mac) | off | 66% (33/50) | 10/10 | 14/20 | 9/20 | 13.1 | 57.4 | 0 | 12.0 | 0 | ~9.3 GB weights, footprint ~8.5-10 GB (Mac) |
| KMac Bonsai-2 PQ2_0+DFlash2 (Mac) | on | 98% (49/50) | 10/10 | 19/20 | 20/20 | 71.6 | 172.2 | 11 | 11.6 | 0 | ~9.3 GB weights, footprint ~8.5-10 GB (Mac) |
| N54 HauhauCS Qwen3.8-27B IQ3_XS+MTP (.54, partial offload) | off | 60% (30/50) | 10/10 | 12/20 | 8/20 | 48.4 | 184.8 | 8 | 2.0 | 0 | 11221 MiB VRAM (1067 free; -ngl 44/65, rest on CPU) |
| NMac HauhauCS Qwen3.8-27B IQ3_XS+MTP (Mac) | off | 62% (31/50) | 10/10 | 13/20 | 8/20 | 17.6 | 60.7 | 0 | 7.6 | 0 | Mac RSS 12.8 GB (footprint 2.1 GB, weights mmap) |
| NMac HauhauCS Qwen3.8-27B IQ3_XS+MTP (Mac) | on | 86% (43/50) | 10/10 | 18/20 | 15/20 | 135.5 | 467.6 | 27 | 6.8 | 5 | Mac RSS 12.8 GB (footprint 2.1 GB, weights mmap) |
Final Table
Source: llm-bench-2026-09-27/final_table.md · revision 6550ead3945b