Five inference engines, one Strix Halo: Qwen3.8 Flash-Next at 256k context

A Framework Desktop with AMD’s Strix Halo APU has 128 GB of memory that the CPU and the GPU share, which makes it one of the few small machines that can hold a 100 GB model. What it does not have is one obvious way to run that model. Alongside llama.cpp, a handful of engines written specifically for this chip have appeared over the last few months, each with its own numbers in its README – measured on its author’s machine, with its author’s prompts.

We wanted numbers we could compare: every engine on the same box, behind the same endpoint, driven by the same client. This post is the result – what we measured, what moved the numbers, and what tripped us up along the way.

The setup

• Hardware: Framework Desktop, Ryzen AI Max+ 395 with the Radeon 8060S iGPU (gfx1151), 128 GB LPDDR5X-8000 (about 256 GB/s). UMA frame buffer at the BIOS minimum of 512 MB, so nearly all memory stays dynamically available to the GPU.
• OS: Ubuntu 26.04, kernel 7.0, deployed with Ansible. The GPU may use up to 124 GiB of system memory (amdgpu.gttsize=126976 ttm.pages_limit=32505856), and the ACPI platform profile is set to performance (the APU package may draw about 120 W sustained, 140 W boost).
• Serving: llama-swap exposes one OpenAI-compatible endpoint and starts the requested engine in its own container on demand. Only one model is loaded at a time – Flash-Next alone fills most of the memory.
• Model: Qwen3.8 Flash-Next, a mixture-of-experts model with a native context of 262,144 tokens (256k). Each engine runs the quantization it was built for.

Gufo, strixite, crucible and talus all draft tokens with the model’s multi-token-prediction (MTP) head – speculative decoding that the full model then verifies, so the output stays the same while several tokens can be accepted per step. llama.cpp decodes one token at a time.

How we measured

A small Python client on the box itself talks to llama-swap’s streaming chat API:

• Prefill is prompt tokens divided by time to first token; decode is generated tokens over the time between the first and the last one. 256 tokens are generated per request, temperature 0.
• Every prompt starts with a random nonce, so no engine can answer from a prompt cache.
• Two kinds of prompts: a story (prose, hard to predict) and code (a Python module with tests). The difference matters a lot for engines that draft.
• Parallel load: 4 or 8 identical requests at once, reported as total generated tokens per wall-clock second.
• Prompt lengths below are actual token counts: about 1k, 7k, 27k and 215k tokens.
• Each point is the median of three runs (two for the 215k-token prompt).
• The engine results were measured with amd_iommu=off; with the IOMMU on, expect prefill a few percent lower (see tuning). The power measurements ran with the IOMMU on.

This measures speed, not quality. The engines run different quantizations – between 94 and 124 GB of weights – so a faster engine is not automatically the better choice.

Results at short context

After tuning (more on that below), with prompts of up to 27k tokens:

The chip-specific engines prefill about three to four times faster than llama.cpp and decode roughly twice as fast. Among them, the spread is small on prose and large on code: crucible goes from 53 to 90 tok/s when the text becomes predictable, strixite from 50 to 79. Gufo’s MTP drafting gains little on either – though switching it off costs Gufo about 30 % of its decode speed.

Results at 256k context

We need the model’s full 256k context, which changes what fits: every parallel slot reserves room for its context, next to roughly 100 GB of weights. At 262,144 tokens:

Two things stand out. First, the long context costs surprisingly little: decode after a 215k-token prompt is within 10 % of the short-context figure for every engine, and prefill drops by about 10 %. Such a prompt still takes two minutes before the first token – fine for an agent reading a code base, not for chat.

Second, the parallel slots are where the engines differ at 256k. strixite keeps four slots of 256k each and scales best under load; crucible 4.05 fits three, Gufo only two (four and eight sessions run out of memory at 256k). llama.cpp holds 256k too, but only with few context checkpoints (see below) and then prefills at about 270 tok/s – the 215k prompt takes 13 minutes.

talus, the newest engine in the field, stays below its own published figures (65.6 tok/s decode at 32k, 73.8 on its coding mix). Its coding benchmark is not ours, and it serves one request at a time, so the gap may say more about the workload than about the engine.

What moved the numbers

Parallel slots. By default Gufo serves one request at a time, crucible one stream, strixite two slots. Raising them lifted the total throughput for four parallel requests by 50–75 %, without slowing down a single request:

If more than one client talks to the box – several agents, or a team behind a gateway – this is the single most valuable setting, and it is off by default.

amd_iommu=off. Prefill got 3–9 % faster and decode 1–4 % faster, consistently, on every engine. strixite’s author recommends it – but it comes at a real price. Without the IOMMU, the kernel can no longer restrict which memory a device may access by DMA: any PCIe device, including anything plugged into one of the USB4 ports, can read and write all of the machine’s memory, and a device with buggy or compromised firmware is no longer contained. Device passthrough to VMs (VFIO) stops working as well. We measured the gain, but run our machine with the IOMMU on.

llama.cpp batch sizes. -b 2048 -ub 2048 raised prefill by 10–20 %. 4096 was slower again on Vulkan.

What did not help: forcing the GPU to its highest clock state (power_dpm_force_performance_level=high), strixite’s lower-precision prefill options and its MTP draft depth, and Gufo’s draft depth (seven tokens, the default, is also its maximum). All within noise.

Memory pitfalls

With 100 GB of weights in 125 GiB of usable memory, a few gigabytes decide whether a model loads at all.

The crash kernel reservation. Ubuntu’s kdump-tools reserves memory for a crash kernel via crashkernel= – 2.3 GiB on this machine. That was enough to make crucible’s 4.05 pack fail while loading: the GPU allocation reached 102 GiB, the page cache still held the pack files being read, and the next allocation failed instead of evicting the cache. After apt purge kdump-tools and a reboot, the pack loads entirely on the GPU.

The n-gram table. Flash-Next has a large n-gram embedding table, and crucible’s 4.18 pack stores it in 8 bits: 51.8 GB on its own. Copied to the GPU, the pack runs out of memory with any context size. CRUCIBLE_PLE_HOST=1 leaves the table memory-mapped on the host, reading the 16 rows each token needs through the page cache: GPU memory drops to 75 GiB, and the pack runs. At 256k context the 4.05 pack needs the same switch, because the context state no longer fits next to the resident table. Its cost is small – decode 50 instead of 52 tok/s.

llama.cpp’s context checkpoints. Flash-Next is a hybrid model: most layers keep a recurrent state instead of a key-value cache. llama-server saves up to 32 checkpoints of that state per slot in host memory by default. At 256k context, that grew to 29 GB in the middle of a long prompt, and the kernel’s out-of-memory killer ended the process. -ctxcp 4 --cache-ram 0 -lm none (four checkpoints, no host-side prompt cache, no memory-mapped weights) fixed it.

Power

The platform profile sets the power limits of the whole APU package – CPU, GPU and the rest of the chip share one budget – so we expected performance to matter. It does, a little, but inference never needs the full budget. The figures below are the package power the APU reports (amdgpu’s PPT sensor); they leave out the RAM chips, fans and power-supply losses, so the draw at the wall is higher. A 27k-token prompt followed by 512 generated tokens on Gufo:

Medians of three runs each. Decode varies by up to 4 tok/s between runs of the same setting – every request generates slightly different text, so a different share of the MTP drafts is accepted – and the profiles’ decode figures differ by less than that. On this machine, balanced and low-power behave identically: both cap the package at 100 W. performance buys about 10 % more prefill, at peaks of 140 W.

In every profile, the GPU sits at its maximum clock of 2.9 GHz. Raising the power limits further – with ryzenadj, say – would not make it faster.

The lever that does work points the other way. Decode is bound by memory bandwidth, not by compute, so a lower GPU clock costs prefill speed but barely touches decode. amdgpu lets you cap the clock through pp_od_clk_voltage (in manual performance level). Same workload, performance profile:

At 2.2 GHz, decode keeps its full speed at 37 % less power, while prefill gets 15 % slower. For a box that mostly generates, that is a better trade than any platform profile.

Conclusion

• For coding and agent work at long context, crucible with its 4.05 pack was the fastest engine in every single-request measurement: about 2,000 tok/s prefill, and 89 tok/s decode on code.
• For several clients at once, strixite scales best: four 256k slots and the highest total throughput under load.
• Gufo was close on prefill but decodes slower than both; talus is promising, but not yet ahead on our workload; llama.cpp is three to four times slower on this model than the engines written for the chip.
• The defaults leave a lot on the table: parallel slots above all, and making sure nothing else reserves the memory the model needs.

All of these engines are 0.x releases that change weekly; crucible alone went from 0.26 to 0.30 in the ten days before these runs. The numbers above are a snapshot of early November 2026 – the methodology is the part that will last.