I’ve been running a 4x Intel Arc Pro B70 box for local inference, and I wanted to try RecursiveSI/flint on it. Flint is a big MoE model: 39.8B total parameters but only 2.5B active per token, which is exactly the kind of thing that should run fine on four 32 GiB GPUs. It should have been a simple “point vLLM at the weights” job. It was not. Not even freaking close. Nope.
Sigh, okay so here we go..
This is a writeup of everything I had to do to get it actually serving, and why. I put the whole thing together as a drop-in stack on Github over here in the vllm-xpu-flint-cookbook, so if you just want to walk through everything I needed to do, you can, because you’re also clinically insane and like blowing things up, apparently. Idk, it’s your stuff, not mine.
Yes, you’re goddamn right I did use AI to write the patches and fix things. Would you expect me not to? Come on, duh!
Setup
My main inference box has four Intel Arc Pro B70 GPUs. The server itself is running a Threadripper 3945WX on a Supermicro M12SWA-TF Motherboard with 256GB of DDR4 non-ECC ram, becuase I’m not THAT rich!! Each B70 is running at a full PCIe 4.0 x16. It’s a good little custom box, and I like the Supermicro better than my previous Gigabyte MC62-G40 board which developed a small case of COMPLETE AND UTTER DEATH, and when I went to try and buy a replacement, they were over 1500 dollars on eBay! Hell no. Almost went with a Xeon Scalable 3rd gen motherboard and processor, but decided against it as I didn’t want to be extra poor from having to buy more RAM in this AI Bubble Hellscape we currently live in.
Yes, I am completely congnisant of the fact that I contributed to this hellscape by buying and building this server. I just wish that us little guys had better access to this hardware and that it wasn’t so completely overpriced.
I digress..
I’m using the vLLM XPU nightly image, BF16 weights, tensor parallelism across all four GPUs, and Triton for attention. The model is a Qwen3-MoE-style decoder: 49 layers (2 dense, 47 MoE), 256 routed experts with top-6 sigmoid routing plus one shared expert, a 262k context, and a hybrid attention layout (14 full-attention layers and 35 sliding-window layers with learned attention sinks).
The weights are about 75 GB across 15 safetensors shards. That fits comfortably across four 32 GiB cards, so memory was never the real problem. The problems were all in Every single goddamn XPU-specific corner of vLLM.
Patches for days, Bro
The XPU nightly recently shipped with Flint support, but it’s written for NVIDIA SM100 first. I don’t make that kind of money, so duct tape and paperclips it is! On XPU a few things break, and I had to patch three files in the installed vLLM copy.
norm.py: Triton kernels that assumed CUDA
Flint’s Triton kernels for RMSNorm and SiLU-and-mul had two bits of code that only build on CUDA. The _rsqrt helper used CUDA PTX inline asm (rsqrt.approx.ftz.f32), which the Intel Triton backend just cannot compile. I replaced it with the portable 1.0 / tl.sqrt(x). The SiLU kernel used tl.math.div_rn(...), which is also CUDA-specific, so I swapped it for plain g / (1.0 + tldevice.exp(-g)). The output is bf16 anyway, so the rounding difference is a non-issue. I also picked up an upstream fix for the block-size calculation in the TP-shard path. This sucked to try and figure out and took me about an hour.
moe.py: an import-time crash
This one was annoying. On XPU, torch.cuda is a lazy accessor function, not the real CUDA module. So a type annotation like torch.cuda.Event | None evaluates to function | None and blows up with a TypeError the moment the module is imported. The fix is a single line at the top: from __future__ import annotations. That turns all the annotations into strings so they’re never evaluated at import time. The fused MoE decode path isn’t even active on XPU, so the Event is never used at runtime. Only the import-time annotation was the problem.
model.py: an unconditional FlashInfer import
FlintAttention.__init__ imported the FlashInfer backend and asserted that the implementation was a FlashInfer one, unconditionally. But the XPU image doesn’t ship flashinfer at all; it uses Triton. The patch checks whether the fused Flint kernels are actually active first (they’re not on XPU, since they target SM100 with TP=1) and only imports and asserts FlashInfer in that case. Otherwise it just aliases the active Triton implementation and the forward path uses the stock attention call.
Two Flags That Mattered
Two serve flags did most of the heavy lifting, and both are easy to get wrong.
The first is --attention-backend TRITON_ATTN. The XPU Flash-Attention kernel is precompiled with a fixed table of configs, and Flint needs a sliding-window plus attention-sink config that isn’t in that table. So FA2 crashes at graph capture. Triton handles sliding windows and sinks natively, which is why I force it.
The second is --kv-cache-dtype auto. Flint’s attention layer asserts that the KV dtype is auto or bfloat16, because its fused kernel writes K and V in bf16. If you try fp8 KV you hit that assert immediately. auto resolves to bf16, which is what we want.
The Site-Package Gotchas…
This one cost me a full run. The XPU image has two copies of vLLM in it: a source checkout at /workspace/vllm/vllm/ and the installed copy at /opt/venv/lib/python3.12/site-packages/vllm/. Which one import vllm resolves to depends on the working directory. The vllm serve worker runs with its working directory set to /, so it imports the site-packages copy. My first attempt mounted the patches onto the /workspace/vllm copy, which was a silent no-op. The model loaded, looked fine, and then failed in a way that was hard to trace. Once I pointed all three mounts at the site-packages path, it worked.
There’s also a smaller compose gotcha worth knowing: the command block is a single backslash-continued shell command, and a bare # comment line in the middle of that continuation silently ends the command. Any flags after the comment get dropped. So all the comments in the compose file sit above the exec vllm serve line, not inside it.
It’s ALIVE!!
With the three patches mounted onto site-packages and the two flags set, Flint comes up clean on the Triton backend. The health endpoint returns 200, and real traffic flows. The engine allocates about 904k tokens of KV cache total (roughly 7 GiB per GPU) and reports about 3.4x concurrency at the full 262k context.
Benchmark Outputs
Test Setup
- Models:
- Parameters
- Medians are across 3x reps
- Spread = run-to-run noise
- Benchmark script is in the repo under
/benchmark - Read the file before you run it!!
| Scenario | Concurrency | Metric | Qwen3.8 | Flint | Change | Noise | Verdict |
|---|---|---|---|---|---|---|---|
| Chat | 1 | Total output tok/s | 141.6 | 100.5 | -29.1% | 2.6% | WORSE |
| Chat | 1 | Per-user decode ms/token | 5.0 | 9.7 | +95.3% | 14.4% | WORSE |
| Chat | 1 | Median time to first token ms | 460.8 | 93.9 | -79.6% | 3.6% | BETTER |
| Chat | 1 | p99 time to first token ms | 471.2 | 100.5 | -78.7% | 7.5% | BETTER |
| Chat | 4 | Total output tok/s | 291.8 | 331.8 | +13.7% | 1.7% | BETTER |
| Chat | 4 | Per-user decode ms/token | 9.7 | 11.7 | +21.6% | 1.3% | WORSE |
| Chat | 4 | Median time to first token ms | 531.0 | 160.5 | -69.8% | 0.9% | BETTER |
| Chat | 4 | p99 time to first token ms | 1607.1 | 185.9 | -88.4% | 5.2% | BETTER |
| Chat | 8 | Total output tok/s | 423.1 | 566.9 | +34.0% | 1.0% | BETTER |
| Chat | 8 | Per-user decode ms/token | 15.3 | 13.7 | -10.2% | 1.0% | BETTER |
| Chat | 8 | Median time to first token ms | 578.9 | 169.7 | -70.7% | 2.2% | BETTER |
| Chat | 8 | p99 time to first token ms | 2837.0 | 204.6 | -92.8% | 6.6% | BETTER |
| Chat | 16 | Total output tok/s | 428.8 | 892.6 | +108.2% | 15.3% | BETTER |
| Chat | 16 | Per-user decode ms/token | 32.0 | 17.4 | -45.5% | 1.7% | BETTER |
| Chat | 16 | Median time to first token ms | 1198.9 | 224.4 | -81.3% | 82.1% | BETTER |
| Chat | 16 | p99 time to first token ms | 5492.3 | 401.9 | -92.7% | 591.4% | BETTER |
| Prefill | 1 | Total output tok/s | 104.1 | 77.7 | -25.4% | 19.8% | WORSE |
| Prefill | 1 | Per-user decode ms/token | 5.5 | 12.1 | +121.6% | 1.0% | WORSE |
| Prefill | 1 | Median time to first token ms | 823.4 | 158.3 | -80.8% | 2.9% | BETTER |
| Prefill | 1 | p99 time to first token ms | 851.0 | 176.4 | -79.3% | 7.2% | BETTER |
| Prefill | 4 | Total output tok/s | 162.9 | 244.1 | +49.8% | 0.4% | BETTER |
| Prefill | 4 | Per-user decode ms/token | 16.3 | 15.1 | -7.4% | 1.0% | BETTER |
| Prefill | 4 | Median time to first token ms | 1302.4 | 273.2 | -79.0% | 3.7% | BETTER |
| Prefill | 4 | p99 time to first token ms | 2737.7 | 427.3 | -84.4% | 7.8% | BETTER |
As you can see, Flint is incredibly fast, especially in concurrent workloads. It runs circles around my current production model in the speed department in concurrency.
Honest Feedback
This whole experience was a weekend’s worth of work, but I think it was worth it.
Overall I’m seriously impressed. That’s a full fat MoE model running on 4x arc b70s with like.. minor hacks to get it running. Not even the quantized model. That’s absolutely insane. I’m using a quantized model for Qwen3.8-27B (johannrplaster/Qwen3.8-27B-Uncensored-int4-AutoRound, specifically). and it’s not even coming close to keeping up throughput-wise.
The biggest problem is that it just doesn’t work as well for me yet, especially with tools and Chat.
My daily paperclip end-of-day run report takes about 5 minutes on my normal Qwen3.8 model with 6 tool calls, and took flint about 15 minutes with 87 tool calls to give me effectively the same report. So even though it’s cranking tokens, it doesn’t help if the output isn’t useful.
If they fix the tool calling, the output handlers (my 2 biggest complaints. Open does NOT work as output with this model), and give this another pass, this is gonna be insane. Comparing this to something like deepseek R1 when it first came out, this is an incredibly good first foray into “first model release” territory.
This one may not be usable for me right now, but who knows where it’s going to land in a month or two once folks have had time to actually build around it. I had to pull patches from vllm GitHub to get this damn thing to even boot right, and it’s BARELY running as it is. For a brand spanking new model that has been on the market for 2 weeks and has almost zero support, I’m stoked to see what’s coming next.
Caveats
This is built against a specific XPU nightly, so if you’re following along, check whether a newer image already includes the norm fix (in which case you can drop the norm.py patch) or a dedicated flint reasoning and tool parser (in which case you can swap the glm47 parsers for flint). The whole stack, including the compose file and the three patches and my benchmark tool, is in the cookbook repo. Hopefully this becomes better soon. The results were extremely promising. I’ll try again in a few months when smarter people than me fix a lot of the issues I have.