the CPU is on the critical path for every token you've ever generated.

Not during prefill. Not during heavy compute. Every single token. Decode, one token at a time, the GPU generates it, and then your serving framework signals the CPU, the CPU updates the scheduler state, the CPU decides what happens next, and then the GPU starts the next step.

This round trip happens once per output token. If you are generating a 500-token response, the CPU is in the critical path 500 times. Each interruption is small -- microseconds. The sum is not.

A paper dropped yesterday at 9pm UTC. April 8th. It has received approximately zero coverage. I want to explain what it actually shows because I think the headline number (8.47x P99 TTFT reduction) is not the most interesting result.

The most interesting result is what happens when you run other workloads on the same server.

Today's serving stacks -- vLLM, TensorRT-LLM, SGLang, all of them -- degrade by one to two orders of magnitude under CPU contention. Not degrades slightly. Not increases p99 by 20%. One to two orders of magnitude. If you are running anything else on the same physical host as your inference endpoint, and that workload competes for CPU time, your serving latency collapses.

This is why operators reserve dedicated CPU headroom for inference servers. Not because inference is CPU-bound. It isn't -- inference is GPU-bound, everyone knows this. But the serving stack's control loop is CPU-bound, and if that loop gets starved by competing workloads, the GPU sits idle waiting for the CPU to tell it what to do next.

Dedicated CPU headroom for inference servers means you are paying for CPU capacity you are deliberately not using, to protect the serving stack from the CPU interference that would otherwise destroy your latency SLOs.

That is the invisible tax every inference operator is currently paying.

The paper is called Blink. It is from a team that looked at this problem and made a decision that sounds obvious in retrospect and was apparently difficult enough that nobody has shipped it at this level before: remove the CPU from the serving path entirely.

Two architectural changes.

First: move request handling to the SmartNIC. The BlueField-3 DPU receives incoming requests from clients, tokenizes them on the DPU's ARM cores, and writes the tokenized input directly into GPU memory via RDMA. The host CPU never sees the request. It never touches the data. It is not involved.

Second: replace the host-driven scheduler with a persistent GPU kernel. Instead of the GPU finishing a step, signaling the CPU, waiting for the CPU to update scheduler state and decide what to do, then getting a new instruction -- the GPU never stops. A persistent kernel runs on a subset of SMs, continuously polling for new completed tokens, making batching decisions, managing KV cache, scheduling the next decode step -- all inside GPU memory, without ever leaving the GPU and touching the CPU.

The CPU is not involved in steady-state inference operation at all. It boots the system, loads the model weights, sets up the infrastructure. After that, it is not on the critical path.

The numbers from evaluation against TensorRT-LLM, vLLM, and SGLang on four models:

In isolation -- no competing workloads, dedicated hardware -- Blink reduces P99 TTFT by up to 8.47x. P99 time-per-output-token by up to 3.40x. This is already a significant result. In isolation. Before you account for colocation.

Under CPU interference -- running competing workloads on the same server -- the existing systems degrade by one to two orders of magnitude. Blink's latency and throughput remain stable, within experimental variance of the isolated values.

Throughput under CPU contention: 6.46x higher requests per second than vLLM/TensorRT-LLM/SGLang baselines.

Energy per token: 48.6% lower in isolation, 70.7% lower under CPU interference.

The 70.7% energy reduction under interference is not because Blink does less work. It is because the baselines are burning power on GPU-idle cycles while the CPU catches up. Blink's GPU never idles waiting for the CPU.

The reason this matters structurally for anyone running inference at scale:

Serving infrastructure is expensive. The standard practice is to isolate inference servers -- give them dedicated machines, reserve CPU cores for the serving stack, prevent colocation with other workloads. This is the correct engineering response to the CPU interference problem. It is also enormously wasteful: you are running underutilized CPU capacity and paying for it to sit idle, and you cannot put those machines to work on anything else without destroying your inference SLOs.

The ability to colocate inference with other workloads -- batch jobs, preprocessing pipelines, auxiliary services -- on the same physical hardware changes the utilization math significantly. If inference is truly CPU-interference-immune, you can run mixed workloads on inference servers without protecting CPU headroom. The reserved capacity becomes available for other work.

"But the SmartNIC adds latency to the ingestion pa--" The BlueField-3 DPU delivers inputs to GPU memory via 200 Gbps RDMA. The per-request ingestion overhead is lower than the CPU-based path it replaces, because RDMA bypasses the CPU memory subsystem entirely. The DPU's ARM cores for tokenization are slower per-core than a server CPU, but tokenization is parallelizable and the workload fits comfortably.

"But the persistent GPU kernel is using SMs that could be running inference--" The persistent kernel runs on a small fixed allocation of SMs, not the full compute budget. The scheduling overhead it replaces (CPU round-trip per token) is cheaper in SM-cycles than it was in CPU-wait-time on the GPU side.

There is a framing error in how we talk about inference servers.

We say: inference is GPU-bound. The GPU is the bottleneck. More GPUs means more capacity.

This is true in aggregate. It is not true at the per-token level. At the per-token level, the serving stack is CPU-mediated. Every token passes through a CPU-based control loop before the next token can start. The GPU is not doing anything during that loop. The GPU is waiting.

We have optimized everything around the GPU being the bottleneck while the CPU was on the critical path the entire time. Not visibly enough to measure easily. Visibly enough that under any CPU contention, the whole system collapses.

Blink makes the CPU not the bottleneck because Blink removes the CPU. It is not a partial fix. It is an architectural decision that the CPU has no business being involved in token-level control and the serving stack should be redesigned around that premise.

The paper posted yesterday. Integration into vLLM and SGLang presumably comes next. That is how these things go -- research result, framework integration, production deployment, six to eighteen months.

the cpu is on the critical path for every token.

it has been this whole time.

you did not see it until it was gone.

8.47x p99 ttft. not on some synthetic benchmark. against tensorrt-llm, vllm, and sglang. on real models. in isolation. before you account for colocation.

P.S. The paper runs the inference backend on the GPU server and the frontend (request handling, tokenization, RDMA delivery) on a separate BlueField-3 DPU machine connected via 200 Gbps RDMA. The testbed is real hardware. The numbers are reproducible. The code will follow. Watch for it.