Serve GLM-5.3 Coding Agents Faster and Cheaper on GB200 NVL72

With dstack presets, the preset agent optimized a GB200 NVL72 fleet to serve GLM-5.3 coding agents as fast as the three fastest endpoints on OpenRouter, with lower p50 TTFT. Self-hosting them this way costs under a third of buying the same tokens from those endpoints. By a coding agent we mean one session of a tool like Claude Code: a multi-turn conversation that resends its whole context, about 135k tokens, on every request and spawns subagents along the way.

This work was done jointly with Crusoe, who provided the GB200 NVL72 capacity.

On 14 GB200 nodes, 48 agents get the same p50 output speed per user as the three fastest GLM-5.3 endpoints on OpenRouter, about 3x lower p50 TTFT, and 2x or more higher output speed at p99, where a coding agent's slowest turns are. Serving the same work ourselves costs $12.77 per million output tokens, against $43–66 when buying it from these endpoints.

tok/s/user p50tok/s/user p99TTFT p50TTFT p99Cost

Output speed follows OpenRouter's definition: output tokens per second from sending the request to the end of the response, TTFT included. The self-hosting cost is 56 chips at $4.00 per chip-hour divided by the 17.54 million output tokens the fleet produces per hour. The cost of buying applies each endpoint's list prices, including cache-read prices, to our exact token counts. Provider figures are OpenRouter's for 19–26 October.

Purpose

We want to optimize a GB200 NVL72 fleet for GLM-5.3 coding agents and find out three things: how fast it serves them and what that costs against buying the same tokens from providers, how many agents it holds, and which layout fits an agentic load best.

The last question is open because GB200 NVL72 is a multi-node machine. A fleet can be aggregated or prefill-decode (PD) disaggregated, split into a few wide engines or many narrow ones (TP4, TP8, TP16), parallelized in different ways (expert parallelism, data-parallel attention, decode context parallelism), and keep the cache in GPU memory or extend it into host memory.

Setup

	Aggregated	PD disaggregated
GB200 nodes	14 (56 GPUs)	16 (64 GPUs)
Inference engine	SGLang	SGLang
Router	SGLang Model Gateway	Dynamo

• Model: GLM-5.3, NVFP4 checkpoint (nvidia/GLM-5.3-NVFP4)
• Orchestration: dstack
• Optimization: dstack presets with model: Claude Opus 5

Each GB200 node has 4 GPUs, 744 GB of GPU memory and 881 GB of CPU memory. The router ran on a separate CPU node with 32 vCPUs and 128 GB of RAM, in the same private network. The GB200 NVL72 cluster was provided by Crusoe:

Methodology

Workload

The agents are replayed from SemiAnalysis's AgentX benchmark, built from 393 recorded Claude Code sessions. One agent is one session, and it fans out into subagents, so several of its requests can be in flight at once. A request carries about 135k input tokens, 96% of them a prefix the server has already seen, and returns about 1k tokens.

Each benchmark run lasts 900 seconds. Real sessions pause between requests while the user reads a reply or a tool runs. We capped those pauses at 5 seconds to keep the runs short, which makes the load heavier than real agents produce: 48 agents kept about 39 requests in flight.

Optimization

The preset agent runs every candidate configuration as a benchmarked trial and verifies the winner as a deployable service. Sessions are chained, so later sessions inherit the trial records of earlier ones. Every session ran five trials under the same constraints: a 1M-token context and a p50 TTFT of at most 1 s, raised to 3 s at 128 and 192 agents.

The objective was to optimize the layout: we did not ask the preset agent to patch SGLang or write kernels. The whole study took about two days.

The baseline is the SGLang cookbook recipe for GLM-5.3 NVFP4. The cookbook publishes it for a single GB300 node; on GB200 it ran as seven two-node TP8 engines.

Results

Preset agent converged on TP8 with HiCache

The preset agent was not told which layout to use. Its PD attempts failed to reach a benchmark because of setup problems, so it stayed with aggregated and, over three sessions, converged on seven two-node TP8 engines with HiCache, which keeps context evicted from GPU memory in host memory. Throughput rose from 2,188 to 5,114 tok/s (+134%) and p50 TTFT fell from 383 to 293 ms (−23%), with no source patches.

1. Wider engines with data-parallel attention held more KV cache but paid for it in communication between GPUs: +25% in session 1.
2. Decode context parallelism split each session's cache across eight GPUs for about 5x the capacity, but combining partial attention results on every decode step ate the gain. Seven TP8 engines with 140 GB of host-memory cache per GPU rank kept decode steps fast and won session 2 at 4,412 tok/s. The session's baseline dropped to 1,642 tok/s because the preset agent first removed session 1's odd fourth engine.
3. Single-node TP4 engines decoded faster but processed long prompts slower, losing 28%. Back on TP8, a write-back HiCache policy and SGLang's newer speculative decoding path brought session 3 to 5,114 tok/s at 293 ms.

PD scales throughput, not TTFT

We then asked the preset agent to make PD work and use it on every trial. At 48 agents its best was 3,693 tok/s at 887 ms p50 TTFT on 16 nodes, still behind aggregated's 5,114 tok/s at 293 ms on 14. With data-parallel attention its decode steps took longer, and only four engines generated replies against aggregated's seven.

1. Rebalancing prefill and decode from 1:3 to 1:1 nodes brought p50 TTFT from 1,447 to 762 ms, under the 1 s budget.
2. At 128 agents, the preset agent started from SemiAnalysis's published layout with the engine count doubled to fill 64 GPUs. Splitting the DEP16 decode engines into DEP8 cut the decode step from 35.7 to 24.5 ms and raised throughput 32%, to 8,599 tok/s at 2,035 ms.
3. At 192 agents, giving decode more KV memory reached 10,808 tok/s, 1.8x aggregated's throughput per chip, but p50 TTFT rose to 3,170 ms, more than 10x aggregated's and over the 3 s budget.

At 128 agents, the PD configuration matches SemiAnalysis's InferenceX run on GB200 NVL72 per chip: 134.4 against 136.4 output tok/s, at a similar 2.07 against 2.27 requests in flight per chip. Their run served GLM-5.2, which has the same architecture as GLM-5.3.

Aggregated stops scaling once its cache fills

We then raised the agent count for each winner. Aggregated, replayed at normal pace with pauses capped at 300 s instead of 5 s, rose from 2,150 tok/s at 48 agents to 3,722 at 120, then fell to 2,307 at 240 as cache reuse dropped from 97.2% to 76.6% and p99 TTFT reached 690 s. PD, at the accelerated pace, kept cache reuse around 95% and rose from 3,530 tok/s at 48 agents to 10,545 at 192, while p50 TTFT climbed from 727 ms to 3,250 ms.

Conclusion

Self-hosting is economically viable

Optimized for 48 agents, 14 GB200 nodes match the fastest GLM-5.3 endpoints on p50 output speed and beat them on p50 TTFT and on output speed at p99. Self-hosting costs $12.77 per million output tokens, at least 70% less than the $43–66 it costs to buy the same tokens from these endpoints.

Cache fit decides aggregated vs PD

Aggregated is faster while the sessions' caches fit in memory: there is no KV transfer, all seven engines generate replies, and cache-aware routing sends each session back to the engine that holds its context. PD keeps more context reusable as load grows, so it produces more tokens per chip, but each agent waits longer for its first token.

Duplicated KV caps aggregated

How many agents a fleet holds depends on whether their caches fit and reuse stays high, not on a fixed number; for our aggregated fleet it was 48 at the accelerated pace. Aggregated duplicates the KV cache across the GPUs of each TP8 engine, so once GPU and host memory fill up, evicted context has to be recomputed and throughput falls. PD scaled to 192 agents on throughput, but not on TTFT. The aggregated configuration was optimized for 48 agents, so its higher-load runs test that configuration rather than one tuned for them.

Custom kernels can add further gains

Almost all of the gain came from layout decisions: whether to disaggregate, how many engines, how wide, and whether to extend the cache into host memory. The preset agent found them in about two days, without patching the engine or writing kernels, which remain open for further gains.

Limitations

1. The recorded sessions are replayed with pauses capped at 5 s and generated prompt text, so production traffic will differ in cache reuse, timing and concurrency. For production, give presets your own traffic as the workload.
2. Provider figures are OpenRouter's measurements of its users' requests to each endpoint, not of our workload, and we serve the NVFP4 checkpoint while Baseten's fast endpoints serve FP8.
3. The cost compares self-hosting with buying the same tokens, not with what providers pay to serve them. It counts only GPU rental and assumes the 56 chips produce 17.54 million output tokens in every hour they are rented, at $4.00 per chip-hour, SemiAnalysis's retail price for GB200 NVL72. It leaves out the router node, storage, operations and idle hours.
4. The preset agent spent its sessions on layout rather than engine settings, source patches or kernels, so more gains are likely.
5. The comparison with SemiAnalysis is approximate. Their run served the GLM-5.2 NVFP4 checkpoint, which has the same architecture but a different quantization recipe. The SGLang builds also differ, we used 64 chips against their 32, and our benchmark window was 929 s against their 3,625 s, so we compare only throughput per chip.

Deploy

Both presets are public: aggregated and PD disaggregated. They can be deployed via dstack to any GPU cloud, Kubernetes, or bare-metal cluster with GB200 NVL72 nodes and a CPU node for the router.

To deploy on Crusoe, use dstack's native integration: it can provision VMs through the crusoe backend or use a Kubernetes cluster through the kubernetes backend. Any other nodes can be added as an SSH fleet.

Once the fleet is created, pull a preset, export it as a service configuration, and apply it:

AggregatedPD disaggregated

$ dstack preset pull dstackai/glm53-gb200-nvl72-agentic-c48
$ dstack preset export dstackai/glm53-gb200-nvl72-agentic-c48 -f glm53-c48.dstack.yml
$ dstack apply -f glm53-c48.dstack.yml

$ dstack preset pull dstackai/glm53-gb200-nvl72-agentic-pd-c128
$ dstack preset export dstackai/glm53-gb200-nvl72-agentic-pd-c128 -f glm53-pd-c128.dstack.yml
$ dstack apply -f glm53-pd-c128.dstack.yml

Thanks to our partners

Crusoe

This work was done jointly with Crusoe, who provided the GB200 NVL72 capacity for it. We are sincerely grateful for their compute and support.

If you'd like to rent high-quality GPU clusters, including GB200 NVL72, for training or inference, we recommend going with Crusoe.

Special thanks to Martin Cala and Michael YenChi Ho from the Crusoe team for their help and technical assistance throughout this work.