Skip to content

Need help?

Reach out to us on Discord

Discord

vLLM

This example shows how to deploy Qwen/Qwen3.6-27B using vLLM and dstack.

Apply a configuration

Here's an example of a service that deploys Qwen/Qwen3.6-27B using vLLM.

type: service
name: qwen36

image: vllm/vllm-openai:v0.19.1

commands:
  - |
    vllm serve Qwen/Qwen3.6-27B \
      --host 0.0.0.0 \
      --port 8000 \
      --tensor-parallel-size $DSTACK_GPUS_NUM \
      --max-model-len 262144 \
      --reasoning-parser qwen3

port: 8000
model: Qwen/Qwen3.6-27B

volumes:
  - instance_path: /root/.cache
    path: /root/.cache
    optional: true

resources:
  shm_size: 16GB
  gpu: H100:4

type: service
name: qwen36

image: vllm/vllm-openai-rocm:v0.19.1

commands:
  - |
    vllm serve Qwen/Qwen3.6-27B \
      --host 0.0.0.0 \
      --port 8000 \
      --tensor-parallel-size $DSTACK_GPUS_NUM \
      --max-model-len 262144 \
      --reasoning-parser qwen3

port: 8000
model: Qwen/Qwen3.6-27B

volumes:
  - instance_path: /root/.cache
    path: /root/.cache
    optional: true

resources:
  cpu: 52..
  memory: 896GB..
  shm_size: 16GB
  disk: 450GB..
  gpu: MI300X:4

Qwen3.6-27B is a multimodal model. For text-only workloads, add --language-model-only to free more memory for the KV cache. To enable tool calling, add --enable-auto-tool-choice --tool-call-parser qwen3_coder.

Save one of the configurations above as service.dstack.yml, then use the dstack apply command.

$ dstack apply -f service.dstack.yml

If no gateway is created, the service endpoint will be available at <dstack server URL>/proxy/services/<project name>/<run name>/.

curl http://127.0.0.1:3000/proxy/services/main/qwen36/v1/chat/completions \
    -X POST \
    -H 'Authorization: Bearer &lt;user token&gt;' \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "Qwen/Qwen3.6-27B",
      "messages": [
        {
          "role": "user",
          "content": "A bat and a ball cost $1.10 total. The bat costs $1.00 more than the ball. How much does the ball cost?"
        }
      ],
      "max_tokens": 1024
    }'

If a gateway is configured (e.g. to enable auto-scaling, HTTPS, rate limits, etc.), the service endpoint will be available at https://qwen36.<gateway domain>/.

Configuration options

PD disaggregation

To run vLLM with PD disaggregation, use replica groups: one for Shepherd Model Gateway (SMG), one for prefill workers, and one for decode workers.

type: service
name: prefill-decode

env:
  - HF_TOKEN
  - MODEL_ID=zai-org/GLM-4.5-Air-FP8

groups:
  - replicas: 1
    python: "3.12"
    commands:
      - pip install smg==1.11.0
      - |
        smg launch \
          --pd-disaggregation \
          --model-path $MODEL_ID \
          --enable-igw \
          --host 0.0.0.0 \
          --port 8000 \
          --prefill-policy cache_aware
    router:
      type: smg
    resources:
      cpu: 4

  - replicas: 1..4
    scaling:
      metric: rps
      target: 3
    image: ghcr.io/smg-project/smg:1.11.0-vllm-v0.27.1
    commands:
      - export VLLM_NIXL_SIDE_CHANNEL_HOST=$DSTACK_MASTER_NODE_IP
      - |
        python3 -m vllm.entrypoints.grpc_server \
          --model "$MODEL_ID" \
          --host 0.0.0.0 \
          --port 8000 \
          --max-model-len 65536 \
          --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
    resources:
      gpu: H200
      disk: 200GB..

  - replicas: 1..8
    scaling:
      metric: rps
      target: 2
    image: ghcr.io/smg-project/smg:1.11.0-vllm-v0.27.1
    commands:
      - export VLLM_NIXL_SIDE_CHANNEL_HOST=$DSTACK_MASTER_NODE_IP
      - |
        python3 -m vllm.entrypoints.grpc_server \
          --model "$MODEL_ID" \
          --host 0.0.0.0 \
          --port 8000 \
          --max-model-len 65536 \
          --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
    resources:
      gpu: H200
      disk: 200GB..

port: 8000

Use the same SMG version for the router and the worker images: the worker image bundles the SMG gRPC servicer, which must return the KV transfer parameters that the router relays from prefill to decode.

VLLM_NIXL_SIDE_CHANNEL_HOST sets the address of the NIXL handshake listener, which the prefill worker also passes to the decode worker. It defaults to localhost, in which case the decode worker connects to itself instead of the prefill worker. DSTACK_MASTER_NODE_IP is the internal IP of the replica's instance.

To use the Mooncake Transfer backend, set "kv_connector": "MooncakeConnector" in --kv-transfer-config.

Currently, auto-scaling only supports rps as the metric. TTFT and ITL metrics are coming soon.

Cluster

PD disaggregation requires the service to run in a fleet with placement set to cluster, because the replicas require an interconnect between instances.

While the prefill and decode replicas run on GPUs, the router replica requires a CPU instance in the same cluster.

For production, use instances with RDMA (InfiniBand or RoCE). Without RDMA, NIXL transfers the KV cache over TCP, which works but may be slower than recomputing the prompt on the decode worker.

Instances without RDMA

If an instance has both a private and a public network interface and no RDMA, UCX (the default NIXL transport) may advertise the public address, which may be unreachable from other instances. In this case, restrict UCX to the interface that has the instance's internal IP by adding the following command before starting the worker:

- |
  export UCX_NET_DEVICES=$(python3 -c '
  import os, psutil
  ip = os.environ["DSTACK_MASTER_NODE_IP"]
  ifaces = psutil.net_if_addrs().items()
  print(next(name for name, addrs in ifaces if any(a.address == ip for a in addrs)))
  ')

Don't use this on instances with RDMA, as it prevents UCX from using RDMA devices.

What's next?

  1. Read about services and gateways
  2. Browse the Qwen 3.5 & 3.6 vLLM recipe and the SGLang example