# docs/index.md --- title: What is dstack? description: Introduction to dstack and how it works --- # What is dstack? `dstack` is a unified control plane for GPU provisioning and orchestration that works with any GPU cloud, Kubernetes, or on-prem clusters. It streamlines development, training, and inference, and is compatible with any hardware, open-source tools, and frameworks. !!! info "Accelerators" `dstack` supports `NVIDIA`, `AMD`, `TPU`, and `Tenstorrent` accelerators out of the box. ## How does it work? ### Set up the server > Before using `dstack`, ensure you've [installed](installation.md) the server, or signed up for [dstack Sky](https://sky.dstack.ai). ### Define configurations `dstack` supports the following configurations: * [Fleets](concepts/fleets.md) — for managing cloud and on-prem clusters * [Dev environments](concepts/dev-environments.md) — for interactive development using a desktop IDE * [Tasks](concepts/tasks.md) — for scheduling jobs, incl. distributed ones (or running web apps) * [Services](concepts/services.md) — for deploying models (or web apps) * [Volumes](concepts/volumes.md) — for managing network volumes (to persist data) Configuration can be defined as YAML files within your repo. ### Apply configurations Apply the configuration either via the `dstack apply` CLI command (or through a programmatic API.) `dstack` automatically manages infrastructure provisioning and job scheduling, while also handling auto-scaling, port-forwarding, ingress, and more. !!! info "Where do I start?" 1. Proceed to [installation](installation.md) 2. See [quickstart](quickstart.md) 3. Browse [examples](/examples) 4. Join [Discord](https://discord.gg/u8SmfwPpMd) # docs/installation.md --- title: Installation description: How to install the dstack server and CLI --- # Installation ## Launch the server { #server } The server can run on your laptop or any environment with access to the cloud and on-prem clusters you plan to use. === "uv" > The server can be set up via `uv` on Linux, macOS, and Windows (via WSL 2). > It requires Git and OpenSSH.
```shell $ uv tool install "dstack[all]" -U $ dstack server Applying ~/.dstack/server/config.yml... The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ ```
=== "pip" > The server can be set up via `pip` on Linux, macOS, and Windows (via WSL 2). > It requires Git and OpenSSH.
```shell $ pip install "dstack[all]" -U $ dstack server Applying ~/.dstack/server/config.yml... The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ ```
=== "Docker"
```shell $ docker run -p 3000:3000 \ -v $HOME/.dstack/server/:/root/.dstack/server \ dstackai/dstack Applying ~/.dstack/server/config.yml... The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ ```
For more details on server deployment options, see the [Server deployment](guides/server-deployment.md) guide. !!! info "Configure backends" To orchestrate compute across GPU clouds or Kubernetes clusters, you need to configure [backends](concepts/backends.md). ## Install the CLI { #cli } Once the server is up, you can access it via the `dstack` CLI. > The CLI can be used on Linux, macOS, and Windows. It requires Git and OpenSSH. === "uv"
```shell $ uv tool install dstack -U ```
=== "pip"
```shell $ pip install dstack -U ```
??? info "Windows" To use the CLI on Windows, ensure you've installed Git and OpenSSH via [Git for Windows](https://git-scm.com/download/win). When installing it, ensure you've checked `Git from the command line and also from 3-rd party software` (or `Use Git and optional Unix tools from the Command Prompt`), and `Use bundled OpenSSH`. ??? info "Shell autocompletion" `dstack` supports shell autocompletion for `bash` and `zsh`. === "bash" First, validate if completion scripts load correctly in your current shell session:
```shell $ eval "$(dstack completion bash)" ```
If completions work as expected and you would like them to persist across shell sessions, add the completion script to your shell profile using these commands:
```shell $ mkdir -p ~/.dstack $ dstack completion bash > ~/.dstack/completion.sh $ echo 'source ~/.dstack/completion.sh' >> ~/.bashrc ```
=== "zsh" First, validate if completion scripts load correctly in your current shell session:
```shell $ eval "$(dstack completion zsh)" ```
If completions work as expected and you would like them to persist across shell sessions, you can install them via Oh My Zsh using these commands:
```shell $ mkdir -p ~/.oh-my-zsh/completions $ dstack completion zsh > ~/.oh-my-zsh/completions/_dstack ```
And if you don't use Oh My Zsh:
```shell $ mkdir -p ~/.dstack $ dstack completion zsh > ~/.dstack/completion.sh $ echo 'source ~/.dstack/completion.sh' >> ~/.zshrc ```
> If you get an error similar to `2: command not found: compdef`, then add the following line to the beginning of your `~/.zshrc` file: > `autoload -Uz compinit && compinit`. ### Configure the project When the server is started, by default, it creates the `main` project and the `admin` user. To point the CLI to the `dstack` server, configure it with the server address, user token, and project name:
```shell $ dstack project add \ --name main \ --url http://127.0.0.1:3000 \ --token bbae0f28-d3dd-4820-bf61-8f4bb40815da Configuration is updated at ~/.dstack/config.yml ```
This configuration is stored in `~/.dstack/config.yml`. Later, you can create additional projects and users. ### Use CLI or API Once the project is configured, you can use the `dstack` CLI or API. > See the [CLI & API](guides/cli-api.md) guide for using the CLI and HTTP API. ## Install agent skills { #skills } If you'd like to use `dstack` with AI agents like Claude, Codex, and Cursor, install [`dstack` skills](https://skills.sh/dstackai/dstack/dstack) to help them use the CLI and edit configuration files.
```shell $ npx skills add dstackai/dstack ```
We're actively improving Skills and would love your feedback in [GitHub issues](https://github.com/dstackai/dstack/issues). !!! info "What's next?" 1. See [Backends](concepts/backends.md) 2. Follow [Quickstart](quickstart.md) 3. Check the [Server deployment](guides/server-deployment.md) guide # docs/quickstart.md --- title: Quickstart description: Quick guide to creating fleets and submitting runs --- # Quickstart !!! info "Prerequisites" Ensure the server and CLI are [installed](installation.md). To use `dstack` with AI agents, install [skills](installation.md#skills). ## Create a fleet > Before submitting runs, you must create a [fleet](concepts/fleets.md). === "Backend fleet" If you're using cloud providers or Kubernetes clusters and have configured the corresponding [backends](concepts/backends.md), create a fleet as follows:
```yaml type: fleet name: default # Allow to provision of up to 2 instances nodes: 0..2 # Deprovision instances above the minimum if they remain idle idle_duration: 1h resources: # Allow to provision up to 8 GPUs gpu: 0..8 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f fleet.dstack.yml # BACKEND REGION RESOURCES SPOT PRICE 1 gcp us-west4 2xCPU, 8GB, 100GB (disk) yes $0.010052 2 azure westeurope 2xCPU, 8GB, 100GB (disk) yes $0.0132 3 gcp europe-central2 2xCPU, 8GB, 100GB (disk) yes $0.013248 Create the fleet? [y/n]: y FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED defalut - - - - - 10:36 ```
If the `nodes` range starts with `0`, `dstack apply` creates only a template. Instances are provisioned only when you submit runs. If the fleet needs to be a cluster, the [placement](concepts/fleets.md#cluster-placement) property must be set to `cluster`. === "SSH fleet" If you have a group of on-prem servers accessible via SSH, you can create an SSH fleet as follows:
```yaml type: fleet name: my-fleet ssh_config: user: ubuntu identity_file: ~/.ssh/id_rsa hosts: - 3.255.177.51 - 3.255.177.52 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f fleet.dstack.yml Provisioning... ---> 100% FLEET INSTANCE GPU PRICE STATUS CREATED my-fleet 0 L4:24GB (spot) $0 idle 3 mins ago 1 L4:24GB (spot) $0 idle 3 mins ago ```
> Hosts must have Docker and GPU drivers installed and meet the other [requirements](concepts/fleets.md#ssh-fleets). If the fleet needs to be a cluster, the [placement](concepts/fleets.md#ssh-placement) property must be set to `cluster`. ## Submit your first run `dstack` supports three types of run configurations. === "Dev environment" A [dev environment](concepts/dev-environments.md) lets you provision an instance and access it with your desktop IDE. Create the following run configuration:
```yaml type: dev-environment name: vscode # If `image` is not specified, dstack uses its default image python: "3.11" #image: dstackai/base:py3.13-0.7-cuda-12.1 ide: vscode # Uncomment to request resources #resources: # gpu: 24GB ```
Apply the configuration via `dstack apply`:
```shell $ dstack apply -f .dstack.yml # BACKEND REGION RESOURCES SPOT PRICE 1 gcp us-west4 2xCPU, 8GB, 100GB (disk) yes $0.010052 2 azure westeurope 2xCPU, 8GB, 100GB (disk) yes $0.0132 3 gcp europe-central2 2xCPU, 8GB, 100GB (disk) yes $0.013248 Submit the run vscode? [y/n]: y Launching `vscode`... ---> 100% To open in VS Code Desktop, use this link: vscode://vscode-remote/ssh-remote+vscode/workflow ```
Open the link to access the dev environment using your desktop IDE. Alternatively, you can access it via `ssh `. === "Task" A [task](concepts/tasks.md) allows you to schedule a job or run a web app. Tasks can be distributed and can forward ports. Create the following run configuration:
```yaml type: task name: streamlit # If `image` is not specified, dstack uses its default image python: "3.11" #image: dstackai/base:py3.13-0.7-cuda-12.1 # Commands of the task commands: - pip install streamlit - streamlit hello # Ports to forward ports: - 8501 # Uncomment to request resources #resources: # gpu: 24GB ```
By default, tasks run on a single instance. To run a distributed task, specify [`nodes`](concepts/tasks.md#distributed-tasks), and `dstack` will run it on a cluster. Run the configuration via `dstack apply`:
```shell $ dstack apply -f task.dstack.yml # BACKEND REGION RESOURCES SPOT PRICE 1 gcp us-west4 2xCPU, 8GB, 100GB (disk) yes $0.010052 2 azure westeurope 2xCPU, 8GB, 100GB (disk) yes $0.0132 3 gcp europe-central2 2xCPU, 8GB, 100GB (disk) yes $0.013248 Submit the run streamlit? [y/n]: y Provisioning `streamlit`... ---> 100% Welcome to Streamlit. Check out our demo in your browser. Local URL: http://localhost:8501 ```
If you specified `ports`, they will be automatically forwarded to `localhost` for convenient access. === "Service" A [service](concepts/services.md) allows you to deploy a model or any web app as an endpoint. Create the following run configuration:
```yaml type: service name: qwen36-service image: lmsysorg/sglang:v0.5.10.post1 commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --reasoning-parser qwen3 # Expose the SGLang server port port: 30000 # Specify a name if it's an OpenAI-compatible model model: Qwen/Qwen3.6-27B # Required resources resources: gpu: H100 ```
Run the configuration via `dstack apply`:
```shell $ dstack apply -f service.dstack.yml # BACKEND REGION INSTANCE RESOURCES SPOT PRICE 1 nebius eu-north1 gpu-h100-sxm 16xCPU, 250GB, 1xH100 (80GB) no $2.95 2 runpod US-CA-2 NVIDIA H100 80GB HBM3 64xCPU, 1004GB, 1xH100 (80GB) no $2.99 Submit the run qwen36-service? [y/n]: y Provisioning `qwen36-service`... ---> 100% Service is published at: http://localhost:3000/proxy/services/main/qwen36-service/ Model Qwen/Qwen3.6-27B is published at: http://localhost:3000/proxy/models/main/ ```
> To enable auto-scaling rate limits, or use a custom domain with HTTPS, set up a [gateway](concepts/gateways.md) before running the service. `dstack apply` automatically provisions instances with created fleets and runs the workload according to the configuration. ## Troubleshooting Something not working? See the [troubleshooting](guides/troubleshooting.md) guide. !!! info "What's next?" 1. Read about [backends](concepts/backends.md), [dev environments](concepts/dev-environments.md), [tasks](concepts/tasks.md), [services](concepts/services.md), and [fleets](concepts/services.md) 2. Browse [examples](examples.md) 3. Join [Discord](https://discord.gg/u8SmfwPpMd) # docs/concepts/backends.md --- title: Backends description: Configuring cloud providers and Kubernetes clusters --- # Backends Backends allow `dstack` to provision fleets across GPU clouds or Kubernetes clusters. `dstack` supports two types of backends: * [VM-based](#vm-based) – use `dstack`'s native integration with cloud providers to provision VMs, manage clusters, and orchestrate container-based runs. * [Container-based](#container-based) – use either `dstack`'s native integration with cloud providers or Kubernetes to orchestrate container-based runs; provisioning in this case is delegated to the cloud provider or Kubernetes. !!! info "SSH fleets" When using `dstack` with on-prem servers, backend configuration isn’t required. Simply create [SSH fleets](../concepts/fleets.md#ssh-fleets) once the server is up. Backends can be configured via `~/.dstack/server/config.yml` or through the [project settings page](../concepts/projects.md#backends) in the UI. See the examples of backend configuration below. > If you update `~/.dstack/server/config.yml`, you have to restart the server. ## VM-based VM-based backends allow `dstack` users to manage clusters and orchestrate container-based runs across a wide range of cloud providers. Under the hood, `dstack` uses native integrations with these providers to provision clusters on demand. Compared to [container-based](#container-based) backends, this approach offers finer-grained, simpler control over cluster provisioning and eliminates the dependency on a Kubernetes layer. ### AWS There are two ways to configure AWS: using an access key or using the default credentials. === "Default credentials" If you have default credentials set up (e.g. in `~/.aws/credentials`), configure the backend like this:
```yaml projects: - name: main backends: - type: aws creds: type: default ```
=== "Access key" Create an access key by following the [this guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-authentication-user.html#cli-authentication-user-get). Once you've downloaded the `.csv` file with your IAM user's Access key ID and Secret access key, proceed to configure the backend.
```yaml projects: - name: main backends: - type: aws creds: type: access_key access_key: KKAAUKLIZ5EHKICAOASV secret_key: pn158lMqSBJiySwpQ9ubwmI6VUU3/W2fdJdFwfgO ```
??? info "Required permissions" The following AWS policy permissions are sufficient for `dstack` to work: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:AttachVolume", "ec2:AuthorizeSecurityGroupEgress", "ec2:AuthorizeSecurityGroupIngress", "ec2:CreatePlacementGroup", "ec2:CancelSpotInstanceRequests", "ec2:CreateSecurityGroup", "ec2:CreateTags", "ec2:CreateVolume", "ec2:DeletePlacementGroup", "ec2:DeleteVolume", "ec2:DescribeAvailabilityZones", "ec2:DescribeCapacityReservations" "ec2:DescribeImages", "ec2:DescribeInstances", "ec2:DescribeInstanceAttribute", "ec2:DescribeInstanceTypes", "ec2:DescribeRouteTables", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs", "ec2:DescribeVolumes", "ec2:DetachVolume", "ec2:RunInstances", "ec2:TerminateInstances" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "servicequotas:ListServiceQuotas", "servicequotas:GetServiceQuota" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "elasticloadbalancing:CreateLoadBalancer", "elasticloadbalancing:CreateTargetGroup", "elasticloadbalancing:CreateListener", "elasticloadbalancing:RegisterTargets", "elasticloadbalancing:AddTags", "elasticloadbalancing:DeleteLoadBalancer", "elasticloadbalancing:DeleteTargetGroup", "elasticloadbalancing:DeleteListener", "elasticloadbalancing:DeregisterTargets" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "acm:DescribeCertificate", "acm:ListCertificates" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "iam:GetInstanceProfile", "iam:GetRole", "iam:PassRole" ], "Resource": "*" } ] } ``` The `elasticloadbalancing:*` and `acm:*` permissions are only needed for provisioning gateways with ACM (AWS Certificate Manager) certificates. The `iam:*` permissions are only needed if you specify `iam_instance_profile` to assign to EC2 instances. The following additional permissions are required when running [multi-EFA instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#network-cards) with `public_ips: true`: ``` { "Effect": "Allow", "Action": [ "ec2:AllocateAddress", "ec2:AssociateAddress", "ec2:DescribeAddresses", "ec2:DisassociateAddress", "ec2:ReleaseAddress" ], "Resource": "*" } ``` You can also limit permissions to specific resources in your account: ``` { "Version": "2012-10-17", "Statement": [ ... { "Effect": "Allow", "Action": [ "iam:GetInstanceProfile", "iam:GetRole", "iam:PassRole" ], "Resource": "arn:aws:iam::account-id:role/EC2-roles-for-XYZ-*" } ] } ``` ??? info "VPC" By default, `dstack` uses the default VPC. It's possible to customize it: === "vpc_name" ```yaml projects: - name: main backends: - type: aws creds: type: default vpc_name: my-vpc ``` === "vpc_ids" ```yaml projects: - name: main backends: - type: aws creds: type: default default_vpcs: true vpc_ids: us-east-1: vpc-0a2b3c4d5e6f7g8h us-east-2: vpc-9i8h7g6f5e4d3c2b us-west-1: vpc-4d3c2b1a0f9e8d7 ``` For the regions without configured `vpc_ids`, enable default VPCs by setting `default_vpcs` to `true`. ??? info "Private subnets" By default, `dstack` provisions instances with public IPs and permits inbound SSH traffic. If you want `dstack` to use private subnets and provision instances without public IPs, set `public_ips` to `false`. ```yaml projects: - name: main backends: - type: aws creds: type: default public_ips: false ``` Using private subnets assumes that both the `dstack` server and users can access the configured VPC's private subnets. Additionally, private subnets must have outbound internet connectivity provided by NAT Gateway, Transit Gateway, or other mechanism. ??? info "OS images" By default, `dstack` uses its own [AMI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) optimized for `dstack`. To use your own or other third-party images, set the `os_images` property: ```yaml projects: - name: main backends: - type: aws creds: type: default os_images: cpu: name: my-ami-for-cpu-instances owner: self user: dstack nvidia: name: 'Some ThirdParty CUDA image' owner: 123456789012 user: ubuntu ``` Here, both `cpu` and `nvidia` properties are optional, but if the property is not set, you won´t be able to use the corresponding instance types. The `name` is an AMI name. The `owner` is either an AWS account ID (a 12-digit number) or a special value `self` indicating the current account. The `user` specifies an OS user for instance provisioning. !!! info "Image requirements" * SSH server listening on port 22 * `user` with passwordless sudo access * Docker is installed * (For NVIDIA instances) NVIDIA/CUDA drivers and NVIDIA Container Toolkit are installed * The firewall (`iptables`, `ufw`, etc.) must allow external traffic to port 22 and all traffic within the private subnet, and should forbid any other incoming external traffic. ### Azure There are two ways to configure Azure: using a client secret or using the default credentials. === "Default credentials" If you have default credentials set up, configure the backend like this:
```yaml projects: - name: main backends: - type: azure subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808 tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1 creds: type: default ```
If you don't know your `subscription_id` and `tenant_id`, use [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli): ```shell az account show --query "{subscription_id: id, tenant_id: tenantId}" ``` === "Client secret" A client secret can be created using the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli): ```shell SUBSCRIPTION_ID=... az ad sp create-for-rbac --name dstack-app \ --role $DSTACK_ROLE \ --scopes /subscriptions/$SUBSCRIPTION_ID \ --query "{ tenant_id: tenant, client_id: appId, client_secret: password }" ``` Once you have `tenant_id`, `client_id`, and `client_secret`, go ahead and configure the backend.
```yaml projects: - name: main backends: - type: azure subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808 tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1 creds: type: client client_id: acf3f73a-597b-46b6-98d9-748d75018ed0 client_secret: 1Kb8Q~o3Q2hdEvrul9yaj5DJDFkuL3RG7lger2VQ ```
If you don't know your `subscription_id`, use [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli): ```shell az account show --query "{subscription_id: id}" ``` ??? info "Required permissions" The following Azure permissions are sufficient for `dstack` to work: ```json { "properties": { "roleName": "dstack-role", "description": "Minimal required permissions for using Azure with dstack", "assignableScopes": [ "/subscriptions/${YOUR_SUBSCRIPTION_ID}" ], "permissions": [ { "actions": [ "Microsoft.Authorization/*/read", "Microsoft.Compute/availabilitySets/*", "Microsoft.Compute/locations/*", "Microsoft.Compute/virtualMachines/*", "Microsoft.Compute/virtualMachineScaleSets/*", "Microsoft.Compute/cloudServices/*", "Microsoft.Compute/disks/write", "Microsoft.Compute/disks/read", "Microsoft.Compute/disks/delete", "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action", "Microsoft.ManagedIdentity/userAssignedIdentities/read", "Microsoft.Network/networkSecurityGroups/*", "Microsoft.Network/locations/*", "Microsoft.Network/virtualNetworks/*", "Microsoft.Network/networkInterfaces/*", "Microsoft.Network/publicIPAddresses/*", "Microsoft.Resources/subscriptions/resourceGroups/read", "Microsoft.Resources/subscriptions/resourceGroups/write", "Microsoft.Resources/subscriptions/read" ], "notActions": [], "dataActions": [], "notDataActions": [] } ] } } ``` The `"Microsoft.Resources/subscriptions/resourceGroups/write"` permission is not required if [`resource_group`](/docs/reference/server/config.yml/#azure) is specified. ??? info "VPC" By default, `dstack` creates new Azure networks and subnets for every configured region. It's possible to use custom networks by specifying `vpc_ids`: ```yaml projects: - name: main backends: - type: azure creds: type: default regions: [westeurope] vpc_ids: westeurope: myNetworkResourceGroup/myNetworkName ``` ??? info "Private subnets" By default, `dstack` provisions instances with public IPs and permits inbound SSH traffic. If you want `dstack` to use private subnets and provision instances without public IPs, specify custom networks using `vpc_ids` and set `public_ips` to `false`. ```yaml projects: - name: main backends: - type: azure creds: type: default regions: [westeurope] vpc_ids: westeurope: myNetworkResourceGroup/myNetworkName public_ips: false ``` Using private subnets assumes that both the `dstack` server and users can access the configured VPC's private subnets. Additionally, private subnets must have outbound internet connectivity provided by [NAT Gateway or other mechanism](https://learn.microsoft.com/en-us/azure/nat-gateway/nat-overview). ### GCP There are two ways to configure GCP: using a service account or using the default credentials. === "Default credentials" Enable GCP application default credentials: ```shell gcloud auth application-default login ``` Then configure the backend like this:
```yaml projects: - name: main backends: - type: gcp project_id: gcp-project-id creds: type: default ```
=== "Service account" To create a service account, follow [this guide](https://cloud.google.com/iam/docs/service-accounts-create). After setting up the service account [create a key](https://cloud.google.com/iam/docs/keys-create-delete) for it and download the corresponding JSON file. Then go ahead and configure the backend by specifying the downloaded file path.
```yaml projects: - name: main backends: - type: gcp project_id: my-gcp-project creds: type: service_account filename: ~/.dstack/server/gcp-024ed630eab5.json ```
??? info "User interface" If you are configuring the `gcp` backend on the [project settings page](projects.md#backends), specify the contents of the JSON file in `data`:
```yaml type: gcp project_id: my-gcp-project creds: type: service_account data: | { "type": "service_account", "project_id": "my-gcp-project", "private_key_id": "abcd1234efgh5678ijkl9012mnop3456qrst7890", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEv...rest_of_key...IDAQAB\n-----END PRIVATE KEY-----\n", "client_email": "my-service-account@my-gcp-project.iam.gserviceaccount.com", "client_id": "123456789012345678901", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-service-account%40my-gcp-project.iam.gserviceaccount.com", "universe_domain": "googleapis.com" } ```
If you don't know your GCP project ID, use [Google Cloud CLI](https://cloud.google.com/sdk/docs/install-sdk): ```shell gcloud projects list --format="json(projectId)" ``` ??? info "Required permissions" The following GCP permissions are sufficient for `dstack` to work: ``` compute.disks.create compute.disks.delete compute.disks.get compute.disks.list compute.disks.setLabels compute.disks.use compute.firewalls.create compute.images.useReadOnly compute.instances.attachDisk compute.instances.create compute.instances.delete compute.instances.detachDisk compute.instances.get compute.instances.setLabels compute.instances.setMetadata compute.instances.setServiceAccount compute.instances.setTags compute.networks.get compute.networks.updatePolicy compute.projects.get compute.regions.get compute.regions.list compute.reservations.list compute.resourcePolicies.create compute.resourcePolicies.delete compute.routers.list compute.subnetworks.list compute.subnetworks.use compute.subnetworks.useExternalIp compute.zoneOperations.get ``` If you plan to use TPUs, additional permissions are required: ``` tpu.nodes.create tpu.nodes.get tpu.nodes.update tpu.nodes.delete tpu.operations.get tpu.operations.list ``` Also, the use of TPUs requires the `serviceAccountUser` role. For TPU VMs, dstack will use the default service account. If you plan to use shared reservations, the `compute.reservations.list` permission is required in the project that owns the reservations. ??? info "Required APIs" First, ensure the required APIs are enabled in your GCP `project_id`. ```shell PROJECT_ID=... gcloud config set project $PROJECT_ID gcloud services enable cloudapis.googleapis.com gcloud services enable compute.googleapis.com ``` ??? info "VPC" === "VPC"
```yaml projects: - name: main backends: - type: gcp project_id: gcp-project-id creds: type: default vpc_name: my-custom-vpc ```
If you specify a non-default VPC, ensure it has a firewall rule allowing all traffic within the VPC. This is needed for multi-node tasks to work. The default VPC already permits traffic within the VPC. === "Shared VPC"
```yaml projects: - name: main backends: - type: gcp project_id: gcp-project-id creds: type: default vpc_name: my-custom-vpc vpc_project_id: another-project-id ```
When using a Shared VPC, ensure there is a firewall rule allowing `INGRESS` traffic on port `22`. You can limit this rule to `dstack` instances using the `dstack-runner-instance` target tag. When using GCP gateways with a Shared VPC, also ensure there is a firewall rule allowing `INGRESS` traffic on ports `22`, `80`, `443`. You can limit this rule to `dstack` gateway instances using the `dstack-gateway-instance` target tag. To use TPUs with a Shared VPC, you need to grant the TPU Service Account in your service project permissions to manage resources in the host project by granting the "TPU Shared VPC Agent" (roles/tpu.xpnAgent) role ([more in the GCP docs](https://cloud.google.com/tpu/docs/shared-vpc-networks#vpc-shared-vpc)). ??? info "Private subnets" By default, `dstack` provisions instances with public IPs and permits inbound SSH traffic. If you want `dstack` to use private subnets and provision instances without public IPs, set `public_ips` to `false`. ```yaml projects: - name: main backends: - type: gcp creds: type: default public_ips: false ``` Using private subnets assumes that both the `dstack` server and users can access the configured VPC's private subnets. Additionally, [Cloud NAT](https://cloud.google.com/nat/docs/overview) must be configured to provide access to external resources for provisioned instances. ### Lambda Log into your [Lambda Cloud](https://lambdalabs.com/service/gpu-cloud) account, click API keys in the sidebar, and then click the `Generate API key` button to create a new API key. Then, go ahead and configure the backend:
```yaml projects: - name: main backends: - type: lambda creds: type: api_key api_key: eersct_yrpiey-naaeedst-tk-_cb6ba38e1128464aea9bcc619e4ba2a5.iijPMi07obgt6TZ87v5qAEj61RVxhd0p ```
### Nebius Log into your [Nebius AI Cloud](https://console.eu.nebius.com/) account, navigate to Access, and select Service Accounts. Create a service account, add it to the editors group, and upload its authorized key. Then configure the backend:
```yaml projects: - name: main backends: - type: nebius creds: type: service_account service_account_id: serviceaccount-e00dhnv9ftgb3cqmej public_key_id: publickey-e00ngaex668htswqy4 private_key_file: ~/path/to/key.pem ```
??? info "Credentials file" It's also possible to configure the `nebius` backend using a credentials file [generated](https://docs.nebius.com/iam/service-accounts/authorized-keys#create) by the `nebius` CLI:
```shell $ nebius iam auth-public-key generate \ --service-account-id \ --output ~/.nebius/sa-credentials.json ```
```yaml projects: - name: main backends: - type: nebius creds: type: service_account filename: ~/.nebius/sa-credentials.json ``` ??? info "User interface" If you are configuring the `nebius` backend on the [project settings page](projects.md#backends), specify the contents of the private key file in `private_key_content`:
```yaml type: nebius creds: type: service_account service_account_id: serviceaccount-e00dhnv9ftgb3cqmej public_key_id: publickey-e00ngaex668htswqy4 private_key_content: | -----BEGIN PRIVATE KEY----- MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQChwQ5OOhy60N7m cPx/9M0oRUyJdRRv2nCALbdU/wSDOo8o5N7sP63zCaxXPeKwLNEzneMd/U0gWSv2 [...] 8y1qYDPKQ8LR+DPCUmyhM2I8t6673Vz3GrtEjkLhgQo/KqOVb3yiBFVfkA5Jov5s kO7y4T0ynsI8b6wlhCukQTLpIYJ5 -----END PRIVATE KEY----- ```
??? info "Projects" If you have multiple projects per region, specify which ones to use, at most one per region.
```yaml type: nebius projects: - project-e00jt6t095t1ahrg4re30 - project-e01iahuh3cklave4ao1nv creds: type: service_account service_account_id: serviceaccount-e00dhnv9ftgb3cqmej public_key_id: publickey-e00ngaex668htswqy4 private_key_file: ~/path/to/key.pem ```
### Crusoe Log into your [Crusoe](https://console.crusoecloud.com/) console and create an API key under your account settings. Note your project ID from the project settings page. Then, go ahead and configure the backend:
```yaml projects: - name: main backends: - type: crusoe project_id: your-project-id creds: type: access_key access_key: your-access-key secret_key: your-secret-key regions: - us-east1-a - us-southcentral1-a ```
`regions` is optional. If not specified, all available Crusoe regions are used. ### Verda (formerly DataCrunch) { #verda } Log into your [Verda](https://console.verda.com/signin) account, click Keys in the sidebar, find `REST API Credentials` area and then click the `Generate Credentials` button. Then, go ahead and configure the backend:
```yaml projects: - name: main backends: - type: verda creds: type: api_key client_id: xfaHBqYEsArqhKWX-e52x3HH7w8T client_secret: B5ZU5Qx9Nt8oGMlmMhNI3iglK8bjMhagTbylZy4WzncZe39995f7Vxh8 ```
### AMD Developer Cloud Log into your [AMD Developer Cloud](https://amd.digitalocean.com/login) account. Click `API` in the sidebar and click the button `Generate New Token`. Then, go ahead and configure the backend:
```yaml projects: - name: main backends: - type: amddevcloud project_name: my-amd-project creds: type: api_key api_key: ... ```
??? info "Project" If `project_name` is not set, the default project will be used. ??? info "Required permissions" The API key must have the following scopes assigned: * `account` - read * `droplet` - create, read, update, delete, admin * `project` - create, read, update, delete * `regions` - read * `sizes` - read * `ssh_key` - create, read, update, delete ### Digital Ocean Log into your [Digital Ocean](https://cloud.digitalocean.com/login) account. Click `API` in the sidebar and click the button `Generate New Token`. Then, go ahead and configure the backend:
```yaml projects: - name: main backends: - type: digitalocean project_name: my-digital-ocean-project creds: type: api_key api_key: ... ```
??? info "Project" If `project_name` is not set, the default project will be used. ??? info "Required permissions" The API key must have the following scopes assigned: * `account` - read * `droplet` - create, read, update, delete, admin * `project` - create, read, update, delete * `regions` - read * `sizes` - read * `ssh_key` - create, read, update,delete ### Hot Aisle Log in to the SSH TUI as described in the [Hot Aisle Quick Start](https://hotaisle.xyz/quick-start/). Create a new team and generate an API key for the member in the team. Then, go ahead and configure the backend:
```yaml projects: - name: main backends: - type: hotaisle team_handle: hotaisle-team-handle creds: type: api_key api_key: 9c27a4bb7a8e472fae12ab34.3f2e3c1db75b9a0187fd2196c6b3e56d2b912e1c439ba08d89e7b6fcd4ef1d3f ```
??? info "Required permissions" The API key must have the following roles assigned: * **Owner role for the user** - Required for creating and managing SSH keys * **Operator role for the team** - Required for managing virtual machines within the team ??? info "Pricing" `dstack` shows the hourly price for Hot Aisle instances. Some instances also require an upfront payment for a minimum reservation period, which is usually a few hours. You will be charged for the full minimum period even if you stop the instance early. See the Hot Aisle API for the minimum reservation period for each instance type:
```shell $ curl -H "Authorization: Token $API_KEY" https://admin.hotaisle.app/api/teams/$TEAM_HANDLE/virtual_machines/available/ | jq ".[] | {gpus: .Specs.gpus, MinimumReservationMinutes}" ```
### CloudRift Log into your [CloudRift](https://console.cloudrift.ai/) console, click `API Keys` in the sidebar and click the button to create a new API key. Ensure you've created a project with CloudRift. Then proceed to configuring the backend.
```yaml projects: - name: main backends: - type: cloudrift creds: type: api_key api_key: rift_2prgY1d0laOrf2BblTwx2B2d1zcf1zIp4tZYpj5j88qmNgz38pxNlpX3vAo ```
### Vultr Log into your [Vultr](https://www.vultr.com/) account, click `Account` in the sidebar, select `API`, find the `Personal Access Token` panel and click the `Enable API` button. In the `Access Control` panel, allow API requests from all addresses or from the subnet where your `dstack` server is deployed. Then, go ahead and configure the backend:
```yaml projects: - name: main backends: - type: vultr creds: type: api_key api_key: B57487240a466624b48de22865589 ```
### OCI There are two ways to configure OCI: using client credentials or using the default credentials. === "Default credentials" If you have default credentials set up in `~/.oci/config`, configure the backend like this:
```yaml projects: - name: main backends: - type: oci creds: type: default ```
=== "Client credentials" Log into the [OCI Console](https://cloud.oracle.com), go to `My profile`, select `API keys`, and click `Add API key`. Once you add a key, you'll see the configuration file. Copy its values to configure the backend as follows:
```yaml projects: - name: main backends: - type: oci creds: type: client user: ocid1.user.oc1..g5vlaeqfu47akmaafq665xsgmyaqjktyfxtacfxc4ftjxuca7aohnd2ev66m tenancy: ocid1.tenancy.oc1..ajqsftvk4qarcfaak3ha4ycdsaahxmaita5frdwg3tqo2bcokpd3n7oizwai region: eu-frankfurt-1 fingerprint: 77:32:77:00:49:7c:cb:56:84:75:8e:77:96:7d:53:17 key_file: ~/.oci/private_key.pem ```
Make sure to include either the path to your private key via `key_file` or the contents of the key via `key_content`. ??? info "Required permissions" This is an example of a restrictive policy for a group of `dstack` users: ``` Allow group to read compartments in tenancy where target.compartment.name = '' Allow group to read marketplace-community-listings in compartment Allow group to manage app-catalog-listing in compartment Allow group to manage instances in compartment Allow group to manage compute-capacity-reports in compartment Allow group to manage volumes in compartment Allow group to manage volume-attachments in compartment Allow group to manage virtual-network-family in compartment ``` To use this policy, create a compartment for `dstack` and specify it in `~/.dstack/server/config.yml`. ```yaml projects: - name: main backends: - type: oci creds: type: default compartment_id: ocid1.compartment.oc1..aaaaaaaa ``` SSH fleets support the same features as [VM-based](#vm-based) backends. !!! info "What's next" 1. See the [`~/.dstack/server/config.yml`](../reference/server/config.yml.md) reference 2. Check [Projects](../concepts/projects.md) ## Container-based Container-based backends allow `dstack` to orchestrate container-based runs either directly on cloud providers that support containers or on Kubernetes. In this case, `dstack` delegates provisioning to the cloud provider or Kubernetes. Compared to [VM-based](#vm-based) backends, they offer less fine-grained control over provisioning but rely on the native logic of the underlying environment, whether that’s a cloud provider or Kubernetes. ### Kubernetes Regardless of whether it’s on-prem Kubernetes or managed, `dstack` can orchestrate container-based runs across your clusters. To use the `kubernetes` backend with `dstack`, you need to configure it with the path to the kubeconfig file, the IP address of any node in the cluster, and the port that `dstack` will use for proxying SSH traffic.
```yaml projects: - name: main backends: - type: kubernetes kubeconfig: filename: ~/.kube/config proxy_jump: hostname: 204.12.171.137 port: 32000 ```
!!! info "Proxy jump" To allow the `dstack` server and CLI to access runs via SSH, `dstack` requires a node that acts as a jump host to proxy SSH traffic into containers. To configure this node, specify `hostname` and `port` under the `proxy_jump` property: - `hostname` — the IP address of any cluster node selected as the jump host. Both the `dstack` server and CLI must be able to reach it. This node can be either a GPU node or a CPU-only node — it makes no difference. - `port` — any accessible port on that node, which `dstack` uses to forward SSH traffic. No additional setup is required — `dstack` configures and manages the proxy automatically. ??? info "User interface" If you are configuring the `kubernetes` backend on the [project settings page](projects.md#backends), specify the contents of the `kubeconfig` file in `data`:
```yaml type: kubernetes kubeconfig: data: | apiVersion: v1 kind: Config current-context: kubernetes-admin@gpu-cluster clusters: - name: gpu-cluster cluster: server: https://gpu-cluster.internal.example.com:6443 certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...LS0tLQo= users: - name: kubernetes-admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...LS0tLQo= client-key-data: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0t...LS0tLQo= contexts: - name: kubernetes-admin@gpu-cluster context: cluster: gpu-cluster user: kubernetes-admin proxy_jump: hostname: 204.12.171.137 port: 32000 ```
??? info "Required operators" === "NVIDIA" For `dstack` to correctly detect GPUs in your Kubernetes cluster, the cluster must have the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) pre-installed. === "AMD" For `dstack` to correctly detect GPUs in your Kubernetes cluster, the cluster must have the [AMD GPU Operator](https://github.com/ROCm/gpu-operator) pre-installed. ??? info "Required permissions" The following Kubernetes permissions are sufficient for `dstack` to work: * Cluster-scoped resources: ```yaml --8<-- "snippets/kubernetes/dstack-backend-clusterrole.yaml" ``` * Namespaced resources (replace the `${NAMESPACE}` placeholder with an actual value): ```yaml --8<-- "snippets/kubernetes/dstack-backend-role.yaml" ``` Ensure you've created a ClusterRoleBinding to grant the role to the user or the service account you're using. ??? info "Resources and offers" If you use ranges with [`resources`](../concepts/tasks.md#resources) (e.g. `gpu: 1..8` or `memory: 64GB..`) in fleet or run configurations, other backends collect and try all offers that satisfy the range. The `kubernetes` backend handles it differently. * For `gpu`, if you specify a range (e.g. `gpu: 4..8`), the `kubernetes` backend only provisions pods with the GPU count equal to the lower limit (`4`). The upper limit of the GPU range is always ignored. * For other resources such as `cpu`, `memory`, and `disk`, the `kubernetes` backend passes the lower and upper limits of the range as Kubernetes [requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits) respectively. If the upper limit is not set, the Kubernetes limit is also not set. Example:
```yaml type: dev-environment ide: vscode resources: cpu: 32..64 memory: 1024GB disk: 100GB.. gpu: nvidia:4..8 ```
This translates to the following Kubernetes resource spec: | Resource | Request | Limit | |---------------------|----------|-----------| | `cpu` | `32` | `64` | | `memory` | `1024Gi` | `1024Gi` | | `ephemeral-storage` | `100Gi` | _not set_ | | `nvidia.com/gpu` | `4` | `4` | This applies to offers shown in `dstack apply` (run plans), during provisioning, and in `dstack offer`. Unlike other backends, offers for the `kubernetes` backend always reflect the lower limit of the range. > To learn more, see the [Lambda](../examples/clusters/lambda/#kubernetes) and [Crusoe](../examples/clusters/crusoe/#kubernetes) examples. ### Runpod Log into your [Runpod](https://www.runpod.io/console/) console, click Settings in the sidebar, expand the `API Keys` section, and click the button to create a Read & Write key. Then proceed to configuring the backend.
```yaml projects: - name: main backends: - type: runpod creds: type: api_key api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9 ```
??? info "Community Cloud" By default, `dstack` considers instance offers only from the Secure Cloud. To also include the [Community Cloud](https://docs.runpod.io/references/faq/#secure-cloud-vs-community-cloud), set `community_cloud: true` in the backend settings.
```yaml projects: - name: main backends: - type: runpod creds: type: api_key api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9 community_cloud: true ```
You can tell Secure Cloud and Community Cloud apart by their regions. Secure Cloud regions contain datacenter IDs such as `CA-MTL-3`. Community Cloud regions contain country codes such as `CA`.
```shell $ dstack apply -f .dstack.yml -b runpod # BACKEND REGION INSTANCE SPOT PRICE 1 runpod CA NVIDIA A100 80GB PCIe yes $0.6 2 runpod CA-MTL-3 NVIDIA A100 80GB PCIe yes $0.82 ```
### Vast.ai Log into your [Vast.ai](https://cloud.vast.ai/) account, click Account in the sidebar, and copy your API Key. Then, go ahead and configure the backend:
```yaml projects: - name: main backends: - type: vastai creds: type: api_key api_key: d75789f22f1908e0527c78a283b523dd73051c8c7d05456516fc91e9d4efd8c5 ```
??? info "Community Cloud" By default, `dstack` includes both Server Cloud (datacenter) and Community Cloud offers. To restrict offers to Server Cloud only, set `community_cloud: false` in the backend settings.
```yaml projects: - name: main backends: - type: vastai creds: type: api_key api_key: d75789f22f1908e0527c78a283b523dd73051c8c7d05456516fc91e9d4efd8c5 community_cloud: false ```
Also, the `vastai` backend supports on-demand instances only. Spot instance support coming soon. # docs/concepts/fleets.md --- title: Fleets description: Managing pools of compute instances --- # Fleets Before submitting runs, you must create a fleet. Fleets act as both pools of instances and templates for how those instances are provisioned. > `dstack` supports two fleet types: [backend fleets](#backend-fleet) (which are provisioned dynamically in the cloud or on Kubernetes), and [SSH fleets](#ssh-fleet) (which use existing on-prem servers). ## Apply a configuration To create a fleet, define its configuration in a YAML file. The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `fleet.dstack.yml`), regardless of fleet type. === "Backend fleets" If you're using cloud providers or Kubernetes clusters and have configured the corresponding [backends](backends.md), create a backend fleet as follows:
```yaml type: fleet name: my-fleet # Allow to provision of up to 2 instances nodes: 0..2 # Uncomment to ensure instances are inter-connected #placement: cluster # Deprovision instances above the minimum if they remain idle idle_duration: 1h resources: # Allow to provision up to 8 GPUs gpu: 0..8 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f fleet.dstack.yml # BACKEND REGION RESOURCES SPOT PRICE 1 gcp us-west4 2xCPU, 8GB, 100GB (disk) yes $0.010052 2 azure westeurope 2xCPU, 8GB, 100GB (disk) yes $0.0132 3 gcp europe-central2 2xCPU, 8GB, 100GB (disk) yes $0.013248 Create the fleet? [y/n]: y FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED my-fleet 0 gcp (europe-west-1) L4:24GB (spot) $0.1624 idle 3 mins ago 1 gcp (europe-west-1) L4:24GB (spot) $0.1624 idle 3 mins ago ```
If the `nodes` range starts with `0`, `dstack apply` creates only a template. Instances are provisioned only when you submit runs. === "SSH fleets" If you have a group of on-prem servers accessible via SSH, you can create an SSH fleet as follows:
```yaml type: fleet name: my-fleet # Uncomment if instances are interconnected #placement: cluster ssh_config: user: ubuntu identity_file: ~/.ssh/id_rsa hosts: - 3.255.177.51 - 3.255.177.52 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f fleet.dstack.yml Provisioning... ---> 100% FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED my-fleet 0 ssh (remote) L4:24GB $0 idle 3 mins ago 1 ssh (remote) L4:24GB $0 idle 3 mins ago ```
`dstack apply` automatically connects to on-prem servers, installs the required dependencies, and adds them to the created fleet. ??? info "Host requirements" 1. Hosts must be pre-installed with Docker. === "NVIDIA" 2. Hosts with NVIDIA GPUs must also be pre-installed with CUDA 12.1 and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). === "AMD" 2. Hosts with AMD GPUs must also be pre-installed with AMDGPU-DKMS kernel driver (e.g. via [native package manager](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/native-install/index.html) or [AMDGPU installer](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/amdgpu-install.html).) === "Intel Gaudi" 2. Hosts with Intel Gaudi accelerators must be pre-installed with [Gaudi software and drivers](https://docs.habana.ai/en/latest/Installation_Guide/Driver_Installation.html#driver-installation). This must include the drivers, `hl-smi`, and Habana Container Runtime. === "Tenstorrent" 2. Hosts with Tenstorrent accelerators must be pre-installed with [Tenstorrent software](https://docs.tenstorrent.com/getting-started/README.html#software-installation). This must include the drivers, `tt-smi`, and HugePages. 3. The user specified must have passwordless `sudo` access. 4. The SSH server must be running and configured with `AllowTcpForwarding yes` in `/etc/ssh/sshd_config`. 5. The firewall must allow SSH and should forbid any other connections from external networks. For `placement: cluster` fleets, it should also allow any communication between fleet nodes. > Once the fleet is created, you can run [dev environments](dev-environments.md), [tasks](tasks.md), and [services](services.md). ## Configuration options Backend fleets support [many options](../reference/dstack.yml/fleet.md); see some major configuration examples below. ### Cluster placement Both [backend fleets](#backend-fleet) and [SSH fleets](#ssh-fleet) allow the `placement` property to be set to `cluster`. This property ensures that instances are interconnected. This is required for running [distributed tasks](tasks.md#distributed-tasks). === "Backend fleets" Backend fleets allow to provision interconnected clusters across supported backends.
```yaml type: fleet name: my-fleet nodes: 2 placement: cluster resources: gpu: H100:8 ```
#### Backends Fast interconnect is supported on the `aws`, `gcp`, `nebius`, `crusoe`, and `kubernetes` backends. Some backends may require additional configuration. === "GCP" On GCP, you may need to configure `extra_vpcs` and `roce_vpcs` in the `gcp` backend configuration. Refer to the [GCP](../examples/clusters/gcp.md) examples for more details. === "Nebius" On [Nebius](https://docs.nebius.com/compute/clusters/gpu), `dstack` automatically configures InfiniBand networking if it is supported by the selected instance type. === "Crusoe" On [Crusoe](https://docs.crusoecloud.com/networking/infiniband/managing-infiniband-networks), `dstack` automatically configures InfiniBand networking if it is supported by the selected instance type. Refer to the [Crusoe](../examples/clusters/crusoe.md#vms) example for more details. === "Kubernetes" If the Kubernetes cluster has interconnect configured, `dstack` can use it without additional setup. See the [Lambda](../examples/clusters/lambda.md#kubernetes) or [Crusoe](../examples/clusters/crusoe.md#kubernetes) examples. === "SSH fleets" If the hosts in the SSH fleet have interconnect configured, you only need to set `placement` to `cluster`.
```yaml type: fleet name: my-fleet placement: cluster ssh_config: user: ubuntu identity_file: ~/.ssh/id_rsa hosts: - 3.255.177.51 - 3.255.177.52 ```
!!! info "Network" By default, `dstack` automatically detects the network shared by the hosts. However, it's possible to configure it explicitly via the [`network`](../reference/dstack.yml/fleet.md#network) property. !!! info "Examples" See the cluster examples for [AWS](../examples/clusters/aws.md), [GCP](../examples/clusters/gcp.md), [Lambda](../examples/clusters/lambda.md), [Crusoe](../examples/clusters/crusoe.md), [Nebius](../examples/clusters/nebius.md), and [NCCL/RCCL tests](../examples/clusters/nccl-rccl-tests.md). ### Nodes The `nodes` property is supported only by backend fleets and specifies how many nodes `dstack` must or can provision.
```yaml type: fleet name: my-fleet # Allow to provision of up to 2 instances nodes: 0..2 # Uncomment to ensure instances are inter-connected #placement: cluster # Deprovision instances above the minimum if they remain idle idle_duration: 1h resources: # Allow to provision up to 8 GPUs gpu: 0..8 ```
#### Pre-provisioning If the `nodes` range starts with `0`, `dstack apply` creates only a template, and instances are provisioned when you submit runs. To provision instances up front, set the `nodes` range to start above `0`. This pre-creates the initial number of instances; additional instances (if any) are provisioned on demand.
```yaml type: fleet name: my-fleet nodes: 2..10 # Uncomment to ensure instances are inter-connected #placement: cluster resources: gpu: H100:8 ```
Pre-provisioning is supported only for [VM-based backends](backends.md#vm-based). ??? info "Target number" To pre-provision more than the minimum number of instances, set the `target` parameter.
```yaml type: fleet name: my-fleet nodes: min: 2 max: 10 target: 6 # Deprovision instances above the minimum if they remain idle idle_duration: 1h ```
`dstack apply` pre-provisions up to `target` and scales back to `min` after `idle_duration`. ### Resources Backend fleets allow you to specify the resource requirements for the instances to be provisioned. The `resources` property syntax is the same as for [run configurations](dev-environments.md#resources). ### Spot policy Backend fleets allow you to specify a `spot policy`. By default, it is set to `on-demand`. If you want to use spot instances, you must set it to `auto` if you plan to use both on-demand and spot instances, or to `spot` if only spot instances are allowed.
```yaml type: fleet name: my-fleet nodes: 0..2 # Uncomment to ensure instances are inter-connected #placement: cluster # Allows both on-demand and spot spot_policy: auto idle_duration: 1h resources: gpu: 0..8 ```
Note that run configurations must specify their own `spot policy` which is also set to `on-demand` by default. ### Backends Backend fleets allow you to set `backends` to specify which backends are allowed to be used. ### Idle duration By default, instances of a backend fleet stay `idle` for 3 days and can be reused within that time. If an instance is not reused within this period, it is automatically terminated. To change the default idle duration, set [`idle_duration`](../reference/dstack.yml/fleet.md#idle_duration) in the fleet configuration (e.g., `0s`, `1m`, or `off` for unlimited).
```yaml type: fleet name: my-fleet nodes: 2 # Terminate instances idle for more than 1 hour idle_duration: 1h resources: gpu: 24GB ```
### Blocks By default, a job uses the entire instance—e.g., all 8 GPUs. To allow multiple jobs on the same instance, set the `blocks` property to divide the instance. Each job can then use one or more blocks, up to the full instance. === "Backend fleets"
```yaml type: fleet name: my-fleet nodes: 0..2 resources: gpu: H100:8 # Split into 4 blocks, each with 2 GPUs blocks: 4 ```
=== "SSH fleets"
```yaml type: fleet name: my-fleet ssh_config: user: ubuntu identity_file: ~/.ssh/id_rsa hosts: - hostname: 3.255.177.51 blocks: 4 - hostname: 3.255.177.52 # As many as possible, according to numbers of GPUs and CPUs blocks: auto - hostname: 3.255.177.53 # Do not slice. This is the default value, may be omitted blocks: 1 ```
All resources (GPU, CPU, memory) are split evenly across blocks, while disk is shared. For example, with 8 GPUs, 128 CPUs, and 2TB RAM, setting `blocks` to `8` gives each block 1 GPU, 16 CPUs, and 256 GB RAM. Set `blocks` to `auto` to match the number of blocks to the number of GPUs. !!! info "Distributed tasks" Distributed tasks require exclusive access to all host resources and therefore must use all blocks on each node. ### SSH config #### Proxy jump If hosts are behind a head node (aka "login node"), configure [`proxy_jump`](../reference/dstack.yml/fleet.md#proxy_jump):
```yaml type: fleet name: my-fleet ssh_config: user: ubuntu identity_file: ~/.ssh/worker_node_key hosts: - 3.255.177.51 - 3.255.177.52 proxy_jump: hostname: 3.255.177.50 user: ubuntu identity_file: ~/.ssh/head_node_key ```
To be able to attach to runs, both explicitly with `dstack attach` and implicitly with `dstack apply`, you must either add a front node key (`~/.ssh/head_node_key`) to an SSH agent or configure a key path in `~/.ssh/config`:
``` Host 3.255.177.50 IdentityFile ~/.ssh/head_node_key ```
where `Host` must match `ssh_config.proxy_jump.hostname` or `ssh_config.hosts[n].proxy_jump.hostname` if you configure head nodes on a per-worker basis. ### Environment variables If needed, you can specify environment variables that will be automatically passed to any jobs running on this fleet. For example, these variables can be used to configure a proxy: ```yaml type: fleet name: my-fleet env: - HTTP_PROXY=http://proxy.example.com:80 - HTTPS_PROXY=http://proxy.example.com:80 - NO_PROXY=localhost,127.0.0.1 ssh_config: user: ubuntu identity_file: ~/.ssh/id_rsa hosts: - 3.255.177.51 - 3.255.177.52 ``` !!! info "Reference" The fleet configuration file supports additional options, including [`instance_types`](../reference/dstack.yml/fleet.md#instance_types), [`max_price`](../reference/dstack.yml/fleet.md#max_price), [`regions`](../reference/dstack.yml/fleet.md#max_price), among others. For the complete list, see the [reference](../reference/dstack.yml/fleet.md). ## Tenant isolation Users running workloads on a fleet have access to the host, including the folders that may be used as instance volumes, and containers use host network mode unless the host has multiple [blocks](#blocks) configured and the job uses only a subset of them. Tighter isolation is on the roadmap, including [SSH reverse proxy](https://github.com/dstackai/dstack/issues/3644){:target="_blank"} and rootless access to the host. When [exporting fleets](exports.md) to other projects, the same access model applies to members of the importer projects. ## Export fleets Fleets can be exported to other projects, allowing those projects to use the exported fleets for running dev environments, tasks, and services. See [Exports](exports.md) for more details. ## Manage fleets ### List fleets The [`dstack fleet`](../reference/cli/dstack/fleet.md#dstack-fleet-list) command lists fleet instances and their status:
```shell $ dstack fleet FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED my-fleet 0 gcp (europe-west-1) L4:24GB (spot) $0.1624 idle 3 mins ago 1 gcp (europe-west-1) L4:24GB (spot) $0.1624 idle 3 mins ago ```
### Delete fleets When a fleet isn't used by a run, you can delete it by passing the fleet configuration to `dstack delete`:
```shell $ dstack delete -f cluster.dstack.yaml Delete the fleet my-gcp-fleet? [y/n]: y Fleet my-gcp-fleet deleted ```
Alternatively, you can delete a fleet by passing the fleet name to `dstack fleet delete`. To terminate and delete specific instances from a fleet, pass `-i INSTANCE_NUM`. ### List offers To inspect offers available through a fleet, pass `--fleet` to `dstack offer`.
```shell $ dstack offer --gpu H100 --fleet my-fleet ```
Use `--group-by gpu,backend` to aggregate offers. !!! info "What's next?" 1. Check [dev environments](dev-environments.md), [tasks](tasks.md), and [services](services.md) 2. Read about [Backends](backends.md) guide 3. Learn how to [export fleets](exports.md) to other projects 4. Explore the [`.dstack.yml` reference](../reference/dstack.yml/fleet.md) 5. See the cluster examples for [AWS](../examples/clusters/aws.md), [GCP](../examples/clusters/gcp.md), [Lambda](../examples/clusters/lambda.md), [Crusoe](../examples/clusters/crusoe.md), [Nebius](../examples/clusters/nebius.md), and [NCCL/RCCL tests](../examples/clusters/nccl-rccl-tests.md) # docs/concepts/dev-environments.md --- title: Dev environments description: Provisioning remote instances for cloud-based development --- # Dev environments A dev environment lets you provision an instance and access it with your desktop IDE or SSH. ??? info "Prerequisites" Before running a dev environment, make sure you’ve [installed](../installation.md) the server and CLI, and created a [fleet](fleets.md). ## Apply a configuration First, define a dev environment configuration as a YAML file. The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml` are both acceptable).
```yaml type: dev-environment # The name is optional, if not specified, generated randomly name: vscode python: "3.11" # Uncomment to use a custom Docker image #image: huggingface/trl-latest-gpu # Comment if not required ide: vscode # Uncomment to leverage spot instances #spot_policy: auto resources: gpu: 24GB ```
To run a dev environment, pass the configuration to [`dstack apply`](../reference/cli/dstack/apply.md):
```shell $ dstack apply -f examples/.dstack.yml # BACKEND REGION RESOURCES SPOT PRICE 1 runpod CA-MTL-1 9xCPU, 48GB, A5000:24GB yes $0.11 2 runpod EU-SE-1 9xCPU, 43GB, A5000:24GB yes $0.11 3 gcp us-west4 4xCPU, 16GB, L4:24GB yes $0.214516 Submit the run vscode? [y/n]: y Launching `vscode`... ---> 100% To open in VS Code Desktop, use this link: vscode://vscode-remote/ssh-remote+vscode/workflow To connect via SSH, use: `ssh vscode` ```
`dstack apply` automatically provisions an instance and sets up an IDE on it. ??? info "SSH-only" The `ide` property is optional. If omitted, no IDE is pre-installed, but the dev environment is still accessible via SSH:
```yaml type: dev-environment name: my-env python: "3.11" resources: gpu: 24GB ```
??? info "Windows" On Windows, `dstack` works both natively and inside WSL. But, for dev environments, it's recommended _not to use_ `dstack apply` _inside WSL_ due to a [VS Code issue](https://github.com/microsoft/vscode-remote-release/issues/937). To open the dev environment in your desktop IDE, use the link from the output (such as `vscode://vscode-remote/ssh-remote+fast-moth-1/workflow`). ![](../../assets/images/dstack-vscode-jupyter.png){ width=800 } ??? info "SSH" Alternatively, while the CLI is attached to the run, you can connect to the dev environment via SSH:
```shell $ ssh vscode ```
## Configuration options ### Initialization If you want to pre-configure the dev environment, specify the [`init`](../reference/dstack.yml/dev-environment.md#init) property with a list of commands to run at startup:
```yaml type: dev-environment name: vscode python: "3.11" ide: vscode init: - pip install wandb ```
### Resources When you specify a resource value like `cpu` or `memory`, you can either use an exact value (e.g. `24GB`) or a range (e.g. `24GB..`, or `24GB..80GB`, or `..80GB`).
```yaml type: dev-environment # The name is optional, if not specified, generated randomly name: vscode ide: vscode resources: # 16 or more x86_64 cores cpu: 16.. # 200GB or more RAM memory: 200GB.. # 4 GPUs from 40GB to 80GB gpu: 40GB..80GB:4 # Shared memory (required by multi-gpu) shm_size: 16GB # Disk size disk: 500GB ```
The `cpu` property lets you set the architecture (`x86` or `arm`) and core count — e.g., `x86:16` (16 x86 cores), `arm:8..` (at least 8 ARM cores). If not set, `dstack` infers it from the GPU or defaults to `x86`. The `gpu` property lets you specify vendor, model, memory, and count — e.g., `nvidia` (one NVIDIA GPU), `A100` (one A100), `A10G,A100` (either), `A100:80GB` (one 80GB A100), `A100:2` (two A100), `24GB..40GB:2` (two GPUs with 24–40GB), `A100:40GB:2` (two 40GB A100s). If vendor is omitted, `dstack` infers it from the model or defaults to `nvidia`. ??? info "Shared memory" If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure `shm_size`, e.g. set it to `16GB`. > If you’re unsure which offers (hardware configurations) are available from the configured backends, use the > [`dstack offer`](../reference/cli/dstack/offer.md#list-gpu-offers) command to list them. ### Docker #### Default image If you don't specify `image`, `dstack` uses its [base](https://github.com/dstackai/dstack/tree/master/docker/base) Docker image pre-configured with `uv`, `python`, `pip`, essential CUDA drivers, `mpirun`, and NCCL tests (under `/opt/nccl-tests/build`). Set the `python` property to pre-install a specific version of Python.
```yaml type: dev-environment name: vscode python: 3.12 ide: vscode ```
#### NVCC By default, the base Docker image doesn’t include `nvcc`, which is required for building custom CUDA kernels. If you need `nvcc`, set the [`nvcc`](../reference/dstack.yml/dev-environment.md#nvcc) property to true.
```yaml type: dev-environment name: vscode python: 3.12 nvcc: true ide: vscode init: - uv pip install flash_attn --no-build-isolation ```
#### Custom image If you want, you can specify your own Docker image via `image`.
```yaml type: dev-environment name: vscode image: huggingface/trl-latest-gpu ide: vscode ```
#### Docker in Docker Set `docker` to `true` to enable the `docker` CLI in your dev environment, e.g., to run or build Docker images, or use Docker Compose.
```yaml type: dev-environment name: vscode docker: true ide: vscode init: - docker run --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi ```
Cannot be used with `python` or `image`. Not supported on `runpod`, `vastai`, or `kubernetes`. #### Privileged mode To enable privileged mode, set [`privileged`](../reference/dstack.yml/dev-environment.md#privileged) to `true`. Not supported with `runpod`, `vastai`, and `kubernetes`. #### Private registry Use the [`registry_auth`](../reference/dstack.yml/dev-environment.md#registry_auth) property to provide credentials for a private Docker registry.
```yaml type: dev-environment name: vscode env: - NGC_API_KEY image: nvcr.io/nim/deepseek-ai/deepseek-r1-distill-llama-8b registry_auth: username: $oauthtoken password: ${{ env.NGC_API_KEY }} ide: vscode ```
### Environment variables
```yaml type: dev-environment name: vscode env: - HF_TOKEN - HF_HUB_ENABLE_HF_TRANSFER=1 ide: vscode ```
If you don't assign a value to an environment variable (see `HF_TOKEN` above), `dstack` will require the value to be passed via the CLI or set in the current process. ??? info "System environment variables" The following environment variables are available in any run by default: | Name | Description | |-------------------------|--------------------------------------------------| | `DSTACK_RUN_NAME` | The name of the run | | `DSTACK_REPO_ID` | The ID of the repo | | `DSTACK_GPUS_NUM` | The total number of GPUs in the run | | `DSTACK_WORKING_DIR` | The working directory of the run | | `DSTACK_REPO_DIR` | The directory where the repo is mounted (if any) | ### Working directory If `working_dir` is not specified, it defaults to the working directory set in the Docker image. For example, the [default image](#default-image) uses `/dstack/run` as its working directory. If the Docker image does not have a working directory set, `dstack` uses `/` as the `working_dir`. The `working_dir` must be an absolute path. The tilde (`~`) is supported (e.g., `~/my-working-dir`). ### Files Sometimes, when you run a dev environment, you may want to mount local files. This is possible via the [`files`](../reference/dstack.yml/task.md#_files) property. Each entry maps a local directory or file to a path inside the container.
```yaml type: dev-environment name: vscode files: - .:examples # Maps the directory with `.dstack.yml` to `/examples` - ~/.ssh/id_rsa:/root/.ssh/id_rsa # Maps `~/.ssh/id_rsa` to `/root/.ssh/id_rsa` ide: vscode ```
If the local path is relative, it’s resolved relative to the configuration file. If the container path is relative, it’s resolved relative to the [working directory](#working-directory). The container path is optional. If not specified, it will be automatically calculated:
```yaml type: dev-environment name: vscode files: - ../examples # Maps the parent directory of `.dstack.yml` to `/../examples` - ~/.ssh/id_rsa # Maps `~/.ssh/id_rsa` to `/root/.ssh/id_rsa` ide: vscode ```
??? info "File size" Whether its a file or folder, each entry is limited to 2MB. To avoid exceeding this limit, make sure to exclude unnecessary files by listing it via `.gitignore` or `.dstackignore`. The 2MB upload limit can be increased by setting the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable. ### Repos Sometimes, you may want to clone an entire Git repo inside the container. Imagine you have a Git repo (clonned locally) containing an `examples` subdirectory with a `.dstack.yml` file:
```yaml type: dev-environment name: vscode repos: # Clones the repo from the parent directory (`examples/..`) to `` - .. ide: vscode ```
When you run it, `dstack` clones the repo on the instance, applies your local changes, and mounts it—so the container matches your local repo. The local path can be either relative to the configuration file or absolute. ??? info "Repo directory" By default, `dstack` clones the repo to the [working directory](#working-directory). You can override the repo directory using either a relative or an absolute path:
```yaml type: dev-environment name: vscode repos: # Clones the repo in the parent directory (`examples/..`) to `/my-repo` - ..:/my-repo ide: vscode ```
> If the repo directory is relative, it is resolved against [working directory](#working-directory). If the repo directory is not empty, the run will fail with a runner error. To override this behavior, you can set `if_exists` to `skip`: ```yaml type: dev-environment name: vscode repos: - local_path: .. path: /my-repo if_exists: skip ide: vscode ``` ??? info "Repo size" The repo size is not limited. However, local changes are limited to 2MB. To avoid exceeding this limit, exclude unnecessary files using `.gitignore` or `.dstackignore`. You can increase the 2MB limit by setting the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable. ??? info "Repo URL" Sometimes you may want to clone a Git repo within the container without cloning it locally. In this case, simply provide a URL in `repos`:
```yaml type: dev-environment name: vscode repos: # Clone the repo to `` - https://github.com/dstackai/dstack ide: vscode ```
??? info "Private repos" If a Git repo is private, `dstack` will automatically try to use your default Git credentials (from `~/.ssh/config` or `~/.config/gh/hosts.yml`). > If you want to use custom credentials, ensure to pass them via [`dstack init`](../reference/cli/dstack/init.md) before submitting a run. Currently, you can configure up to one repo per run configuration. ### Retry policy By default, if `dstack` can't find capacity or the instance is interrupted, the run will fail. If you'd like `dstack` to automatically retry, configure the [retry](../reference/dstack.yml/dev-environment.md#retry) property accordingly:
```yaml type: dev-environment # The name is optional, if not specified, generated randomly name: vscode ide: vscode retry: # Retry on specific events on_events: [no-capacity, error, interruption] # Retry for up to 1 hour duration: 1h ```
!!! info "Retry duration" The duration period is calculated as a run age for `no-capacity` event and as a time passed since the last `interruption` and `error` for `interruption` and `error` events. ### Inactivity duration Set [`inactivity_duration`](../reference/dstack.yml/dev-environment.md#inactivity_duration) to automatically stop the dev environment after a configured period of inactivity.
```yaml type: dev-environment name: vscode ide: vscode # Stop if inactive for 2 hours inactivity_duration: 2h ```
The dev environment becomes inactive when you close the remote VS Code window, close any `ssh ` shells, and stop the `dstack apply` or `dstack attach` command. If you go offline without stopping anything manually, the dev environment will also become inactive within about 3 minutes. If `inactivity_duration` is configured for your dev environment, you can see how long it has been inactive in `dstack ps --verbose` (or `-v`).
```shell $ dstack ps -v NAME BACKEND RESOURCES PRICE STATUS SUBMITTED vscode runpod 2xCPU, 8GB, $0.0286 running 8 mins ago 100.0GB (disk) (inactive for 2m 34s) ```
If you reattach to the dev environment using [`dstack attach`](../reference/cli/dstack/attach.md), the inactivity timer will be reset within a few seconds. ??? info "In-place update" As long as the configuration defines the `name` property, the value of `inactivity_duration` can be changed for a running dev environment without a restart. Just change the value in the configuration and run `dstack apply` again.
```shell $ dstack apply -f .dstack.yml Detected configuration changes that can be updated in-place: ['inactivity_duration'] Update the run? [y/n]: ```
> `inactivity_duration` is not to be confused with [`idle_duration`](#idle-duration). > The latter determines how soon the underlying cloud instance will be terminated > _after_ the dev environment is stopped. ### Utilization policy Sometimes it’s useful to track whether a dev environment is fully utilizing all GPUs. While you can check this with [`dstack metrics`](../reference/cli/dstack/metrics.md), `dstack` also lets you set a policy to auto-terminate the run if any GPU is underutilized. Below is an example of a dev environment that auto-terminate if any GPU stays below 10% utilization for 1 hour.
```yaml type: dev-environment name: my-dev python: 3.12 ide: cursor resources: gpu: H100:8 utilization_policy: min_gpu_utilization: 10 time_window: 1h ```
### Schedule Specify `schedule` to start a dev environment periodically at specific UTC times using the cron syntax:
```yaml type: dev-environment ide: vscode schedule: cron: "0 8 * * mon-fri" # at 8:00 UTC from Monday through Friday ```
The `schedule` property can be combined with `max_duration` or `utilization_policy` to shutdown the dev environment automatically when it's not needed. ??? info "Cron syntax" `dstack` supports [POSIX cron syntax](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/crontab.html#tag_20_25_07). One exception is that days of the week are started from Monday instead of Sunday so `0` corresponds to Monday. The month and day of week fields accept abbreviated English month and weekday names (`jan–dec` and `mon–sun`) respectively. A cron expression consists of five fields: ``` ┌───────────── minute (0-59) │ ┌───────────── hour (0-23) │ │ ┌───────────── day of the month (1-31) │ │ │ ┌───────────── month (1-12 or jan-dec) │ │ │ │ ┌───────────── day of the week (0-6 or mon-sun) │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ * * * * * ``` The following operators can be used in any of the fields: | Operator | Description | Example | |----------|-----------------------|-------------------------------------------------------------------------| | `*` | Any value | `0 * * * *` runs every hour at minute 0 | | `,` | Value list separator | `15,45 10 * * *` runs at 10:15 and 10:45 every day. | | `-` | Range of values | `0 1-3 * * *` runs at 1:00, 2:00, and 3:00 every day. | | `/` | Step values | `*/10 8-10 * * *` runs every 10 minutes during the hours 8:00 to 10:59. | ### Spot policy By default, `dstack` uses on-demand instances. However, you can change that via the [`spot_policy`](../reference/dstack.yml/dev-environment.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`. --8<-- "docs/concepts/snippets/manage-fleets.ext" !!! info "Reference" Dev environments support many more configuration options, incl. [`backends`](../reference/dstack.yml/dev-environment.md#backends), [`regions`](../reference/dstack.yml/dev-environment.md#regions), [`max_price`](../reference/dstack.yml/dev-environment.md#max_price), and [`max_duration`](../reference/dstack.yml/dev-environment.md#max_duration), among [others](../reference/dstack.yml/dev-environment.md). --8<-- "docs/concepts/snippets/manage-runs.ext" !!! info "What's next?" 1. Read about [tasks](tasks.md) and [services](services.md) 2. Learn how to manage [fleets](fleets.md) # docs/concepts/tasks.md --- title: Tasks description: Running commands for training and batch processing --- # Tasks A task allows you to run arbitrary commands on one or more nodes. They are best suited for jobs like training or batch processing. ??? info "Prerequisites" Before running a task, make sure you’ve [installed](../installation.md) the server and CLI, and created a [fleet](fleets.md). ## Apply a configuration First, define a task configuration as a YAML file. The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml` are both acceptable). [//]: # (TODO: Make tabs - single machine & distributed tasks & web app)
```yaml type: task # The name is optional, if not specified, generated randomly name: trl-sft python: 3.12 # Uncomment to use a custom Docker image #image: huggingface/trl-latest-gpu env: - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: # One to two H100 GPUs gpu: H100:1..2 shm_size: 24GB ```
To run a task, pass the configuration to [`dstack apply`](../reference/cli/dstack/apply.md):
```shell $ dstack apply -f .dstack.yml # BACKEND REGION RESOURCES SPOT PRICE 1 runpod CA-MTL-1 18xCPU, 100GB, A5000:24GB:2 yes $0.22 2 runpod EU-SE-1 18xCPU, 100GB, A5000:24GB:2 yes $0.22 3 gcp us-west4 27xCPU, 150GB, A5000:24GB:3 yes $0.33 Submit the run trl-sft? [y/n]: y Launching `axolotl-train`... ---> 100% {'loss': 1.4967, 'grad_norm': 1.2734375, 'learning_rate': 1.0000000000000002e-06, 'epoch': 0.0} 0% 1/24680 [00:13<95:34:17, 13.94s/it] 6% 73/1300 [00:48<13:57, 1.47it/s] ```
`dstack apply` automatically provisions instances and runs the task. ## Configuration options !!! info "No commands" If `commands` are not specified, `dstack` runs `image`’s entrypoint (or fails if none is set). ### Ports A task can configure ports. In this case, if the task is running an application on a port, `dstack apply` will securely allow you to access this port from your local machine through port forwarding.
```yaml type: task name: streamlit-hello python: 3.12 commands: - uv pip install streamlit - streamlit hello ports: - 8501 ```
When running it, `dstack apply` forwards `8501` port to `localhost:8501`, enabling secure access to the running application. ### Distributed tasks By default, a task runs on a single node. However, you can run it on a cluster of nodes by specifying `nodes`.
```yaml type: task name: train-distrib nodes: 2 python: 3.12 env: - NCCL_DEBUG=INFO commands: - git clone https://github.com/pytorch/examples.git pytorch-examples - cd pytorch-examples/distributed/ddp-tutorial-series - uv pip install -r requirements.txt - | torchrun \ --nproc-per-node=$DSTACK_GPUS_PER_NODE \ --node-rank=$DSTACK_NODE_RANK \ --nnodes=$DSTACK_NODES_NUM \ --master-addr=$DSTACK_MASTER_NODE_IP \ --master-port=12345 \ multinode.py 50 10 resources: gpu: 24GB:1..2 shm_size: 24GB ```
!!! info "Cluster placement" To submit a distributed task, you must create at least one fleet with a [cluster placement](fleets.md#cluster-placement). Jobs on each node communicate using their private IP addresses. Use `DSTACK_MASTER_NODE_IP`, `DSTACK_NODES_IPS`, `DSTACK_NODE_RANK`, and other [system environment variables](#system-environment-variables) for inter-node communication. `dstack` is easy to use with `accelerate`, `torchrun`, Ray, Spark, and any other distributed frameworks. !!! info "Examples" See the training examples for [TRL](../examples/training/trl.md#distributed-training), [Axolotl](../examples/training/axolotl.md#distributed-training), and [Ray+RAGEN](../examples/training/ray-ragen.md). See the cluster examples for [AWS](../examples/clusters/aws.md), [GCP](../examples/clusters/gcp.md), [Lambda](../examples/clusters/lambda.md), [Crusoe](../examples/clusters/crusoe.md), [Nebius](../examples/clusters/nebius.md), and [NCCL/RCCL tests](../examples/clusters/nccl-rccl-tests.md). ??? info "Network interface" Distributed frameworks usually detect the correct network interface automatically, but sometimes you need to specify it explicitly. For example, with PyTorch and the NCCL backend, you may need to add these commands to tell NCCL to use the private interface: ```yaml commands: - apt-get install -y iproute2 - > if [[ $DSTACK_NODE_RANK == 0 ]]; then export NCCL_SOCKET_IFNAME=$(ip -4 -o addr show | fgrep $DSTACK_MASTER_NODE_IP | awk '{print $2}') else export NCCL_SOCKET_IFNAME=$(ip route get $DSTACK_MASTER_NODE_IP | sed -E 's/.*?dev (\S+) .*/\1/;t;d') fi # ... The rest of the commands ``` ??? info "SSH" You can log in to any node from any node via SSH on port 10022 using the `~/.ssh/dstack_job` private key. For convenience, `~/.ssh/config` is preconfigured with these options, so a simple `ssh ` is enough. For a list of nodes IPs check the `DSTACK_NODES_IPS` environment variable. ### Resources When you specify a resource value like `cpu` or `memory`, you can either use an exact value (e.g. `24GB`) or a range (e.g. `24GB..`, or `24GB..80GB`, or `..80GB`).
```yaml type: task name: trl-sft python: 3.12 env: - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: # 16 or more x86_64 cores cpu: 16.. # 200GB or more RAM memory: 200GB.. # 4 GPUs from 40GB to 80GB gpu: 40GB..80GB:4 # Shared memory (required by multi-gpu) shm_size: 24GB # Disk size disk: 500GB ```
The `cpu` property lets you set the architecture (`x86` or `arm`) and core count — e.g., `x86:16` (16 x86 cores), `arm:8..` (at least 8 ARM cores). If not set, `dstack` infers it from the GPU or defaults to `x86`. The `gpu` property lets you specify vendor, model, memory, and count — e.g., `nvidia` (one NVIDIA GPU), `A100` (one A100), `A10G,A100` (either), `A100:80GB` (one 80GB A100), `A100:2` (two A100), `24GB..40GB:2` (two GPUs with 24–40GB), `A100:40GB:2` (two 40GB A100s). If vendor is omitted, `dstack` infers it from the model or defaults to `nvidia`. ??? info "Shared memory" If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure `shm_size`, e.g. set it to `24GB`. > If you’re unsure which offers (hardware configurations) are available from the configured backends, use the > [`dstack offer`](../reference/cli/dstack/offer.md#list-gpu-offers) command to list them. ### Docker #### Default image If you don't specify `image`, `dstack` uses its [base](https://github.com/dstackai/dstack/tree/master/docker/base) Docker image pre-configured with `uv`, `python`, `pip`, essential CUDA drivers, `mpirun`, and NCCL tests (under `/opt/nccl-tests/build`). Set the `python` property to pre-install a specific version of Python.
```yaml type: task name: train python: 3.12 env: - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1..2 shm_size: 24GB ```
#### NVCC By default, the base Docker image doesn’t include `nvcc`, which is required for building custom CUDA kernels. If you need `nvcc`, set the [`nvcc`](../reference/dstack.yml/dev-environment.md#nvcc) property to true. ```yaml type: task name: train python: 3.12 nvcc: true env: - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - uv pip install flash_attn --no-build-isolation - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --attn_implementation=flash_attention_2 \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1 ``` #### Custom image If you want, you can specify your own Docker image via `image`.
```yaml type: task name: trl-sft image: huggingface/trl-latest-gpu env: - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb # if shell is not specified, `sh` is used for custom images shell: bash commands: - source activate trl - | trl sft --model_name_or_path $MODEL \ --dataset_name $DATASET \ --output_dir /output \ --torch_dtype bfloat16 \ --use_peft true resources: gpu: H100:1 ```
#### Docker in Docker Set `docker` to `true` to enable the `docker` CLI in your task, e.g., to run or build Docker images, or use Docker Compose.
```yaml type: task name: docker-nvidia-smi docker: true commands: - docker run --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi resources: gpu: 1 ```
Cannot be used with `python` or `image`. Not supported on `runpod`, `vastai`, or `kubernetes`. #### Privileged mode To enable privileged mode, set [`privileged`](../reference/dstack.yml/dev-environment.md#privileged) to `true`. Not supported with `runpod`, `vastai`, and `kubernetes`. #### Private registry Use the [`registry_auth`](../reference/dstack.yml/dev-environment.md#registry_auth) property to provide credentials for a private Docker registry. ```yaml type: task name: train env: - NGC_API_KEY image: nvcr.io/nvidia/pytorch:25.05-py3 registry_auth: username: $oauthtoken password: ${{ env.NGC_API_KEY }} commands: - git clone https://github.com/pytorch/examples.git pytorch-examples - cd pytorch-examples/distributed/ddp-tutorial-series - pip install -r requirements.txt - | torchrun \ --nproc-per-node=$DSTACK_GPUS_PER_NODE \ --nnodes=$DSTACK_NODES_NUM \ multinode.py 50 10 resources: gpu: H100:1..2 shm_size: 24GB ``` ### Environment variables
```yaml type: task name: trl-sft python: 3.12 env: - HF_TOKEN - HF_HUB_ENABLE_HF_TRANSFER=1 - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1 ```
If you don't assign a value to an environment variable (see `HF_TOKEN` above), `dstack` will require the value to be passed via the CLI or set in the current process. ??? info "System environment variables" The following environment variables are available in any run by default: | Name | Description | |-------------------------|------------------------------------------------------------------| | `DSTACK_RUN_NAME` | The name of the run | | `DSTACK_REPO_ID` | The ID of the repo | | `DSTACK_GPUS_NUM` | The total number of GPUs in the run | | `DSTACK_NODES_NUM` | The number of nodes in the run | | `DSTACK_GPUS_PER_NODE` | The number of GPUs per node | | `DSTACK_NODE_RANK` | The rank of the node | | `DSTACK_MASTER_NODE_IP` | The internal IP address of the master node | | `DSTACK_NODES_IPS` | The list of internal IP addresses of all nodes delimited by "\n" | | `DSTACK_MPI_HOSTFILE` | The path to a pre-populated MPI hostfile | | `DSTACK_WORKING_DIR` | The working directory of the run | | `DSTACK_REPO_DIR` | The directory where the repo is mounted (if any) | ### Working directory If `working_dir` is not specified, it defaults to the working directory set in the Docker image. For example, the [default image](#default-image) uses `/dstack/run` as its working directory. If the Docker image does not have a working directory set, `dstack` uses `/` as the `working_dir`. The `working_dir` must be an absolute path. The tilde (`~`) is supported (e.g., `~/my-working-dir`). ### Files Sometimes, when you run a task, you may want to mount local files. This is possible via the [`files`](../reference/dstack.yml/task.md#_files) property. Each entry maps a local directory or file to a path inside the container.
```yaml type: task name: trl-sft files: - .:examples # Maps the directory with `.dstack.yml` to `/examples` - ~/.ssh/id_rsa:/root/.ssh/id_rsa # Maps `~/.ssh/id_rsa` to `/root/.ssh/id_rs python: 3.12 env: - HF_TOKEN - HF_HUB_ENABLE_HF_TRANSFER=1 - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1 ```
If the local path is relative, it’s resolved relative to the configuration file. If the container path is relative, it’s resolved relative to the [working directory](#working-directory). The container path is optional. If not specified, it will be automatically calculated:
```yaml type: task name: trl-sft files: - ../examples # Maps the parent directory of `.dstack.yml` to `/../examples` - ~/.cache/huggingface/token # Maps `~/.cache/huggingface/token` to `/root/.cache/huggingface/token` python: 3.12 env: - HF_TOKEN - HF_HUB_ENABLE_HF_TRANSFER=1 - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1 ```
??? info "File size" Whether its a file or folder, each entry is limited to 2MB. To avoid exceeding this limit, make sure to exclude unnecessary files by listing it via `.gitignore` or `.dstackignore`. The 2MB upload limit can be increased by setting the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable. ### Repos Sometimes, you may want to clone an entire Git repo inside the container. Imagine you have a Git repo (clonned locally) containing an `examples` subdirectory with a `.dstack.yml` file:
```yaml type: task name: trl-sft repos: # Clones the repo from the parent directory (`examples/..`) to `` - .. python: 3.12 env: - HF_TOKEN - HF_HUB_ENABLE_HF_TRANSFER=1 - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1 ```
When you run it, `dstack` clones the repo on the instance, applies your local changes, and mounts it—so the container matches your local repo. The local path can be either relative to the configuration file or absolute. ??? info "Repo directory" By default, `dstack` clones the repo to the [working directory](#working-directory). You can override the repo directory using either a relative or an absolute path:
```yaml type: task name: trl-sft repos: # Clones the repo in the parent directory (`examples/..`) to `/my-repo` - ..:/my-repo python: 3.12 env: - HF_TOKEN - HF_HUB_ENABLE_HF_TRANSFER=1 - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1 ```
> If the repo directory is relative, it is resolved against [working directory](#working-directory). If the repo directory is not empty, the run will fail with a runner error. To override this behavior, you can set `if_exists` to `skip`: ```yaml type: task name: trl-sft repos: - local_path: .. path: /my-repo if_exists: skip python: 3.12 env: - HF_TOKEN - HF_HUB_ENABLE_HF_TRANSFER=1 - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1 ``` ??? info "Repo size" The repo size is not limited. However, local changes are limited to 2MB. To avoid exceeding this limit, exclude unnecessary files using `.gitignore` or `.dstackignore`. You can increase the 2MB limit by setting the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable. ??? info "Repo URL" Sometimes you may want to clone a Git repo within the container without cloning it locally. In this case, simply provide a URL in `repos`:
```yaml type: task name: trl-sft repos: # Clone the repo to `` - https://github.com/dstackai/dstack python: 3.12 env: - HF_TOKEN - HF_HUB_ENABLE_HF_TRANSFER=1 - MODEL=Qwen/Qwen2.5-0.5B - DATASET=stanfordnlp/imdb commands: - uv pip install trl - | trl sft \ --model_name_or_path $MODEL --dataset_name $DATASET \ --num_processes $DSTACK_GPUS_PER_NODE resources: gpu: H100:1 ```
??? info "Private repos" If a Git repo is private, `dstack` will automatically try to use your default Git credentials (from `~/.ssh/config` or `~/.config/gh/hosts.yml`). > If you want to use custom credentials, you can provide them with [`dstack init`](../reference/cli/dstack/init.md). Currently, you can configure up to one repo per run configuration. ### Retry policy By default, if `dstack` can't find capacity, or the task exits with an error, or the instance is interrupted, the run will fail. If you'd like `dstack` to automatically retry, configure the [retry](../reference/dstack.yml/task.md#retry) property accordingly:
```yaml type: task name: train python: 3.12 commands: - uv pip install -r fine-tuning/qlora/requirements.txt - python fine-tuning/qlora/train.py retry: on_events: [no-capacity, error, interruption] # Retry for up to 1 hour duration: 1h ```
If one job of a multi-node task fails with retry enabled, `dstack` will stop all the jobs and resubmit the run. !!! info "Retry duration" The duration period is calculated as a run age for `no-capacity` event and as a time passed since the last `interruption` and `error` for `interruption` and `error` events. ### Priority Be default, submitted runs are scheduled in the order they were submitted. When compute resources are limited, you may want to prioritize some runs over others. This can be done by specifying the [`priority`](../reference/dstack.yml/task.md) property in the run configuration:
```yaml type: task name: train python: 3.12 commands: - uv pip install -r fine-tuning/qlora/requirements.txt - python fine-tuning/qlora/train.py priority: 50 ```
`dstack` tries to provision runs with higher priority first. Note that if a high priority run cannot be scheduled, it does not block other runs with lower priority from scheduling. ### Utilization policy Sometimes it’s useful to track whether a task is fully utilizing all GPUs. While you can check this with [`dstack metrics`](../reference/cli/dstack/metrics.md), `dstack` also lets you set a policy to auto-terminate the run if any GPU is underutilized. Below is an example of a task that auto-terminate if any GPU stays below 10% utilization for 1 hour.
```yaml type: task name: train python: 3.12 commands: - uv pip install -r fine-tuning/qlora/requirements.txt - python fine-tuning/qlora/train.py resources: gpu: H100:8 utilization_policy: min_gpu_utilization: 10 time_window: 1h ```
### Schedule Specify `schedule` to start a task periodically at specific UTC times using the cron syntax:
```yaml type: task name: train python: 3.12 commands: - uv pip install -r fine-tuning/qlora/requirements.txt - python fine-tuning/qlora/train.py resources: gpu: H100:8 schedule: cron: "15 23 * * *" # everyday at 23:15 UTC ```
??? info "Cron syntax" `dstack` supports [POSIX cron syntax](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/crontab.html#tag_20_25_07). One exception is that days of the week are started from Monday instead of Sunday so `0` corresponds to Monday. The month and day of week fields accept abbreviated English month and weekday names (`jan–dec` and `mon–sun`) respectively. A cron expression consists of five fields: ``` ┌───────────── minute (0-59) │ ┌───────────── hour (0-23) │ │ ┌───────────── day of the month (1-31) │ │ │ ┌───────────── month (1-12 or jan-dec) │ │ │ │ ┌───────────── day of the week (0-6 or mon-sun) │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ * * * * * ``` The following operators can be used in any of the fields: | Operator | Description | Example | |----------|-----------------------|-------------------------------------------------------------------------| | `*` | Any value | `0 * * * *` runs every hour at minute 0 | | `,` | Value list separator | `15,45 10 * * *` runs at 10:15 and 10:45 every day. | | `-` | Range of values | `0 1-3 * * *` runs at 1:00, 2:00, and 3:00 every day. | | `/` | Step values | `*/10 8-10 * * *` runs every 10 minutes during the hours 8:00 to 10:59. | ### Spot policy By default, `dstack` uses on-demand instances. However, you can change that via the [`spot_policy`](../reference/dstack.yml/task.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`. --8<-- "docs/concepts/snippets/manage-fleets.ext" !!! info "Reference" Tasks support many more configuration options, incl. [`backends`](../reference/dstack.yml/task.md#backends), [`regions`](../reference/dstack.yml/task.md#regions), [`max_price`](../reference/dstack.yml/task.md#max_price), and [`max_duration`](../reference/dstack.yml/task.md#max_duration), among [others](../reference/dstack.yml/task.md). --8<-- "docs/concepts/snippets/manage-runs.ext" !!! info "What's next?" 1. Read about [dev environments](dev-environments.md) and [services](services.md) 2. Learn how to manage [fleets](fleets.md) 3. Check the [Axolotl](../examples/training/axolotl.md) example # docs/concepts/services.md --- title: Services description: Deploying models and web apps as endpoints --- # Services Services allow you to deploy models or web apps as secure and scalable endpoints. ??? info "Prerequisites" Before running a service, make sure you’ve [installed](../installation.md) the server and CLI, and created a [fleet](fleets.md). ## Apply a configuration First, define a service configuration as a YAML file in your project folder. The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `dev.dstack.yml` are both acceptable). === "NVIDIA"
```yaml type: service name: qwen36 image: lmsysorg/sglang:v0.5.10.post1 commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --mem-fraction-static 0.8 \ --context-length 262144 \ --reasoning-parser qwen3 port: 30000 model: Qwen/Qwen3.6-27B volumes: # Optional instance volume for model and runtime caches - instance_path: /root/.cache path: /root/.cache optional: true resources: shm_size: 16GB gpu: H100:4 ```
=== "AMD"
```yaml type: service name: qwen36 image: lmsysorg/sglang:v0.5.10-rocm720-mi30x commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --mem-fraction-static 0.8 \ --context-length 262144 \ --reasoning-parser qwen3 port: 30000 model: Qwen/Qwen3.6-27B volumes: # Optional instance volume for model and runtime caches - instance_path: /root/.cache path: /root/.cache optional: true resources: cpu: 52.. memory: 896GB.. shm_size: 16GB disk: 450GB.. gpu: MI300X:4 ```
The first startup on MI300X can take longer while SGLang compiles ROCm kernels. To run a service, pass the configuration to [`dstack apply`](../reference/cli/dstack/apply.md):
```shell $ dstack apply -f .dstack.yml Submit the run qwen36? [y/n]: y Provisioning... ---> 100% Service is published at: http://localhost:3000/proxy/services/main/qwen36/ Model Qwen/Qwen3.6-27B is published at: http://localhost:3000/proxy/models/main/ ```
`dstack apply` automatically provisions instances and runs the service. If you do not have a [gateway](gateways.md) created, the service endpoint will be accessible at `/proxy/services///`.
```shell $ curl http://localhost:3000/proxy/services/main/qwen36/v1/chat/completions \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer <user token>' \ -d '{ "model": "Qwen/Qwen3.6-27B", "messages": [ { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming." } ] }' ```
The request and response format depends on the serving framework used by the service. Even for OpenAI-compatible endpoints, the format may vary slightly across frameworks. If [authorization](#authorization) is not disabled, the service endpoint requires the `Authorization` header with `Bearer `. ## Configuration options ### Gateway Here are cases where a service may need a [gateway](gateways.md): * To use [auto-scaling](#replicas-and-scaling) or [rate limits](#rate-limits) * To enable HTTPS for the endpoint and map it to your domain * If your service requires WebSockets * If your service cannot work with a [path prefix](#path-prefix) If you want `dstack` to explicitly validate that a gateway is used, you can set the [`gateway`](../reference/dstack.yml/service.md#gateway) property in the service configuration to `true`. In this case, `dstack` will raise an error during `dstack apply` if a default gateway is not created. You can also set the `gateway` property to the name of a specific gateway, if required. If you have a [gateway](gateways.md) created, the service endpoint will be accessible at `https://./`:
```shell $ curl https://llama31.example.com/v1/chat/completions \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer <user token>' \ -d '{ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [ { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming." } ] }' ```
### Replicas and scaling By default, `dstack` runs a single replica of the service. You can configure the number of replicas as well as the auto-scaling rules. === "NVIDIA"
```yaml type: service name: qwen36-service image: lmsysorg/sglang:v0.5.10.post1 commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --reasoning-parser qwen3 \ --mem-fraction-static 0.8 \ --context-length 262144 port: 30000 model: Qwen/Qwen3.6-27B volumes: # Optional instance volume for model and runtime caches - instance_path: /root/.cache path: /root/.cache optional: true resources: shm_size: 16GB gpu: H100:4 replicas: 1..2 scaling: metric: rps target: 1 ```
=== "AMD"
```yaml type: service name: qwen36-service image: lmsysorg/sglang:v0.5.10-rocm720-mi30x commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --reasoning-parser qwen3 \ --mem-fraction-static 0.8 \ --context-length 262144 port: 30000 model: Qwen/Qwen3.6-27B volumes: # Optional instance volume for model and runtime caches - instance_path: /root/.cache path: /root/.cache optional: true resources: cpu: 52.. memory: 896GB.. shm_size: 16GB disk: 450GB.. gpu: MI300X:4 replicas: 1..2 scaling: metric: rps target: 1 ```
The [`replicas`](../reference/dstack.yml/service.md#replicas) property can be a number or a range. The [`metric`](../reference/dstack.yml/service.md#metric) property of [`scaling`](../reference/dstack.yml/service.md#scaling) only supports the `rps` metric (requests per second). In this case `dstack` adjusts the number of replicas (scales up or down) automatically based on the load. Setting the minimum number of replicas to `0` allows the service to scale down to zero when there are no requests. > The `scaling` property requires creating a [gateway](gateways.md). ??? info "Replica groups" A service can include multiple replica groups. Each group can define its own `commands`, `resources` requirements, and `scaling` rules.
```yaml type: service name: llama-8b-service image: lmsysorg/sglang:v0.5.10.post1 env: - MODEL_ID=deepseek-ai/DeepSeek-R1-Distill-Llama-8B replicas: - count: 1..2 scaling: metric: rps target: 10 commands: - | python -m sglang.launch_server \ --model-path $MODEL_ID \ --port 8000 \ --trust-remote-code resources: gpu: 48GB - count: 1..4 scaling: metric: rps target: 5 commands: - | python -m sglang.launch_server \ --model-path $MODEL_ID \ --port 8000 \ --trust-remote-code resources: gpu: 24GB port: 8000 model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B ```
> Properties such as `regions`, `port`, `image`, `env` and some other cannot be configured per replica group. This support is coming soon. ### PD disaggregation Since 0.20.17, `dstack` supports serving a model using Prefill-Decode disaggregation. To use it, configure three replica groups: one for the router, one for prefill workers, and one for decode workers. `dstack` integrates with two routers for PD disaggregation: [Shepherd Model Gateway (SMG)](https://docs.sglang.io/advanced_features/sgl_model_gateway.html) and [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo). Below is an example for running `zai-org/GLM-4.5-Air-FP8`: === "SMG"
```yaml type: service name: prefill-decode image: lmsysorg/sglang:v0.5.10.post1 env: - HF_TOKEN - MODEL_ID=zai-org/GLM-4.5-Air-FP8 replicas: - count: 1 # For now replica group with router must have count: 1 commands: - pip install smg - | smg launch \ --host 0.0.0.0 \ --port 8000 \ --pd-disaggregation \ --prefill-policy cache_aware resources: cpu: 4 router: type: sglang - count: 1..4 scaling: metric: rps target: 3 commands: - | python -m sglang.launch_server \ --model-path $MODEL_ID \ --disaggregation-mode prefill \ --disaggregation-transfer-backend nixl \ --port 8000 \ --disaggregation-bootstrap-port 8998 resources: gpu: H200 - count: 1..8 scaling: metric: rps target: 2 commands: - | python -m sglang.launch_server \ --model-path $MODEL_ID \ --disaggregation-mode decode \ --disaggregation-transfer-backend nixl \ --port 8000 resources: gpu: H200 port: 8000 model: zai-org/GLM-4.5-Air-FP8 # Custom probe is required for PD disaggregation. probes: - type: http url: /health interval: 15s ```
> With the `sglang` router, you can use SGLang prefill and decode workers. Support for vLLM and TensorRT-LLM workers is coming soon. === "Dynamo"
```yaml type: service name: dynamo-pd env: - HF_TOKEN - MODEL_ID=zai-org/GLM-4.5-Air-FP8 replicas: - count: 1 docker: true commands: - apt-get update - apt-get install -y python3-dev python3-venv - python3 -m venv ~/dyn-venv - source ~/dyn-venv/bin/activate - pip install -U pip - pip install "ai-dynamo[sglang]==1.1.1" - git clone https://github.com/ai-dynamo/dynamo.git # Brings up the NATS / etcd compose stack and runs the Dynamo HTTP frontend. - docker compose -f dynamo/deploy/docker-compose.yml up -d - | python3 -m dynamo.frontend \ --http-host 0.0.0.0 --http-port 8000 \ --discovery-backend etcd --router-mode kv \ --kv-cache-block-size 64 resources: cpu: 4 router: type: dynamo - count: 1..4 scaling: metric: rps target: 3 python: "3.12" nvcc: true commands: # dstack injects DSTACK_ROUTER_INTERNAL_IP after the router replica # is provisioned. Compose the etcd/NATS endpoints from it. - export ETCD_ENDPOINTS="http://$DSTACK_ROUTER_INTERNAL_IP:2379" - export NATS_SERVER="nats://$DSTACK_ROUTER_INTERNAL_IP:4222" # Set to enable /health endpoint required by dstack probes. - export DYN_SYSTEM_PORT="8000" # Wait until the router's etcd and NATS ports are actually accepting connections. - | until (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/2379) 2>/dev/null \ && (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/4222) 2>/dev/null; do echo "waiting for etcd/NATS on $DSTACK_ROUTER_INTERNAL_IP..."; sleep 3 done - pip install "ai-dynamo[sglang]==1.1.1" - | python3 -m dynamo.sglang \ --model-path $MODEL_ID --served-model-name $MODEL_ID \ --discovery-backend etcd --host 0.0.0.0 \ --page-size 64 \ --disaggregation-mode prefill --disaggregation-transfer-backend nixl resources: gpu: H200 - count: 1..8 scaling: metric: rps target: 2 python: "3.12" nvcc: true commands: - export ETCD_ENDPOINTS="http://$DSTACK_ROUTER_INTERNAL_IP:2379" - export NATS_SERVER="nats://$DSTACK_ROUTER_INTERNAL_IP:4222" - export DYN_SYSTEM_PORT="8000" - | until (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/2379) 2>/dev/null \ && (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/4222) 2>/dev/null; do echo "waiting for etcd/NATS on $DSTACK_ROUTER_INTERNAL_IP..."; sleep 3 done - pip install "ai-dynamo[sglang]==1.1.1" - | python3 -m dynamo.sglang \ --model-path $MODEL_ID --served-model-name $MODEL_ID \ --discovery-backend etcd --host 0.0.0.0 \ --page-size 64 \ --disaggregation-mode decode --disaggregation-transfer-backend nixl resources: gpu: H200 port: 8000 model: zai-org/GLM-4.5-Air-FP8 # Custom probe is required for PD disaggregation. probes: - type: http url: /health interval: 15s ```
> With the the `dynamo` router, you can use SGLang, vLLM, and TensorRT-LLM prefill and decode workers. !!! info "Cluster" PD disaggregation requires the service to run in a fleet with `placement` set to `cluster`, because the replicas require an interconnect between instances. While the prefill and decode replicas run on GPUs, the router replica requires a CPU instance in the same cluster. ### Authorization By default, the service enables authorization, meaning the service endpoint requires a `dstack` user token. This can be disabled by setting `auth` to `false`.
```yaml type: service name: http-server-service # Disable authorization auth: false python: 3.12 commands: - python3 -m http.server port: 8000 ```
### Probes Configure one or more HTTP probes to periodically check the health of the service.
```yaml type: service name: my-service port: 80 image: my-app:latest probes: - type: http url: /health interval: 15s ```
You can track probe statuses in `dstack ps --verbose`.
```shell $ dstack ps --verbose NAME BACKEND STATUS PROBES SUBMITTED my-service deployment=1 running 11 mins ago replica=0 job=0 deployment=0 aws (us-west-2) running ✓ 11 mins ago replica=1 job=0 deployment=1 aws (us-west-2) running × 1 min ago ```
??? info "Status" The following symbols are used for probe statuses: - `×` — the last probe execution failed. - `~` — the last probe execution succeeded, but the [`ready_after`](../reference/dstack.yml/service.md#ready_after) threshold is not yet reached. - `✓` — the last `ready_after` probe executions succeeded. If multiple probes are configured for the service, their statuses are displayed in the order in which the probes appear in the configuration. Probes are executed for each service replica while the replica is `running`. A probe execution is considered successful if the replica responds with a `2xx` status code. Probe statuses do not affect how `dstack` handles replicas, except during [rolling deployments](#rolling-deployment). ??? info "HTTP request configuration" You can configure the HTTP request method, headers, and other properties. To include secret values in probe requests, use environment variable interpolation, which is enabled for the `url`, `headers[i].value`, and `body` properties.
```yaml type: service name: my-service port: 80 image: my-app:latest env: - PROBES_API_KEY probes: - type: http method: post url: /check-health headers: - name: X-API-Key value: ${{ env.PROBES_API_KEY }} - name: Content-Type value: application/json body: '{"level": 2}' timeout: 20s ```
??? info "Model" If you set the [`model`](#model) property but don't explicitly configure `probes`, `dstack` automatically configures a default probe that tests the model using the `/v1/chat/completions` API. To disable probes entirely when `model` is set, explicitly set `probes` to an empty list. See the [reference](../reference/dstack.yml/service.md#probes) for more probe configuration options. ### Path prefix { #path-prefix } If your `dstack` project doesn't have a [gateway](gateways.md), services are hosted with the `/proxy/services///` path prefix in the URL. When running web apps, you may need to set some app-specific settings so that browser-side scripts and CSS work correctly with the path prefix.
```yaml type: service name: dash gateway: false auth: false # Do not strip the path prefix strip_prefix: false env: # Configure Dash to work with a path prefix # Replace `main` with your dstack project name - DASH_ROUTES_PATHNAME_PREFIX=/proxy/services/main/dash/ commands: - uv pip install dash # Assuming the Dash app is in your repo at app.py - python app.py port: 8050 ```
By default, `dstack` strips the prefix before forwarding requests to your service, so to the service it appears as if the prefix isn't there. This allows some apps to work out of the box. If your app doesn't expect the prefix to be stripped, set [`strip_prefix`](../reference/dstack.yml/service.md#strip_prefix) to `false`. If your app cannot be configured to work with a path prefix, you can host it on a dedicated domain name by setting up a [gateway](gateways.md). ### Rate limits If you have a [gateway](gateways.md), you can configure rate limits for your service using the [`rate_limits`](../reference/dstack.yml/service.md#rate_limits) property.
```yaml type: service image: my-app:latest port: 80 rate_limits: # For /api/auth/* - 1 request per second, no bursts - prefix: /api/auth/ rps: 1 # For other URLs - 4 requests per second + bursts of up to 9 requests - rps: 4 burst: 9 ```
The rps limit sets the max requests per second, tracked in milliseconds (e.g., `rps: 4` means 1 request every 250 ms). Use `burst` to allow short spikes while keeping the average within `rps`. Limits apply to the whole service (all replicas) and per client (by IP). Clients exceeding the limit get a 429 error. ??? info "Partitioning key" Instead of partitioning requests by client IP address, you can choose to partition by the value of a header.
```yaml type: service image: my-app:latest port: 80 rate_limits: - rps: 4 burst: 9 # Apply to each user, as determined by the `Authorization` header key: type: header header: Authorization ```
### Model If the service runs a model with an OpenAI-compatible interface, you can set the [`model`](#model) property to make the model accessible through `dstack`'s chat UI on the `Models` page. In this case, `dstack` will use the service's `/v1/chat/completions` service. When `model` is set, `dstack` automatically configures [`probes`](#probes) to verify model health. To customize or disable this, set `probes` explicitly. ### Resources If you specify memory size, you can either specify an explicit size (e.g. `24GB`) or a range (e.g. `24GB..`, or `24GB..80GB`, or `..80GB`).
```yaml type: service name: llama31-service python: 3.12 env: - HF_TOKEN - MODEL_ID=meta-llama/Meta-Llama-3.1-8B-Instruct - MAX_MODEL_LEN=4096 commands: - uv pip install vllm - | vllm serve $MODEL_ID --max-model-len $MAX_MODEL_LEN --tensor-parallel-size $DSTACK_GPUS_NUM port: 8000 resources: # 16 or more x86_64 cores cpu: 16.. # 2 GPUs of 80GB gpu: 80GB:2 # Minimum disk size disk: 200GB ```
The `cpu` property lets you set the architecture (`x86` or `arm`) and core count — e.g., `x86:16` (16 x86 cores), `arm:8..` (at least 8 ARM cores). If not set, `dstack` infers it from the GPU or defaults to `x86`. The `gpu` property lets you specify vendor, model, memory, and count — e.g., `nvidia` (one NVIDIA GPU), `A100` (one A100), `A10G,A100` (either), `A100:80GB` (one 80GB A100), `A100:2` (two A100), `24GB..40GB:2` (two GPUs with 24–40GB), `A100:40GB:2` (two 40GB A100s). If vendor is omitted, `dstack` infers it from the model or defaults to `nvidia`. ??? info "Shared memory" If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure `shm_size`, e.g. set it to `16GB`. > If you’re unsure which offers (hardware configurations) are available from the configured backends, use the > [`dstack offer`](../reference/cli/dstack/offer.md#list-gpu-offers) command to list them. ### Docker #### Default image If you don't specify `image`, `dstack` uses its [base](https://github.com/dstackai/dstack/tree/master/docker/base) Docker image pre-configured with `uv`, `python`, `pip`, essential CUDA drivers, `mpirun`, and NCCL tests (under `/opt/nccl-tests/build`). Set the `python` property to pre-install a specific version of Python.
```yaml type: service name: http-server-service python: 3.12 commands: - python3 -m http.server port: 8000 ```
#### NVCC By default, the base Docker image doesn’t include `nvcc`, which is required for building custom CUDA kernels. If you need `nvcc`, set the [`nvcc`](../reference/dstack.yml/dev-environment.md#nvcc) property to true.
```yaml type: service name: http-server-service python: 3.12 nvcc: true commands: - python3 -m http.server port: 8000 ```
#### Custom image If you want, you can specify your own Docker image via `image`.
```yaml type: service name: http-server-service image: python commands: - python3 -m http.server port: 8000 ```
#### Docker in Docker Set `docker` to `true` to enable the `docker` CLI in your service, e.g., to run Docker images or use Docker Compose.
```yaml type: service name: compose-service auth: false docker: true commands: - | cat > compose.yaml <<'EOF' services: web: image: python:3.11-slim command: python -m http.server 9000 ports: - "9000:9000" EOF - docker compose up port: 9000 ```
Cannot be used with `python` or `image`. Not supported on `runpod`, `vastai`, or `kubernetes`. #### Privileged mode To enable privileged mode, set [`privileged`](../reference/dstack.yml/dev-environment.md#privileged) to `true`. Not supported with `runpod`, `vastai`, and `kubernetes`. #### Private registry Use the [`registry_auth`](../reference/dstack.yml/dev-environment.md#registry_auth) property to provide credentials for a private Docker registry. ```yaml type: service name: serve-distill-deepseek env: - NGC_API_KEY - NIM_MAX_MODEL_LEN=4096 image: nvcr.io/nim/deepseek-ai/deepseek-r1-distill-llama-8b registry_auth: username: $oauthtoken password: ${{ env.NGC_API_KEY }} port: 8000 model: deepseek-ai/deepseek-r1-distill-llama-8b resources: gpu: H100:1 ``` ### Environment variables
```yaml type: service name: llama-2-7b-service python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB ```
> If you don't assign a value to an environment variable (see `HF_TOKEN` above), `dstack` will require the value to be passed via the CLI or set in the current process. ??? info "System environment variables" The following environment variables are available in any run by default: | Name | Description | |-------------------------|--------------------------------------------------| | `DSTACK_RUN_NAME` | The name of the run | | `DSTACK_REPO_ID` | The ID of the repo | | `DSTACK_GPUS_NUM` | The total number of GPUs in the run | | `DSTACK_WORKING_DIR` | The working directory of the run | | `DSTACK_REPO_DIR` | The directory where the repo is mounted (if any) | ### Working directory If `working_dir` is not specified, it defaults to the working directory set in the Docker image. For example, the [default image](#default-image) uses `/dstack/run` as its working directory. If the Docker image does not have a working directory set, `dstack` uses `/` as the `working_dir`. The `working_dir` must be an absolute path. The tilde (`~`) is supported (e.g., `~/my-working-dir`). ### Files Sometimes, when you run a service, you may want to mount local files. This is possible via the [`files`](../reference/dstack.yml/task.md#_files) property. Each entry maps a local directory or file to a path inside the container.
```yaml type: service name: llama-2-7b-service files: - .:examples # Maps the directory with `.dstack.yml` to `/examples` - ~/.ssh/id_rsa:/root/.ssh/id_rsa # Maps `~/.ssh/id_rsa` to `/root/.ssh/id_rsa` python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB ```
If the local path is relative, it’s resolved relative to the configuration file. If the container path is relative, it’s resolved relative to the [working directory](#working-directory). The container path is optional. If not specified, it will be automatically calculated:
```yaml type: service name: llama-2-7b-service files: - ../examples # Maps the parent directory of `.dstack.yml` to `/../examples` - ~/.ssh/id_rsa # Maps `~/.ssh/id_rsa` to `/root/.ssh/id_rsa` python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB ```
??? info "File size" Whether its a file or folder, each entry is limited to 2MB. To avoid exceeding this limit, make sure to exclude unnecessary files by listing it via `.gitignore` or `.dstackignore`. The 2MB upload limit can be increased by setting the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable. ### Repos Sometimes, you may want to clone an entire Git repo inside the container. Imagine you have a Git repo (clonned locally) containing an `examples` subdirectory with a `.dstack.yml` file:
```yaml type: service name: llama-2-7b-service repos: # Clones the repo from the parent directory (`examples/..`) to `` - .. python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB ```
When you run it, `dstack` clones the repo on the instance, applies your local changes, and mounts it—so the container matches your local repo. The local path can be either relative to the configuration file or absolute. ??? info "Repo directory" By default, `dstack` clones the repo to the [working directory](#working-directory). You can override the repo directory using either a relative or an absolute path:
```yaml type: service name: llama-2-7b-service repos: # Clones the repo in the parent directory (`examples/..`) to `/my-repo` - ..:/my-repo python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB ```
> If the repo directory is relative, it is resolved against [working directory](#working-directory). If the repo directory is not empty, the run will fail with a runner error. To override this behavior, you can set `if_exists` to `skip`: ```yaml type: service name: llama-2-7b-service repos: - local_path: .. path: /my-repo if_exists: skip python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB ``` ??? info "Repo size" The repo size is not limited. However, local changes are limited to 2MB. To avoid exceeding this limit, exclude unnecessary files using `.gitignore` or `.dstackignore`. You can increase the 2MB limit by setting the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable. ??? info "Repo URL" Sometimes you may want to clone a Git repo within the container without cloning it locally. In this case, simply provide a URL in `repos`:
```yaml type: service name: llama-2-7b-service repos: # Clone the repo to `` - https://github.com/dstackai/dstack python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB ```
??? info "Private repos" If a Git repo is private, `dstack` will automatically try to use your default Git credentials (from `~/.ssh/config` or `~/.config/gh/hosts.yml`). > If you want to use custom credentials, you can provide them with [`dstack init`](../reference/cli/dstack/init.md). Currently, you can configure up to one repo per run configuration. ### Retry policy By default, if `dstack` can't find capacity, or the service exits with an error, or the instance is interrupted, the run will fail. If you'd like `dstack` to automatically retry, configure the [retry](../reference/dstack.yml/service.md#retry) property accordingly:
```yaml type: service image: my-app:latest port: 80 retry: on_events: [no-capacity, error, interruption] # Retry for up to 1 hour duration: 1h ```
If one replica of a multi-replica service fails with retry enabled, `dstack` will resubmit only the failed replica while keeping active replicas running. !!! info "Retry duration" The duration period is calculated as a run age for `no-capacity` event and as a time passed since the last `interruption` and `error` for `interruption` and `error` events. ### Spot policy By default, `dstack` uses on-demand instances. However, you can change that via the [`spot_policy`](../reference/dstack.yml/service.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`. ### Utilization policy Sometimes it’s useful to track whether a service is fully utilizing all GPUs. While you can check this with [`dstack metrics`](../reference/cli/dstack/metrics.md), `dstack` also lets you set a policy to auto-terminate the run if any GPU is underutilized. Below is an example of a service that auto-terminate if any GPU stays below 10% utilization for 1 hour.
```yaml type: service name: llama-2-7b-service python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB utilization_policy: min_gpu_utilization: 10 time_window: 1h ```
### Schedule Specify `schedule` to start a service periodically at specific UTC times using the cron syntax:
```yaml type: service name: llama-2-7b-service python: 3.12 env: - HF_TOKEN - MODEL=NousResearch/Llama-2-7b-chat-hf commands: - uv pip install vllm - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 resources: gpu: 24GB schedule: cron: "0 8 * * mon-fri" # at 8:00 UTC from Monday through Friday ```
The `schedule` property can be combined with `max_duration` or `utilization_policy` to shutdown the service automatically when it's not needed. ??? info "Cron syntax" `dstack` supports [POSIX cron syntax](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/crontab.html#tag_20_25_07). One exception is that days of the week are started from Monday instead of Sunday so `0` corresponds to Monday. The month and day of week fields accept abbreviated English month and weekday names (`jan–dec` and `mon–sun`) respectively. A cron expression consists of five fields: ``` ┌───────────── minute (0-59) │ ┌───────────── hour (0-23) │ │ ┌───────────── day of the month (1-31) │ │ │ ┌───────────── month (1-12 or jan-dec) │ │ │ │ ┌───────────── day of the week (0-6 or mon-sun) │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ * * * * * ``` The following operators can be used in any of the fields: | Operator | Description | Example | |----------|-----------------------|-------------------------------------------------------------------------| | `*` | Any value | `0 * * * *` runs every hour at minute 0 | | `,` | Value list separator | `15,45 10 * * *` runs at 10:15 and 10:45 every day. | | `-` | Range of values | `0 1-3 * * *` runs at 1:00, 2:00, and 3:00 every day. | | `/` | Step values | `*/10 8-10 * * *` runs every 10 minutes during the hours 8:00 to 10:59. | --8<-- "docs/concepts/snippets/manage-fleets.ext" !!! info "Reference" Services support many more configuration options, incl. [`backends`](../reference/dstack.yml/service.md#backends), [`regions`](../reference/dstack.yml/service.md#regions), [`max_price`](../reference/dstack.yml/service.md#max_price), and among [others](../reference/dstack.yml/service.md). ## Rolling deployment To deploy a new version of a service that is already `running`, use `dstack apply`. `dstack` will automatically detect changes and suggest a rolling deployment update.
```shell $ dstack apply -f my-service.dstack.yml Active run my-service already exists. Detected changes that can be updated in-place: - Repo state (branch, commit, or other) - File archives - Configuration properties: - env - files Update the run? [y/n]: ```
If approved, `dstack` gradually updates the service replicas. To update a replica, `dstack` starts a new replica, waits for it to become `running` and for all of its [probes](#probes) to pass, then terminates the old replica. This process is repeated for each replica, one at a time. You can track the progress of rolling deployment in both `dstack apply` or `dstack ps`. Older replicas have lower `deployment` numbers; newer ones have higher. ```shell $ dstack apply -f my-service.dstack.yml ⠋ Launching my-service... NAME BACKEND PRICE STATUS SUBMITTED my-service deployment=1 running 11 mins ago replica=0 job=0 deployment=0 aws (us-west-2) $0.0026 terminating 11 mins ago replica=1 job=0 deployment=1 aws (us-west-2) $0.0026 running 1 min ago ``` The rolling deployment stops when all replicas are updated or when a new deployment is submitted. ??? info "Supported properties" Rolling deployment supports changes to the following properties: `port`, `probes`, `resources`, `volumes`, `docker`, `files`, `image`, `user`, `privileged`, `entrypoint`, `working_dir`, `python`, `nvcc`, `single_branch`, `env`, `shell`, `commands`, as well as changes to [repo](#repos) or [file](#files) contents. Changes to `replicas` and `scaling` can be applied without redeploying replicas. Changes to other properties require a full service restart. To trigger a rolling deployment when no properties have changed (e.g., after updating [secrets](secrets.md) or to restart all replicas), make a minor config change, such as adding a dummy [environment variable](#environment-variables). --8<-- "docs/concepts/snippets/manage-runs.ext" !!! info "What's next?" 1. Read about [dev environments](dev-environments.md) and [tasks](tasks.md) 2. Learn how to manage [fleets](fleets.md) 3. See how to set up [gateways](gateways.md) 4. Check the [vLLM](../examples/inference/vllm.md) and [NIM](../examples/inference/nim.md) examples # docs/concepts/volumes.md --- title: Volumes description: Managing persistent data storage --- # Volumes Volumes enable data persistence between runs of dev environments, tasks, and services. `dstack` supports two kinds of volumes: * [Network volumes](#network-volumes) — provisioned via backends and mounted to specific container directories. Ideal for persistent storage. * [Instance volumes](#instance-volumes) — bind directories on the host instance to container directories. Useful as a cache for cloud fleets or for persistent storage with SSH fleets. ## Network volumes > Network volumes are currently supported for the `aws`, `gcp`, `runpod`, and `kubernetes` backends. ### Apply a configuration First, define a volume configuration as a YAML file in your project folder. The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `volume.dstack.yml` are both acceptable).
```yaml type: volume # A name of the volume name: my-volume # Volumes are bound to a specific backend and region backend: aws region: eu-central-1 # Required size size: 100GB ```
If you use this configuration, `dstack` will create a new volume based on the specified options. ??? info "Kubernetes" With the `kubernetes` backend, you don't have to specify `region`, but you can optionally specify `storage_class_name` and/or `access_modes`:
```yaml type: volume backend: kubernetes name: my-volume size: 100GB ```
This automatically creates a `PersistentVolumeClaim` and associates it with the volume. If you don't specify `storage_class_name`, the decision is delegated to the `DefaultStorageClass` admission controller, if enabled. If you don't specify `access_modes`, it defaults to `[ReadWriteOnce]`. To attach volumes to multiple runs at the same time, set it to `[ReadWriteMany]` or `[ReadWriteMany, ReadOnlyMany]`. To create, update, or register the volume, pass the volume configuration to `dstack apply`:
```shell $ dstack apply -f volume.dstack.yml Volume my-volume does not exist yet. Create the volume? [y/n]: y NAME BACKEND REGION STATUS CREATED my-volume aws eu-central-1 submitted now ```
Once created, the volume can be attached to dev environments, tasks, and services. > When creating a new network volume, `dstack` automatically creates an `ext4` filesystem on it. #### Register existing volumes If you prefer not to create a new volume but to reuse an existing one (e.g., created manually), you can specify its ID via [`volume_id`](../reference/dstack.yml/volume.md#volume_id). In this case, `dstack` will register the specified volume so that you can use it with dev environments, tasks, and services.
```yaml type: volume # The name of the volume name: my-volume # Volumes are bound to a specific backend and region backend: aws region: eu-central-1 # The ID of the volume in AWS volume_id: vol1235 ```
If you register an existing volume, you must ensure the volume already has a filesystem. ??? info "Kubernetes" With the `kubernetes` backend, to reuse an existing `PersistentVolumeClaim`, specify its name in `claim_name`:
```yaml type: volume backend: kubernetes name: my-volume claim_name: existing-pvc ```
For all volume configuration options, refer to the [reference](../reference/dstack.yml/volume.md). ### Attach a volume { #attach-network-volume } Dev environments, tasks, and services let you attach any number of network volumes. To attach a network volume, simply specify its name using the `volumes` property and specify where to mount its contents:
```yaml type: dev-environment # A name of the dev environment name: vscode-vol ide: vscode # Map the name of the volume to any path volumes: - name: my-volume path: /volume_data # You can also use the short syntax in the `name:path` form # volumes: # - my-volume:/volume_data ```
Once you run this configuration, the contents of the volume will be attached to `/volume_data` inside the dev environment, and its contents will persist across runs. ??? info "Multiple regions or backends" If you're unsure in advance which region or backend you'd like to use (or which is available), you can specify multiple volumes for the same path.
```yaml volumes: - name: [my-aws-eu-west-1-volume, my-aws-us-east-1-volume] path: /volume_data ```
`dstack` will attach one of the volumes based on the region and backend of the run. ??? info "Distributed tasks" When using single-attach volumes such as AWS EBS with distributed tasks, you can attach different volumes to different nodes using `dstack` variable interpolation:
```yaml type: task nodes: 8 commands: - ... volumes: - name: data-volume-${{ dstack.node_rank }} path: /volume_data ```
This way, every node will use its own volume. Tip: To create volumes for all nodes using one volume configuration, specify volume name with `-n`: ```shell $ for i in {0..7}; do dstack apply -f vol.dstack.yml -n data-volume-$i -y; done ``` ### Detach a volume { #detach-network-volume } `dstack` automatically detaches volumes from instances when a run stops. !!! info "Force detach" In some clouds such as AWS a volume may stuck in the detaching state. To fix this, you can abort the run, and `dstack` will force detach the volume. `dstack` will also force detach the stuck volume automatically after `stop_duration`. Note that force detaching a volume is a last resort measure and may corrupt the file system. Contact your cloud support if you experience volumes getting stuck in the detaching state. ### Manage volumes { #manage-network-volumes } #### List volumes The [`dstack volume list`](../reference/cli/dstack/volume.md#dstack-volume-list) command lists created and registered volumes:
```shell $ dstack volume list NAME BACKEND REGION STATUS CREATED my-volume aws eu-central-1 active 3 weeks ago ```
#### Delete volumes When the volume isn't attached to any active dev environment, task, or service, you can delete it by passing the volume configuration to `dstack delete`:
```shell $ dstack delete -f vol.dstack.yaml ```
Alternatively, you can delete a volume by passing the volume name to `dstack volume delete`. If the volume was created using `dstack`, it will be physically destroyed along with the data. If you've registered an existing volume, it will be de-registered with `dstack` but will keep the data. ### FAQs ??? info "Can I use network volumes across backends?" Since volumes are backed up by cloud network disks, you can only use them within the same cloud. If you need to access data across different backends, you should either use object storage or replicate the data across multiple volumes. ??? info "Can I use network volumes across regions?" Typically, network volumes are associated with specific regions, so you can't use them in other regions. Often, volumes are also linked to availability zones, but some providers support volumes that can be used across different availability zones within the same region. If you don't want to limit a run to one particular region, you can create different volumes for different regions and specify them for the same mount point as [documented above](#attach-network-volume). ??? info "Can I attach network volumes to multiple runs or instances?" You can mount a volume in multiple runs. This feature is currently supported only by the `runpod` backend. ## Instance volumes Instance volumes allow mapping any directory on the instance where the run is executed to any path inside the container. This means that the data in instance volumes is persisted only if the run is executed on the same instance. > Instance volumes are currently supported for all backends except `runpod` and `vastai`, and can also be used with [SSH fleets](fleets.md#ssh-fleets). ### Attach a volume A run can configure any number of instance volumes. To attach an instance volume, specify the `instance_path` and `path` in the `volumes` property:
```yaml type: dev-environment # A name of the dev environment name: vscode-vol ide: vscode # Map the instance path to any container path volumes: - instance_path: /mnt/volume path: /volume_data # You can also use the short syntax in the `instance_path:path` form # volumes: # - /mnt/volume:/volume_data ```
Since persistence isn't guaranteed (instances may be interrupted or runs may occur on different instances), use instance volumes only for caching or with directories manually mounted to network storage. ??? info "Optional volumes" If the volume is not critical for your workload, you can mark it as `optional`.
```yaml type: task volumes: - instance_path: /dstack-cache path: /root/.cache/ optional: true ``` Configurations with optional volumes can run in any backend, but the volume is only mounted if the selected backend supports it.
### Use instance volumes for caching For example, if a run regularly installs packages with `pip install`, you can mount the `/root/.cache/pip` folder inside the container to a folder on the instance for reuse.
```yaml type: task volumes: - /dstack-cache/pip:/root/.cache/pip ```
### Use instance volumes with SSH fleets If you control the instances (e.g. they are on-prem servers configured via [SSH fleets](fleets.md#ssh-fleets)), you can mount network storage (e.g., NFS or SMB) and use the mount points as instance volumes. For example, if you mount a network storage to `/mnt/nfs-storage` on all hosts of your SSH fleet, you can map this directory via instance volumes and be sure the data is persisted.
```yaml type: task volumes: - /mnt/nfs-storage:/storage ```
# docs/concepts/gateways.md --- title: Gateways description: Managing ingress traffic and endpoints for services --- # Gateways Gateways manage ingress traffic for running [services](services.md), handle auto-scaling and rate limits, enable HTTPS, and allow you to configure a custom domain. ## Apply a configuration First, define a gateway configuration as a YAML file in your project folder. The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `gateway.dstack.yml` are both acceptable).
```yaml type: gateway # A name of the gateway name: example-gateway # Gateways are bound to a specific backend and region backend: aws region: eu-west-1 # This domain will be used to access the endpoint domain: example.com ```
To create or update the gateway, simply call the [`dstack apply`](../reference/cli/dstack/apply.md) command:
```shell $ dstack apply -f gateway.dstack.yml The example-gateway doesn't exist. Create it? [y/n]: y Provisioning... ---> 100% BACKEND REGION NAME HOSTNAME DOMAIN DEFAULT STATUS aws eu-west-1 example-gateway example.com ✓ submitted ```
## Configuration options ### Domain A gateway requires a `domain` to be specified in the configuration before creation. The domain is used to generate service endpoints (e.g. `.`). Once the gateway is created and assigned a hostname, configure your DNS by adding a wildcard record for `*.` (e.g. `*.example.com`). The record should point to the gateway's hostname and should be of type `A` if the hostname is an IP address (most cases), or of type `CNAME` if the hostname is another domain (some private gateways and Kubernetes). ??? info "Project name interpolation" You can use the `${{ run.project_name }}` variable to include the service’s project name in the domain name. This is especially useful when [exporting](exports.md) the gateway to multiple projects, as it ensures each importer receives a unique domain name. ```yaml type: gateway name: global-gateway backend: aws region: eu-west-1 domain: ${{ run.project_name }}.mycompany.example ``` ### Backend You can create gateways with the `aws`, `azure`, `gcp`, or `kubernetes` backends, but that does not limit where services run. A gateway can use one backend while services run on any other backend supported by dstack, including backends where gateways themselves cannot be created. ??? info "Kubernetes" Gateways in `kubernetes` backend require an external load balancer. Managed Kubernetes solutions usually include a load balancer. For self-hosted Kubernetes, you must provide a load balancer by yourself. ### Router > In previous releases, `dstack` allowed configuring `router` the gateway, which was required for PD disaggregation. Since 0.20.17, the `router` configuration has moved to [services](services.md#pd-disaggregation), and the gateway no longer needs to configure router. ### Certificate By default, when you run a service with a gateway, `dstack` provisions an SSL certificate via Let's Encrypt for the configured domain. This automatically enables HTTPS for the service endpoint. If you disable [public IP](#public-ip) (e.g. to make the gateway private) or if you simply don't need HTTPS, you can set `certificate` to `null`. > Note, by default services set [`https`](../reference/dstack.yml/service.md#https) to `true` which requires a certificate. You can set `https` to `auto` to detect if the gateway supports HTTPS or not automatically. ??? info "Certificate types" `dstack` supports the following certificate types: * `lets-encrypt` (default) — Automatic certificates via [Let's Encrypt](https://letsencrypt.org/). Requires a [public IP](#public-ip). * `acm` — Certificates managed by [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/). AWS-only. TLS is terminated at the load balancer, not at the gateway, and HTTP requests are redirected to HTTPS by the ALB. Requires a VPC with at least two subnets in different availability zones to provision a load balancer. If `public_ip: False`, subnets must be private and have a route to NAT gateway. * `null` — No certificate. Services will use HTTP. ### Public IP If you don't need a public IP for the gateway, you can set `public_ip` to `false` (the default is `true`), making the gateway private. Private gateways are currently supported in `aws` and `gcp` backends.
```yaml type: gateway name: private-gateway backend: aws region: eu-west-1 domain: example.com public_ip: false certificate: null ```
### Instance type By default, `dstack` provisions a small, low-cost instance for the gateway. If you expect to run high-traffic services, you can configure a larger instance type using the `instance_type` property.
```yaml type: gateway name: example-gateway backend: aws region: eu-west-1 instance_type: t3.large domain: example.com ```
!!! info "Reference" For all gateway configuration options, refer to the [reference](../reference/dstack.yml/gateway.md). ## Export gateways Gateways can be exported to other projects, allowing those projects to use the exported gateways for running services. See [Exports](exports.md) for more details. ## Manage gateways ### List gateways The [`dstack gateway list`](../reference/cli/dstack/gateway.md#dstack-gateway-list) command lists existing gateways and their status. ### Delete a gateway To delete a gateway, pass the gateway configuration to [`dstack delete`](../reference/cli/dstack/delete.md):
```shell $ dstack delete -f examples/inference/gateway.dstack.yml ```
Alternatively, you can delete a gateway by passing the gateway name to `dstack gateway delete`. [//]: # (TODO: Elaborate on default) [//]: # (TODO: ## Accessing endpoints) !!! info "What's next?" 1. See [services](services.md) on how to run services # docs/concepts/secrets.md --- title: Secrets description: Managing sensitive values and credentials --- # Secrets Secrets allow centralized management of sensitive values such as API keys and credentials. They are project-scoped, managed by project admins, and can be referenced in run configurations to pass sensitive values to runs in a secure manner. !!! info "Secrets encryption" By default, secrets are stored in plaintext in the DB. Configure [server encryption](../guides/server-deployment.md#encryption) to store secrets encrypted. ## Manage secrets ### Set Use the `dstack secret set` command to create a new secret:
```shell $ dstack secret set my_secret some_secret_value OK ```
The same command can be used to update an existing secret:
```shell $ dstack secret set my_secret another_secret_value OK ```
### List Use the `dstack secret list` command to list all secrets set in a project:
```shell $ dstack secret NAME VALUE hf_token ****** my_secret ****** ```
### Get The `dstack secret list` does not show secret values. To see a secret value, use the `dstack secret get` command:
```shell $ dstack secret get my_secret NAME VALUE my_secret some_secret_value ```
### Delete Secrets can be deleted using the `dstack secret delete` command:
```shell $ dstack secret delete my_secret Delete the secret my_secret? [y/n]: y OK ```
## Use secrets You can use the `${{ secrets. }}` syntax to reference secrets in run configurations. Currently, secrets interpolation is supported in `env` and `registry_auth` properties. ### `env` Suppose you need to pass a sensitive environment variable to a run such as `HF_TOKEN`. You'd first create a secret holding the environment variable value:
```shell $ dstack secret set hf_token {hf_token_value} OK ```
and then reference the secret in `env`:
```yaml type: service env: - HF_TOKEN=${{ secrets.hf_token }} commands: ... ```
### `registry_auth` If you need to pull a private Docker image, you can store registry credentials as secrets and reference them in `registry_auth`:
```yaml type: service image: nvcr.io/nim/deepseek-ai/deepseek-r1-distill-llama-8b registry_auth: username: $oauthtoken password: ${{ secrets.ngc_api_key }} ```
# docs/concepts/projects.md --- title: Projects description: Organizing teams and isolating resources --- # Projects Projects enable the isolation of different teams and their resources. Each project can configure its own backends and control which users have access to it. > While project backends can be configured via [`~/.dstack/server/config.yml`](../reference/server/config.yml.md), > use the UI to fully manage projects, users, and user permissions. ## Project backends { #backends } In addition to [`~/.dstack/server/config.yml`](../reference/server/config.yml.md), a global admin or a project admin can configure backends on the project settings page. ## Global admins A user can be assigned or unassigned a global admin role on the user account settings page. This can only be done by another global admin. The global admin role allows a user to manage all projects and users. ## Project members A user can be added to a project and assigned or unassigned as a project role on the project settings page. ### Project roles * **Admin** – The project admin role allows a user to manage the project's settings, including backends, gateways, and members. * **Manager** – The project manager role allows a user to manage project members. Unlike admins, managers cannot configure backends and gateways. * **User** – A user can manage project resources including runs, fleets, and volumes. ## Project exports Projects can export resources such as fleets to other projects, allowing them to be used across team boundaries. See [Exports](exports.md) for more details. ## Authorization ### User token Once created, a user is issued a token. This token can be found on the user account settings page. The token must be used for authentication when logging into the control plane UI and when using the CLI or API. ### Setting up the CLI You can configure multiple projects on the client and set the default project using the [`dstack project`](../reference/cli/dstack/project.md) CLI command. You can find the command on the project’s settings page: ??? info "API" In addition to the UI, managing projects, users, and user permissions can also be done via the [HTTP API](../reference/http/index.md). # docs/concepts/metrics.md --- title: Metrics description: Tracking and monitoring system metrics --- # Metrics `dstack` automatically tracks essential metrics, which you can access via the CLI and UI. You can also configure the `dstack` server to export metrics to Prometheus—this is required to access advanced metrics such as those from DCGM. ## UI To access metrics via the UI, open the page of the corresponding run or job and switch to the `Metrics` tab: ![](https://dstack.ai/static-assets/static-assets/images/dstack-newsletter-metrics.png){ width=800 } This tab displays key CPU, memory, and GPU metrics collected during the last hour of the run or job. ## CLI As an alternative to the UI, you can track real-time essential metrics via the CLI. The `dstack metrics` command displays the most recently tracked CPU, memory, and GPU metrics.
```shell dstack metrics gentle-mayfly-1 NAME STATUS CPU MEMORY GPU gentle-mayfly-1 done 0% 16.27GB/2000GB gpu=0 mem=72.48GB/80GB util=0% gpu=1 mem=64.99GB/80GB util=0% gpu=2 mem=580MB/80GB util=0% gpu=3 mem=4MB/80GB util=0% gpu=4 mem=4MB/80GB util=0% gpu=5 mem=4MB/80GB util=0% gpu=6 mem=4MB/80GB util=0% gpu=7 mem=292MB/80GB util=0% ```
## Prometheus To enable exporting metrics to Prometheus, set the `DSTACK_ENABLE_PROMETHEUS_METRICS` environment variable and configure Prometheus to scrape metrics from `/metrics`. In addition to the essential metrics available via the CLI and UI, `dstack` exports additional metrics to Prometheus, including data on fleets, runs, jobs, and DCGM metrics. ??? info "NVIDIA DCGM" NVIDIA DCGM metrics are automatically collected for `aws`, `azure`, `gcp`, and `oci` backends, as well as for [SSH fleets](../concepts/fleets.md#ssh-fleets). To ensure NVIDIA DCGM metrics are collected from SSH fleets, ensure the `datacenter-gpu-manager-4-core`, `datacenter-gpu-manager-4-proprietary`, and `datacenter-gpu-manager-exporter` packages are installed on the hosts. ### Fleets Fleet metrics include metrics for each instance within a fleet. This includes information such as the instance's running time, price, GPU name, and more. === "Metrics" | Name | Type | Description | Examples | |------------------------------------------|-----------|-----------------------------------|--------------| | `dstack_instance_duration_seconds_total` | *counter* | Total instance runtime in seconds | `1123763.22` | | `dstack_instance_price_dollars_per_hour` | *gauge* | Instance price, USD/hour | `16.0` | | `dstack_instance_gpu_count` | *gauge* | Instance GPU count | `4.0`, `0.0` | === "Labels" | Name | Type | Description | Examples | |------------------------|-----------|:--------------|----------------------------------------| | `dstack_project_name` | *string* | Project name | `main` | | `dstack_fleet_name` | *string?* | Fleet name | `my-fleet` | | `dstack_fleet_id` | *string?* | Fleet ID | `51e837bf-fae9-4a37-ac9c-85c005606c22` | | `dstack_instance_name` | *string* | Instance name | `my-fleet-0` | | `dstack_instance_id` | *string* | Instance ID | `8c28c52c-2f94-4a19-8c06-12f1dfee4dd2` | | `dstack_instance_type` | *string?* | Instance type | `g4dn.xlarge` | | `dstack_backend` | *string?* | Backend | `aws`, `runpod` | | `dstack_gpu` | *string?* | GPU name | `H100` | ### Runs Run metrics include run counters for each user in each project. === "Metrics" | Name | Type | Description | Examples | |-------------------------------------|-----------|-------------------------------|----------| | `dstack_run_count_total` | *counter* | The total number of runs | `537` | | `dstack_run_count_terminated_total` | *counter* | The number of terminated runs | `118` | | `dstack_run_count_failed_total` | *counter* | The number of failed runs | `27` | | `dstack_run_count_done_total` | *counter* | The number of successful runs | `218` | === "Labels" | Name | Type | Description | Examples | |-----------------------|-----------|:--------------|-------------| | `dstack_project_name` | *string* | Project name | `main` | | `dstack_user_name` | *string* | User name | `alice` | ### Jobs A run consists of one or more jobs, each mapped to an individual container. For distributed workloads or auto-scalable services, a run spans multiple jobs. Job metrics provide detailed insights into each job within a run, including execution time, cost, GPU model, DCGM telemetry, and more. === "Metrics" | Name | Type | Description | Examples | |-------------------------------------------------|-----------|--------------------------------------------------------------------------------------------|----------------| | `dstack_job_duration_seconds_total` | *counter* | Total job runtime in seconds | `520.37` | | `dstack_job_price_dollars_per_hour` | *gauge* | Job instance price, USD/hour | `8.0` | | `dstack_job_gpu_count` | *gauge* | Job GPU count | `2.0`, `0.0` | | `dstack_job_cpu_count` | *gauge* | Job CPU count | `32.0` | | `dstack_job_cpu_time_seconds_total` | *counter* | Total CPU time consumed by the job, seconds | `11.727975` | | `dstack_job_memory_total_bytes` | *gauge* | Total memory allocated for the job, bytes | `4009754624.0` | | `dstack_job_memory_usage_bytes` | *gauge* | Memory used by the job (including cache), bytes | `339017728.0` | | `dstack_job_memory_working_set_bytes` | *gauge* | Memory used by the job (not including cache), bytes | `147251200.0` | | `dstack_job_gpu_usage_ratio` | *gauge* | Job GPU usage, percent (as 0.0-1.0) | `0.93` | | `dstack_job_gpu_memory_total_bytes` | *gauge* | Total GPU memory allocated for the job, bytes | `8589934592.0` | | `dstack_job_gpu_memory_usage_bytes` | *gauge* | GPU memory used by the job, bytes | `1048576.0` | | `DCGM_FI_DEV_GPU_UTIL` | *gauge* | GPU utilization (in %) | | | `DCGM_FI_DEV_MEM_COPY_UTIL` | *gauge* | Memory utilization (in %) | | | `DCGM_FI_DEV_ENC_UTIL` | *gauge* | Encoder utilization (in %) | | | `DCGM_FI_DEV_DEC_UTIL` | *gauge* | Decoder utilization (in %) | | | `DCGM_FI_DEV_FB_FREE` | *gauge* | Framebuffer memory free (in MiB) | | | `DCGM_FI_DEV_FB_USED` | *gauge* | Framebuffer memory used (in MiB) | | | `DCGM_FI_PROF_GR_ENGINE_ACTIVE` | *gauge* | The ratio of cycles during which a graphics engine or compute engine remains active | | | `DCGM_FI_PROF_SM_ACTIVE` | *gauge* | The ratio of cycles an SM has at least 1 warp assigned | | | `DCGM_FI_PROF_SM_OCCUPANCY` | *gauge* | The ratio of number of warps resident on an SM | | | `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` | *gauge* | Ratio of cycles the tensor (HMMA) pipe is active | | | `DCGM_FI_PROF_PIPE_FP64_ACTIVE` | *gauge* | Ratio of cycles the fp64 pipes are active | | | `DCGM_FI_PROF_PIPE_FP32_ACTIVE` | *gauge* | Ratio of cycles the fp32 pipes are active | | | `DCGM_FI_PROF_PIPE_FP16_ACTIVE` | *gauge* | Ratio of cycles the fp16 pipes are active | | | `DCGM_FI_PROF_PIPE_INT_ACTIVE` | *gauge* | Ratio of cycles the integer pipe is active | | | `DCGM_FI_PROF_DRAM_ACTIVE` | *gauge* | Ratio of cycles the device memory interface is active sending or receiving data | | | `DCGM_FI_PROF_PCIE_TX_BYTES` | *counter* | The number of bytes of active PCIe tx (transmit) data including both header and payload | | | `DCGM_FI_PROF_PCIE_RX_BYTES` | *counter* | The number of bytes of active PCIe rx (read) data including both header and payload | | | `DCGM_FI_DEV_SM_CLOCK` | *gauge* | SM clock frequency (in MHz) | | | `DCGM_FI_DEV_MEM_CLOCK` | *gauge* | Memory clock frequency (in MHz) | | | `DCGM_FI_DEV_MEMORY_TEMP` | *gauge* | Memory temperature (in C) | | | `DCGM_FI_DEV_GPU_TEMP` | *gauge* | GPU temperature (in C) | | | `DCGM_FI_DEV_POWER_USAGE` | *gauge* | Power draw (in W) | | | `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` | *counter* | Total energy consumption since boot (in mJ) | | | `DCGM_FI_DEV_PCIE_REPLAY_COUNTER` | *counter* | Total number of PCIe retries | | | `DCGM_FI_DEV_XID_ERRORS` | *gauge* | Value of the last XID error encountered | | | `DCGM_FI_DEV_POWER_VIOLATION` | *counter* | Throttling duration due to power constraints (in us) | | | `DCGM_FI_DEV_THERMAL_VIOLATION` | *counter* | Throttling duration due to thermal constraints (in us) | | | `DCGM_FI_DEV_SYNC_BOOST_VIOLATION` | *counter* | Throttling duration due to sync-boost constraints (in us) | | | `DCGM_FI_DEV_BOARD_LIMIT_VIOLATION` | *counter* | Throttling duration due to board limit constraints (in us) | | | `DCGM_FI_DEV_LOW_UTIL_VIOLATION` | *counter* | Throttling duration due to low utilization (in us) | | | `DCGM_FI_DEV_RELIABILITY_VIOLATION` | *counter* | Throttling duration due to reliability constraints (in us) | | | `DCGM_FI_DEV_ECC_SBE_VOL_TOTAL` | *counter* | Total number of single-bit volatile ECC errors | | | `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL` | *counter* | Total number of double-bit volatile ECC errors | | | `DCGM_FI_DEV_ECC_SBE_AGG_TOTAL` | *counter* | Total number of single-bit persistent ECC errors | | | `DCGM_FI_DEV_ECC_DBE_AGG_TOTAL` | *counter* | Total number of double-bit persistent ECC errors | | | `DCGM_FI_DEV_RETIRED_SBE` | *counter* | Total number of retired pages due to single-bit errors | | | `DCGM_FI_DEV_RETIRED_DBE` | *counter* | Total number of retired pages due to double-bit errors | | | `DCGM_FI_DEV_RETIRED_PENDING` | *counter* | Total number of pages pending retirement | | | `DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS` | *counter* | Number of remapped rows for uncorrectable errors | | | `DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS` | *counter* | Number of remapped rows for correctable errors | | | `DCGM_FI_DEV_ROW_REMAP_FAILURE` | *gauge* | Whether remapping of rows has failed | | | `DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL` | *counter* | Total number of NVLink flow-control CRC errors | | | `DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL` | *counter* | Total number of NVLink data CRC errors | | | `DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL` | *counter* | Total number of NVLink retries | | | `DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL` | *counter* | Total number of NVLink recovery errors | | | `DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL` | *counter* | Total number of NVLink bandwidth counters for all lanes | | | `DCGM_FI_DEV_NVLINK_BANDWIDTH_L0` | *counter* | The number of bytes of active NVLink rx or tx data including both header and payload | | | `DCGM_FI_PROF_NVLINK_RX_BYTES` | *counter* | The number of bytes of active PCIe rx (read) data including both header and payload | | | `DCGM_FI_PROF_NVLINK_TX_BYTES` | *counter* | The number of bytes of active NvLink tx (transmit) data including both header and payload | | === "Labels" | Label | Type | Description | Examples | |-----------------------|-----------|:-----------------------|----------------------------------------| | `dstack_project_name` | *string* | Project name | `main` | | `dstack_user_name` | *string* | User name | `alice` | | `dstack_run_name` | *string* | Run name | `nccl-tests` | | `dstack_run_id` | *string* | Run ID | `51e837bf-fae9-4a37-ac9c-85c005606c22` | | `dstack_job_name` | *string* | Job name | `nccl-tests-0-0` | | `dstack_job_id` | *string* | Job ID | `8c28c52c-2f94-4a19-8c06-12f1dfee4dd2` | | `dstack_job_num` | *integer* | Job number | `0` | | `dstack_replica_num` | *integer* | Replica number | `0` | | `dstack_run_type` | *string* | Run configuration type | `task`, `dev-environment` | | `dstack_backend` | *string* | Backend | `aws`, `runpod` | | `dstack_gpu` | *string?* | GPU name | `H100` | | `dstack_gpu_num`[^1] | *integer* | GPU number (0-based) | `0` | [^1]: For `dstack_gpu_*` metrics only. ### Server health metrics These are operational metrics to monitor the health of the dstack server. For now, these only include HTTP metrics, but more will be added later. === "Metrics" | Name | Type | Description | Examples | |------------------------------------------|-----------|-----------------------------------|--------------| | `dstack_server_requests_total` | *counter* | Total number of HTTP requests | `100.0` | | `dstack_server_request_duration_seconds` | *histogram* | HTTP request duration in seconds | `1.0`| === "Labels" | Name | Type | Description | Examples | |------------------------|-----------|:--------------|----------------------------------------| | `method` | *string* | HTTP method | `POST` | | `endpoint` | *string* | Endpoint path | `/api/project/main/repos/get` | | `http_status` | *string* | HTTP status code | `200` | | `project_name` | *string?* | Project name | `main` | # docs/concepts/events.md --- title: Events description: Auditing resource state changes and operations --- # Events Events provide a chronological record of notable state changes and operations affecting `dstack` resources. They are designed for auditing, debugging, and understanding the lifecycle of runs, jobs, fleets, and other resources. Each event includes the following fields: | Field | Description | | --------- | ----------------------------------------------------------- | | Timestamp | When the event occurred | | Actor | The user or system that initiated the change, if applicable | | Targets | The resources affected by the event | | Message | A description of the change or additional event details | Events can be queried by targeting a specific resource or within a group of related resources. For example, you can query events targeting a particular job, or query events within a run, including the run itself and all of its jobs. Events are accessible through the UI, CLI, and API. ## UI The UI allows you to query events either globally on the dedicated `Events` page or within a specific group on the page of a run, job, fleet, and other resources. ### Global page The global page shows events from all projects that the user has access to and allows filtering by many fields. ![](https://dstack.ai/static-assets/static-assets/images/dstack-ui-events-global.png){ width=800 } This page allows you to query events targeting a specific resource or within a particular group. ### Resource page The resource page shows events within that specific group. For example, if you open a run and switch to the `Events` tab, you will see all events about that run and its jobs. ![](https://dstack.ai/static-assets/static-assets/images/dstack-ui-events-run.png){ width=800 } ## CLI To query events via the CLI, use the `dstack event` command. This command provides several arguments that allow filtering by target and within scopes. Here is an example of querying all events within a particular run:
```shell $ dstack event --within-run cursor [2026-01-21 13:09:37] [👤admin] [run cursor] Run submitted. Status: SUBMITTED [2026-01-21 13:09:37] [job cursor-0-0] Job created on run submission. Status: SUBMITTED [2026-01-21 13:09:57] [job cursor-0-0] Job status changed SUBMITTED -> PROVISIONING [2026-01-21 13:09:58] [job cursor-0-0, instance some-fleet-0] Instance created for job. Instance status: PROVISIONING [2026-01-21 13:09:59] [run cursor] Run status changed SUBMITTED -> PROVISIONING [2026-01-21 13:11:22] [job cursor-0-0] Job status changed PROVISIONING -> PULLING [2026-01-21 13:11:49] [job cursor-0-0] Job status changed PULLING -> RUNNING [2026-01-21 13:11:51] [run cursor] Run status changed PROVISIONING -> RUNNING [2026-01-21 13:18:41] [👤admin] [run cursor] Run status changed RUNNING -> TERMINATING. Termination reason: STOPPED_BY_USER [2026-01-21 13:18:48] [job cursor-0-0] Job status changed RUNNING -> TERMINATING. Termination reason: TERMINATED_BY_USER [2026-01-21 13:19:05] [instance some-fleet-0, job cursor-0-0] Job unassigned from instance. Instance blocks: 0/1 busy [2026-01-21 13:19:05] [job cursor-0-0] Job status changed TERMINATING -> TERMINATED [2026-01-21 13:19:07] [run cursor] Run status changed TERMINATING -> TERMINATED ```
To see all supported arguments, check the [reference](../reference/cli/dstack/event.md). If you invoke the command without arguments, it will include all events targeting resources in the project. ## TTL By default, `dstack` stores each event for 30 days and then deletes it. This can be overridden by server administrators using the `DSTACK_SERVER_EVENTS_TTL_SECONDS` environment variable. # docs/concepts/exports.md --- title: Exports description: Exporting resources across projects --- # Exports Exports allow making resources from one project available to other projects. When a project exports a resource, the specified importer projects can see and use it as if it were their own. !!! warning "Experimental" Exports are an experimental feature. Currently, [SSH fleets](fleets.md#ssh-fleets) and [gateways](gateways.md) can be exported. An export is created in the exporter project and specifies the resources to export and the importer projects that will gain access to them. Once an export is created, the importer projects can see the exported resources in their resource lists and use them for running tasks, dev environments, and services. Imported resources appear with a project prefix (e.g., `team-a/my-fleet`) to distinguish them from the project's own resources. !!! info "Required project role" The user creating or updating an export must have the project admin role on both the exporter project and any importer project they add. Alternatively, a global admin can add any project as an importer. ## Manage exports ### Create exports Use the `dstack export create` command to create a new export. Specify the fleets to export with `--fleet`, the gateways to export with `--gateway`, and the importer projects with `--importer`:
```shell $ dstack export create my-export --fleet my-fleet --gateway my-gateway --importer team-b NAME FLEETS GATEWAYS IMPORTERS my-export my-fleet my-gateway team-b ```
`--fleet`, `--gateway`, and `--importer` can be specified multiple times:
```shell $ dstack export create shared-gpus --fleet gpu-fleet-1 --fleet gpu-fleet-2 --importer team-b --importer team-c NAME FLEETS GATEWAYS IMPORTERS shared-gpus gpu-fleet-1, gpu-fleet-2 - team-b, team-c ```
### List exports Use `dstack export list` (or simply `dstack export`) to list all exports in the project:
```shell $ dstack export list NAME FLEETS GATEWAYS IMPORTERS my-export my-fleet my-gateway team-b shared-gpus gpu-fleet-1, gpu-fleet-2 - team-b, team-c ```
### Update exports Use the `dstack export update` command to add or remove fleets, gateways, and importers from an existing export:
```shell $ dstack export update my-export --add-fleet another-fleet --add-importer team-c NAME FLEETS GATEWAYS IMPORTERS my-export my-fleet, another-fleet my-gateway team-b, team-c ```
To remove a fleet, gateway, or importer:
```shell $ dstack export update my-export --remove-importer team-b NAME FLEETS GATEWAYS IMPORTERS my-export my-fleet, another-fleet my-gateway team-c ```
### Delete exports Use the `dstack export delete` command to delete an export. This revokes access for all importer projects:
```shell $ dstack export delete my-export Delete the export my-export? [y/n]: y Export my-export deleted ```
Use `-y` to skip the confirmation prompt. ### Global exports Users with the global admin role can mark any export as a global export. Global exports are automatically imported into all projects, and their imports cannot be deleted.
```shell $ dstack export create global-export --gateway shared-gateway --global NAME FLEETS GATEWAYS IMPORTERS global-export - shared-gateway * ``` Only promoting an export to global requires the global admin role. Regular project admins can add or remove resources, remove global status, or delete the export.
## Access imported resources From the importer project's perspective, use `dstack import list` (or simply `dstack import`) to list all imports in the project — i.e., all exports from other projects that this project has been granted access to:
```shell $ dstack import list NAME FLEETS GATEWAYS team-a/my-export my-fleet, another-fleet my-gateway ```
Imported fleets and gateways also appear in `dstack fleet list` and `dstack gateway list` in the `/` format:
```shell $ dstack fleet list NAME NODES GPU SPOT BACKEND PRICE STATUS CREATED my-local-fleet 1 - - ssh - active 3 days ago team-a/my-fleet 2 A100:80GB:8 - ssh - active 1 week ago team-a/another-fleet 1 H100:80GB:4 - ssh - active 2 days ago $ dstack gateway list NAME BACKEND HOSTNAME DOMAIN DEFAULT STATUS team-a/my-gateway aws (eu-west-1) 10.0.0.4 gtw.mycompany.example running ```
Imported resources can be used for runs just like the project's own resources.
```yaml type: service image: nginx port: 80 gateway: team-a/my-gateway fleets: - my-local-fleet - team-a/my-fleet ```
!!! info "Tenant isolation" Exported fleets share the same access model as regular fleets. See [Tenant isolation](fleets.md#tenant-isolation) for details. !!! info "What's next?" 1. Check the [`dstack export` CLI reference](../reference/cli/dstack/export.md) 1. Check the [`dstack import` CLI reference](../reference/cli/dstack/import.md) 1. Learn how to manage [fleets](fleets.md) 1. Learn how to manage [gateways](gateways.md) 1. Read about [projects](projects.md) and project roles # docs/guides/cli-api.md --- title: CLI & API description: How to use the dstack CLI and HTTP API --- # CLI & API !!! info "Prerequisites" Ensure the [server](../installation.md#server) is up and running. To use `dstack` with AI agents, install [skills](../installation.md#skills). The primary way to use `dstack` is the CLI. It can be used to manage [fleets](../concepts/fleets.md), [dev environments](../concepts/dev-environments.md), [tasks](../concepts/tasks.md), [services](../concepts/services.md), [volumes](../concepts/volumes.md), and [gateways](../concepts/gateways.md), view logs, and inspect [events](../concepts/events.md). Use the HTTP API for functionality not available in the CLI or for integrations that need to call the server directly. ## CLI > See [installation](../installation.md#cli) on how to install the CLI. ### Configuration The CLI requires a [project](../concepts/projects.md) configuration with the project name, server URL, and user token in `~/.dstack/config.yml`.
```yaml projects: - name: main url: http://127.0.0.1:3000 token: default: true - name: octocat url: https://sky.dstack.ai token: ```
Use [`dstack project`](../reference/cli/dstack/project.md) to list, [add](../installation.md#configure-the-project), delete, and set the default project configurations. To run a command against a non-default project, pass `--project NAME`, or set `DSTACK_PROJECT` in the current shell. ??? info "Projects" [Projects](../concepts/projects.md) enable the isolation of different teams and their resources. Users can be added to projects and assigned roles. Each user has a user token for authentication. ### Manage fleets Before submitting runs, you must create at least one [fleet](../concepts/fleets.md). Fleets act as both pools of instances and templates for how those instances are provisioned. Use [`dstack fleet`](../reference/cli/dstack/fleet.md#dstack-fleet-list) to list existing fleets, their configurations, and instances (if any):
```shell $ dstack fleet ```
??? info "Offers" Offers are available instance configurations that match resource requirements.
```shell $ dstack offer --gpu H100 --max-offers 10 ```
If no fleet is specified, [`dstack offer`](../reference/cli/dstack/offer.md) shows offers from all configured backends. Use `--fleet NAME` to restrict offers to a fleet. Listing offers does not create capacity. Define a fleet configuration in a YAML file. The filename must end with `.dstack.yml`, for example `fleet.dstack.yml`:
```yaml type: fleet name: default nodes: 0..1 idle_duration: 1h resources: gpu: 0 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f fleet.dstack.yml ```
If the `nodes` range starts with `0`, `dstack` creates a fleet template. Instances are provisioned when matching runs are submitted. ### Submit runs To submit a run, define a [dev environment](../concepts/dev-environments.md), [task](../concepts/tasks.md), or [service](../concepts/services.md) configuration. The example below submits a task.
```yaml type: task name: hello commands: - echo hello world ```
Submit the run:
```shell $ dstack apply -f .dstack.yml ```
!!! info "Plan and confirmation" `dstack apply` shows the plan and asks for confirmation before submitting the run. To only see the plan, answer `n` at the prompt:
```shell $ echo "n" | dstack apply -f .dstack.yml ```
Use `-y` to skip confirmation. !!! info "Attached by default" For run configurations, `dstack apply` automatically attaches after submitting the run. This configures SSH access, forwards declared ports, and streams logs. See [Attach to runs](#attach-to-runs). Use `-d` to submit in detached mode. ### Attach to runs If the run was submitted with `-d`, or if you need to attach to another job in a multi-job run, use `dstack attach`:
```shell $ dstack attach <run name> ```
!!! info "SSH" During `dstack apply` in attached mode and during `dstack attach `, the CLI downloads the current user's built-in private SSH key if needed and stores it under `~/.dstack/ssh/`. While attached, the CLI updates `~/.dstack/ssh/config` and ensures this file is included from `~/.ssh/config`. The file contains a `Host ` alias used by `ssh ` to SSH into the run container:
```ssh-config Host HostName Port User IdentityFile IdentitiesOnly yes ```
> For [VM-based backends](../concepts/backends.md#vm-based) and > [SSH fleets](../concepts/fleets.md), `dstack` may add an additional alias > `-host` and use it as a proxy jump for `ssh `. While attached, connect to the run with:
```shell $ ssh <run name> ```
Use `--job JOB_NUMBER` with `dstack attach` to attach to another job. Ports declared in the run configuration are forwarded while attached. ??? info "User SSH keys" The server stores a built-in SSH key pair for each user. Users can add custom public SSH keys via the UI or the [users](../reference/http/users.md) API. To use a custom private key for a particular run, pass `--ssh-identity` to `dstack apply` or `dstack attach`. ### Browse logs When `dstack apply` is attached, it streams logs for job `0` automatically. Use [`dstack logs`](../reference/cli/dstack/logs.md) to view logs in detached mode, or to view logs for a specific job:
```shell $ dstack logs <run name> ```
Use `--job JOB_NUMBER` to select a job and `--since` to filter by time. ??? info "Attached logs" Use `--logs` with `dstack attach` to stream logs while attaching:
```shell $ dstack attach <run name> --logs ```
### Commands Other common CLI commands include [`dstack ps`](../reference/cli/dstack/ps.md), [`dstack stop`](../reference/cli/dstack/stop.md), and [`dstack event`](../reference/cli/dstack/event.md). !!! info "Verbose and JSON modes" Use `-v` for more details where supported. For automation, use `--json`, e.g. `dstack ps --json`, `dstack run get --json`, or `dstack fleet get --json`. ## API The `dstack` API is represented by the HTTP API. Use it for functionality not available in the CLI or for integrations that need to call the server directly. ### Authenticate The HTTP API requires the `Authorization` header for user authentication: ```text Authorization: Bearer ``` ### Manage fleets The [fleets](../reference/http/fleets.md) API can list existing fleets, their configurations, and instances (if any):
```shell $ curl "<server URL>/api/project/<project name>/fleets/list" \ -X POST \ -H "Authorization: Bearer <user token>" \ -H 'Content-Type: application/json' \ -d '{"include_imported": true}' ```
??? info "Offers" To check available offers via the HTTP API, call [`/runs/get_plan`](../reference/http/runs.md) with the same lightweight task specification used by `dstack offer`:
```shell $ curl "<server URL>/api/project/<project name>/runs/get_plan" \ -X POST \ -H "Authorization: Bearer <user token>" \ -H 'Content-Type: application/json' \ -d '{ "run_spec": { "configuration": { "type": "task", "commands": [":"], "image": "scratch", "user": "root", "resources": { "gpu": 0 } } }, "max_offers": 5 }' ```
If `fleets` is not set in the run configuration, offers are returned from all configured backends. Use `"fleets": ["default"]` to restrict offers to a fleet. To group offers by GPU and other fields, use the [gpus](../reference/http/gpus.md) API. Creating fleets uses `/fleets/get_plan` followed by `/fleets/apply`:
```shell $ curl "<server URL>/api/project/<project name>/fleets/get_plan" \ -X POST \ -H "Authorization: Bearer <user token>" \ -H 'Content-Type: application/json' \ -d '{ "spec": { "configuration": { "type": "fleet", "name": "cpu-fleet", "nodes": "0..1", "idle_duration": "1h", "resources": { "gpu": 0 } }, "profile": {} } }' ```
Then apply the fleet plan:
```shell $ curl "<server URL>/api/project/<project name>/fleets/apply" \ -X POST \ -H "Authorization: Bearer <user token>" \ -H 'Content-Type: application/json' \ -d '{ "plan": { "spec": { "configuration": { "type": "fleet", "name": "cpu-fleet", "nodes": "0..1", "idle_duration": "1h", "resources": { "gpu": 0 } }, "profile": {} } }, "force": false }' ```
### Submit runs Use the [runs](../reference/http/runs.md) API to submit [dev environments](../concepts/dev-environments.md), [tasks](../concepts/tasks.md), and [services](../concepts/services.md). The example below submits a task:
```shell $ curl "<server URL>/api/project/<project name>/runs/apply" \ -X POST \ -H "Authorization: Bearer <user token>" \ -H 'Content-Type: application/json' \ -d '{ "plan": { "run_spec": { "run_name": "hello-api", "configuration": { "type": "task", "commands": ["echo hello world"] } } }, "force": false }' ```
Set `run_name` if a stable run name is needed. Otherwise, the server can generate a run name. Poll `/runs/get` to check the run status:
```shell $ curl "<server URL>/api/project/<project name>/runs/get" \ -X POST \ -H "Authorization: Bearer <user token>" \ -H 'Content-Type: application/json' \ -d '{"run_name": "hello-api"}' ```
### Poll logs Use the [logs](../reference/http/logs.md) API to poll logs. Get `job_submission_id` from `/runs/get`, e.g. from `latest_job_submission.id`.
```shell $ curl "<server URL>/api/project/<project name>/logs/poll" \ -X POST \ -H "Authorization: Bearer <user token>" \ -H 'Content-Type: application/json' \ -d '{ "run_name": "hello-api", "job_submission_id": "<job submission id>", "limit": 100 }' ```
Use `next_token` from the response to continue polling. ## Reference For complete details on specific CLI commands and HTTP APIs, see the [`dstack server`](../reference/cli/dstack/server.md) and [server](../reference/http/server.md) references. !!! info "OpenAPI" For complete information on the HTTP API, or to generate native clients, refer to [openapi.json](../reference/http/openapi.json). !!! info "What's next?" 1. Follow the [installation guide](../installation.md) 2. Read about [projects](../concepts/projects.md) 3. Check [fleets](../concepts/fleets.md), [dev environments](../concepts/dev-environments.md), [tasks](../concepts/tasks.md), and [services](../concepts/services.md) # docs/guides/server-deployment.md --- title: Server Deployment description: Deploying the dstack server --- The `dstack` server can run on your laptop or any environment with access to the cloud and on-prem clusters you plan to use. The minimum hardware requirements for running the server are 1 CPU and 1GB of RAM. === "pip" > The server can be set up via `pip` on Linux, macOS, and Windows (via WSL 2). It requires Git and OpenSSH.
```shell $ pip install "dstack[all]" -U $ dstack server Applying ~/.dstack/server/config.yml... The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ ```
=== "uv" > The server can be set up via `uv` on Linux, macOS, and Windows (via WSL 2). It requires Git and OpenSSH.
```shell $ uv tool install 'dstack[all]' -U $ dstack server Applying ~/.dstack/server/config.yml... The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ ```
=== "Docker" > To deploy the server most reliably, it's recommended to use `dstackai/dstack` Docker image.
```shell $ docker run -p 3000:3000 \ -v $HOME/.dstack/server/:/root/.dstack/server \ dstackai/dstack Applying ~/.dstack/server/config.yml... The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ ```
??? info "AWS CloudFormation" If you'd like to deploy the server to a private AWS VPC, you can use our CloudFormation [template](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://get-dstack.s3.eu-west-1.amazonaws.com/cloudformation/template.yaml). First, ensure you've set up a private VPC with public and private subnets. ![](https://dstack.ai/static-assets/static-assets/images/dstack-aws-private-vpc-example-v2.png) Create a stack using the template, and specify the VPC and private subnets. Once, the stack is created, go to `Outputs` for the server URL and admin token. To access the server URL, ensure you're connected to the VPC, e.g. via VPN client. > If you'd like to adjust anything, the source code of the template can be found at [`examples/server-deployment/cloudformation/template.yaml`](https://github.com/dstackai/dstack/blob/master/examples/server-deployment/cloudformation/template.yaml). ## Backend configuration To use `dstack` with cloud providers, configure [backends](../concepts/backends.md) via the `~/.dstack/server/config.yml` file. The server loads this file on startup. Alternatively, you can configure backends on the [project settings page](../concepts/projects.md#backends) via UI. > For using `dstack` with on-prem servers, no backend configuration is required. > Use [SSH fleets](../concepts/fleets.md#ssh-fleets) instead. ## State persistence The `dstack` server can store its internal state in SQLite or Postgres. By default, it stores the state locally in `~/.dstack/server` using SQLite. With SQLite, you can run at most one server replica. Postgres has no such limitation and is recommended for production deployment. ??? info "Replicate SQLite to cloud storage" You can configure automatic replication of your SQLite state to a cloud object storage using Litestream. This allows persisting the server state across re-deployments when using SQLite. To enable Litestream replication, set the following environment variables: - `LITESTREAM_REPLICA_URL` - The url of the cloud object storage. Examples: `s3:///`, `gcs:///`, `abs://@/`, etc. You also need to configure cloud storage credentials. **AWS S3** To persist state into an AWS S3 bucket, provide the following environment variables: - `AWS_ACCESS_KEY_ID` - The AWS access key ID - `AWS_SECRET_ACCESS_KEY` - The AWS secret access key **GCP Storage** To persist state into a GCP Storage bucket, provide one of the following environment variables: - `GOOGLE_APPLICATION_CREDENTIALS` - The path to the GCP service account key JSON file - `GOOGLE_APPLICATION_CREDENTIALS_JSON` - The GCP service account key JSON **Azure Blob Storage** To persist state into an Azure blog storage, provide the following environment variable. - `LITESTREAM_AZURE_ACCOUNT_KEY` - The Azure storage account key More [details](https://litestream.io/guides/) on options for configuring replication. ### PostgreSQL To store the server state in Postgres, set the `DSTACK_DATABASE_URL` environment variable: ```shell $ DSTACK_DATABASE_URL=postgresql+asyncpg://user:password@db-host:5432/dstack dstack server ``` The minimum requirements for the DB instance are 2 CPU, 2GB of RAM, and at least 50 `max_connections` per server replica or a configured connection pooler to handle that many connections. If you're using a smaller DB instance, you may need to set lower `DSTACK_DB_POOL_SIZE` and `DSTACK_DB_MAX_OVERFLOW`, e.g. `DSTACK_DB_POOL_SIZE=10` and `DSTACK_DB_MAX_OVERFLOW=0`. ??? info "Migrate from SQLite to PostgreSQL" You can migrate the existing state from SQLite to PostgreSQL using `pgloader`: 1. Create a new PostgreSQL database 2. Clone the `dstack` repo and [install](https://github.com/dstackai/dstack/blob/master/contributing/DEVELOPMENT.md) `dstack` from source. Ensure you've checked out the tag that corresponds to your server version (e.g. `git checkout 0.18.10`). 3. Apply database migrations to the new database: ```bash cd src/dstack/_internal/server/ export DSTACK_DATABASE_URL="postgresql+asyncpg://..." alembic upgrade head ``` 4. Install [pgloader :material-arrow-top-right-thin:{.external }](https://github.com/dimitri/pgloader) 5. Pass the path to the `~/.dstack/server/data/sqlite.db` file to `SOURCE_PATH` and set `TARGET_PATH` with the URL of the PostgreSQL database. Example: ```bash cd scripts/ export SOURCE_PATH=sqlite:///Users/me/.dstack/server/data/sqlite.db export TARGET_PATH=postgresql://postgres:postgres@localhost:5432/postgres pgloader sqlite_to_psql.load ``` The `pgloader` script will migrate the SQLite data to PostgreSQL. It may emit warnings that are safe to ignore. If you encounter errors, please [submit an issue](https://github.com/dstackai/dstack/issues/new/choose). ## Logs storage By default, `dstack` stores workload logs locally in `~/.dstack/server/projects//logs`. For multi-replica server deployments, it's required to store logs externally. `dstack` supports storing logs using AWS CloudWatch, GCP Logging, or Fluent-bit with Elasticsearch / Opensearch. ### AWS CloudWatch To store logs in AWS CloudWatch, set the `DSTACK_SERVER_CLOUDWATCH_LOG_GROUP` and the `DSTACK_SERVER_CLOUDWATCH_LOG_REGION` environment variables. The log group must be created beforehand. `dstack` won't try to create it. ??? info "Required permissions" ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DstackLogStorageAllow", "Effect": "Allow", "Action": [ "logs:DescribeLogStreams", "logs:CreateLogStream", "logs:GetLogEvents", "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:::log-group:", "arn:aws:logs:::log-group::*" ] } ] } ``` ### GCP Logging To store logs using GCP Logging, set the `DSTACK_SERVER_GCP_LOGGING_PROJECT` environment variable. ??? info "Required permissions" Ensure you've configured Application Default Credentials with the following permissions: ``` logging.logEntries.create logging.logEntries.list ``` ??? info "Logs management" `dstack` writes all the logs to the `projects/[PROJECT]/logs/dstack-run-logs` log name. If you want to set up a custom retention policy for `dstack` logs, create a new bucket and configure a sink:
```shell $ gcloud logging buckets create dstack-bucket \ --location=global \ --description="Bucket for storing dstack run logs" \ --retention-days=10 $ gcloud logging sinks create dstack-sink \ logging.googleapis.com/projects/[PROJECT]/locations/global/buckets/dstack-bucket \ --log-filter='logName = "projects/[PROJECT]/logs/dstack-run-logs"' ```
### Fluent-bit To store logs using Fluent-bit, set the `DSTACK_SERVER_FLUENTBIT_HOST` environment variable. Fluent-bit supports two modes depending on how you want to access logs. === "Full mode" Logs are shipped to Fluent-bit and can be read back through the `dstack` UI and CLI via Elasticsearch or OpenSearch. Use this mode when you want a complete integration with log viewing in `dstack`: ```shell $ DSTACK_SERVER_FLUENTBIT_HOST=fluentbit.example.com \ DSTACK_SERVER_ELASTICSEARCH_HOST=https://elasticsearch.example.com:9200 \ dstack server ``` === "Ship-only mode" Logs are forwarded to Fluent-bit but cannot be read through `dstack`. The dstack UI/CLI will show empty logs. Use this mode when: - You have an existing logging infrastructure (Kibana, Grafana, Datadog, etc.) - You only need to forward logs without reading them back through `dstack` - You want to reduce operational complexity by not running Elasticsearch/OpenSearch ```shell $ DSTACK_SERVER_FLUENTBIT_HOST=fluentbit.example.com \ dstack server ``` ??? info "Additional configuration" The following optional environment variables can be used to customize the Fluent-bit integration: **Fluent-bit settings:** - `DSTACK_SERVER_FLUENTBIT_PORT` – The Fluent-bit port. Defaults to `24224`. - `DSTACK_SERVER_FLUENTBIT_PROTOCOL` – The protocol to use: `forward` or `http`. Defaults to `forward`. - `DSTACK_SERVER_FLUENTBIT_TAG_PREFIX` – The tag prefix for logs. Defaults to `dstack`. **Elasticsearch/OpenSearch settings (for full mode only):** - `DSTACK_SERVER_ELASTICSEARCH_HOST` – The Elasticsearch/OpenSearch host for reading logs. If not set, runs in ship-only mode. - `DSTACK_SERVER_ELASTICSEARCH_INDEX` – The Elasticsearch/OpenSearch index pattern. Defaults to `dstack-logs`. - `DSTACK_SERVER_ELASTICSEARCH_API_KEY` – The Elasticsearch/OpenSearch API key for authentication. ??? info "Fluent-bit configuration" Configure Fluent-bit to receive logs and forward them to Elasticsearch or OpenSearch. Example configuration: ```ini [INPUT] Name forward Listen 0.0.0.0 Port 24224 [OUTPUT] Name es Match dstack.* Host elasticsearch.example.com Port 9200 Index dstack-logs Suppress_Type_Name On ``` ??? info "Required dependencies" To use Fluent-bit log storage, install the `fluentbit` extras: ```shell $ pip install "dstack[all]" -U # or $ pip install "dstack[fluentbit]" -U ``` ## File storage When using [files](../concepts/dev-environments.md#files) or [repos](../concepts/dev-environments.md#repos), `dstack` uploads local files and diffs to the server so that you can have access to them within runs. By default, the files are stored in the DB and each upload is limited to 2MB. You can configure an object storage to be used for uploads and increase the default limit by setting the `DSTACK_SERVER_CODE_UPLOAD_LIMIT` environment variable ### S3 To use S3 for storing uploaded files, set the `DSTACK_SERVER_S3_BUCKET` and `DSTACK_SERVER_S3_BUCKET_REGION` environment variables. The bucket must be created beforehand. `dstack` won't try to create it. ??? info "Required permissions" ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] } ] } ``` ### GCS To use GCS for storing uploaded files, set the `DSTACK_SERVER_GCS_BUCKET` environment variable. The bucket must be created beforehand. `dstack` won't try to create it. ??? info "Required permissions" Ensure you've configured Application Default Credentials with the following permissions: ``` storage.buckets.get storage.buckets.list storage.objects.get storage.objects.list storage.objects.create storage.objects.delete storage.objects.update ``` ## SSH proxy [`dstack-sshproxy`](https://github.com/dstackai/sshproxy) is an optional component that provides direct SSH access to workloads. Without SSH proxy, in order to connect to a job via SSH or use an IDE URL, the `dstack attach` CLI command must be used, which configures user's SSH client in a backend-specific way for each job. When SSH proxy is deployed, there is one well-known entry point – a proxy address – for all `dstack` jobs, which can be used for SSH access without any additional steps on the user's side (such as installing `dstack` and executing `dstack attach` each time). All the user has to do is to upload their public key to the `dstack` server once – there is a dedicated “SSH keys” tab on the user's page of the control plane UI. To deploy SSH proxy, see `dstack-sshproxy` [Deployment guide](https://github.com/dstackai/sshproxy/blob/main/DEPLOYMENT.md). To enable SSH proxy integration on the `dstack` server side, set the following environment variables: * `DSTACK_SSHPROXY_API_TOKEN` – a token used to authenticate SSH proxy API requests, must be the same value as when deploying `dstack-sshproxy`. * `DSTACK_SERVER_SSHPROXY_ADDRESS` – an address where SSH proxy is available to `dstack` users, in the `HOSTNAME[:PORT]` form, where `HOSTNAME` is a domain name or an IP address, and `PORT`, if not specified, defaults to 22. ## Encryption By default, `dstack` stores data in plaintext. To enforce encryption, you specify one or more encryption keys. `dstack` currently supports AES and identity (plaintext) encryption keys. Support for external providers like HashiCorp Vault and AWS KMS is planned. === "AES" The `aes` encryption key encrypts data using [AES-256](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in GCM mode. To configure the `aes` encryption, generate a random 32-byte key:
```shell $ head -c 32 /dev/urandom | base64 opmx+r5xGJNVZeErnR0+n+ElF9ajzde37uggELxL ```
And specify it as `secret`: ```yaml # ... encryption: keys: - type: aes name: key1 secret: opmx+r5xGJNVZeErnR0+n+ElF9ajzde37uggELxL ``` === "Identity" The `identity` encryption performs no encryption and stores data in plaintext. You can specify an `identity` encryption key explicitly if you want to decrypt the data:
```yaml # ... encryption: keys: - type: identity - type: aes name: key1 secret: opmx+r5xGJNVZeErnR0+n+ElF9ajzde37uggELxL ```
With this configuration, the `aes` key will still be used to decrypt the old data, but new writes will store the data in plaintext. ??? info "Key rotation" If multiple keys are specified, the first is used for encryption, and all are tried for decryption. This enables key rotation by specifying a new encryption key.
```yaml # ... encryption: keys: - type: aes name: key2 secret: cR2r1JmkPyL6edBQeHKz6ZBjCfS2oWk87Gc2G3wHVoA= - type: aes name: key1 secret: E5yzN6V3XvBq/f085ISWFCdgnOGED0kuFaAkASlmmO4= ```
Old keys may be deleted once all existing records have been updated to re-encrypt sensitive data. Encrypted values are prefixed with key names, allowing DB admins to identify the keys used for encryption. ## Default permissions By default, all users can create and manage their own projects. You can specify `default_permissions` to `false` so that only global admins can create and manage projects:
```yaml # ... default_permissions: allow_non_admins_create_projects: false ```
## Backward compatibility `dstack` follows the `{major}.{minor}.{patch}` versioning scheme. Backward compatibility is maintained based on these principles: * The server backward compatibility is maintained on a best-effort basis across minor and patch releases. The specific features can be removed, but the removal is preceded with deprecation warnings for several minor releases. This means you can use older client versions with newer server versions. * The client backward compatibility is maintained across patch releases. A new minor release indicates that the release breaks client backward compatibility. This means you don't need to update the server when you update the client to a new patch release. Still, upgrading a client to a new minor version requires upgrading the server too. ## Server limits A single `dstack` server replica can support at least * 1000 active instances * 1000 active runs * 1000 active jobs. If you hit server performance limits, try scale up server instances and/or configure Postgres with multiple server replicas. Also, please [submit a GitHub issue](https://github.com/dstackai/dstack/issues) describing your setup – we strive to improve `dstack` scalability and efficiency. ## Server upgrades When upgrading the `dstack` server, follow these guidelines to ensure a smooth transition and minimize downtime. ### Before upgrading 1. **Check the changelog**: Review the [release notes](https://github.com/dstackai/dstack/releases) for breaking changes, new features, and migration notes. 2. **Review backward compatibility**: Understand the [backward compatibility](#backward-compatibility) policy. 3. **Back up your data**: Ensure you always create a backup before upgrading. ### Best practices - **Test in staging**: Always test upgrades in a non-production environment first. - **Monitor logs**: Watch server logs during and after the upgrade for any errors or warnings. - **Keep backups**: Retain backups for at least a few days after a successful upgrade. ### Troubleshooting **Deadlock when upgrading a multi-replica PostgreSQL deployment** If a deployment is stuck due to a deadlock when applying DB migrations, try scaling server replicas to 1 and retry the deployment multiple times. Some releases may not support rolling deployments, which is always noted in the release notes. If you think there is a bug, please [file an issue](https://github.com/dstackai/dstack/issues). ## FAQs ??? info "Can I run multiple replicas of dstack server?" Yes, you can if you configure `dstack` to use [PostgreSQL](#postgresql) and an external log storage such as [AWS CloudWatch](#aws-cloudwatch), [GCP Logging](#gcp-logging), or [Fluent-bit](#fluent-bit). ??? info "Does dstack server support blue-green or rolling deployments?" Yes, it does if you configure `dstack` to use [PostgreSQL](#postgresql) and an external log storage such as [AWS CloudWatch](#aws-cloudwatch), [GCP Logging](#gcp-logging), or [Fluent-bit](#fluent-bit). # docs/guides/troubleshooting.md --- title: Troubleshooting description: Common issues and how to resolve them --- # Troubleshooting ## Reporting issues When you encounter a problem, please report it as a [GitHub issue](https://github.com/dstackai/dstack/issues/new/choose). If you have a question or need help, feel free to ask it in our [Discord server](https://discord.gg/u8SmfwPpMd). > When bringing up issues, always include the steps to reproduce. ### Steps to reproduce Make sure to provide clear, detailed steps to reproduce the issue. Include server logs, CLI outputs, and configuration samples. Avoid using screenshots for logs or errors—use text instead. #### Server logs To get more detailed server logs, set the `DSTACK_SERVER_LOG_LEVEL` environment variable to `DEBUG`. By default, it is set to `INFO`. #### CLI logs CLI logs are located in `~/.dstack/logs/cli`, and the default log level is `DEBUG`. > See these examples for well-reported issues: [this](https://github.com/dstackai/dstack/issues/1640) and [this](https://github.com/dstackai/dstack/issues/1551). ## Typical issues ### No fleets { #no-fleets } [//]: # (NOTE: This section is referenced in the CLI. Do not change its URL.) If you run `dstack apply` and see `No fleets` status it can mean two things: === "The project has no fleets" In this case, ensure you've created one before submitting runs. This can be either a [backend fleet](../concepts/fleets.md#backend-fleets) (if you are using cloud or Kubernetes) or an [SSH fleet](../concepts/fleets.md#ssh-fleets) (if you're using on-prem clusters). !!! info "Backend fleets" Note that creating [backend fleet](../concepts/fleets.md#backend-fleets) doesn't necessarily require provisioning instances upfront. If you set `nodes` to a range, `dstack` will be able to provision instances as required. See [backend fleet](../concepts/fleets.md#backend-fleets) for examples. === "No matching fleet found" This means fleets exist but run requirements do not match the configuration of the fleet. Review your fleets, and ensure that both run and fleet configuration are correct. ### No offers { #no-offers } [//]: # (NOTE: This section is referenced in the CLI. Do not change its URL.) If you run `dstack apply` and don't see any instance offers, it means that `dstack` could not find instances that match the requirements in your configuration. Below are some of the reasons why this might happen. Feel free to use `dstack offer` to inspect available offers: ```shell # All matching offers, ignoring fleet configurations $ dstack offer --gpu H100 # Offers available through a specific fleet $ dstack offer --gpu H100 --fleet my-fleet ``` #### Cause 1: No backends If you are not using [SSH fleets](../concepts/fleets.md#ssh-fleets), make sure you have configured at least one [backends](../concepts/backends.md). If you have configured a backend but still cannot use it, check the output of `dstack server` for backend configuration errors. > You can find a list of successfully configured backends on the [project settings page](../concepts/projects.md#backends) in the UI. #### Cause 2: Requirements mismatch When you apply a configuration, `dstack` tries to find instances that match the [`resources`](../reference/dstack.yml/task.md#resources), [`backends`](../reference/dstack.yml/task.md#backends), [`regions`](../reference/dstack.yml/task.md#regions), [`availability_zones`](../reference/dstack.yml/task.md#availability_zones), [`instance_types`](../reference/dstack.yml/task.md#instance_types), [`spot_policy`](../reference/dstack.yml/task.md#spot_policy), and [`max_price`](../reference/dstack.yml/task.md#max_price) properties from the configuration. `dstack` will only select instances that meet all the requirements. Make sure your configuration doesn't set any conflicting requirements, such as `regions` that don't exist in the specified `backends`, or `instance_types` that don't match the specified `resources`. #### Cause 3: Too specific resources If you set a resource requirement to an exact value, `dstack` will only select instances that have exactly that amount of resources. For example, `cpu: 5` and `memory: 10GB` will only match instances that have exactly 5 CPUs and exactly 10GB of memory. Typically, you will want to set resource ranges to match more instances. For example, `cpu: 4..8` and `memory: 10GB..` will match instances with 4 to 8 CPUs and at least 10GB of memory. #### Cause 4: Default resources By default, `dstack` uses these resource requirements: `cpu: 2..`, `memory: 8GB..`, `disk: 100GB..`. If you want to use smaller instances, override the `cpu`, `memory`, or `disk` properties in your configuration. #### Cause 5: GPU requirements By default, `dstack` only selects instances with no GPUs or a single NVIDIA GPU. If you want to use non-NVIDIA GPUs or multi-GPU instances, set the `gpu` property in your configuration. Examples: `gpu: amd` (one AMD GPU), `gpu: A10:4..8` (4 to 8 A10 GPUs), `gpu: 8:Gaudi2` (8 Gaudi2 accelerators). > If you don't specify the number of GPUs, `dstack` will only select single-GPU instances. #### Cause 6: Network volumes If your run configuration uses [network volumes](../concepts/volumes.md#network-volumes), `dstack` will only select instances from the same backend and region as the volumes. For AWS, the availability zone of the volume and the instance should also match. #### Cause 7: Feature support Some `dstack` features are not supported by all backends. If your configuration uses one of these features, `dstack` will only select offers from the backends that support it. - [Backend fleets](../concepts/fleets.md#backend-fleets) configurations, [Instance volumes](../concepts/volumes.md#instance-volumes), and [Privileged containers](../reference/dstack.yml/dev-environment.md#privileged) are supported by all backends except `runpod`, `vastai`, and `kubernetes`. - [Clusters](../concepts/fleets.md#cluster-placement) and [distributed tasks](../concepts/tasks.md#distributed-tasks) are only supported by the `aws`, `azure`, `gcp`, `nebius`, `oci`, and `vultr` backends, as well as SSH fleets. - [Reservations](../reference/dstack.yml/fleet.md#reservation) are only supported by the `aws` and `gcp` backends. #### Cause 8: dstack Sky balance If you are using [dstack Sky](https://sky.dstack.ai), you will not see marketplace offers until you top up your balance. Alternatively, you can configure your own cloud accounts on the [project settings page](../concepts/projects.md#backends) or use [SSH fleets](../concepts/fleets.md#ssh-fleets). ### Provisioning fails In certain cases, running `dstack apply` may show instance offers, but then produce the following output: ```shell wet-mangust-1 provisioning completed (failed) All provisioning attempts failed. This is likely due to cloud providers not having enough capacity. Check CLI and server logs for more details. ``` #### Cause 1: Insufficient service quotas If some runs fail to provision, it may be due to an insufficient service quota. For cloud providers like AWS, GCP, Azure, and OCI, you often need to request an increased [service quota](protips.md#service-quotas) before you can use specific instances. ### Run starts but fails There could be several reasons for a run failing after successful provisioning. !!! info "Termination reason" To find out why a run terminated, use `--verbose` (or `-v`) with `dstack ps`. This will show the run's status and any failure reasons. !!! info "Diagnostic logs" You can get more information on why a run fails with diagnostic logs. Pass `--diagnose` (or `-d`) to `dstack logs` and you'll see logs of the run executor. #### Cause 1: Spot interruption If a run fails after provisioning with the termination reason `INTERRUPTED_BY_NO_CAPACITY`, it is likely that the run was using spot instances and was interrupted. To address this, you can either set the [`spot_policy`](../reference/dstack.yml/task.md#spot_policy) to `on-demand` or specify the [`retry`](../reference/dstack.yml/task.md#retry) property. [//]: # (#### Other) [//]: # (TODO: Explain how to get the shim logs) ### Services fail to start #### Cause 1: Gateway misconfiguration If all services fail to start with a specific gateway, make sure a [correct DNS record](../concepts/gateways.md#update-dns-records) pointing to the gateway's hostname is configured. ### Service endpoint doesn't work #### Cause 1: Bad Authorization If the service endpoint returns a 403 error, it is likely because the [`Authorization`](../concepts/services.md#access-the-endpoint) header with the correct `dstack` token was not provided. [//]: # (#### Other) [//]: # (TODO: Explain how to get the gateway logs) ### Cannot access dev environment or task ports #### Cause 1: Detached from run When running a dev environment or task with configured ports, `dstack apply` automatically forwards remote ports to `localhost` via SSH for easy and secure access. If you interrupt the command, the port forwarding will be disconnected. To reattach, use `dstack attach ```yaml type: task name: streamlit-task python: 3.12 commands: - uv pip install streamlit - streamlit hello ports: - 8501 ``` While you run a task, `dstack apply` forwards the remote ports to `localhost`.
```shell $ dstack apply -f app.dstack.yml Welcome to Streamlit. Check out our demo in your browser. Local URL: http://localhost:8501 ```
This allows you to access the remote `8501` port on `localhost:8501` while the CLI is attached. ??? info "Port mapping" If you want to override the local port, use the `--port` option:
```shell $ dstack apply -f app.dstack.yml --port 3000:8501 ```
This will forward the remote `8501` port to `localhost:3000`. !!! info "Tasks vs. services" [Services](../concepts/services.md) provide external access, `https`, replicas with autoscaling, OpenAI-compatible endpoint and other service features. If you don't need them, you can use [tasks](../concepts/tasks.md) for running apps. ## Utilization policy If you want your run to automatically terminate if any of GPUs are underutilized, you can specify `utilization_policy`. Below is an example of a dev environment that auto-terminate if any GPU stays below 10% utilization for 1 hour.
```yaml type: dev-environment name: my-dev python: 3.12 ide: cursor resources: gpu: H100:8 utilization_policy: min_gpu_utilization: 10 time_window: 1h ```
## Docker in Docker Set `docker` to `true` to enable the `docker` CLI in your dev environment, e.g., to run or build Docker images, or use Docker Compose. === "Dev environment"
```yaml type: dev-environment name: vscode docker: true ide: vscode init: - docker run --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi ```
=== "Task"
```yaml type: task name: docker-nvidia-smi docker: true commands: - docker run --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi resources: gpu: 1 ```
??? info "Volumes" To persist Docker data between runs (e.g. images, containers, volumes, etc), create a `dstack` [volume](../concepts/volumes.md) and add attach it in your run configuration. === "Network volums" ```yaml type: dev-environment name: vscode docker: true ide: vscode volumes: - name: docker-volume path: /var/lib/docker ``` === "Instance volumes" ```yaml type: dev-environment name: vscode docker: true ide: vscode volumes: - name: /docker-volume path: /var/lib/docker optional: true ``` ## Fleets ### Creation policy By default, when you run `dstack apply` with a dev environment, task, or service, if no `idle` instances from the available fleets meet the requirements, `dstack` provisions a new instance using configured backends. To ensure `dstack apply` doesn't provision a new instance but reuses an existing one, pass `-R` (or `--reuse`) to `dstack apply`.
```shell $ dstack apply -R -f examples/.dstack.yml ```
Or, set [`creation_policy`](../reference/dstack.yml/dev-environment.md#creation_policy) to `reuse` in the run configuration. ### Idle duration If the run is submitted to a fleet with `nodes` set to a range and a new instance is provisioned, the shorter of the fleet's and run's `idle_duration` is used. If the run reuses an existing fleet instance, only the fleet's [`idle_duration`](../reference/dstack.yml/fleet.md#idle_duration) applies. If an instance remains `idle`, it is automatically terminated after `idle_duration`. > Not applied for container-based backends (Kubernetes, Vast.ai, Runpod). ## Volumes To persist data across runs, it is recommended to use volumes. `dstack` supports two types of volumes: [network](../concepts/volumes.md#network-volumes) (for persisting data even if the instance is interrupted) and [instance](../concepts/volumes.md#instance-volumes) (useful for persisting cached data across runs while the instance remains active). > If you use [SSH fleets](../concepts/fleets.md#ssh-fleets), you can mount network storage (e.g., NFS or SMB) to the hosts and access it in runs via instance volumes. ## Environment variables If a configuration requires an environment variable that you don't want to hardcode in the YAML, you can define it without assigning a value:
```yaml type: dev-environment name: vscode python: 3.12 env: - HF_TOKEN ide: vscode ```
Then, you can pass the environment variable either via the shell:
```shell $ HF_TOKEN=... $ dstack apply -f .dstack.yml ```
Or via the `-e` option of the `dstack apply` command:
```shell $ dstack apply -e HF_TOKEN=... -f .dstack.yml ```
??? info ".envrc" A better way to configure environment variables not hardcoded in YAML is by specifying them in a `.envrc` file:
```shell export HF_TOKEN=... ```
If you install [`direnv`](https://direnv.net/), it will automatically apply the environment variables from the `.envrc` file to the `dstack apply` command. Remember to add `.envrc` to `.gitignore` to avoid committing it to the repo. [//]: # (## Profiles) [//]: # () [//]: # (If you don't want to specify the same parameters for each configuration, you can define them once via [profiles](../reference/profiles.yml.md)) [//]: # (and reuse them across configurations.) [//]: # () [//]: # (This can be handy, for example, for configuring parameters such as `max_duration`, `max_price`, `termination_idle_time`,) [//]: # (`regions`, etc.) [//]: # () [//]: # (Set `default` to `true` in your profile, and it will be applied automatically to any run.) ## Retry policy By default, if `dstack` can't find available capacity, the run will fail. If you'd like `dstack` to automatically retry, configure the [retry](../reference/dstack.yml/task.md#retry) property accordingly:
```yaml type: task name: train python: 3.12 commands: - uv pip install -r fine-tuning/qlora/requirements.txt - python fine-tuning/qlora/train.py retry: on_events: [no-capacity] # Retry for up to 1 hour duration: 1h ```
## Profiles Sometimes, you may want to reuse parameters across runs or set defaults so you don’t have to repeat them in every configuration. You can do this by defining a profile. ??? info ".dstack/profiles.yml" A profile file can be created either globally in `~/.dstack/profiles.yml` or locally in `.dstack/profiles.yml`: ```yaml profiles: - name: my-profile # If set to true, this profile will be applied automatically default: true # The spot pololicy can be "spot", "on-demand", or "auto" spot_policy: auto # Limit the maximum price of the instance per hour max_price: 1.5 # Stop any run if it runs longer that this duration max_duration: 1d # Use only these backends backends: [azure, lambda] ``` Check [`.dstack/profiles.yml`](../reference/profiles.yml.md) to see what properties can be defined there. A profile can be set as `default` to apply automatically to any run, or specified with `--profile NAME` in `dstack apply`. ## Projects If you're using multiple `dstack` projects (e.g., from different `dstack` servers), you can switch between them using the [`dstack project`](../reference/cli/dstack/project.md) command. ??? info ".envrc" Alternatively, you can install [`direnv`](https://direnv.net/) to automatically apply environment variables from the `.envrc` file in your project directory.
```shell export DSTACK_PROJECT=main ```
Now, `dstack` will always use this project within this directory. Remember to add `.envrc` to `.gitignore` to avoid committing it to the repo. ## Attached mode By default, `dstack apply` runs in attached mode. This means it streams the logs as they come in and, in the case of a task, forwards its ports to `localhost`. To run in detached mode, use `-d` with `dstack apply`. > If you detached the CLI, you can always re-attach to a run via [`dstack attach`](../reference/cli/dstack/attach.md). ## GPU specification `dstack` natively supports NVIDIA GPU, AMD GPU, and Google Cloud TPU accelerator chips. The `gpu` property within [`resources`](../reference/dstack.yml/dev-environment.md#resources) (or the `--gpu` option with [`dstack apply`](../reference/cli/dstack/apply.md) or [`dstack offer`](../reference/cli/dstack/offer.md)) allows specifying not only memory size but also GPU vendor, names, their memory, and quantity. The general format is: `:::`. Each component is optional. Ranges can be: * **Closed** (e.g. `24GB..80GB` or `1..8`) * **Open** (e.g. `24GB..` or `1..`) * **Single values** (e.g. `1` or `24GB`). Examples: - `1` (any GPU) - `amd:2` (two AMD GPUs) - `A100` (A100) - `24GB..` (any GPU starting from 24GB) - `24GB..40GB:2` (two GPUs between 24GB and 40GB) - `A10G,A100` (either A10G or A100) - `A100:80GB` (one A100 of 80GB) - `A100:2` (two A100) - `MI300X:4` (four MI300X) - `A100:40GB:2` (two A100 40GB) - `tpu:v2-8` (`v2` Google Cloud TPU with 8 cores) The GPU vendor is indicated by one of the following case-insensitive values: - `nvidia` (NVIDIA GPUs) - `amd` (AMD GPUs) - `tpu` (Google Cloud TPUs) ??? info "AMD" Currently, when an AMD GPU is specified, either by name or by vendor, the `image` property must be specified as well. ??? info "TPU" Currently, you can't specify other than 8 TPU cores. This means only single host workloads are supported. Support for multiple hosts is coming soon. ## Offers If you're not sure which offers (hardware configurations) are available with the configured backends, use the [`dstack offer`](../reference/cli/dstack/offer.md#list-gpu-offers) command.
```shell $ dstack offer --gpu H100 --max-offers 10 Getting offers... ---> 100% # BACKEND REGION INSTANCE TYPE RESOURCES SPOT PRICE 1 verda FIN-01 1H100.80S.30V 30xCPU, 120GB, 1xH100 (80GB), 100.0GB (disk) no $2.19 2 verda FIN-02 1H100.80S.30V 30xCPU, 120GB, 1xH100 (80GB), 100.0GB (disk) no $2.19 3 verda FIN-02 1H100.80S.32V 32xCPU, 185GB, 1xH100 (80GB), 100.0GB (disk) no $2.19 4 verda ICE-01 1H100.80S.32V 32xCPU, 185GB, 1xH100 (80GB), 100.0GB (disk) no $2.19 5 runpod US-KS-2 NVIDIA H100 PCIe 16xCPU, 251GB, 1xH100 (80GB), 100.0GB (disk) no $2.39 6 runpod CA NVIDIA H100 80GB HBM3 24xCPU, 251GB, 1xH100 (80GB), 100.0GB (disk) no $2.69 7 nebius eu-north1 gpu-h100-sxm 16xCPU, 200GB, 1xH100 (80GB), 100.0GB (disk) no $2.95 8 runpod AP-JP-1 NVIDIA H100 80GB HBM3 20xCPU, 251GB, 1xH100 (80GB), 100.0GB (disk) no $2.99 9 runpod CA-MTL-1 NVIDIA H100 80GB HBM3 28xCPU, 251GB, 1xH100 (80GB), 100.0GB (disk) no $2.99 10 runpod CA-MTL-2 NVIDIA H100 80GB HBM3 26xCPU, 125GB, 1xH100 (80GB), 100.0GB (disk) no $2.99 ... Shown 10 of 99 offers, $127.816 max ```
By default, `dstack offer` ignores fleet configurations and shows all available offers that match the request. To inspect offers available through a specific fleet, pass `--fleet NAME`. ??? info "Grouping offers" Use `--group-by` to aggregate offers. Accepted values: `gpu`, `backend`, `region`, and `count`.
```shell dstack offer --gpu b200 --group-by gpu,backend,region Project main User admin Resources cpu=2.. mem=8GB.. disk=100GB.. b200:1.. Spot policy auto Max price - Reservation - Group by gpu, backend, region # GPU SPOT $/GPU BACKEND REGION 1 B200:180GB:1..8 spot, on-demand 3.59..5.99 runpod EU-RO-1 2 B200:180GB:1..8 spot, on-demand 3.59..5.99 runpod US-CA-2 3 B200:180GB:8 on-demand 4.99 lambda us-east-1 4 B200:180GB:8 on-demand 5.5 nebius us-central1 ```
When using `--group-by`, `gpu` must always be `included`. The `region` value can only be used together with `backend`. The `offer` command allows you to filter and group offers with various [advanced options](../reference/cli/dstack/offer.md#usage). ## Metrics `dstack` tracks essential metrics accessible via the CLI and UI. To access advanced metrics like DCGM, configure the server to export metrics to Prometheus. See [Metrics](../concepts/metrics.md) for details. ## Service quotas If you're using your own AWS, GCP, Azure, or OCI accounts, before you can use GPUs or spot instances, you have to request the corresponding service quotas for each type of instance in each region. ??? info "AWS" Check this [guide ](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html) on EC2 service quotas. The relevant service quotas include: - `Running On-Demand P instances` (on-demand V100, A100 80GB x8) - `All P4, P3 and P2 Spot Instance Requests` (spot V100, A100 80GB x8) - `Running On-Demand G and VT instances` (on-demand T4, A10G, L4) - `All G and VT Spot Instance Requests` (spot T4, A10G, L4) - `Running Dedicated p5 Hosts` (on-demand H100) - `All P5 Spot Instance Requests` (spot H100) ??? info "GCP" Check this [guide ](https://cloud.google.com/compute/resource-usage) on Compute Engine service quotas. The relevant service quotas include: - `NVIDIA V100 GPUs` (on-demand V100) - `Preemtible V100 GPUs` (spot V100) - `NVIDIA T4 GPUs` (on-demand T4) - `Preemtible T4 GPUs` (spot T4) - `NVIDIA L4 GPUs` (on-demand L4) - `Preemtible L4 GPUs` (spot L4) - `NVIDIA A100 GPUs` (on-demand A100) - `Preemtible A100 GPUs` (spot A100) - `NVIDIA A100 80GB GPUs` (on-demand A100 80GB) - `Preemtible A100 80GB GPUs` (spot A100 80GB) - `NVIDIA H100 GPUs` (on-demand H100) - `Preemtible H100 GPUs` (spot H100) ??? info "Azure" Check this [guide ](https://learn.microsoft.com/en-us/azure/quotas/quickstart-increase-quota-portal) on Azure service quotas. The relevant service quotas include: - `Total Regional Spot vCPUs` (any spot instances) - `Standard NCASv3_T4 Family vCPUs` (on-demand T4) - `Standard NVADSA10v5 Family vCPUs` (on-demand A10) - `Standard NCADS_A100_v4 Family vCPUs` (on-demand A100 80GB) - `Standard NDASv4_A100 Family vCPUs` (on-demand A100 40GB x8) - `Standard NDAMSv4_A100Family vCPUs` (on-demand A100 80GB x8) - `Standard NCadsH100v5 Family vCPUs` (on-demand H100) - `Standard NDSH100v5 Family vCPUs` (on-demand H100 x8) ??? info "OCI" Check this [guide ](https://docs.oracle.com/en-us/iaas/Content/General/Concepts/servicelimits.htm#Requesti) on requesting OCI service limits increase. The relevant service category is compute. The relevant resources include: - `GPUs for GPU.A10 based VM and BM instances` (on-demand A10) - `GPUs for GPU2 based VM and BM instances` (on-demand P100) - `GPUs for GPU3 based VM and BM instances` (on-demand V100) Note, for AWS, GCP, and Azure, service quota values are measured with the number of CPUs rather than GPUs. [//]: # (TODO: Mention spot policy) # docs/guides/upgrade.md --- title: Upgrade description: Upgrading to newer versions of dstack --- # Upgrade guide ## 0.20.* { #0_20 } ### CLI compatibility - CLI versions `0.19.*` and earlier remain backward compatible with the `0.20.*` `dstack` server. - CLI versions `0.20.` are not compatible with server versions prior to `0.20.*`. > Do not upgrade the CLI to `0.20.*` until the server has been upgraded. ### Fleets * Prior to `0.20`, `dstack` automatically provisioned a fleet if one did not exist at run time. Beginning with `0.20`, `dstack` will only use existing fleets. > Create fleets before submitting runs. To enable on-demand instance provisioning, configure `nodes` as a range in the [backend fleet](../concepts/fleets.md#backend-fleets) configuration. ### Working directory - Previously, when `working_dir` was not specified, `dstack` defaulted to `/workflow`. As of `0.20`, `dstack` uses the working directory defined in the Docker image. If the image does not define a working directory, `dstack` falls back to `/`. - The default image introduced in `0.20` uses `/dstack/run` as its default working directory. > To override the directory defined in the Docker image, specify [`working_dir`](../concepts/dev-environments.md#working-directory) explicitly. ### Repo directory - Previously, if no [repo directory](../concepts/dev-environments.md#repos) was specified, `dstack` cloned the repository into `/workflow`. With `0.20`, the working directory becomes the default repo directory. - In earlier versions, cloning was skipped if the repo directory was non-empty. Starting with `0.20`, this results in a `runner error` unless `if_exists` is set to `skip` in the repo configuration. > Ensure repo directories are empty, or explicitly set `if_exists` to `skip`. ### Deprecated feature removal The following deprecated commands have been removed in **0.20**: - `dstack config` - `dstack stats` - `dstack gateway create` Use the corresponding replacements: - `dstack project` - `dstack metrics` - `dstack apply` > For more details on the changes, see the [release notes](https://github.com/dstackai/dstack/releases). # docs/guides/migration/slurm.md --- title: Migrate from Slurm description: This guide compares Slurm and dstack, and shows how to orchestrate equivalent GPU-based workloads using dstack. --- # Migrate from Slurm Both Slurm and `dstack` are open-source workload orchestration systems designed to manage compute resources and schedule jobs. This guide compares Slurm and `dstack`, maps features between the two systems, and shows their `dstack` equivalents. !!! tip "Slurm vs dstack" Slurm is a battle-tested system with decades of production use in HPC environments. `dstack` is designed for modern ML/AI workloads with cloud-native provisioning and container-first architecture. Slurm is better suited for traditional HPC centers with static clusters; `dstack` is better suited for cloud-native ML teams working with cloud GPUs. Both systems can handle distributed training and batch workloads. | | Slurm | dstack | |---|-------|--------| | **Provisioning** | Pre-configured static clusters; cloud requires third-party integrations with potential limitations | Native integration with top GPU clouds; automatically provisions clusters on demand | | **Containers** | Optional via plugins | Built around containers from the ground up | | **Use cases** | Batch job scheduling and distributed training | Interactive development, distributed training, and production inference services | | **Personas** | HPC centers, academic institutions, research labs | ML engineering teams, AI startups, cloud-native organizations | While `dstack` is designed to be use-case agnostic and supports both development and production-grade inference, this guide focuses specifically on training workloads. ## Architecture Both Slurm and `dstack` follow a client-server architecture with a control plane and a compute plane running on cluster instances. | | Slurm | dstack | |---|---------------|-------------------| | **Control plane** | `slurmctld` (controller) | `dstack-server` | | **State persistence** | `slurmdbd` (database) | `dstack-server` (SQLite/PostgreSQL) | | **API** | `slurmrestd` (REST API) | `dstack-server` (HTTP API) | | **Compute plane** | `slurmd` (compute agent) | `dstack-shim` (on VMs/hosts) and/or `dstack-runner` (inside containers) | | **Client** | CLI from login nodes | CLI from anywhere | | **High availability** | Active-passive failover (typically 2 controller nodes) | Horizontal scaling with multiple server replicas (requires PostgreSQL) | ## Job configuration and submission Both Slurm and `dstack` allow defining jobs as files and submitting them via CLI. ### Slurm Slurm uses shell scripts with `#SBATCH` directives embedded in the script:
```bash #!/bin/bash #SBATCH --job-name=train-model #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=8 #SBATCH --gres=gpu:1 #SBATCH --mem=32G #SBATCH --time=2:00:00 #SBATCH --partition=gpu #SBATCH --output=train-%j.out #SBATCH --error=train-%j.err export HF_TOKEN export LEARNING_RATE=0.001 module load python/3.9 srun python train.py --batch-size=64 ```
Submit the job from a login node (with environment variables that override script defaults):
```shell $ sbatch --export=ALL,LEARNING_RATE=0.002 train.sh Submitted batch job 12346 ```
### dstack `dstack` uses declarative YAML configuration files:
```yaml type: task name: train-model python: 3.9 repos: - . env: - HF_TOKEN - LEARNING_RATE=0.001 commands: - python train.py --batch-size=64 resources: gpu: 1 memory: 32GB cpu: 8 shm_size: 8GB max_duration: 2h ```
Submit the job from anywhere (laptop, CI/CD) via the CLI. `dstack apply` allows overriding various options and runs in attached mode by default, streaming job output in real-time:
```shell $ dstack apply -f .dstack.yml --env LEARNING_RATE=0.002 # BACKEND REGION RESOURCES SPOT PRICE 1 aws us-east-1 4xCPU, 16GB, T4:1 yes $0.10 Submit the run train-model? [y/n]: y Launching `train-model`... ---> 100% ```
### Configuration comparison | | Slurm | dstack | |---|-------|--------| | **File type** | Shell script with `#SBATCH` directives | YAML configuration file (`.dstack.yml`) | | **GPU** | `--gres=gpu:N` or `--gres=gpu:type:N` | `gpu: A100:80GB:4` or `gpu: 40GB..80GB:2..8` (supports ranges) | | **Memory** | `--mem=M` (per node) or `--mem-per-cpu=M` | `memory: 200GB..` (range, per node, minimum requirement) | | **CPU** | `--cpus-per-task=C` or `--ntasks` | `cpu: 32` (per node) | | **Shared memory** | Configured on host | `shm_size: 24GB` (explicit) | | **Duration** | `--time=2:00:00` | `max_duration: 2h` (both enforce walltime) | | **Cluster** | `--partition=gpu` | `fleets: [gpu]` (see Partitions and fleets below) | | **Output** | `--output=train-%j.out` (writes files) | `dstack logs` or UI (streams via API) | | **Working directory** | `--chdir=/path/to/dir` or defaults to submission directory | `working_dir: /path/to/dir` (defaults to image's working directory, typically `/dstack/run`) | | **Environment variables** | `export VAR` or `--export=ALL,VAR=value` | `env: - VAR` or `--env VAR=value` | | **Node exclusivity** | `--exclusive` (entire node) | Automatic if `blocks` is not used or job uses all blocks; required for distributed tasks (`nodes` > 1) | > For multi-node examples, see [Distributed training](#distributed-training) below. ## Containers ### Slurm By default, Slurm runs jobs on compute nodes using the host OS with cgroups for resource isolation and full access to the host filesystem. Container execution is optional via plugins but require explicit filesystem mounts. === "Singularity/Apptainer" Container image must exist on shared filesystem. Mount host directories with `--container-mounts`: ```bash #!/bin/bash #SBATCH --nodes=1 #SBATCH --gres=gpu:1 #SBATCH --mem=32G #SBATCH --time=2:00:00 srun --container-image=/shared/images/pytorch-2.0-cuda11.8.sif \ --container-mounts=/shared/datasets:/datasets,/shared/checkpoints:/checkpoints \ python train.py --batch-size=64 ``` === "Pyxis with Enroot" Pyxis plugin pulls images from Docker registry. Mount host directories with `--container-mounts`: ```bash #!/bin/bash #SBATCH --nodes=1 #SBATCH --gres=gpu:1 #SBATCH --mem=32G #SBATCH --time=2:00:00 srun --container-image=pytorch/pytorch:2.0.0-cuda11.8-cudnn8-runtime \ --container-mounts=/shared/datasets:/datasets,/shared/checkpoints:/checkpoints \ python train.py --batch-size=64 ``` === "Enroot" Pulls images from registry. Mount host directories with `--container-mounts`: ```bash #!/bin/bash #SBATCH --nodes=1 #SBATCH --gres=gpu:1 #SBATCH --mem=32G #SBATCH --time=2:00:00 srun --container-image=docker://pytorch/pytorch:2.0.0-cuda11.8-cudnn8-runtime \ --container-mounts=/shared/datasets:/datasets,/shared/checkpoints:/checkpoints \ python train.py --batch-size=64 ``` ### dstack `dstack` always uses container. If `image` is not specified, `dstack` uses a base Docker image with `uv`, `python`, essential CUDA drivers, and other dependencies. You can also specify your own Docker image: === "Public registry" ```yaml type: task name: train-with-image image: pytorch/pytorch:2.0.0-cuda11.8-cudnn8-runtime repos: - . commands: - python train.py --batch-size=64 resources: gpu: 1 memory: 32GB ``` === "Private registry" ```yaml type: task name: train-ngc image: nvcr.io/nvidia/pytorch:24.01-py3 registry_auth: username: $oauthtoken password: ${{ secrets.nvidia_ngc_api_key }} repos: - . commands: - python train.py --batch-size=64 resources: gpu: 1 memory: 32GB ``` `dstack` can automatically upload files via `repos` or `files`, or mount filesystems via `volumes`. See [Filesystems and data access](#filesystems-and-data-access) below. ## Distributed training Both Slurm and `dstack` schedule distributed workloads over clusters with fast interconnect, automatically propagating environment variables required by distributed frameworks (PyTorch DDP, DeepSpeed, FSDP, etc.). ### Slurm Slurm explicitly controls both `nodes` and processes/tasks. === "PyTorch DDP" ```bash #!/bin/bash #SBATCH --job-name=distributed-train #SBATCH --nodes=4 #SBATCH --ntasks-per-node=1 # One task per node #SBATCH --gres=gpu:8 # 8 GPUs per node #SBATCH --mem=200G #SBATCH --time=24:00:00 #SBATCH --partition=gpu # Set up distributed training environment MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1) MASTER_PORT=12345 export MASTER_ADDR MASTER_PORT # Launch training with torchrun (torch.distributed.launch is deprecated) srun torchrun \ --nnodes="$SLURM_JOB_NUM_NODES" \ --nproc_per_node=8 \ --node_rank="$SLURM_NODEID" \ --rdzv_backend=c10d \ --rdzv_endpoint="$MASTER_ADDR:$MASTER_PORT" \ train.py \ --model llama-7b \ --batch-size=32 \ --epochs=10 ``` === "MPI" ```bash #!/bin/bash #SBATCH --nodes=2 #SBATCH --ntasks=16 #SBATCH --gres=gpu:8 #SBATCH --mem=200G #SBATCH --time=24:00:00 export MASTER_ADDR=$(scontrol show hostnames $SLURM_NODELIST | head -n1) export MASTER_PORT=12345 # Convert SLURM_JOB_NODELIST to hostfile format HOSTFILE=$(mktemp) scontrol show hostnames $SLURM_JOB_NODELIST | awk -v slots=$SLURM_NTASKS_PER_NODE '{print $0" slots="slots}' > $HOSTFILE # MPI with NCCL tests or custom MPI application mpirun \ --allow-run-as-root \ --hostfile $HOSTFILE \ -n $SLURM_NTASKS \ --bind-to none \ /opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 rm -f $HOSTFILE ``` ### dstack `dstack` only specifies `nodes`. A run with multiple nodes creates multiple jobs (one per node), each running in a container on a particular instance. Inside the job container, processes are determined by the user's `commands`. === "PyTorch DDP" ```yaml type: task name: distributed-train-pytorch nodes: 4 python: 3.12 repos: - . env: - NCCL_DEBUG=INFO - NCCL_IB_DISABLE=0 - NCCL_SOCKET_IFNAME=eth0 commands: - | torchrun \ --nproc-per-node=$DSTACK_GPUS_PER_NODE \ --node-rank=$DSTACK_NODE_RANK \ --nnodes=$DSTACK_NODES_NUM \ --master-addr=$DSTACK_MASTER_NODE_IP \ --master-port=12345 \ train.py \ --model llama-7b \ --batch-size=32 \ --epochs=10 resources: gpu: A100:80GB:8 memory: 200GB.. shm_size: 24GB max_duration: 24h ``` === "MPI" For MPI workloads that require specific job startup and termination behavior, `dstack` provides `startup_order` and `stop_criteria` properties. The master node (rank 0) runs the MPI command, while worker nodes wait for the master to complete. ```yaml type: task name: nccl-tests nodes: 2 startup_order: workers-first stop_criteria: master-done env: - NCCL_DEBUG=INFO commands: - | if [ $DSTACK_NODE_RANK -eq 0 ]; then mpirun \ --allow-run-as-root \ --hostfile $DSTACK_MPI_HOSTFILE \ -n $DSTACK_GPUS_NUM \ -N $DSTACK_GPUS_PER_NODE \ --bind-to none \ /opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 else sleep infinity fi resources: gpu: nvidia:1..8 shm_size: 16GB ``` If `startup_order` and `stop_criteria` are not configured (as in the PyTorch DDP example above), the master worker starts first and waits until all workers terminate. For MPI workloads, we need to change this. #### Nodes and processes comparison | | Slurm | dstack | |---|-------|--------| | **Nodes** | `--nodes=4` | `nodes: 4` | | **Processes/tasks** | `--ntasks=8` or `--ntasks-per-node=2` (controls process distribution) | Determined by `commands` (relies on frameworks like `torchrun`, `accelerate`, `mpirun`, etc.) | **Environment variables comparison:** | Slurm | dstack | Purpose | |-------|--------|---------| | `SLURM_NODELIST` | `DSTACK_NODES_IPS` | Newline-delimited list of node IPs | | `SLURM_NODEID` | `DSTACK_NODE_RANK` | Node rank (0-based) | | `SLURM_PROCID` | N/A | Process rank (0-based, across all processes) | | `SLURM_NTASKS` | `DSTACK_GPUS_NUM` | Total number of processes/GPUs | | `SLURM_NTASKS_PER_NODE` | `DSTACK_GPUS_PER_NODE` | Number of processes/GPUs per node | | `SLURM_JOB_NUM_NODES` | `DSTACK_NODES_NUM` | Number of nodes | | Manual master address | `DSTACK_MASTER_NODE_IP` | Master node IP (automatically set) | | N/A | `DSTACK_MPI_HOSTFILE` | Pre-populated MPI hostfile | !!! info "Fleets" Distributed tasks may run only on a fleet with `placement: cluster` configured. Refer to [Partitions and fleets](#partitions-and-fleets) for configuration details. ## Queueing and scheduling Both systems support core scheduling features and efficient resource utilization. | | Slurm | dstack | |---------|-------|--------| | **Prioritization** | Multi-factor system (fairshare, age, QOS); influenced via `--qos` or `--partition` flags | Set via `priority` (0-100); plus FIFO within the same priority | | **Queueing** | Automatic via `sbatch`; managed through partitions | Set `on_events` to `[no-capacity]` under `retry` configuration | | **Usage quotas** | Set via `sacctmgr` command per user/account/QOS | Not supported | | **Backfill scheduling** | Enabled via `SchedulerType=sched/backfill` in `slurm.conf` | Not supported | | **Preemption** | Configured via `PreemptType` in `slurm.conf` (QOS or partition-based) | Not supported | | **Topology-aware scheduling** | Configured via `topology.conf` (InfiniBand switches, interconnects) | Not supported | ### Slurm Slurm may use a multi-factor priority system, and limit usage across accounts, users, and runs. #### QOS Quality of Service (QOS) provides a static priority boost. Administrators create QOS levels and assign them to users as defaults:
```shell $ sacctmgr add qos high_priority Priority=1000 $ sacctmgr modify qos high_priority set MaxWall=200:00:00 MaxTRES=gres/gpu=8 ```
Users can override the default QOS when submitting jobs via CLI (`sbatch --qos=high_priority`) or in the job script:
```bash #!/bin/bash #SBATCH --qos=high_priority ```
#### Accounts and usage quotas Usage quotas limit resource consumption and can be set per user, account, or QOS:
```shell $ sacctmgr add account research $ sacctmgr modify user user1 set account=research $ sacctmgr modify user user1 set MaxWall=100:00:00 MaxTRES=gres/gpu=4 $ sacctmgr modify account research set MaxWall=1000:00:00 MaxTRES=gres/gpu=16 ```
#### Monitoring commands Slurm provides several CLI commands to check queue status, job details, and quota usage: === "Queue status" Use `squeue` to check queue status. Jobs are listed in scheduling order by priority:
```shell $ squeue -u $USER JOBID PARTITION NAME USER ST TIME NODES REASON 12345 gpu training user1 PD 0:00 2 Priority ```
=== "Job details" Use `scontrol show job` to show detailed information about a specific job:
```shell $ scontrol show job 12345 JobId=12345 JobName=training UserId=user1(1001) GroupId=users(100) Priority=4294 Reason=Priority (Resources) ```
=== "Quota usage" The `sacct` command can show quota consumption per user, account, or QOS depending on the format options:
```shell $ sacct -S 2024-01-01 -E 2024-01-31 --format=User,Account,TotalCPU,TotalTRES User Account TotalCPU TotalTRES user1 research 100:00:00 gres/gpu=50 ```
#### Topology-aware scheduling Slurm detects network topology (InfiniBand switches, interconnects) and optimizes multi-node job placement to minimize latency. Configured in `topology.conf`, referenced from `slurm.conf`:
```bash SwitchName=switch1 Nodes=node[01-10] SwitchName=switch2 Nodes=node[11-20] ```
When scheduling multi-node jobs, Slurm prioritizes nodes connected to the same switch to minimize network latency. ### dstack `dstack` doesn't have the concept of accounts, QOS, and doesn't support usage quotas yet. #### Priority and retry policy However, `dstack` supports prioritization (integer, no multi-factor or pre-emption) and queueing jobs.
```yaml type: task name: train-with-retry python: 3.12 repos: - . commands: - python train.py --batch-size=64 resources: gpu: 1 memory: 32GB # Priority: 0-100 (FIFO within same level; default: 0) priority: 50 retry: on_events: [no-capacity] # Retry until idle instances are available (enables queueing similar to Slurm) duration: 48h # Maximum retry time (run age for no-capacity, time since last event for error/interruption) max_duration: 2h ```
By default, the `retry` policy is not set, which means run fails immediately if no capacity is available. #### Scheduled runs Unlike Slurm, `dstack` supports scheduled runs using the `schedule` property with cron syntax, allowing tasks to start periodically at specific UTC times.
```yaml type: task name: task-with-cron python: 3.12 repos: - . commands: - python task.py --batch-size=64 resources: gpu: 1 memory: 32GB schedule: cron: "15 23 * * *" # everyday at 23:15 UTC ```
#### Monitoring commands === "Queue status" The `dstack ps` command displays runs and jobs sorted by priority, reflecting the order in which they will be scheduled.
```shell $ dstack ps NAME BACKEND RESOURCES PRICE STATUS SUBMITTED training-job aws H100:1 (spot) $4.50 provisioning 2 mins ago ```
#### Topology-aware scheduling Topology-aware scheduling is not supported in `dstack`. While backend provisioning may respect network topology (e.g., cloud providers may provision instances with optimal inter-node connectivity), `dstack` task scheduling does not leverage topology-aware placement. ## Partitions and fleets Partitions in Slurm and fleets in `dstack` both organize compute nodes for job scheduling. The key difference is that `dstack` fleets natively support dynamic cloud provisioning, whereas Slurm partitions organize pre-configured static nodes. | | Slurm | dstack | |---|-------|--------| | **Provisioning** | Static nodes only | Supports both static clusters (SSH fleets) and dynamic provisioning via backends (cloud or Kubernetes) | | **Overlap** | Nodes can belong to multiple partitions | Each instance belongs to exactly one fleet | | **Accounts and projects** | Multiple accounts can use the same partition; used for quotas and resource accounting | Each fleet belongs to one project | ### Slurm Slurm partitions are logical groupings of static nodes defined in `slurm.conf`. Nodes can belong to multiple partitions:
```bash PartitionName=gpu Nodes=gpu-node[01-10] Default=NO MaxTime=24:00:00 PartitionName=cpu Nodes=cpu-node[01-50] Default=YES MaxTime=72:00:00 PartitionName=debug Nodes=gpu-node[01-10] Default=NO MaxTime=1:00:00 ```
Submit to a specific partition:
```shell $ sbatch --partition=gpu train.sh Submitted batch job 12346 ```
### dstack `dstack` fleets are pools of instances (VMs or containers) that serve as both the organization unit and the provisioning template. `dstack` supports two types of fleets: | Fleet type | Description | |------------|-------------| | **Backend fleets** | Dynamically provisioned via configured backends (cloud or Kubernetes). Specify `resources` and `nodes` range; `dstack apply` provisions matching instances/clusters automatically. | | **SSH fleets** | Use existing on-premises servers/clusters via `ssh_config`. `dstack apply` connects via SSH, installs dependencies. | === "Backend fleets"
```yaml type: fleet name: gpu-fleet nodes: 0..8 resources: gpu: A100:80GB:8 # Optional: Enables inter-node connectivity; required for distributed tasks placement: cluster # Optional: Split GPUs into blocks for multi-tenant sharing # Optional: Allows to share the instance across up to 8 workloads blocks: 8 backends: [aws] # Spot instances for cost savings spot_policy: auto ```
=== "SSH fleets"
```yaml type: fleet name: on-prem-gpu-fleet # Optional: Enables inter-node connectivity; required for distributed tasks placement: cluster # Optional: Allows to share the instance across up to 8 workloads blocks: 8 ssh_config: user: dstack identity_file: ~/.ssh/id_rsa hosts: - gpu-node01.example.com - gpu-node02.example.com # Optional: Only required if hosts are behind a login node (bastion host) proxy_jump: hostname: login-node.example.com user: dstack identity_file: ~/.ssh/login_node_key ```
Tasks with multiple nodes require a fleet with `placement: cluster` configured, otherwise they cannot run. Submit to a specific fleet:
```shell $ dstack apply -f train.dstack.yml --fleet gpu-fleet BACKEND REGION RESOURCES SPOT PRICE 1 aws us-east-1 4xCPU, 16GB, T4:1 yes $0.10 Submit the run train-model? [y/n]: y Launching `train-model`... ---> 100% ```
Create or update a fleet:
```shell $ dstack apply -f fleet.dstack.yml Provisioning... ---> 100% ```
List fleets:
```shell $ dstack fleet FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED gpu-fleet 0 aws (us-east-1) A100:80GB (spot) $0.50 idle 3 mins ago ```
## Filesystems and data access Both Slurm and `dstack` allow workloads to access filesystems (including shared filesystems) and copy files. | | Slurm | dstack | |---|-------|--------| | **Host filesystem access** | Full access by default (native processes); mounting required only for containers | Always uses containers; requires explicit mounting via `volumes` (instance or network) | | **Shared filesystems** | Assumes global namespace (NFS, Lustre, GPFS); same path exists on all nodes | Supported via SSH fleets with instance volumes (pre-mounted network storage); network volumes for backend fleets (limited support for shared filesystems) | | **Instance disk size** | Fixed by cluster administrator | Configurable via `disk` property in `resources` (tasks) or fleet configuration; supports ranges (e.g., `disk: 500GB` or `disk: 200GB..1TB`) | | **Local/temporary storage** | `$SLURM_TMPDIR` (auto-cleaned on job completion) | Container filesystem (auto-cleaned on job completion; except instance volumes or network volumes) | | **File transfer** | `sbcast` for broadcasting files to allocated nodes | `repos` and `files` properties; `rsync`/`scp` via SSH (when attached) | ### Slurm Slurm assumes a shared filesystem (NFS, Lustre, GPFS) with a global namespace. The same path exists on all nodes, and `$SLURM_TMPDIR` provides local scratch space that is automatically cleaned. === "Native processes"
```bash #!/bin/bash #SBATCH --nodes=4 #SBATCH --gres=gpu:8 #SBATCH --time=24:00:00 # Global namespace - same path on all nodes # Dataset accessible at same path on all nodes DATASET_PATH=/shared/datasets/imagenet # Local scratch (faster I/O, auto-cleaned) # Copy dataset to local SSD for faster access cp -r $DATASET_PATH $SLURM_TMPDIR/dataset # Training with local dataset python train.py \ --data=$SLURM_TMPDIR/dataset \ --checkpoint-dir=/shared/checkpoints \ --epochs=100 # $SLURM_TMPDIR automatically cleaned when job ends # Checkpoints saved to shared filesystem persist ```
=== "Containers" When using containers, shared filesystems must be explicitly mounted via bind mounts:
```bash #!/bin/bash #SBATCH --nodes=4 #SBATCH --gres=gpu:8 #SBATCH --time=24:00:00 # Shared filesystem mounted at /datasets and /checkpoints DATASET_PATH=/datasets/imagenet # Local scratch accessible via $SLURM_TMPDIR (host storage mounted into container) # Copy dataset to local scratch, then train srun --container-image=/shared/images/pytorch-2.0-cuda11.8.sif \ --container-mounts=/shared/datasets:/datasets,/shared/checkpoints:/checkpoints \ cp -r $DATASET_PATH $SLURM_TMPDIR/dataset srun --container-image=/shared/images/pytorch-2.0-cuda11.8.sif \ --container-mounts=/shared/datasets:/datasets,/shared/checkpoints:/checkpoints \ python train.py \ --data=$SLURM_TMPDIR/dataset \ --checkpoint-dir=/checkpoints \ --epochs=100 # \$SLURM_TMPDIR automatically cleaned when job ends # Checkpoints saved to mounted shared filesystem persist ```
#### File broadcasting (sbcast) Slurm provides `sbcast` to distribute files efficiently using its internal network topology, avoiding filesystem contention:
```bash #!/bin/bash #SBATCH --nodes=4 #SBATCH --ntasks=32 # Broadcast file to all allocated nodes srun --ntasks=1 --nodes=1 sbcast /shared/data/input.txt /tmp/input.txt # Use broadcasted file on all nodes srun python train.py --input=/tmp/input.txt ```
### dstack `dstack` supports both accessing filesystems (including shared filesystems) and uploading/downloading code/data from the client. #### Instance volumes Instance volumes mount host directories into containers. With distributed tasks, the host can use a shared filesystem (NFS, Lustre, GPFS) to share data across jobs within the same task:
```yaml type: task name: distributed-train nodes: 4 python: 3.12 repos: - . volumes: # Host directory (can be on shared filesystem) mounted into container - /mnt/shared/datasets:/data - /mnt/shared/checkpoints:/checkpoints commands: - | torchrun \ --nproc-per-node=$DSTACK_GPUS_PER_NODE \ --node-rank=$DSTACK_NODE_RANK \ --nnodes=$DSTACK_NODES_NUM \ --master-addr=$DSTACK_MASTER_NODE_IP \ --master-port=12345 \ train.py \ --data=/data \ --checkpoint-dir=/checkpoints resources: gpu: A100:80GB:8 memory: 200GB ```
#### Network volumes Network volumes are persistent cloud storage (AWS EBS, GCP persistent disks, Runpod volumes). Single-node task:
```yaml type: task name: train-model python: 3.9 repos: - . volumes: - name: imagenet-dataset path: /data commands: - python train.py --data=/data --batch-size=64 resources: gpu: 1 memory: 32GB ```
Network volumes cannot be used with distributed tasks (no multi-attach support), except where multi-attach is supported (Runpod) or via volume interpolation. For distributed tasks, use interpolation to attach different volumes to each node.
```yaml type: task name: distributed-train nodes: 4 python: 3.12 repos: - . volumes: # Each node gets its own volume - name: dataset-${{ dstack.node_rank }} path: /data commands: - | torchrun \ --nproc-per-node=$DSTACK_GPUS_PER_NODE \ --node-rank=$DSTACK_NODE_RANK \ --nnodes=$DSTACK_NODES_NUM \ --master-addr=$DSTACK_MASTER_NODE_IP \ --master-port=12345 \ train.py \ --data=/data resources: gpu: A100:80GB:8 memory: 200GB ```
Volume name interpolation is not the same as a shared filesystem—each node has its own separate volume. `dstack` currently has limited support for shared filesystems when using backend fleets. #### Repos and files The `repos` and `files` properties allow uploading code or data into the container. === "Repos" The `repos` property clones Git repositories into the container. `dstack` clones the repo on the instance, applies local changes, and mounts it into the container. This is useful for code that needs to be version-controlled and synced.
```yaml type: task name: train-model python: 3.9 repos: - . # Clone current directory repo commands: - python train.py --batch-size=64 resources: gpu: 1 memory: 32GB cpu: 8 ```
=== "Files" The `files` property mounts local files or directories into the container. Each entry maps a local path to a container path.
```yaml type: task name: train-model python: 3.9 files: - ../configs:~/configs - ~/.ssh/id_rsa:~/ssh/id_rsa commands: - python train.py --config ~/configs/model.yaml --batch-size=64 resources: gpu: 1 memory: 32GB cpu: 8 ```
Files are uploaded to the instance and mounted into the container, but are not persisted across runs (2MB limit per file, configurable). #### SSH file transfer While attached to a run, you can transfer files via `rsync` or `scp` using the run name alias: === "rsync"
```shell $ rsync -avz ./data/ :/path/inside/container/data/ ```
=== "scp"
```shell $ scp large-dataset.h5 :/path/inside/container/ ```
> Uploading code/data from/to the client is not recommended as transfer speed greatly depends on network bandwidth between the CLI and the instance. ## Interactive development Both Slurm and `dstack` allow allocating resources for interactive development. | | Slurm | dstack | |---|-------|--------| | **Configuration** | Uses `salloc` command to allocate resources with a time limit; resources are automatically released when time expires | Uses `type: dev-environment` configurations as first-class citizen; provisions compute and runs until explicitly stopped (optional inactivity-based termination) | | **IDE access** | Requires SSH access to allocated nodes | Native access using desktop IDEs (VS Code, Cursor, Windsurf, etc.) or SSH | | **SSH access** | SSH to allocated nodes (host OS) using `SLURM_NODELIST` or `srun --pty` | SSH automatically configured; access via run name alias (inside container) | ### Slurm Slurm uses `salloc` to allocate resources with a time limit. `salloc` returns a shell on the login node with environment variables set; use `srun` or SSH to access compute nodes. After the time limit expires, resources are automatically released:
```shell $ salloc --nodes=1 --gres=gpu:1 --time=4:00:00 salloc: Granted job allocation 12346 $ srun --pty bash [user@compute-node-01 ~]$ python train.py --epochs=1 Training epoch 1... [user@compute-node-01 ~]$ exit exit $ exit exit salloc: Relinquishing job allocation 12346 ```
Alternatively, SSH directly to allocated nodes using hostnames from `SLURM_NODELIST`:
```shell $ ssh $SLURM_NODELIST [user@compute-node-01 ~]$ ```
### dstack `dstack` uses `dev-environment` configuration type that automatically provisions an instance and runs until explicitly stopped, with optional inactivity-based termination. Access is provided via native desktop IDEs (VS Code, Cursor, Windsurf, etc.) or SSH:
```yaml type: dev-environment name: ml-dev python: 3.12 ide: vscode resources: gpu: A100:80GB:1 memory: 200GB # Optional: Maximum runtime duration (stops after this time) max_duration: 8h # Optional: Auto-stop after period of inactivity (no SSH/IDE connections) inactivity_duration: 2h # Optional: Auto-stop if GPU utilization is below threshold utilization_policy: min_gpu_utilization: 10 # Percentage time_window: 1h ```
Start the dev environment:
```shell $ dstack apply -f dev.dstack.yml BACKEND REGION RESOURCES SPOT PRICE 1 runpod CA-MTL-1 9xCPU, 48GB, A5000:24GB yes $0.11 Submit the run ml-dev? [y/n]: y Launching `ml-dev`... ---> 100% To open in VS Code Desktop, use this link: vscode://vscode-remote/ssh-remote+ml-dev/workflow ```
#### Port forwarding `dstack` tasks support exposing `ports` for running interactive applications like Jupyter notebooks or Streamlit apps: === "Jupyter"
```yaml type: task name: jupyter python: 3.12 commands: - pip install jupyterlab - jupyter lab --allow-root ports: - 8888 resources: gpu: 1 memory: 32GB ```
=== "Streamlit"
```yaml type: task name: streamlit-app python: 3.12 commands: - pip install streamlit - streamlit hello ports: - 8501 resources: gpu: 1 memory: 32GB ```
While `dstack apply` is attached, ports are automatically forwarded to `localhost` (e.g., `http://localhost:8888` for Jupyter, `http://localhost:8501` for Streamlit). ## Job arrays ### Slurm job arrays Slurm provides native job arrays (`--array=1-100`) that create multiple job tasks from a single submission. Job arrays can be specified via CLI argument or in the job script.
```shell $ sbatch --array=1-100 train.sh Submitted batch job 1001 ```
Each task can use the `$SLURM_ARRAY_TASK_ID` environment variable within the job script to determine its configuration. Output files can use `%A` for the job ID and `%a` for the task ID in `#SBATCH --output` and `--error` directives. ### dstack `dstack` does not support native job arrays. Submit multiple runs programmatically via CLI or API. Pass a custom environment variable (e.g., `TASK_ID`) to identify each run:
```shell $ for i in {1..100}; do dstack apply -f train.dstack.yml \ --name "train-array-task-${i}" \ --env TASK_ID=${i} \ --detach done ```
## Environment variables and secrets Both Slurm and `dstack` handle sensitive data (API keys, tokens, passwords) for ML workloads. Slurm uses environment variables or files, while `dstack` provides encrypted secrets management in addition to environment variables. ### Slurm Slurm uses OS-level authentication. Jobs run with the user's UID/GID and inherit the environment from the login node. No built-in secrets management; users manage credentials in their environment or shared files. Set environment variables in the shell before submitting (requires `--export=ALL`):
```shell $ export HF_TOKEN=$(cat ~/.hf_token) $ sbatch --export=ALL train.sh Submitted batch job 12346 ```
### dstack In addition to environment variables (`env`), `dstack` provides a secrets management system with encryption. Secrets are referenced in configuration using `${{ secrets.name }}` syntax. Set secrets:
```shell $ dstack secret set huggingface_token $ dstack secret set wandb_api_key ```
Use secrets in configuration:
```yaml type: task name: train-with-secrets python: 3.12 repos: - . env: - HF_TOKEN=${{ secrets.huggingface_token }} - WANDB_API_KEY=${{ secrets.wandb_api_key }} commands: - pip install huggingface_hub - huggingface-cli download meta-llama/Llama-2-7b-hf - wandb login - python train.py resources: gpu: A100:80GB:8 ```
## Authentication ### Slurm Slurm uses OS-level authentication. Users authenticate via SSH to login nodes using their Unix accounts. Jobs run with the user's UID/GID, ensuring user isolation—users cannot access other users' files or processes. Slurm enforces file permissions based on Unix UID/GID and association limits (MaxJobs, MaxSubmitJobs) configured per user or account. ### dstack `dstack` uses token-based authentication. Users are registered within projects on the server, and each user is issued a token. This token is used for authentication with all CLI and API commands. Access is controlled at the project level with user roles: | Role | Permissions | |------|-------------| | **Admin** | Can manage project settings, including backends, gateways, and members | | **Manager** | Can manage project members but cannot configure backends and gateways | | **User** | Can manage project resources including runs, fleets, and volumes | `dstack` manages SSH keys on the server for secure access to runs and instances. User SSH keys are automatically generated and used when attaching to runs via `dstack attach` or `dstack apply`. Project SSH keys are used by the server to establish SSH connections to provisioned instances. !!! note "Multi-tenancy isolation" `dstack` currently does not offer full isolation for multi-tenancy. Users may access global resources within the host. ## Monitoring and observability Both systems provide tools to monitor job/run status, cluster/node status, resource metrics, and logs: | | Slurm | dstack | |---|-------|--------| | **Job/run status** | `squeue` lists jobs in queue | `dstack ps` lists active runs | | **Cluster/node status** | `sinfo` shows node availability | `dstack fleet` lists instances | | **CPU/memory metrics** | `sstat` for running jobs | `dstack metrics` for real-time metrics | | **GPU metrics** | Requires SSH to nodes, `nvidia-smi` per node | Automatic collection via `nvidia-smi`/`amd-smi`, `dstack metrics` | | **Job history** | `sacct` for completed jobs | `dstack ps -n NUM` shows run history | | **Logs** | Written to files (`--output`, `--error`) | Streamed via API, `dstack logs` | ### Slurm Slurm provides command-line tools for monitoring cluster state, jobs, and history. Check node status:
```shell $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST gpu up 1-00:00:00 10 idle gpu-node[01-10] ```
Check job queue:
```shell $ squeue -u $USER JOBID PARTITION NAME USER ST TIME NODES 12345 gpu training user1 R 2:30 2 ```
Check job details:
```shell $ scontrol show job 12345 JobId=12345 JobName=training UserId=user1(1001) GroupId=users(100) NumNodes=2 NumCPUs=64 NumTasks=32 Gres=gpu:8(IDX:0,1,2,3,4,5,6,7) ```
Check resource usage for running jobs (`sstat` only works for running jobs):
```shell $ sstat --job=12345 --format=JobID,MaxRSS,MaxVMSize,CPUUtil JobID MaxRSS MaxVMSize CPUUtil 12345.0 2048M 4096M 95.2% ```
Check GPU usage (requires SSH to node):
```shell $ srun --jobid=12345 --pty nvidia-smi GPU 0: 95% utilization, 72GB/80GB memory ```
Check job history for completed jobs:
```shell $ sacct --job=12345 --format=JobID,Elapsed,MaxRSS,State,ExitCode JobID Elapsed MaxRSS State ExitCode 12345 2:30:00 2048M COMPLETED 0:0 ```
View logs (written to files via `--output` and `--error` flags; typically in the submission directory on a shared filesystem):
```shell $ cat slurm-12345.out Training started... Epoch 1/10: loss=0.5 ```
If logs are on compute nodes, find the node from `scontrol show job`, then access via `srun --jobid` (running jobs) or SSH (completed jobs):
```shell $ srun --jobid=12345 --nodelist=gpu-node01 --pty bash $ cat slurm-12345.out ```
### dstack `dstack` automatically collects essential metrics (CPU, memory, GPU utilization) using vendor utilities (`nvidia-smi`, `amd-smi`, etc.) and provides real-time monitoring via CLI. List runs:
```shell $ dstack ps NAME BACKEND GPU PRICE STATUS SUBMITTED training-job aws H100:1 (spot) $4.50 running 5 mins ago ```
List fleets and instances (shows GPU health status):
```shell $ dstack fleet FLEET INSTANCE BACKEND RESOURCES STATUS PRICE CREATED my-fleet 0 aws (us-east-1) T4:16GB:1 idle $0.526 11 mins ago 1 aws (us-east-1) T4:16GB:1 idle (warning) $0.526 11 mins ago ```
Check real-time metrics:
```shell $ dstack metrics training-job NAME STATUS CPU MEMORY GPU training-job running 45% 16.27GB/200GB gpu=0 mem=72.48GB/80GB util=95% ```
Stream logs (stored centrally using external storage services like CloudWatch Logs or GCP Logging, accessible via CLI and UI):
```shell $ dstack logs training-job Training started... Epoch 1/10: loss=0.5 ```
#### Prometheus integration `dstack` exports additional metrics to Prometheus: | Metric type | Description | |-------------|-------------| | **Fleet metrics** | Instance duration, price, GPU count | | **Run metrics** | Run counters (total, terminated, failed, done) | | **Job metrics** | Execution time, cost, CPU/memory/GPU usage | | **DCGM telemetry** | Temperature, ECC errors, PCIe replay counters, NVLink errors | | **Server health** | HTTP request metrics | To enable Prometheus export, set the `DSTACK_ENABLE_PROMETHEUS_METRICS` environment variable and configure Prometheus to scrape metrics from `/metrics`. > GPU health monitoring is covered in the [GPU health monitoring](#gpu-health-monitoring) section below. ## Fault tolerance, checkpointing, and retry Both systems support fault tolerance for long-running training jobs that may be interrupted by hardware failures, spot instance terminations, or other issues: | | Slurm | dstack | |---|-------|--------| | **Retry** | `--requeue` flag requeues jobs on node failure (hardware crash) or preemption, not application failures (software crashes); all nodes requeued together (all-or-nothing) | `retry` property with `on_events` (`error`, `interruption`) and `duration`; all jobs stopped and run resubmitted if any job fails (all-or-nothing) | | **Graceful stop** | Grace period with `SIGTERM` before `SIGKILL`; `--signal` sends signal before time limit (e.g., `--signal=B:USR1@300`) | Not supported | | **Checkpointing** | Application-based; save to shared filesystem | Application-based; save to persistent volumes | | **Instance health** | `HealthCheckProgram` in `slurm.conf` runs custom scripts (DCGM/RVS); non-zero exit drains node (excludes from new scheduling, running jobs continue) | Automatic GPU health monitoring via DCGM; unhealthy instances excluded from scheduling | ### Slurm Slurm handles three types of failures: system failures (hardware crash), application failures (software crash), and preemption. Enable automatic requeue on node failure (not application failures). For distributed jobs, if one node fails, the entire job is requeued (all-or-nothing):
```bash #!/bin/bash #SBATCH --job-name=train-with-checkpoint #SBATCH --nodes=4 #SBATCH --gres=gpu:8 #SBATCH --time=48:00:00 #SBATCH --requeue # Requeue on node failure only srun python train.py ```
Preempted jobs receive `SIGTERM` during a grace period before `SIGKILL` and are typically requeued automatically. Use `--signal` to send a custom signal before the time limit expires:
```bash #!/bin/bash #SBATCH --job-name=train-with-checkpoint #SBATCH --nodes=4 #SBATCH --gres=gpu:8 #SBATCH --time=48:00:00 #SBATCH --signal=B:USR1@300 # Send USR1 5 minutes before time limit trap 'python save_checkpoint.py --checkpoint-dir=/shared/checkpoints' USR1 if [ -f /shared/checkpoints/latest.pt ]; then RESUME_FLAG="--resume /shared/checkpoints/latest.pt" fi srun python train.py \ --checkpoint-dir=/shared/checkpoints \ $RESUME_FLAG ```
Checkpoints are saved to a shared filesystem. Applications must implement checkpointing logic. Custom health checks are configured via `HealthCheckProgram` in `slurm.conf`:
```bash HealthCheckProgram=/shared/scripts/gpu_health_check.sh ```
The health check script should exit with non-zero code to drain the node:
```bash #!/bin/bash dcgmi diag -r 1 if [ $? -ne 0 ]; then exit 1 # Non-zero exit drains node fi ```
Drained nodes are excluded from new scheduling, but running jobs continue until completion. ### dstack `dstack` handles three types of failures: provisioning failures (`no-capacity`), job failures (`error`), and interruptions (`interruption`). The `error` event is triggered by application failures (non-zero exit code) and instance unreachable issues. The `interruption` event is triggered by spot instance terminations and network/hardware issues. By default, runs fail immediately. Enable retry via the `retry` property to handle these events:
```yaml type: task name: train-with-checkpoint-retry nodes: 4 python: 3.12 repos: - . volumes: # Use instance volumes (host directories) or network volumes (cloud-managed persistent storage) - name: checkpoint-volume path: /checkpoints commands: - | if [ -f /checkpoints/latest.pt ]; then RESUME_FLAG="--resume /checkpoints/latest.pt" fi python train.py \ --checkpoint-dir=/checkpoints \ $RESUME_FLAG resources: gpu: A100:80GB:8 memory: 200GB spot_policy: auto retry: on_events: [error, interruption] duration: 48h ```
For distributed tasks, if any job fails and retry is enabled, all jobs are stopped and the run is resubmitted (all-or-nothing). Unlike Slurm, `dstack` does not support graceful shutdown signals. Applications must implement proactive checkpointing (periodic saves) and check for existing checkpoints on startup to resume after retries. ## GPU health monitoring Both systems monitor GPU health to prevent degraded hardware from affecting workloads: | | Slurm | dstack | |---|-------|--------| | **Health checks** | Custom scripts (DCGM/RVS) via `HealthCheckProgram` in `slurm.conf`; typically active diagnostics (`dcgmi diag`) or passive health watches | Automatic DCGM health watches (passive, continuous monitoring) | | **Failure handling** | Non-zero exit drains node (excludes from new scheduling, running jobs continue); status: DRAIN/DRAINED | Unhealthy instances excluded from scheduling; status shown in `dstack fleet`: `idle` (healthy), `idle (warning)`, `idle (failure)` | ### Slurm Configure custom health check scripts via `HealthCheckProgram` in `slurm.conf`. Scripts typically use DCGM diagnostics (`dcgmi diag`) for NVIDIA GPUs or RVS for AMD GPUs:
```bash HealthCheckProgram=/shared/scripts/gpu_health_check.sh ```
```bash #!/bin/bash dcgmi diag -r 1 # DCGM diagnostic for NVIDIA GPUs if [ $? -ne 0 ]; then exit 1 # Non-zero exit drains node fi ```
Drained nodes are excluded from new scheduling, but running jobs continue until completion. ### dstack `dstack` automatically monitors GPU health using DCGM background health checks on instances with NVIDIA GPUs. Supported on cloud backends where DCGM is pre-installed automatically (or comes with users' `os_images`) and SSH fleets where DCGM packages (`datacenter-gpu-manager-4-core`, `datacenter-gpu-manager-4-proprietary`, `datacenter-gpu-manager-exporter`) are installed on hosts. > AMD GPU health monitoring is not supported yet. Health status is displayed in `dstack fleet`:
```shell $ dstack fleet FLEET INSTANCE BACKEND RESOURCES STATUS PRICE CREATED my-fleet 0 aws (us-east-1) T4:16GB:1 idle $0.526 11 mins ago 1 aws (us-east-1) T4:16GB:1 idle (warning) $0.526 11 mins ago 2 aws (us-east-1) T4:16GB:1 idle (failure) $0.526 11 mins ago ```
Health status: | Status | Description | |--------|-------------| | `idle` | Healthy, no issues detected | | `idle (warning)` | Non-fatal issues (e.g., correctable ECC errors); instance still usable | | `idle (failure)` | Fatal issues (uncorrectable ECC, PCIe failures); instance excluded from scheduling | GPU health metrics are also exported to Prometheus (see [Prometheus integration](#prometheus-integration)). ## Job dependencies Job dependencies enable chaining tasks together, ensuring that downstream jobs only run after upstream jobs complete. ### Slurm dependencies Slurm provides native dependency support via `--dependency` flags. Dependencies are managed by Slurm: | Dependency type | Description | |----------------|-------------| | **`afterok`** | Runs only if the dependency job finishes with Exit Code 0 (success) | | **`afterany`** | Runs regardless of success or failure (useful for cleanup jobs) | | **`aftercorr`** | For array jobs, allows corresponding tasks to start as soon as the matching task in the dependency array completes (e.g., Task 1 of Array B starts when Task 1 of Array A finishes, without waiting for the entire Array A) | | **`singleton`** | Based on job name and user (not job IDs), ensures only one job with the same name runs at a time for that user (useful for serializing access to shared resources) | Submit a job that depends on another job completing successfully:
```shell $ JOB_TRAIN=$(sbatch train.sh | awk '{print $4}') Submitted batch job 1001 $ sbatch --dependency=afterok:$JOB_TRAIN evaluate.sh Submitted batch job 1002 ```
Submit a job with singleton dependency (only one job with this name runs at a time):
```shell $ sbatch --job-name=ModelTraining --dependency=singleton train.sh Submitted batch job 1004 ```
### dstack { #dstack-workflow-orchestration } `dstack` does not support native job dependencies. Use external workflow orchestration tools (Airflow, Prefect, etc.) to implement dependencies. === "Prefect" ```python from prefect import flow, task import subprocess @task def train_model(): """Submit training job and wait for completion""" subprocess.run( ["dstack", "apply", "-f", "train.dstack.yml", "--name", "train-run"], check=True # Raises exception if training fails ) return "train-run" @task def evaluate_model(run_name): """Submit evaluation job after training succeeds""" subprocess.run( ["dstack", "apply", "-f", "evaluate.dstack.yml", "--name", f"eval-{run_name}"], check=True ) @flow def ml_pipeline(): train_run = train_model() evaluate_model(train_run) ``` === "Airflow" ```python from airflow.decorators import dag, task from datetime import datetime import subprocess @dag(schedule=None, start_date=datetime(2024, 1, 1), catchup=False) def ml_training_pipeline(): @task def train(context): """Submit training job and wait for completion""" run_name = f"train-{context['ds']}" subprocess.run( ["dstack", "apply", "-f", "train.dstack.yml", "--name", run_name], check=True # Raises exception if training fails ) return run_name @task def evaluate(run_name, context): """Submit evaluation job after training succeeds""" eval_name = f"eval-{run_name}" subprocess.run( ["dstack", "apply", "-f", "evaluate.dstack.yml", "--name", eval_name], check=True ) # Define task dependencies - train() completes before evaluate() starts train_run = train() evaluate(train_run) ml_training_pipeline() ``` ## Heterogeneous jobs Heterogeneous jobs (het jobs) allow a single job to request different resource configurations for different components (e.g., GPU nodes for training, high-memory CPU nodes for preprocessing). This is an edge case used for coordinated multi-component workflows. ### Slurm Slurm supports heterogeneous jobs via `#SBATCH hetjob` and `--het-group` flags. Each component can specify different resources: ```bash #!/bin/bash #SBATCH --job-name=ml-pipeline #SBATCH hetjob #SBATCH --het-group=0 --nodes=2 --gres=gpu:8 --mem=200G #SBATCH --het-group=1 --nodes=1 --mem=500G --partition=highmem # Use SLURM_JOB_COMPONENT_ID to identify the component if [ "$SLURM_JOB_COMPONENT_ID" -eq 0 ]; then srun python train.py elif [ "$SLURM_JOB_COMPONENT_ID" -eq 1 ]; then srun python preprocess.py fi ``` ### dstack `dstack` does not support heterogeneous jobs natively. Use separate runs with [workflow orchestration tools (Prefect, Airflow)](#dstack-workflow-orchestration) or submit multiple runs programmatically to coordinate components with different resource requirements. ## What's next? 1. Check out [Quickstart](../../quickstart.md) 2. Read about [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), and [services](../../concepts/services.md) 3. Browse the [examples](../../examples.md) # docs/examples/training/trl.md --- title: TRL description: Fine-tuning Llama with TRL — single-node SFT with QLoRA, or distributed across multiple nodes with FSDP and DeepSpeed --- # TRL This example walks you through how to use [TRL](https://github.com/huggingface/trl) with `dstack` to fine-tune `Llama-3.1-8B` — on a single node with SFT and QLoRA, or distributed across multiple nodes with [Accelerate](https://github.com/huggingface/accelerate) and [DeepSpeed](https://github.com/deepspeedai/DeepSpeed). ## Single-node training ### Define a configuration Below is a task configuration that does fine-tuning.
```yaml type: task name: trl-train python: 3.12 # Ensure nvcc is installed (req. for Flash Attention) nvcc: true env: - HF_TOKEN - WANDB_API_KEY - HUB_MODEL_ID commands: # Pin torch==2.6.0 to avoid building Flash Attention from source. # Prebuilt Flash Attention wheels are not available for the latest torch==2.7.0. - uv pip install torch==2.6.0 - uv pip install transformers bitsandbytes peft wandb - uv pip install flash_attn --no-build-isolation - git clone https://github.com/huggingface/trl - cd trl - uv pip install . - | accelerate launch \ --config_file=examples/accelerate_configs/multi_gpu.yaml \ --num_processes $DSTACK_GPUS_PER_NODE \ trl/scripts/sft.py \ --model_name meta-llama/Meta-Llama-3.1-8B \ --dataset_name OpenAssistant/oasst_top1_2023-08-25 \ --dataset_text_field="text" \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 4 \ --learning_rate 2e-4 \ --report_to wandb \ --bf16 \ --max_seq_length 1024 \ --lora_r 16 \ --lora_alpha 32 \ --lora_target_modules q_proj k_proj v_proj o_proj \ --load_in_4bit \ --use_peft \ --attn_implementation "flash_attention_2" \ --logging_steps=10 \ --output_dir models/llama31 \ --hub_model_id peterschmidt85/FineLlama-3.1-8B resources: gpu: # 24GB or more VRAM memory: 24GB.. # One or more GPU count: 1.. # Shared memory (for multi-gpu) shm_size: 24GB ```
Change the `resources` property to specify more GPUs. !!! info "AMD" The example above uses NVIDIA accelerators. To use it with AMD, check out [AMD](../accelerators/amd.md#trl). ??? info "DeepSpeed" For more memory-efficient use of multiple GPUs, consider using DeepSpeed and ZeRO Stage 3. To do this, use the `examples/accelerate_configs/deepspeed_zero3.yaml` configuration file instead of `examples/accelerate_configs/multi_gpu.yaml`. ### Run the configuration Once the configuration is ready, run `dstack apply -f `, and `dstack` will automatically provision the cloud resources and run the configuration.
```shell $ HF_TOKEN=... $ WANDB_API_KEY=... $ HUB_MODEL_ID=... $ dstack apply -f train.dstack.yml # BACKEND RESOURCES INSTANCE TYPE PRICE 1 vastai (cz-czechia) cpu=64 mem=128GB H100:80GB:2 18794506 $3.8907 2 vastai (us-texas) cpu=52 mem=64GB H100:80GB:2 20442365 $3.6926 3 vastai (fr-france) cpu=64 mem=96GB H100:80GB:2 20379984 $3.7389 Submit the run trl-train? [y/n]: Provisioning... ---> 100% ```
## Distributed training !!! info "Prerequisites" Before running a distributed task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). ### Define a configuration Once the fleet is created, define a distributed task configuration. Here's an example using either FSDP or DeepSpeed ZeRO-3. === "FSDP"
```yaml type: task name: trl-train-fsdp-distrib nodes: 2 image: nvcr.io/nvidia/pytorch:25.01-py3 env: - HF_TOKEN - ACCELERATE_LOG_LEVEL=info - WANDB_API_KEY - MODEL_ID=meta-llama/Llama-3.1-8B - HUB_MODEL_ID commands: - pip install transformers bitsandbytes peft wandb - git clone https://github.com/huggingface/trl - cd trl - pip install . - | accelerate launch \ --config_file=examples/accelerate_configs/fsdp1.yaml \ --main_process_ip=$DSTACK_MASTER_NODE_IP \ --main_process_port=8008 \ --machine_rank=$DSTACK_NODE_RANK \ --num_processes=$DSTACK_GPUS_NUM \ --num_machines=$DSTACK_NODES_NUM \ trl/scripts/sft.py \ --model_name $MODEL_ID \ --dataset_name OpenAssistant/oasst_top1_2023-08-25 \ --dataset_text_field="text" \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 4 \ --learning_rate 2e-4 \ --report_to wandb \ --bf16 \ --max_seq_length 1024 \ --attn_implementation flash_attention_2 \ --logging_steps=10 \ --output_dir /checkpoints/llama31-ft \ --hub_model_id $HUB_MODEL_ID \ --torch_dtype bfloat16 resources: gpu: 80GB:8 shm_size: 128GB volumes: - /checkpoints:/checkpoints ```
=== "DeepSpeed ZeRO-3"
```yaml type: task name: trl-train-deepspeed-distrib nodes: 2 image: nvcr.io/nvidia/pytorch:25.01-py3 env: - HF_TOKEN - WANDB_API_KEY - HUB_MODEL_ID - MODEL_ID=meta-llama/Llama-3.1-8B - ACCELERATE_LOG_LEVEL=info commands: - pip install transformers bitsandbytes peft wandb deepspeed - git clone https://github.com/huggingface/trl - cd trl - pip install . - | accelerate launch \ --config_file=examples/accelerate_configs/deepspeed_zero3.yaml \ --main_process_ip=$DSTACK_MASTER_NODE_IP \ --main_process_port=8008 \ --machine_rank=$DSTACK_NODE_RANK \ --num_processes=$DSTACK_GPUS_NUM \ --num_machines=$DSTACK_NODES_NUM \ trl/scripts/sft.py \ --model_name $MODEL_ID \ --dataset_name OpenAssistant/oasst_top1_2023-08-25 \ --dataset_text_field="text" \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 4 \ --learning_rate 2e-4 \ --report_to wandb \ --bf16 \ --max_seq_length 1024 \ --attn_implementation flash_attention_2 \ --logging_steps=10 \ --output_dir /checkpoints/llama31-ft \ --hub_model_id $HUB_MODEL_ID \ --torch_dtype bfloat16 resources: gpu: 80GB:8 shm_size: 128GB volumes: - /checkpoints:/checkpoints ```
!!! info "Docker image" We are using `nvcr.io/nvidia/pytorch:25.01-py3` from NGC because it includes the necessary libraries and packages for RDMA and InfiniBand support. ### Run the configuration To run a configuration, use the [`dstack apply`](../../reference/cli/dstack/apply.md) command.
```shell $ HF_TOKEN=... $ WANDB_API_KEY=... $ HUB_MODEL_ID=... $ dstack apply -f train-distrib.dstack.yml # BACKEND RESOURCES INSTANCE TYPE PRICE 1 ssh (remote) cpu=208 mem=1772GB H100:80GB:8 instance $0 idle 2 ssh (remote) cpu=208 mem=1772GB H100:80GB:8 instance $0 idle Submit the run trl-train-fsdp-distrib? [y/n]: y Provisioning... ---> 100% ```
## What's next? 1. Check [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), [services](../../concepts/services.md), and [fleets](../../concepts/fleets.md) 2. Read about [cluster placement](../../concepts/fleets.md#cluster-placement) 3. See the [AMD](../accelerators/amd.md#trl) example # docs/examples/training/axolotl.md --- title: Axolotl description: Fine-tuning Llama models with Axolotl — single-node SFT with FSDP and QLoRA, or distributed across multiple nodes --- # Axolotl This example shows how to use [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) with `dstack` to fine-tune Llama models — on a single node with SFT, FSDP, and QLoRA, or distributed across multiple nodes. ## Single-node training This section walks through fine-tuning 4-bit quantized `Llama-4-Scout-17B-16E` using SFT with FSDP and QLoRA. ### Define a configuration Axolotl reads the model, QLoRA, and dataset arguments, as well as trainer configuration from a [`scout-qlora-flexattn-fsdp2.yaml`](https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/llama-4/scout-qlora-flexattn-fsdp2.yaml) file. The configuration uses 4-bit axolotl quantized version of `meta-llama/Llama-4-Scout-17B-16E`, requiring only ~43GB VRAM/GPU with 4K context length. Below is a task configuration that does fine-tuning.
```yaml type: task # The name is optional, if not specified, generated randomly name: axolotl-nvidia-llama-scout-train # Using the official Axolotl's Docker image image: axolotlai/axolotl:main-latest # Required environment variables env: - HF_TOKEN - WANDB_API_KEY - WANDB_PROJECT - HUB_MODEL_ID # Commands of the task commands: - wget https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/examples/llama-4/scout-qlora-flexattn-fsdp2.yaml - | axolotl train scout-qlora-flexattn-fsdp2.yaml \ --wandb-project $WANDB_PROJECT \ --wandb-name $DSTACK_RUN_NAME \ --hub-model-id $HUB_MODEL_ID resources: # Four GPU (required by FSDP) gpu: H100:4 # Shared memory size for inter-process communication shm_size: 64GB disk: 500GB.. ```
The task uses Axolotl's Docker image, where Axolotl is already pre-installed. !!! info "AMD" The example above uses NVIDIA accelerators. To use it with AMD, check out [AMD](../accelerators/amd.md#axolotl). ### Run the configuration Once the configuration is ready, run `dstack apply -f `, and `dstack` will automatically provision the cloud resources and run the configuration.
```shell $ HF_TOKEN=... $ WANDB_API_KEY=... $ WANDB_PROJECT=... $ HUB_MODEL_ID=... $ dstack apply -f train.dstack.yml # BACKEND RESOURCES INSTANCE TYPE PRICE 1 vastai (cz-czechia) cpu=64 mem=128GB H100:80GB:2 18794506 $3.8907 2 vastai (us-texas) cpu=52 mem=64GB H100:80GB:2 20442365 $3.6926 3 vastai (fr-france) cpu=64 mem=96GB H100:80GB:2 20379984 $3.7389 Submit the run axolotl-nvidia-llama-scout-train? [y/n]: Provisioning... ---> 100% ```
## Distributed training !!! info "Prerequisites" Before running a distributed task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). This section walks through running distributed fine-tuning of `Llama-3.1-70B` with QLoRA and FSDP across multiple nodes. ### Define a configuration Once the fleet is created, define a distributed task configuration. Here's an example of a distributed `QLoRA` task using `FSDP`.
```yaml type: task name: axolotl-multi-node-qlora-llama3-70b nodes: 2 image: nvcr.io/nvidia/pytorch:25.01-py3 env: - HF_TOKEN - WANDB_API_KEY - WANDB_PROJECT - HUB_MODEL_ID - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 - NCCL_DEBUG=INFO - ACCELERATE_LOG_LEVEL=info commands: # Replacing the default Torch and FlashAttention in the NCG container with Axolotl-compatible versions. # The preinstalled versions are incompatible with Axolotl. - pip uninstall -y torch flash-attn - pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/test/cu124 - pip install --no-build-isolation axolotl[flash-attn,deepspeed] - wget https://raw.githubusercontent.com/huggingface/trl/main/examples/accelerate_configs/fsdp1.yaml - wget https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/examples/llama-3/qlora-fsdp-70b.yaml # Axolotl includes hf-xet version 1.1.0, which fails during downloads. Replacing it with the latest version (1.1.2). - pip uninstall -y hf-xet - pip install hf-xet --no-cache-dir - | accelerate launch \ --config_file=fsdp1.yaml \ -m axolotl.cli.train qlora-fsdp-70b.yaml \ --hub-model-id $HUB_MODEL_ID \ --output-dir /checkpoints/qlora-llama3-70b \ --wandb-project $WANDB_PROJECT \ --wandb-name $DSTACK_RUN_NAME \ --main_process_ip=$DSTACK_MASTER_NODE_IP \ --main_process_port=8008 \ --machine_rank=$DSTACK_NODE_RANK \ --num_processes=$DSTACK_GPUS_NUM \ --num_machines=$DSTACK_NODES_NUM resources: gpu: 80GB:8 shm_size: 128GB volumes: - /checkpoints:/checkpoints ```
!!! info "Docker image" We are using `nvcr.io/nvidia/pytorch:25.01-py3` from NGC because it includes the necessary libraries and packages for RDMA and InfiniBand support. ### Run the configuration To run a configuration, use the [`dstack apply`](../../reference/cli/dstack/apply.md) command.
```shell $ HF_TOKEN=... $ WANDB_API_KEY=... $ WANDB_PROJECT=... $ HUB_MODEL_ID=... $ dstack apply -f train-distrib.dstack.yml # BACKEND RESOURCES INSTANCE TYPE PRICE 1 ssh (remote) cpu=208 mem=1772GB H100:80GB:8 instance $0 idle 2 ssh (remote) cpu=208 mem=1772GB H100:80GB:8 instance $0 idle Submit the run axolotl-multi-node-qlora-llama3-70b? [y/n]: y Provisioning... ---> 100% ```
## What's next? 1. Check [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), [services](../../concepts/services.md), and [fleets](../../concepts/fleets.md) 2. Read about [cluster placement](../../concepts/fleets.md#cluster-placement) 3. See the [AMD](../accelerators/amd.md#axolotl) example # docs/examples/training/ray-ragen.md --- title: Ray + RAGEN description: Multi-node agent fine-tuning using RAGEN with Ray and verl for reinforcement learning --- # Ray + RAGEN This example shows how use `dstack` and [RAGEN](https://github.com/RAGEN-AI/RAGEN) to fine-tune an agent on multiple nodes. Under the hood `RAGEN` uses [verl](https://github.com/volcengine/verl) for Reinforcement Learning and [Ray](https://docs.ray.io/en/latest/) for distributed training. !!! info "Prerequisites" Before running a distributed task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). ## Run a Ray cluster If you want to use Ray with `dstack`, you have to first run a Ray cluster. The task below runs a Ray cluster on an existing fleet:
```yaml type: task name: ray-cluster nodes: 2 env: - WANDB_API_KEY image: whatcanyousee/verl:ngc-cu124-vllm0.8.5-sglang0.4.6-mcore0.12.0-te2.2 commands: - wget -O miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh - bash miniconda.sh -b -p /workflow/miniconda - eval "$(/workflow/miniconda/bin/conda shell.bash hook)" - git clone https://github.com/RAGEN-AI/RAGEN.git - cd RAGEN - bash scripts/setup_ragen.sh - conda activate ragen - cd verl - pip install --no-deps -e . - pip install hf_transfer hf_xet - pip uninstall -y ray - pip install -U "ray[default]" - | if [ $DSTACK_NODE_RANK = 0 ]; then ray start --head --port=6379; else ray start --address=$DSTACK_MASTER_NODE_IP:6379 fi # Expose Ray dashboard port ports: - 8265 resources: gpu: 80GB:8 shm_size: 128GB # Save checkpoints on the instance volumes: - /checkpoints:/checkpoints ```
We are using verl's docker image for vLLM with FSDP. See [Installation](https://verl.readthedocs.io/en/latest/start/install.html) for more. The `RAGEN` setup script `scripts/setup_ragen.sh` isolates dependencies within Conda environment. Note that the Ray setup in the RAGEN environment is missing the dashboard, so we reinstall it using `ray[default]`. Now, if you run this task via `dstack apply`, it will automatically forward the Ray's dashboard port to `localhost:8265`.
```shell $ dstack apply -f ray-cluster.dstack.yml ```
As long as the `dstack apply` is attached, you can use `localhost:8265` to submit Ray jobs for execution. If `dstack apply` is detached, you can use `dstack attach` to re-attach. ## Submit Ray jobs Before you can submit Ray jobs, ensure to install `ray` locally:
```shell $ pip install ray ```
Now you can submit the training job to the Ray cluster which is available at `localhost:8265`:
```shell $ RAY_ADDRESS=http://localhost:8265 $ ray job submit \ -- bash -c "\ export PYTHONPATH=/workflow/RAGEN; \ cd /workflow/RAGEN; \ /workflow/miniconda/envs/ragen/bin/python train.py \ --config-name base \ system.CUDA_VISIBLE_DEVICES=[0,1,2,3,4,5,6,7] \ model_path=Qwen/Qwen2.5-7B-Instruct \ trainer.experiment_name=agent-fine-tuning-Qwen2.5-7B \ trainer.n_gpus_per_node=8 \ trainer.nnodes=2 \ micro_batch_size_per_gpu=2 \ trainer.default_local_dir=/checkpoints \ trainer.save_freq=50 \ actor_rollout_ref.rollout.tp_size_check=False \ actor_rollout_ref.rollout.tensor_model_parallel_size=4" ```
!!! info "Training parameters" 1. `actor_rollout_ref.rollout.tensor_model_parallel_size=4`, because `Qwen/Qwen2.5-7B-Instruct` has 28 attention heads and number of attention heads should be divisible by `tensor_model_parallel_size` 2. `actor_rollout_ref.rollout.tp_size_check=False`, if True `tensor_model_parallel_size` should be equal to `trainer.n_gpus_per_node` 3. `micro_batch_size_per_gpu=2`, to keep the RAGEN-paper's `rollout_filter_ratio` and `es_manager` settings as it is for world size `16` Using Ray via `dstack` is a powerful way to get access to the rich Ray ecosystem while benefiting from `dstack`'s provisioning capabilities. !!! info "What's next" 1. Read about [distributed tasks](../../concepts/tasks.md#distributed-tasks), [fleets](../../concepts/fleets.md), and [cluster placement](../../concepts/fleets.md#cluster-placement) 2. Browse Ray's [docs](https://docs.ray.io/en/latest/train/examples.html) for other examples. # docs/examples/clusters/aws.md --- title: AWS description: High-performance distributed training on AWS using Elastic Fabric Adapter (EFA) --- # AWS In this guide, we'll walk through how to run high-performance distributed training on AWS using [Amazon Elastic Fabric Adapter (EFA)](https://aws.amazon.com/hpc/efa/) with `dstack`. ## Overview EFA is a network interface for Amazon EC2 that enables low-latency, high-bandwidth inter-node communication — essential for scaling distributed deep learning. With `dstack`, EFA is automatically enabled when you create fleets with supported instance types. ## Prerequisite Before you start, make sure the `aws` backend is properly configured.
```yaml projects: - name: main backends: - type: aws creds: type: default regions: ["us-west-2"] ```
!!! info "VPC" If you use a custom VPC, verify that it permits all internal traffic between nodes for EFA to function properly ## Create a fleet Once your backend is ready, define a fleet configuration.
```yaml type: fleet name: efa-fleet nodes: 2 placement: cluster resources: gpu: H100:8 ```
Provision the fleet with `dstack apply`:
```shell $ dstack apply -f efa-fleet.dstack.yml Provisioning... ---> 100% FLEET INSTANCE BACKEND INSTANCE TYPE GPU PRICE STATUS CREATED efa-fleet 0 aws (us-west-2) p4d.24xlarge H100:8:80GB $98.32 idle 3 mins ago 1 aws (us-west-2) p4d.24xlarge H100:8:80GB $98.32 idle 3 mins ago ```
??? info "Instance types" `dstack` selects suitable instances automatically, but not [all types support EFA](https://aws.amazon.com/hpc/efa/). To enforce EFA, you can specify `instance_types` explicitly: ```yaml type: fleet name: efa-fleet nodes: 2 placement: cluster resources: gpu: L4 instance_types: ["g6.8xlarge"] # If not specified, g6.xlarge is used (won't have EFA) ``` ## Run NCCL tests To confirm that EFA is working, run NCCL tests:
```yaml type: task name: nccl-tests nodes: 2 startup_order: workers-first stop_criteria: master-done env: - NCCL_DEBUG=INFO commands: - | if [ $DSTACK_NODE_RANK -eq 0 ]; then mpirun \ --allow-run-as-root \ --hostfile $DSTACK_MPI_HOSTFILE \ -n $DSTACK_GPUS_NUM \ -N $DSTACK_GPUS_PER_NODE \ --bind-to none \ /opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 else sleep infinity fi resources: gpu: 1..8 shm_size: 16GB ```
Run it with `dstack apply`:
```shell $ dstack apply -f nccl-tests.dstack.yml Provisioning... ---> 100% ```
!!! info "Docker image" You can use your own container by setting `image`. If omitted, `dstack` uses its default image with drivers, NCCL tests, and tools pre-installed. ## Run distributed training Here’s an example using `torchrun` for a simple multi-node PyTorch job:
```yaml type: task name: train-distrib nodes: 2 python: 3.12 env: - NCCL_DEBUG=INFO commands: - git clone https://github.com/pytorch/examples.git pytorch-examples - cd pytorch-examples/distributed/ddp-tutorial-series - uv pip install -r requirements.txt - | torchrun \ --nproc-per-node=$DSTACK_GPUS_PER_NODE \ --node-rank=$DSTACK_NODE_RANK \ --nnodes=$DSTACK_NODES_NUM \ --master-addr=$DSTACK_MASTER_NODE_IP \ --master-port=12345 \ multinode.py 50 10 resources: gpu: 1..8 shm_size: 16GB ```
Provision and launch it via `dstack apply`.
```shell $ dstack apply -f train-distrib.dstack.yml Provisioning... ---> 100% ```
Instead of setting `python`, you can specify your own Docker image using `image`. Make sure that the image is properly configured for EFA. !!! info "What's next" 1. Learn more about [distributed tasks](../../concepts/tasks.md#distributed-tasks) and [cluster placement](../../concepts/fleets.md#cluster-placement) 2. Check [dev environments](../../concepts/dev-environments.md), [services](../../concepts/services.md), and [fleets](../../concepts/fleets.md) # docs/examples/clusters/gcp.md --- title: GCP description: Creating and using GPU clusters on GCP with GPUDirect-TCPX and RoCE support --- # GCP This example shows how to create and use clusters on GCP. `dstack` supports the following instance types: | Instance type | GPU | Maximum bandwidth | Fabric | | ------------- | ------ | ----------------- | ---------------------------------------------------------------------------------------------------------------- | | **A3 Edge** | H100:8 | 0.8 Tbps | [GPUDirect-TCPX](https://cloud.google.com/compute/docs/gpus/gpudirect) | | **A3 High** | H100:8 | 1 Tbps | [GPUDirect-TCPX](https://cloud.google.com/compute/docs/gpus/gpudirect) | | **A3 Mega** | H100:8 | 1.8 Tbps | [GPUDirect-TCPXO](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx-autopilot) | | **A4** | B200:8 | 3.2 Tbps | RoCE | ## Configure the backend Despite hiding most of the complexity, `dstack` still requires instance-specific backend configuration: === "A4" A4 requires one `extra_vpcs` for inter-node traffic (regular VPC, one subnet) and one `roce_vpcs` for GPU-to-GPU communication (RoCE profile, eight subnets).
```yaml projects: - name: main backends: - type: gcp # Specify your GCP project ID project_id: extra_vpcs: [dstack-gvnic-net-1] roce_vpcs: [dstack-mrdma] # Specify the regions you intend to use regions: [us-west2] creds: type: default ```

Create extra and RoCE VPCs

See GCP's [RoCE network setup guide](https://cloud.google.com/ai-hypercomputer/docs/create/create-vm#setup-network) for the commands to create VPCs and filewall rules. Ensure VPCs allow internal traffic between nodes for MPI/NCCL to function. === "A3 Mega" A3 Edge/High require at least 4 `extra_vpcs` for data NICs.
```yaml projects: - name: main backends: - type: gcp # Specify your GCP project ID project_id: extra_vpcs: - dstack-gpu-data-net-1 - dstack-gpu-data-net-2 - dstack-gpu-data-net-3 - dstack-gpu-data-net-4 - dstack-gpu-data-net-5 - dstack-gpu-data-net-6 - dstack-gpu-data-net-7 - dstack-gpu-data-net-8 # Specify the regions you intend to use regions: [europe-west4] creds: type: default ```

Create extra VPCs

Create the VPC networks for GPUDirect in your project, each with a subnet and a firewall rule: ```shell # Specify the region where you intend to deploy the cluster REGION="europe-west4" for N in $(seq 1 8); do gcloud compute networks create dstack-gpu-data-net-$N \ --subnet-mode=custom \ --mtu=8244 gcloud compute networks subnets create dstack-gpu-data-sub-$N \ --network=dstack-gpu-data-net-$N \ --region=$REGION \ --range=192.168.$N.0/24 gcloud compute firewall-rules create dstack-gpu-data-internal-$N \ --network=dstack-gpu-data-net-$N \ --action=ALLOW \ --rules=tcp:0-65535,udp:0-65535,icmp \ --source-ranges=192.168.0.0/16 done ``` === "A3 High/Edge" A3 Edge/High require at least 4 `extra_vpcs` for data NICs and a `vm_service_account` authorized to pull GPUDirect Docker images.
```yaml projects: - name: main backends: - type: gcp # Specify your GCP project ID project_id: extra_vpcs: - dstack-gpu-data-net-1 - dstack-gpu-data-net-2 - dstack-gpu-data-net-3 - dstack-gpu-data-net-4 # Specify the regions you intend to use regions: [europe-west4] # Specify your GCP project ID vm_service_account: a3cluster-sa@$.iam.gserviceaccount.com creds: type: default ```

Create extra VPCs

Create the VPC networks for GPUDirect in your project, each with a subnet and a firewall rule: ```shell # Specify the region where you intend to deploy the cluster REGION="europe-west4" for N in $(seq 1 4); do gcloud compute networks create dstack-gpu-data-net-$N \ --subnet-mode=custom \ --mtu=8244 gcloud compute networks subnets create dstack-gpu-data-sub-$N \ --network=dstack-gpu-data-net-$N \ --region=$REGION \ --range=192.168.$N.0/24 gcloud compute firewall-rules create dstack-gpu-data-internal-$N \ --network=dstack-gpu-data-net-$N \ --action=ALLOW \ --rules=tcp:0-65535,udp:0-65535,icmp \ --source-ranges=192.168.0.0/16 done ```

Create a service account

Create a VM service account that allows VMs to access the `pkg.dev` registry: ```shell PROJECT_ID=$(gcloud config get-value project) gcloud iam service-accounts create a3cluster-sa \ --display-name "Service Account for pulling GCR images" gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:a3cluster-sa@${PROJECT_ID}.iam.gserviceaccount.com" \ --role="roles/artifactregistry.reader" ``` !!! info "Default VPC" If you set a non-default `vpc_name` in the backend configuration, ensure it allows all inter-node traffic. This is required for MPI and NCCL. The default VPC already allows this. ## Create a fleet Once you've configured the `gcp` backend, create the fleet configuration: === "A4"
```yaml type: fleet name: a4-fleet placement: cluster # Can be a range on a fixed number nodes: 2 # Specify the zone where you have configured the RoCE VPC availability_zones: [us-west2-c] backends: [gcp] # Uncomment to allow spot instances #spot_policy: auto resources: gpu: B200:8 ```
Then apply it with `dstack apply`:
```shell $ dstack apply -f a4-fleet.dstack.yml Provisioning... ---> 100% FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED a4-fleet 0 gcp (us-west2) B200:180GB:8 (spot) $51.552 idle 9 mins ago 1 gcp (us-west2) B200:180GB:8 (spot) $51.552 idle 9 mins ago ```
=== "A3 Mega"
```yaml type: fleet name: a3mega-fleet placement: cluster # Can be a range on a fixed number nodes: 2 instance_types: - a3-megagpu-8g # Uncomment to allow spot instances #spot_policy: auto ```
Pass the configuration to `dstack apply`:
```shell $ dstack apply -f a3mega-fleet.dstack.yml FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED a3mega-fleet 1 gcp (europe-west4) H100:80GB:8 $22.1525 (spot) idle 9 mins ago a3mega-fleet 2 gcp (europe-west4) H100:80GB:8 $64.2718 idle 9 mins ago Create the fleet? [y/n]: y Provisioning... ---> 100% ```
=== "A3 High/Edge"
```yaml type: fleet name: a3high-fleet placement: cluster nodes: 2 instance_types: - a3-highgpu-8g # Uncomment to allow spot instances #spot_policy: auto ```
Pass the configuration to `dstack apply`:
```shell $ dstack apply -f a3high-fleet.dstack.yml FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED a3mega-fleet 1 gcp (europe-west4) H100:80GB:8 $20.5688 (spot) idle 9 mins ago a3mega-fleet 2 gcp (europe-west4) H100:80GB:8 $58.5419 idle 9 mins ago Create the fleet? [y/n]: y Provisioning... ---> 100% ```
Once the fleet is created, you can run distributed tasks, in addition to dev environments, services, and regular tasks. ## Run tasks ### NCCL tests Use a distributed task that runs NCCL tests to validate cluster network bandwidth. === "A4" Pass the configuration to `dstack apply`:
```shell $ dstack apply -f nccl-tests.dstack.yml Provisioning... ---> 100% nccl-tests provisioning completed (running) nThread 1 nGpus 1 minBytes 8 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 size count type redop root time algbw busbw wrong time algbw busbw wrong (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 8388608 2097152 float sum -1 156.9 53.47 100.25 0 167.6 50.06 93.86 0 16777216 4194304 float sum -1 196.3 85.49 160.29 0 206.2 81.37 152.57 0 33554432 8388608 float sum -1 258.5 129.82 243.42 0 261.8 128.18 240.33 0 67108864 16777216 float sum -1 369.4 181.69 340.67 0 371.2 180.79 338.98 0 134217728 33554432 float sum -1 638.5 210.22 394.17 0 587.2 228.57 428.56 0 268435456 67108864 float sum -1 940.3 285.49 535.29 0 950.7 282.36 529.43 0 536870912 134217728 float sum -1 1695.2 316.70 593.81 0 1666.9 322.08 603.89 0 1073741824 268435456 float sum -1 3229.9 332.44 623.33 0 3201.8 335.35 628.78 0 2147483648 536870912 float sum -1 6107.7 351.61 659.26 0 6157.1 348.78 653.97 0 4294967296 1073741824 float sum -1 11952 359.36 673.79 0 11942 359.65 674.34 0 8589934592 2147483648 float sum -1 23563 364.55 683.52 0 23702 362.42 679.54 0 Out of bounds values : 0 OK Avg bus bandwidth : 165.789 ```
=== "A3 Mega"
```yaml type: task name: nccl-tests nodes: 2 image: nvcr.io/nvidia/pytorch:24.04-py3 entrypoint: "bash -c" # Need to use bash instead of default dash for nccl-env-profile.sh commands: - | # Setup TCPXO NCCL env variables NCCL_LIB_DIR="/var/lib/tcpxo/lib64" source ${NCCL_LIB_DIR}/nccl-env-profile-ll128.sh export NCCL_FASTRAK_CTRL_DEV=enp0s12 export NCCL_FASTRAK_IFNAME=enp6s0,enp7s0,enp13s0,enp14s0,enp134s0,enp135s0,enp141s0,enp142s0 export NCCL_SOCKET_IFNAME=enp0s12 export NCCL_FASTRAK_LLCM_DEVICE_DIRECTORY="/dev/aperture_devices" export LD_LIBRARY_PATH="${NCCL_LIB_DIR}:${LD_LIBRARY_PATH}" # Build NCCL Tests git clone https://github.com/NVIDIA/nccl-tests.git cd nccl-tests MPI=1 CC=mpicc CXX=mpicxx make -j cd build # We use FIFO for inter-node communication FIFO=/tmp/dstack_job if [ ${DSTACK_NODE_RANK} -eq 0 ]; then sleep 10 echo "${DSTACK_NODES_IPS}" > hostfile MPIRUN='mpirun --allow-run-as-root --hostfile hostfile' # Wait for other nodes while true; do if ${MPIRUN} -n ${DSTACK_NODES_NUM} -N 1 true >/dev/null 2>&1; then break fi echo 'Waiting for nodes...' sleep 5 done # Run NCCL Tests ${MPIRUN} \ -n ${DSTACK_GPUS_NUM} -N ${DSTACK_GPUS_PER_NODE} \ --mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0 \ $(env | awk -F= '{print "-x", $1}' | xargs) \ ./all_gather_perf -b 8M -e 8G -f 2 -g 1 -w 5 --iters 200 -c 0; # Notify nodes the job is done ${MPIRUN} -n ${DSTACK_NODES_NUM} -N 1 sh -c "echo done > ${FIFO}" else mkfifo ${FIFO} # Wait for a message from the first node cat ${FIFO} fi spot_policy: auto resources: shm_size: 16GB ```
Pass the configuration to `dstack apply`:
```shell $ dstack apply -f nccl-tests.dstack.yml nccl-tests provisioning completed (running) nThread 1 nGpus 1 minBytes 8388608 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 200 agg iters: 1 validation: 0 graph: 0 out-of-place in-place size count type redop root time algbw busbw #wrong time algbw busbw #wrong (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 8388608 131072 float none -1 166.6 50.34 47.19 N/A 164.1 51.11 47.92 N/A 16777216 262144 float none -1 204.6 82.01 76.89 N/A 203.8 82.30 77.16 N/A 33554432 524288 float none -1 284.0 118.17 110.78 N/A 281.7 119.12 111.67 N/A 67108864 1048576 float none -1 447.4 150.00 140.62 N/A 443.5 151.31 141.86 N/A 134217728 2097152 float none -1 808.3 166.05 155.67 N/A 801.9 167.38 156.92 N/A 268435456 4194304 float none -1 1522.1 176.36 165.34 N/A 1518.7 176.76 165.71 N/A 536870912 8388608 float none -1 2892.3 185.62 174.02 N/A 2894.4 185.49 173.89 N/A 1073741824 16777216 float none -1 5532.7 194.07 181.94 N/A 5530.7 194.14 182.01 N/A 2147483648 33554432 float none -1 10863 197.69 185.34 N/A 10837 198.17 185.78 N/A 4294967296 67108864 float none -1 21481 199.94 187.45 N/A 21466 200.08 187.58 N/A 8589934592 134217728 float none -1 42713 201.11 188.54 N/A 42701 201.16 188.59 N/A Out of bounds values : 0 OK Avg bus bandwidth : 146.948 ```
=== "A3 High/Edge"
```yaml type: task name: nccl-tests nodes: 2 image: us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpx/nccl-plugin-gpudirecttcpx commands: - | export NCCL_DEBUG=INFO export LD_LIBRARY_PATH=/usr/local/tcpx/lib64:$LD_LIBRARY_PATH # We use FIFO for inter-node communication FIFO=/tmp/dstack_job if [ ${DSTACK_NODE_RANK} -eq 0 ]; then mkdir -p /scripts/hostfiles2 : > /scripts/hostfiles2/hostfile8 for ip in ${DSTACK_NODES_IPS}; do echo "${ip} slots=${DSTACK_GPUS_PER_NODE}" >> /scripts/hostfiles2/hostfile8 done MPIRUN='mpirun --allow-run-as-root --hostfile /scripts/hostfiles2/hostfile8' # Wait for other nodes while true; do if ${MPIRUN} -n ${DSTACK_NODES_NUM} -N 1 true >/dev/null 2>&1; then break fi echo 'Waiting for nodes...' sleep 5 done # Run NCCL Tests NCCL_GPUDIRECTTCPX_FORCE_ACK=0 /scripts/run-allgather.sh 8 eth1,eth2,eth3,eth4 8M 8GB 2 # Notify nodes the job is done ${MPIRUN} -n ${DSTACK_NODES_NUM} -N 1 sh -c "echo done > ${FIFO}" else mkfifo ${FIFO} # Wait for a message from the first node cat ${FIFO} fi spot_policy: auto resources: shm_size: 16GB ```
Pass the configuration to `dstack apply`:
```shell $ dstack apply -f nccl-tests.dstack.yml nccl-tests provisioning completed (running) nThread 1 nGpus 1 minBytes 8388608 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 200 agg iters: 1 validation: 0 graph: 0 out-of-place in-place size count type redop root time algbw busbw #wrong time algbw busbw #wrong (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 8388608 131072 float none -1 784.9 10.69 10.02 0 775.9 10.81 10.14 0 16777216 262144 float none -1 1010.3 16.61 15.57 0 999.3 16.79 15.74 0 33554432 524288 float none -1 1161.6 28.89 27.08 0 1152.9 29.10 27.28 0 67108864 1048576 float none -1 1432.6 46.84 43.92 0 1437.8 46.67 43.76 0 134217728 2097152 float none -1 2516.9 53.33 49.99 0 2491.7 53.87 50.50 0 268435456 4194304 float none -1 5066.8 52.98 49.67 0 5131.4 52.31 49.04 0 536870912 8388608 float none -1 10028 53.54 50.19 0 10149 52.90 49.60 0 1073741824 16777216 float none -1 20431 52.55 49.27 0 20214 53.12 49.80 0 2147483648 33554432 float none -1 40254 53.35 50.01 0 39923 53.79 50.43 0 4294967296 67108864 float none -1 80896 53.09 49.77 0 78875 54.45 51.05 0 8589934592 134217728 float none -1 160505 53.52 50.17 0 160117 53.65 50.29 0 Out of bounds values : 0 OK Avg bus bandwidth : 40.6043 ```
### Distributed training === "A4" You can use the standard [distributed task](../../concepts/tasks.md#distributed-tasks) example to run distributed training on A4 instances. === "A3 Mega" You can use the standard [distributed task](../../concepts/tasks.md#distributed-tasks) example to run distributed training on A3 Mega instances. To enable GPUDirect-TCPX, make sure the required [NCCL environment variables](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx-autopilot#environment-variables-nccl) are properly set, for example by adding the following commands at the beginning: ```shell # ... commands: - | NCCL_LIB_DIR="/var/lib/tcpxo/lib64" source ${NCCL_LIB_DIR}/nccl-env-profile-ll128.sh export NCCL_FASTRAK_CTRL_DEV=enp0s12 export NCCL_FASTRAK_IFNAME=enp6s0,enp7s0,enp13s0,enp14s0,enp134s0,enp135s0,enp141s0,enp142s0 export NCCL_SOCKET_IFNAME=enp0s12 export NCCL_FASTRAK_LLCM_DEVICE_DIRECTORY="/dev/aperture_devices" export LD_LIBRARY_PATH="${NCCL_LIB_DIR}:${LD_LIBRARY_PATH}" # ... ``` === "A3 High/Edge" You can use the standard [distributed task](../../concepts/tasks.md#distributed-tasks) example to run distributed training on A3 High/Edge instances. To enable GPUDirect-TCPX0, make sure the required [NCCL environment variables](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx-autopilot#environment-variables-nccl) are properly set, for example by adding the following commands at the beginning: ```shell # ... commands: - | export NCCL_DEBUG=INFO NCCL_LIB_DIR="/usr/local/tcpx/lib64" export LD_LIBRARY_PATH="${NCCL_LIB_DIR}:${LD_LIBRARY_PATH}" export NCCL_SOCKET_IFNAME=eth0 export NCCL_CROSS_NIC=0 export NCCL_ALGO=Ring export NCCL_PROTO=Simple export NCCL_NSOCKS_PERTHREAD=4 export NCCL_SOCKET_NTHREADS=1 export NCCL_NET_GDR_LEVEL=PIX export NCCL_P2P_PXN_LEVEL=0 export NCCL_GPUDIRECTTCPX_SOCKET_IFNAME=eth1,eth2,eth3,eth4 export NCCL_GPUDIRECTTCPX_CTRL_DEV=eth0 export NCCL_DYNAMIC_CHUNK_SIZE=524288 export NCCL_P2P_NET_CHUNKSIZE=524288 export NCCL_P2P_PCI_CHUNKSIZE=524288 export NCCL_P2P_NVL_CHUNKSIZE=1048576 export NCCL_BUFFSIZE=4194304 export NCCL_GPUDIRECTTCPX_TX_BINDINGS="eth1:8-21,112-125;eth2:8-21,112-125;eth3:60-73,164-177;eth4:60-73,164-177" export NCCL_GPUDIRECTTCPX_RX_BINDINGS="eth1:22-35,126-139;eth2:22-35,126-139;eth3:74-87,178-191;eth4:74-87,178-191" export NCCL_GPUDIRECTTCPX_PROGRAM_FLOW_STEERING_WAIT_MICROS=50000 export NCCL_GPUDIRECTTCPX_UNIX_CLIENT_PREFIX="/run/tcpx" # ... ``` In addition to distributed training, you can of course run regular tasks, dev environments, and services. ## What's new 1. Learn about [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), [services](../../concepts/services.md) 2. Read about [cluster placement](../../concepts/fleets.md#cluster-placement) 3. Check GCP's docs on using [A4](https://docs.cloud.google.com/compute/docs/gpus/create-gpu-vm-a3u-a4), and [A3 Mega/High/Edge](https://docs.cloud.google.com/compute/docs/gpus/gpudirect) instances # docs/examples/clusters/lambda.md --- title: Lambda description: Setting up Lambda clusters using Kubernetes or 1-Click Clusters with fast interconnect --- # Lambda `dstack` allows using Lambda clusters with fast interconnect via two ways: * [Kubernetes](#kubernetes) – If you create a Kubernetes cluster on Lambda and configure a `kubernetes` backend and create a backend fleet in `dstack`, `dstack` lets you fully use this cluster through `dstack`. * [VMs](#vms) – If you create a 1CC cluster on Lambda and create an SSH fleet in `dstack`, `dstack` lets you fully use this cluster through `dstack`. ## Kubernetes ### Prerequsisites 1. Follow the instructions in [Lambda's guide](https://docs.lambda.ai/public-cloud/1-click-clusters/managed-kubernetes/#accessing-mk8s) on accessing MK8s. 2. Go to `Firewall` → `Edit rules`, click `Add rule`, and allow ingress traffic on port `30022`. This port will be used by the `dstack` server to access the jump host. ### Configure the backend Follow the standard instructions for setting up a [Kubernetes](../../concepts/backends.md#kubernetes) backend:
```yaml projects: - name: main backends: - type: kubernetes kubeconfig: filename: proxy_jump: port: 30022 ```
### Create a fleet Once the Kubernetes cluster and the `dstack` server are running, you can create a fleet:
```yaml type: fleet name: lambda-fleet placement: cluster nodes: 0.. backends: [kubernetes] resources: # Specify requirements to filter nodes gpu: 1..8 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f lambda-fleet.dstack.yml ```
Once the fleet is created, you can run [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), and [services](../../concepts/services.md). ## 1-Click Clusters Another way to work with Lambda clusters is through [1CC](https://lambda.ai/1-click-clusters). While `dstack` supports automated cluster provisioning via [VM-based backends](../../concepts/backends.md#vm-based), there is currently no programmatic way to provision Lambda 1CCs. As a result, to use a 1CC cluster with `dstack`, you must use [SSH fleets](../../concepts/fleets.md). ### Prerequsisites 1. Follow the instructions in [Lambda's guide](https://docs.lambda.ai/public-cloud/1-click-clusters/) on working with 1-Click Clusters ### Create a fleet Follow the standard instructions for setting up an [SSH fleet](../../concepts/fleets.md#ssh-fleets):
```yaml type: fleet name: lambda-fleet ssh_config: user: ubuntu identity_file: ~/.ssh/id_rsa hosts: - worker-gpu-8x-b200-rplfm-ll9nr - worker-gpu-8x-b200-rplfm-qrcs9 proxy_jump: hostname: 192.222.55.54 user: ubuntu identity_file: ~/.ssh/id_rsa placement: cluster ```
> Under `proxy_jump`, we specify the hostname of the head node along with the private SSH key. Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f lambda-fleet.dstack.yml ```
Once the fleet is created, you can run [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), and [services](../../concepts/services.md). ## Run tasks To run tasks on a cluster, you must use [distributed tasks](../../concepts/tasks.md#distributed-task). ### Run NCCL tests To validate cluster network bandwidth, use the following task:
```yaml type: task name: nccl-tests nodes: 2 startup_order: workers-first stop_criteria: master-done commands: - | if [ $DSTACK_NODE_RANK -eq 0 ]; then mpirun \ --allow-run-as-root \ --hostfile $DSTACK_MPI_HOSTFILE \ -n $DSTACK_GPUS_NUM \ -N $DSTACK_GPUS_PER_NODE \ --bind-to none \ -x NCCL_IB_HCA=^mlx5_0 \ /opt/nccl-tests/build/all_reduce_perf -b 8 -e 2G -f 2 -t 1 -g 1 -c 1 -n 100 else sleep infinity fi # Uncomment if the `kubernetes` backend requires it for `/dev/infiniband` access #privileged: true resources: gpu: nvidia:B200:8 shm_size: 16GB ```
Pass the configuration to `dstack apply`:
```shell $ dstack apply -f lambda-nccl-tests.dstack.yml Provisioning... ---> 100% nccl-tests version 2.17.6 nccl-headers=22602 nccl-library=22602 Collective test starting: all_reduce_perf size count type redop root time algbw busbw #wrong time algbw busbw #wrong (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 8 2 float sum -1 36.50 0.00 0.00 0 36.16 0.00 0.00 0 16 4 float sum -1 35.55 0.00 0.00 0 35.49 0.00 0.00 0 32 8 float sum -1 35.49 0.00 0.00 0 36.28 0.00 0.00 0 64 16 float sum -1 35.85 0.00 0.00 0 35.54 0.00 0.00 0 128 32 float sum -1 37.36 0.00 0.01 0 36.82 0.00 0.01 0 256 64 float sum -1 37.38 0.01 0.01 0 37.80 0.01 0.01 0 512 128 float sum -1 51.05 0.01 0.02 0 37.17 0.01 0.03 0 1024 256 float sum -1 45.33 0.02 0.04 0 37.98 0.03 0.05 0 2048 512 float sum -1 38.67 0.05 0.10 0 38.30 0.05 0.10 0 4096 1024 float sum -1 40.08 0.10 0.19 0 39.18 0.10 0.20 0 8192 2048 float sum -1 42.13 0.19 0.36 0 41.47 0.20 0.37 0 16384 4096 float sum -1 43.66 0.38 0.70 0 41.94 0.39 0.73 0 32768 8192 float sum -1 45.42 0.72 1.35 0 43.29 0.76 1.42 0 65536 16384 float sum -1 44.59 1.47 2.76 0 43.90 1.49 2.80 0 131072 32768 float sum -1 47.44 2.76 5.18 0 46.79 2.80 5.25 0 262144 65536 float sum -1 66.68 3.93 7.37 0 65.36 4.01 7.52 0 524288 131072 float sum -1 240.71 2.18 4.08 0 125.73 4.17 7.82 0 1048576 262144 float sum -1 115.58 9.07 17.01 0 115.48 9.08 17.03 0 2097152 524288 float sum -1 114.44 18.33 34.36 0 114.27 18.35 34.41 0 4194304 1048576 float sum -1 118.25 35.47 66.50 0 117.11 35.82 67.15 0 8388608 2097152 float sum -1 141.39 59.33 111.24 0 134.95 62.16 116.55 0 16777216 4194304 float sum -1 186.86 89.78 168.34 0 184.39 90.99 170.60 0 33554432 8388608 float sum -1 255.79 131.18 245.96 0 253.88 132.16 247.81 0 67108864 16777216 float sum -1 350.41 191.52 359.09 0 350.71 191.35 358.79 0 134217728 33554432 float sum -1 596.75 224.92 421.72 0 595.37 225.44 422.69 0 268435456 67108864 float sum -1 934.67 287.20 538.50 0 931.37 288.22 540.41 0 536870912 134217728 float sum -1 1625.63 330.25 619.23 0 1687.31 318.18 596.59 0 1073741824 268435456 float sum -1 2972.25 361.26 677.35 0 2971.33 361.37 677.56 0 2147483648 536870912 float sum -1 5784.75 371.23 696.06 0 5728.40 374.88 702.91 0 Out of bounds values : 0 OK Avg bus bandwidth : 137.179 ```
## What's next 1. Learn about [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), [services](../../concepts/services.md) 2. Read about the [Kubernetes backend](../../concepts/backends.md#kubernetes) and [cluster placement](../../concepts/fleets.md#cluster-placement) 3. Check Lambda's docs on [Kubernetes](https://docs.lambda.ai/public-cloud/1-click-clusters/managed-kubernetes/#accessing-mk8s) and [1CC](https://docs.lambda.ai/public-cloud/1-click-clusters/) # docs/examples/clusters/crusoe.md --- title: Crusoe description: Using Crusoe clusters with InfiniBand support via VMs or Kubernetes --- # Crusoe `dstack` allows using Crusoe clusters with fast interconnect via two ways: * [VMs](#vms) – If you configure a `crusoe` backend in `dstack` by providing your Crusoe credentials, `dstack` lets you fully provision and use clusters through `dstack`. * [Kubernetes](#kubernetes) – If you create a Kubernetes cluster on Crusoe and configure a `kubernetes` backend and create a backend fleet in `dstack`, `dstack` lets you fully use this cluster through `dstack`. ## VMs Since `dstack` offers a VM-based backend that natively integrates with Crusoe, you only need to provide your Crusoe credentials to `dstack`, and it will allow you to fully provision and use clusters on Crusoe through `dstack`. ### Configure a backend Log into your [Crusoe](https://console.crusoecloud.com/) console, create an API key under your account settings, and note your project ID.
```yaml projects: - name: main backends: - type: crusoe project_id: your-project-id creds: type: access_key access_key: your-access-key secret_key: your-secret-key ```
### Create a fleet Once the backend is configured, you can create a fleet:
```yaml type: fleet name: crusoe-fleet nodes: 2 placement: cluster backends: [crusoe] resources: gpu: A100:80GB:8 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f crusoe-fleet.dstack.yml ```
This will automatically create an IB partition and provision instances with InfiniBand networking. Once the fleet is created, you can run [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), and [services](../../concepts/services.md). > If you want instances to be provisioned on demand, you can set `nodes` to `0..2`. In this case, `dstack` will create instances only when you run workloads. ## Kubernetes ### Create a cluster 1. Go `Networking` → `Firewall Rules`, click `Create Firewall Rule`, and allow ingress traffic on port `30022`. This port will be used by the `dstack` server to access the jump host. 2. Go to `Orchestration` and click `Create Cluster`. Make sure to enable the `NVIDIA GPU Operator` add-on. 3. Go the the cluster, and click `Create Node Pool`. Select the right type of the instance, and `Desired Number of Nodes`. 4. Wait until nodes are provisioned. > Even if you enable `autoscaling`, `dstack` can use only the nodes that are already provisioned. ### Configure the backend Follow the standard instructions for setting up a [`kubernetes`](../../concepts/backends.md#kubernetes) backend:
```yaml projects: - name: main backends: - type: kubernetes kubeconfig: filename: proxy_jump: port: 30022 ```
### Create a fleet Once the Crusoe Managed Kubernetes cluster and the `dstack` server are running, you can create a fleet:
```yaml type: fleet name: crusoe-fleet placement: cluster nodes: 0.. backends: [kubernetes] resources: # Specify requirements to filter nodes gpu: 8 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f crusoe-fleet.dstack.yml ```
Once the fleet is created, you can run [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), and [services](../../concepts/services.md). ## NCCL tests Use a [distributed task](../../concepts/tasks.md#distributed-tasks) that runs NCCL tests to validate cluster network bandwidth. === "VMs" With the Crusoe backend, HPC-X and NCCL topology files are pre-installed on the host VM image. Mount them into the container via [instance volumes](../../concepts/volumes.md#instance-volumes).
```yaml type: task name: nccl-tests nodes: 2 startup_order: workers-first stop_criteria: master-done volumes: - /opt/hpcx:/opt/hpcx - /etc/crusoe/nccl_topo:/etc/crusoe/nccl_topo commands: - . /opt/hpcx/hpcx-init.sh - hpcx_load - | if [ $DSTACK_NODE_RANK -eq 0 ]; then mpirun \ --allow-run-as-root \ --hostfile $DSTACK_MPI_HOSTFILE \ -n $DSTACK_GPUS_NUM \ -N $DSTACK_GPUS_PER_NODE \ --bind-to none \ -mca btl tcp,self \ -mca coll_hcoll_enable 0 \ -x PATH \ -x LD_LIBRARY_PATH \ -x CUDA_DEVICE_ORDER=PCI_BUS_ID \ -x NCCL_SOCKET_NTHREADS=4 \ -x NCCL_NSOCKS_PERTHREAD=8 \ -x NCCL_TOPO_FILE=/etc/crusoe/nccl_topo/a100-80gb-sxm-ib-cloud-hypervisor.xml \ -x NCCL_IB_MERGE_VFS=0 \ -x NCCL_IB_HCA=^mlx5_0:1 \ /opt/nccl-tests/build/all_reduce_perf -b 8 -e 2G -f 2 -t 1 -g 1 -c 1 -n 100 else sleep infinity fi backends: [crusoe] resources: gpu: A100:80GB:8 shm_size: 16GB ```
> Update `NCCL_TOPO_FILE` to match your instance type. Topology files for all supported types are available at `/etc/crusoe/nccl_topo/` on the host. === "Kubernetes" If you're running on Crusoe Managed Kubernetes, make sure to install HPC-X and provide an up-to-date topology file.
```yaml type: task name: nccl-tests nodes: 2 startup_order: workers-first stop_criteria: master-done commands: # Install NCCL topology files - curl -sSL https://gist.github.com/un-def/48df8eea222fa9547ad4441986eb15af/archive/df51d56285c5396a0e82bb42f4f970e7bb0a9b65.tar.gz -o nccl_topo.tar.gz - mkdir -p /etc/crusoe/nccl_topo - tar -C /etc/crusoe/nccl_topo -xf nccl_topo.tar.gz --strip-components=1 # Install and initialize HPC-X - curl -sSL https://content.mellanox.com/hpc/hpc-x/v2.21.3/hpcx-v2.21.3-gcc-doca_ofed-ubuntu22.04-cuda12-x86_64.tbz -o hpcx.tar.bz - mkdir -p /opt/hpcx - tar -C /opt/hpcx -xf hpcx.tar.bz --strip-components=1 --checkpoint=10000 - . /opt/hpcx/hpcx-init.sh - hpcx_load # Run NCCL Tests - | if [ $DSTACK_NODE_RANK -eq 0 ]; then mpirun \ --allow-run-as-root \ --hostfile $DSTACK_MPI_HOSTFILE \ -n $DSTACK_GPUS_NUM \ -N $DSTACK_GPUS_PER_NODE \ --bind-to none \ -mca btl tcp,self \ -mca coll_hcoll_enable 0 \ -x PATH \ -x LD_LIBRARY_PATH \ -x CUDA_DEVICE_ORDER=PCI_BUS_ID \ -x NCCL_SOCKET_NTHREADS=4 \ -x NCCL_NSOCKS_PERTHREAD=8 \ -x NCCL_TOPO_FILE=/etc/crusoe/nccl_topo/a100-80gb-sxm-ib-cloud-hypervisor.xml \ -x NCCL_IB_MERGE_VFS=0 \ -x NCCL_IB_AR_THRESHOLD=0 \ -x NCCL_IB_PCI_RELAXED_ORDERING=1 \ -x NCCL_IB_SPLIT_DATA_ON_QPS=0 \ -x NCCL_IB_QPS_PER_CONNECTION=2 \ -x NCCL_IB_HCA=mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1 \ -x UCX_NET_DEVICES=mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1 \ /opt/nccl-tests/build/all_reduce_perf -b 8 -e 2G -f 2 -t 1 -g 1 -c 1 -n 100 else sleep infinity fi # Required for IB privileged: true resources: gpu: A100:8 shm_size: 16GB ```
> The task above downloads an A100 topology file from a Gist. The most reliable way to obtain the latest topology is to copy it from a Crusoe-provisioned VM (see [VMs](#vms)). ??? info "Privileged" When running on Crusoe Managed Kubernetes, set `privileged` to `true` to ensure access to InfiniBand. Pass the configuration to `dstack apply`:
```shell $ dstack apply -f crusoe-nccl-tests.dstack.yml ```
## What's next 1. Learn about [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), [services](../../concepts/services.md) 2. Check out [backends](../../concepts/backends.md#crusoe-cloud) and [fleets](../../concepts/fleets.md#cloud-fleets) 3. Check the docs on [Crusoe's networking](https://docs.crusoecloud.com/networking/infiniband/) and ["Crusoe Managed" Kubernetes](https://docs.crusoecloud.com/orchestration/cmk/index.html) # docs/examples/clusters/nebius.md --- title: Nebius description: Using Nebius clusters with InfiniBand support via VMs or Kubernetes --- # Nebius `dstack` allows you to use Nebius clusters with fast interconnects in two ways: * [VMs](#vms) – If you configure a `nebius` backend in `dstack` by providing your Nebius credentials, `dstack` lets you fully provision and use clusters through `dstack`. * [Kubernetes](#kubernetes) – If you create a Kubernetes cluster on Nebius and configure a `kubernetes` backend and create a backend fleet in `dstack`, `dstack` lets you fully use this cluster through `dstack`. ## VMs Since `dstack` offers a VM-based backend that natively integrates with Nebius, you only need to provide your Nebius credentials to `dstack`, and it will allow you to fully provision and use clusters on Nebius through `dstack`. ### Configure a backend You can configure the `nebius` backend using a credentials file [generated](https://docs.nebius.com/iam/service-accounts/authorized-keys#create) by the `nebius` CLI:
```shell $ nebius iam auth-public-key generate \ --service-account-id <service account ID> \ --output ~/.nebius/sa-credentials.json ```
```yaml projects: - name: main backends: - type: nebius creds: type: service_account filename: ~/.nebius/sa-credentials.json ```
### Create a fleet Once the backend configured, you can create a fleet:
```yaml type: fleet name: nebius-fleet nodes: 2 placement: cluster backends: [nebius] resources: gpu: H100:8 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f nebius-fleet.dstack.yml ```
This will automatically create a Nebius cluster and provision instances. Once the fleet is created, you can run [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), and [services](../../concepts/services.md). > If you want instances to be provisioned on demand, you can set `nodes` to `0..2`. In this case, `dstack` will create instances only when you run workloads. ## Kubernetes If, for some reason, you’d like to use dstack with Nebius’s managed Kubernetes service, you can point `dstack` to the cluster’s kubeconfig file, and `dstack` will allow you to fully use this cluster through `dstack`. ### Create a cluster 1. Go to `Compute` → `Kubernetes` and click `Create cluster`. Make sure to enable `Public endpoint`. 2. Go to `Node groups` and click `Create node group`. Make sure to enable `Assign public IPv4 addresses` and `Install NVIDIA GPU drivers and other components`. Select the appropriate instance type, specify the `Number of nodes`, and set `Node storage` to at least `120 GiB`. Make sure to click `Create` under `GPU cluster` if you plan to use a fast interconnect. 3. Go to `Applications`, find `NVIDIA Device Plugin`, and click `Deploy`. 4. Wait until the nodes are provisioned. > Even if you enable `autoscaling`, `dstack` can use only the nodes that are already provisioned. To provision instances on demand, use [VMs](#vms) (see above). #### Configure the kubeconfig file 1. Click `How to connect` and copy the `nebius` CLI command that configures the `kubeconfig` file. 2. Install the `nebius` CLI and run the command:
```shell $ nebius mk8s cluster get-credentials --id <cluster id> --external ```
### Configure a backend Follow the standard instructions for setting up a [`kubernetes`](../../concepts/backends.md#kubernetes) backend:
```yaml projects: - name: main backends: - type: kubernetes kubeconfig: filename: ```
### Create a fleet Once the cluster and the `dstack` server are running, you can create a fleet:
```yaml type: fleet name: nebius-fleet placement: cluster nodes: 0.. backends: [kubernetes] resources: # Specify requirements to filter nodes gpu: 8 ```
Pass the fleet configuration to `dstack apply`:
```shell $ dstack apply -f nebius-fleet.dstack.yml ```
Once the fleet is created, you can run [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), and [services](../../concepts/services.md). ## NCCL tests Use a [distributed task](../../concepts/tasks.md#distributed-tasks) to run NCCL tests and validate the cluster’s network bandwidth.
```yaml type: task name: nccl-tests nodes: 2 startup_order: workers-first stop_criteria: master-done env: - NCCL_DEBUG=INFO commands: - | if [ $DSTACK_NODE_RANK -eq 0 ]; then mpirun \ --allow-run-as-root \ --hostfile $DSTACK_MPI_HOSTFILE \ -n $DSTACK_GPUS_NUM \ -N $DSTACK_GPUS_PER_NODE \ --bind-to none \ /opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 else sleep infinity fi # Required for `/dev/infiniband` access privileged: true resources: gpu: 8 shm_size: 16GB ```
Pass the configuration to `dstack apply`:
```shell $ dstack apply -f nebius-nccl-tests.dstack.yml Provisioning... ---> 100% nccl-tests provisioning completed (running) out-of-place in-place size count type redop root time algbw busbw #wrong time algbw busbw #wrong (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 8 2 float sum -1 45.72 0.00 0.00 0 29.78 0.00 0.00 0 16 4 float sum -1 29.92 0.00 0.00 0 29.42 0.00 0.00 0 32 8 float sum -1 30.10 0.00 0.00 0 29.75 0.00 0.00 0 64 16 float sum -1 34.48 0.00 0.00 0 29.36 0.00 0.00 0 128 32 float sum -1 30.38 0.00 0.01 0 29.67 0.00 0.01 0 256 64 float sum -1 30.48 0.01 0.02 0 29.97 0.01 0.02 0 512 128 float sum -1 30.45 0.02 0.03 0 30.85 0.02 0.03 0 1024 256 float sum -1 31.36 0.03 0.06 0 31.29 0.03 0.06 0 2048 512 float sum -1 32.27 0.06 0.12 0 32.26 0.06 0.12 0 4096 1024 float sum -1 36.04 0.11 0.21 0 43.17 0.09 0.18 0 8192 2048 float sum -1 37.24 0.22 0.41 0 35.54 0.23 0.43 0 16384 4096 float sum -1 37.22 0.44 0.83 0 34.55 0.47 0.89 0 32768 8192 float sum -1 43.82 0.75 1.40 0 35.64 0.92 1.72 0 65536 16384 float sum -1 37.85 1.73 3.25 0 37.55 1.75 3.27 0 131072 32768 float sum -1 43.10 3.04 5.70 0 53.08 2.47 4.63 0 262144 65536 float sum -1 58.59 4.47 8.39 0 63.33 4.14 7.76 0 524288 131072 float sum -1 97.88 5.36 10.04 0 83.91 6.25 11.72 0 1048576 262144 float sum -1 87.08 12.04 22.58 0 77.82 13.47 25.26 0 2097152 524288 float sum -1 99.06 21.17 39.69 0 97.67 21.47 40.26 0 4194304 1048576 float sum -1 110.14 38.08 71.40 0 114.66 36.58 68.59 0 8388608 2097152 float sum -1 154.48 54.30 101.82 0 156.03 53.76 100.80 0 16777216 4194304 float sum -1 210.33 79.77 149.56 0 200.98 83.48 156.52 0 33554432 8388608 float sum -1 274.23 122.36 229.43 0 276.45 121.38 227.58 0 67108864 16777216 float sum -1 472.43 142.05 266.35 0 480.00 139.81 262.14 0 134217728 33554432 float sum -1 759.58 176.70 331.31 0 756.21 177.49 332.79 0 268435456 67108864 float sum -1 1305.66 205.59 385.49 0 1303.37 205.95 386.16 0 536870912 134217728 float sum -1 2379.38 225.63 423.06 0 2373.42 226.20 424.13 0 1073741824 268435456 float sum -1 4511.97 237.98 446.21 0 4513.82 237.88 446.02 0 2147483648 536870912 float sum -1 8776.26 244.69 458.80 0 8760.42 245.13 459.63 0 4294967296 1073741824 float sum -1 17407.8 246.73 462.61 0 17302.2 248.23 465.44 0 8589934592 2147483648 float sum -1 34448.4 249.36 467.54 0 34381.0 249.85 468.46 0 Out of bounds values : 0 OK Avg bus bandwidth : 125.499 Collective test concluded: all_reduce_perf ```
## What's next 1. Learn about [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), [services](../../concepts/services.md) 2. Check out [backends](../../concepts/backends.md) and [fleets](../../concepts/fleets.md) 3. Read Nebius' docs on [networking for VMs](https://docs.nebius.com/compute/clusters/gpu) and the [managed Kubernetes service](https://docs.nebius.com/kubernetes). # docs/examples/clusters/nccl-rccl-tests.md --- title: NCCL/RCCL tests description: Running NCCL and RCCL tests to validate cluster network bandwidth --- # NCCL/RCCL tests This example shows how to run [NCCL](https://github.com/NVIDIA/nccl-tests) or [RCCL](https://github.com/ROCm/rccl-tests) tests on a cluster using [distributed tasks](../../concepts/tasks.md#distributed-tasks). !!! info "Prerequisites" Before running a distributed task, make sure to create a fleet with `placement` set to `cluster` (can be a [managed fleet](../../concepts/fleets.md#cluster-placement) or an [SSH fleet](../../concepts/fleets.md#ssh-placement)). ## Running as a task Here's an example of a task that runs AllReduce test on 2 nodes, each with 4 GPUs (8 processes in total). === "NCCL tests"
```yaml type: task name: nccl-tests nodes: 2 startup_order: workers-first stop_criteria: master-done env: - NCCL_DEBUG=INFO commands: - | if [ $DSTACK_NODE_RANK -eq 0 ]; then mpirun \ --allow-run-as-root \ --hostfile $DSTACK_MPI_HOSTFILE \ -n $DSTACK_GPUS_NUM \ -N $DSTACK_GPUS_PER_NODE \ --bind-to none \ /opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 else sleep infinity fi # Uncomment if the `kubernetes` backend requires it for `/dev/infiniband` access #privileged: true resources: gpu: nvidia:1..8 shm_size: 16GB ```
!!! info "Default image" If you don't specify `image`, `dstack` uses its [base](https://github.com/dstackai/dstack/tree/master/docker/base) Docker image pre-configured with `uv`, `python`, `pip`, essential CUDA drivers, `mpirun`, and NCCL tests (under `/opt/nccl-tests/build`). === "RCCL tests"
```yaml type: task name: rccl-tests nodes: 2 startup_order: workers-first stop_criteria: master-done # Mount the system libraries folder from the host volumes: - /usr/local/lib:/mnt/lib image: rocm/dev-ubuntu-22.04:6.4-complete env: - NCCL_DEBUG=INFO - OPEN_MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi commands: # Setup MPI and build RCCL tests - apt-get install -y git libopenmpi-dev openmpi-bin - git clone https://github.com/ROCm/rccl-tests.git - cd rccl-tests - make MPI=1 MPI_HOME=$OPEN_MPI_HOME # Preload the RoCE driver library from the host (for Broadcom driver compatibility) - export LD_PRELOAD=/mnt/lib/libbnxt_re-rdmav34.so # Run RCCL tests via MPI - | if [ $DSTACK_NODE_RANK -eq 0 ]; then mpirun --allow-run-as-root \ --hostfile $DSTACK_MPI_HOSTFILE \ -n $DSTACK_GPUS_NUM \ -N $DSTACK_GPUS_PER_NODE \ --mca btl_tcp_if_include ens41np0 \ -x LD_PRELOAD \ -x NCCL_IB_HCA=mlx5_0/1,bnxt_re0,bnxt_re1,bnxt_re2,bnxt_re3,bnxt_re4,bnxt_re5,bnxt_re6,bnxt_re7 \ -x NCCL_IB_GID_INDEX=3 \ -x NCCL_IB_DISABLE=0 \ ./build/all_reduce_perf -b 8M -e 8G -f 2 -g 1 -w 5 --iters 20 -c 0; else sleep infinity fi resources: gpu: MI300X:8 ```
!!! info "RoCE library" Broadcom RoCE drivers require the `libbnxt_re` userspace library inside the container to be compatible with the host’s Broadcom kernel driver `bnxt_re`. To ensure this compatibility, we mount `libbnxt_re-rdmav34.so` from the host and preload it using `LD_PRELOAD` when running MPI. !!! info "Privileged" In some cases, the backend (e.g., `kubernetes`) may require `privileged: true` to access the high-speed interconnect (e.g., InfiniBand). ### Apply a configuration To run a configuration, use the [`dstack apply`](../../reference/cli/dstack/apply.md) command.
```shell $ dstack apply -f nccl-tests.dstack.yml # BACKEND REGION INSTANCE RESOURCES SPOT PRICE 1 aws us-east-1 g4dn.12xlarge 48xCPU, 192GB, 4xT4 (16GB), 100.0GB (disk) no $3.912 2 aws us-west-2 g4dn.12xlarge 48xCPU, 192GB, 4xT4 (16GB), 100.0GB (disk) no $3.912 3 aws us-east-2 g4dn.12xlarge 48xCPU, 192GB, 4xT4 (16GB), 100.0GB (disk) no $3.912 Submit the run nccl-tests? [y/n]: y ```
## What's next? 1. Check [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), [services](../../concepts/services.md), and [fleets](../../concepts/fleets.md). # docs/examples/inference/sglang.md --- title: SGLang description: Deploying Qwen3.6-27B using SGLang on NVIDIA and AMD GPUs --- # SGLang This example shows how to deploy `Qwen/Qwen3.6-27B` using [SGLang](https://github.com/sgl-project/sglang) and `dstack`. > For a `DeepSeek-V4-Pro` deployment on `B200:8`, see the [DeepSeek V4](../models/deepseek-v4.md) model page. ## Apply a configuration Here's an example of a service that deploys `Qwen/Qwen3.6-27B` using SGLang. === "NVIDIA"
```yaml type: service name: qwen36 image: lmsysorg/sglang:v0.5.10.post1 commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --reasoning-parser qwen3 \ --mem-fraction-static 0.8 \ --context-length 262144 port: 30000 model: Qwen/Qwen3.6-27B volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: shm_size: 16GB gpu: H100:4 ```
=== "AMD"
```yaml type: service name: qwen36 image: lmsysorg/sglang:v0.5.10-rocm720-mi30x commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --reasoning-parser qwen3 \ --mem-fraction-static 0.8 \ --context-length 262144 port: 30000 model: Qwen/Qwen3.6-27B volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: cpu: 52.. memory: 896GB.. shm_size: 16GB disk: 450GB.. gpu: MI300X:4 ```
The AMD example keeps the deployment close to the upstream Qwen and SGLang guidance: a pinned ROCm image, tensor parallelism across all four GPUs, and the standard `qwen3` reasoning parser without extra ROCm-specific tuning flags. Save one of the configurations above as `service.dstack.yml`, then use the [`dstack apply`](../../reference/cli/dstack/apply.md) command.
```shell $ dstack apply -f service.dstack.yml ```
If no gateway is created, the service endpoint will be available at `/proxy/services///`.
```shell curl http://127.0.0.1:3000/proxy/services/main/qwen36/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen/Qwen3.6-27B", "messages": [ { "role": "user", "content": "A bat and a ball cost $1.10 total. The bat costs $1.00 more than the ball. How much does the ball cost? Answer with just the dollar amount." } ], "separate_reasoning": true, "max_tokens": 1024 }' ```
Qwen3.6 uses thinking mode by default. To disable thinking, pass `"chat_template_kwargs": {"enable_thinking": false}` in the request body. To enable tool calling, add `--tool-call-parser qwen3_coder` to the serve command. > If a [gateway](../../concepts/gateways.md) is configured (e.g. to enable auto-scaling, HTTPS, rate limits, etc.), the service endpoint will be available at `https://qwen36./`. ## Configuration options ### PD disaggregation To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/pd_disaggregation.html), use replica groups: one for [Shepherd Model Gateway (SMG)](https://docs.sglang.io/advanced_features/sgl_model_gateway.html), one for prefill workers, and one for decode workers. === "NVIDIA"
```yaml type: service name: prefill-decode image: lmsysorg/sglang:v0.5.10.post1 env: - HF_TOKEN - MODEL_ID=zai-org/GLM-4.5-Air-FP8 replicas: - count: 1 # For now replica group with router must have count: 1 commands: - pip install smg - | smg launch \ --host 0.0.0.0 \ --port 8000 \ --pd-disaggregation \ --prefill-policy cache_aware resources: cpu: 4 router: type: sglang - count: 1..4 scaling: metric: rps target: 3 commands: - | python -m sglang.launch_server \ --model-path $MODEL_ID \ --disaggregation-mode prefill \ --disaggregation-transfer-backend nixl \ --host 0.0.0.0 \ --port 8000 \ --disaggregation-bootstrap-port 8998 resources: gpu: H200 - count: 1..8 scaling: metric: rps target: 2 commands: - | python -m sglang.launch_server \ --model-path $MODEL_ID \ --disaggregation-mode decode \ --disaggregation-transfer-backend nixl \ --host 0.0.0.0 \ --port 8000 resources: gpu: H200 port: 8000 model: zai-org/GLM-4.5-Air-FP8 # Custom probe is required for PD disaggregation. probes: - type: http url: /health interval: 15s ```
> With the `sglang` router, you can use SGLang prefill and decode workers. Support for vLLM and TensorRT-LLM workers is coming soon. Currently, auto-scaling only supports `rps` as the metric. TTFT and ITL metrics are coming soon. !!! info "Cluster" PD disaggregation requires the service to run in a fleet with `placement` set to `cluster`, because the replicas require an interconnect between instances. While the prefill and decode replicas run on GPUs, the router replica requires a CPU instance in the same cluster. ## What's next? 1. Read about [services](../../concepts/services.md) and [gateways](../../concepts/gateways.md) 2. Browse the [Qwen 3.6 SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.6) and the [SGLang server arguments reference](https://docs.sglang.ai/advanced_features/server_arguments.html) # docs/examples/inference/dynamo.md --- title: NVIDIA Dynamo description: Deploying zai-org/GLM-4.5-Air-FP8 using NVIDIA Dynamo with Prefill-Decode disaggregation. --- # Dynamo This example shows how to deploy `zai-org/GLM-4.5-Air-FP8` using [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) and `dstack`. ## Apply a configuration Here's an example of a service that deploys `zai-org/GLM-4.5-Air-FP8` using Dynamo with PD disaggregation.
```yaml type: service name: dynamo-pd env: - HF_TOKEN - MODEL_ID=zai-org/GLM-4.5-Air-FP8 replicas: - count: 1 docker: true commands: - apt-get update - apt-get install -y python3-dev python3-venv - python3 -m venv ~/dyn-venv - source ~/dyn-venv/bin/activate - pip install -U pip - pip install "ai-dynamo[sglang]==1.1.1" - git clone https://github.com/ai-dynamo/dynamo.git # Brings up the NATS / etcd compose stack and runs the Dynamo HTTP frontend. - docker compose -f dynamo/deploy/docker-compose.yml up -d - | python3 -m dynamo.frontend \ --http-host 0.0.0.0 --http-port 8000 \ --discovery-backend etcd --router-mode kv \ --kv-cache-block-size 64 resources: cpu: 4 router: type: dynamo - count: 1..4 scaling: metric: rps target: 3 python: "3.12" nvcc: true commands: # dstack injects DSTACK_ROUTER_INTERNAL_IP after the router replica # is provisioned. Compose the etcd/NATS endpoints from it. - export ETCD_ENDPOINTS="http://$DSTACK_ROUTER_INTERNAL_IP:2379" - export NATS_SERVER="nats://$DSTACK_ROUTER_INTERNAL_IP:4222" # Set to enable /health endpoint required by dstack probes. - export DYN_SYSTEM_PORT="8000" # Wait until the router's etcd and NATS ports are actually accepting connections. - | until (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/2379) 2>/dev/null \ && (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/4222) 2>/dev/null; do echo "waiting for etcd/NATS on $DSTACK_ROUTER_INTERNAL_IP..."; sleep 3 done - pip install "ai-dynamo[sglang]==1.1.1" - | python3 -m dynamo.sglang \ --model-path $MODEL_ID --served-model-name $MODEL_ID \ --discovery-backend etcd --host 0.0.0.0 \ --page-size 64 \ --disaggregation-mode prefill --disaggregation-transfer-backend nixl resources: gpu: H200 - count: 1..8 scaling: metric: rps target: 2 python: "3.12" nvcc: true commands: - export ETCD_ENDPOINTS="http://$DSTACK_ROUTER_INTERNAL_IP:2379" - export NATS_SERVER="nats://$DSTACK_ROUTER_INTERNAL_IP:4222" - export DYN_SYSTEM_PORT="8000" - | until (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/2379) 2>/dev/null \ && (echo > /dev/tcp/$DSTACK_ROUTER_INTERNAL_IP/4222) 2>/dev/null; do echo "waiting for etcd/NATS on $DSTACK_ROUTER_INTERNAL_IP..."; sleep 3 done - pip install "ai-dynamo[sglang]==1.1.1" - | python3 -m dynamo.sglang \ --model-path $MODEL_ID --served-model-name $MODEL_ID \ --discovery-backend etcd --host 0.0.0.0 \ --page-size 64 \ --disaggregation-mode decode --disaggregation-transfer-backend nixl resources: gpu: H200 port: 8000 model: zai-org/GLM-4.5-Air-FP8 # Custom probe is required for PD disaggregation. probes: - type: http url: /health interval: 15s ```
> With the the `dynamo` router, you can use SGLang, vLLM, and TensorRT-LLM prefill and decode workers. Save the configuration as `service.dstack.yml`, then use the [`dstack apply`](../../reference/cli/dstack/apply.md) command.
```shell $ dstack apply -f service.dstack.yml ```
If no gateway is created, the service endpoint will be available at `/proxy/services///`.
```shell curl http://127.0.0.1:3000/proxy/services/main/dynamo-pd/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "zai-org/GLM-4.5-Air-FP8", "messages": [ { "role": "user", "content": "What is prefill-decode disaggregation?" } ], "max_tokens": 1024 }' ```
> If a [gateway](../../concepts/gateways.md) is configured (e.g. to enable auto-scaling, HTTPS, rate limits, etc.), the service endpoint will be available at `https://dynamo-pd./`. ## Configuration options Currently, auto-scaling only supports `rps` as the metric. TTFT and ITL metrics are coming soon. !!! info "Cluster" PD disaggregation requires the service to run in a fleet with `placement` set to `cluster`, because the replicas require an interconnect between instances. While the prefill and decode replicas run on GPUs, the router replica requires a CPU instance in the same cluster. ## What's next? 1. Read about [services](../../concepts/services.md) and [gateways](../../concepts/gateways.md) 2. Browse the [NVIDIA Dynamo GitHub repository](https://github.com/ai-dynamo/dynamo) and the [SGLang](./sglang.md) example # docs/examples/inference/vllm.md --- title: vLLM description: Deploying Qwen3.6-27B using vLLM on NVIDIA and AMD GPUs --- # vLLM This example shows how to deploy `Qwen/Qwen3.6-27B` using [vLLM](https://docs.vllm.ai/en/latest/) and `dstack`. ## Apply a configuration Here's an example of a service that deploys `Qwen/Qwen3.6-27B` using vLLM. === "NVIDIA"
```yaml type: service name: qwen36 image: vllm/vllm-openai:v0.19.1 commands: - | vllm serve Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size $DSTACK_GPUS_NUM \ --max-model-len 262144 \ --reasoning-parser qwen3 port: 8000 model: Qwen/Qwen3.6-27B volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: shm_size: 16GB gpu: H100:4 ```
=== "AMD"
```yaml type: service name: qwen36 image: vllm/vllm-openai-rocm:v0.19.1 commands: - | vllm serve Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size $DSTACK_GPUS_NUM \ --max-model-len 262144 \ --reasoning-parser qwen3 port: 8000 model: Qwen/Qwen3.6-27B volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: cpu: 52.. memory: 896GB.. shm_size: 16GB disk: 450GB.. gpu: MI300X:4 ```
Qwen3.6-27B is a multimodal model. For text-only workloads, add `--language-model-only` to free more memory for the KV cache. To enable tool calling, add `--enable-auto-tool-choice --tool-call-parser qwen3_coder`. Save one of the configurations above as `service.dstack.yml`, then use the [`dstack apply`](../../reference/cli/dstack/apply.md) command.
```shell $ dstack apply -f service.dstack.yml ```
If no gateway is created, the service endpoint will be available at `/proxy/services///`.
```shell curl http://127.0.0.1:3000/proxy/services/main/qwen36/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen/Qwen3.6-27B", "messages": [ { "role": "user", "content": "A bat and a ball cost $1.10 total. The bat costs $1.00 more than the ball. How much does the ball cost?" } ], "max_tokens": 1024 }' ```
> If a [gateway](../../concepts/gateways.md) is configured (e.g. to enable auto-scaling, HTTPS, rate limits, etc.), the service endpoint will be available at `https://qwen36./`. ## What's next? 1. Read about [services](../../concepts/services.md) and [gateways](../../concepts/gateways.md) 2. Browse the [Qwen 3.5 & 3.6 vLLM recipe](https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3.5.html) and the [SGLang](../inference/sglang.md) example # docs/examples/inference/nim.md --- title: NVIDIA NIM description: Deploying Nemotron-3-Super-120B-A12B using NVIDIA NIM --- # NVIDIA NIM This example shows how to deploy Nemotron-3-Super-120B-A12B using [NVIDIA NIM](https://docs.nvidia.com/nim/large-language-models/latest/getting-started.html) and `dstack`. ??? info "Prerequisites" Once `dstack` is [installed](../../installation.md), clone the repo with examples.
```shell $ git clone https://github.com/dstackai/dstack $ cd dstack ```
## Deployment Here's an example of a service that deploys Nemotron-3-Super-120B-A12B using NIM.
```yaml type: service name: nemotron120 image: nvcr.io/nim/nvidia/nemotron-3-super-120b-a12b:1.8.0 env: - NGC_API_KEY registry_auth: username: $oauthtoken password: ${{ env.NGC_API_KEY }} port: 8000 model: nvidia/nemotron-3-super-120b-a12b volumes: - instance_path: /root/.cache/nim path: /opt/nim/.cache optional: true resources: cpu: x86:96.. memory: 512GB.. shm_size: 16GB disk: 500GB.. gpu: H100:80GB:8 ```
### Running a configuration Save the configuration above as `nemotron120.dstack.yml`, then use the [`dstack apply`](../../reference/cli/dstack/apply.md) command.
```shell $ NGC_API_KEY=... $ dstack apply -f service.dstack.yml ```
If no gateway is created, the service endpoint will be available at `/proxy/services///`.
```shell $ curl http://127.0.0.1:3000/proxy/services/main/nemotron120/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "nvidia/nemotron-3-super-120b-a12b", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What is Deep Learning?" } ], "max_tokens": 128 }' ```
When a [gateway](../../concepts/gateways.md) is configured, the service endpoint will be available at `https://nemotron120./`. ## What's next? 1. Check [services](../../concepts/services.md) 2. Browse the [Nemotron-3-Super-120B-A12B model page](https://build.nvidia.com/nvidia/nemotron-3-super-120b-a12b) # docs/examples/inference/trtllm.md --- title: TensorRT-LLM description: Deploying Qwen3-235B-A22B-FP8 using NVIDIA TensorRT-LLM on NVIDIA GPUs --- # TensorRT-LLM This example shows how to deploy `nvidia/Qwen3-235B-A22B-FP8` using [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) and `dstack`. ## Apply a configuration Here's an example of a service that deploys `nvidia/Qwen3-235B-A22B-FP8` using TensorRT-LLM.
```yaml type: service name: qwen235 image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc11 env: - HF_HUB_ENABLE_HF_TRANSFER=1 commands: - pip install hf_transfer - | trtllm-serve serve nvidia/Qwen3-235B-A22B-FP8 \ --host 0.0.0.0 \ --port 8000 \ --backend pytorch \ --tp_size $DSTACK_GPUS_NUM \ --max_batch_size 32 \ --max_num_tokens 4096 \ --kv_cache_free_gpu_memory_fraction 0.75 port: 8000 model: nvidia/Qwen3-235B-A22B-FP8 volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: cpu: 96.. memory: 512GB.. shm_size: 32GB disk: 1000GB.. gpu: H100:8 ```
Apply it with [`dstack apply`](../../reference/cli/dstack/apply.md):
```shell $ dstack apply -f service.dstack.yml ```
## Access the endpoint If no gateway is created, the service endpoint will be available at `/proxy/services///`.
```shell $ curl http://127.0.0.1:3000/proxy/services/main/qwen235/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "nvidia/Qwen3-235B-A22B-FP8", "messages": [ { "role": "user", "content": "A bat and a ball cost $1.10 total. The bat costs $1.00 more than the ball. How much does the ball cost?" } ], "chat_template_kwargs": {"enable_thinking": true}, "max_tokens": 1024, "temperature": 0.0 }' ```
When a [gateway](../../concepts/gateways.md) is configured, the service endpoint will be available at `https://qwen235./`. ## What's next? 1. Read about [services](../../concepts/services.md) and [gateways](../../concepts/gateways.md) 2. Browse the [TensorRT-LLM deployment guides](https://nvidia.github.io/TensorRT-LLM/deployment-guide/index.html) and the [Qwen3 deployment guide](https://nvidia.github.io/TensorRT-LLM/deployment-guide/deployment-guide-for-qwen3-on-trtllm.html) 3. See the [`trtllm-serve` reference](https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve/trtllm-serve.html) # docs/examples/models/deepseek-v4.md --- title: DeepSeek V4 description: Deploying DeepSeek-V4-Pro using SGLang on NVIDIA B200:8 --- # DeepSeek V4 This example shows how to deploy `deepseek-ai/DeepSeek-V4-Pro` as a [service](../../concepts/services.md) using [SGLang](https://github.com/sgl-project/sglang) and `dstack`. ## Apply a configuration Save the following configuration as `deepseek-v4.dstack.yml`.
```yaml type: service name: deepseek-v4 image: lmsysorg/sglang:deepseek-v4-blackwell env: - HF_TOKEN - SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256 - SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 commands: - | sglang serve \ --trust-remote-code \ --model-path deepseek-ai/DeepSeek-V4-Pro \ --tp 8 \ --dp 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.82 \ --cuda-graph-max-bs 64 \ --max-running-requests 256 \ --deepep-config '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}' \ --tool-call-parser deepseekv4 \ --reasoning-parser deepseek-v4 \ --host 0.0.0.0 \ --port 30000 port: 30000 model: deepseek-ai/DeepSeek-V4-Pro volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: gpu: B200:8 shm_size: 32GB disk: 2TB.. ```
This configuration uses the single-node Blackwell `DeepSeek-V4-Pro` recipe shape for `8 x NVIDIA B200`. Export your Hugging Face token and apply the configuration with [`dstack apply`](../../reference/cli/dstack/apply.md).
```shell $ export HF_TOKEN= $ dstack apply -f deepseek-v4.dstack.yml ```
If no gateway is created, the service endpoint will be available at `/proxy/services///`.
```shell curl http://127.0.0.1:3000/proxy/services/main/deepseek-v4/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Pro", "messages": [ { "role": "user", "content": "What is 15% of 240? Reply with just the number." } ], "temperature": 0, "max_tokens": 32 }' ```
## Reasoning mode To separate the model's reasoning into `reasoning_content`, keep `--reasoning-parser deepseek-v4` in the server command and send `chat_template_kwargs` in the request body. For raw HTTP requests, `chat_template_kwargs` and `separate_reasoning` must be top-level JSON fields.
```shell curl http://127.0.0.1:3000/proxy/services/main/deepseek-v4/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Pro", "messages": [ { "role": "user", "content": "Solve step by step: If 3x + 5 = 20, what is x?" } ], "temperature": 0, "max_tokens": 256, "chat_template_kwargs": { "thinking": true }, "separate_reasoning": true }' ```
This returns both: - `reasoning_content`: a separate reasoning trace - `content`: the final user-visible answer ## Deployment notes - The first startup can take several minutes while the model loads and SGLang finishes initialization. - The optional `/root/.cache` instance volume helps reuse the model cache on backends that support instance volumes. ## What's next? 1. Read the [DeepSeek-V4-Pro model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) 2. Read the [DeepSeek-V4 SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4) 3. Browse the dedicated [SGLang](../inference/sglang.md) and [vLLM](../inference/vllm.md) examples # docs/examples/models/qwen36.md --- title: Qwen 3.6 description: Deploying Qwen3.6-27B using SGLang on NVIDIA and AMD GPUs --- # Qwen 3.6 This example shows how to deploy `Qwen/Qwen3.6-27B` as a [service](../../concepts/services.md) using [SGLang](https://github.com/sgl-project/sglang) and `dstack`. ## Apply a configuration Save one of the following configurations as `qwen36.dstack.yml`. === "NVIDIA"
```yaml type: service name: qwen36 image: lmsysorg/sglang:v0.5.10.post1 commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --reasoning-parser qwen3 \ --mem-fraction-static 0.8 \ --context-length 262144 port: 30000 model: Qwen/Qwen3.6-27B volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: shm_size: 16GB gpu: H100:4 ```
=== "AMD"
```yaml type: service name: qwen36 image: lmsysorg/sglang:v0.5.10-rocm720-mi30x commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --reasoning-parser qwen3 \ --mem-fraction-static 0.8 \ --context-length 262144 port: 30000 model: Qwen/Qwen3.6-27B volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: cpu: 52.. memory: 896GB.. shm_size: 16GB disk: 450GB.. gpu: MI300X:4 ```
The NVIDIA and AMD configurations above use pinned SGLang images and the same straightforward 4-GPU layout used across the Qwen 3.6 docs and examples. Apply the configuration with [`dstack apply`](../../reference/cli/dstack/apply.md).
```shell $ dstack apply -f qwen36.dstack.yml ```
If no gateway is created, the service endpoint will be available at `/proxy/services///`.
```shell curl http://127.0.0.1:3000/proxy/services/main/qwen36/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen/Qwen3.6-27B", "messages": [ { "role": "user", "content": "A bat and a ball cost $1.10 total. The bat costs $1.00 more than the ball. How much does the ball cost? Answer with just the dollar amount." } ], "max_tokens": 1024 }' ```
## Thinking mode Qwen3.6 uses thinking mode by default. With SGLang, the reasoning stream is returned separately as `reasoning_content`. To disable thinking, pass `chat_template_kwargs` in the request body.
```shell curl http://127.0.0.1:3000/proxy/services/main/qwen36/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen/Qwen3.6-27B", "messages": [ { "role": "user", "content": "Summarize the benefits of container images in one sentence." } ], "max_tokens": 256, "chat_template_kwargs": { "enable_thinking": false } }' ```
## What's next? 1. Read the [Qwen/Qwen3.6-27B model card](https://huggingface.co/Qwen/Qwen3.6-27B) 2. Read the [Qwen 3.6 SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.6) 3. Read the [Qwen 3.5 & 3.6 vLLM recipe](https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3.5.html) 4. Browse the dedicated [SGLang](../inference/sglang.md) and [vLLM](../inference/vllm.md) examples 5. Check the [AMD](../accelerators/amd.md) example for more AMD deployment and training configurations # docs/examples/accelerators/amd.md --- title: AMD description: Deploying and fine-tuning models on AMD MI300X GPUs using SGLang, vLLM, TRL, and Axolotl --- # AMD `dstack` supports running dev environments, tasks, and services on AMD GPUs. You can do that by setting up an [SSH fleet](../../concepts/fleets.md#ssh-fleets) with on-prem AMD GPUs or configuring a backend that offers AMD GPUs such as the `runpod` backend. ## Deployment Here are examples of a [service](../../concepts/services.md) that deploy `Qwen/Qwen3.6-27B` on AMD MI300X GPUs using [SGLang](https://github.com/sgl-project/sglang) and [vLLM](https://docs.vllm.ai/en/latest/). === "SGLang"
```yaml type: service name: qwen36-service-sglang-amd image: lmsysorg/sglang:v0.5.10-rocm720-mi30x commands: - | sglang serve \ --model-path Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 30000 \ --tp $DSTACK_GPUS_NUM \ --reasoning-parser qwen3 \ --mem-fraction-static 0.8 \ --context-length 262144 port: 30000 model: Qwen/Qwen3.6-27B volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: cpu: 52.. memory: 896GB.. shm_size: 16GB disk: 450GB.. gpu: MI300X:4 ```
=== "vLLM"
```yaml type: service name: qwen36-service-vllm-amd image: vllm/vllm-openai-rocm:v0.19.1 commands: - | vllm serve Qwen/Qwen3.6-27B \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size $DSTACK_GPUS_NUM \ --max-model-len 262144 \ --reasoning-parser qwen3 port: 8000 model: Qwen/Qwen3.6-27B volumes: - instance_path: /root/.cache path: /root/.cache optional: true resources: cpu: 52.. memory: 896GB.. shm_size: 16GB disk: 450GB.. gpu: MI300X:4 ```
!!! info "Docker image" AMD workloads require specifying an image with ROCm-compatible userspace and framework packages. The SGLang and vLLM examples above use pinned ROCm images. If you already have a ROCm-compatible image, use it. Otherwise, choose an image for the framework you use from [ROCm Docker images](https://hub.docker.com/u/rocm), e.g. `rocm/sgl-dev` for SGLang, `rocm/vllm` for vLLM, or `rocm/pytorch` for PyTorch. For generic AMD dev environments or tasks, use `rocm/dev-ubuntu-24.04`. To request multiple GPUs, specify the quantity after the GPU name, separated by a colon, e.g., `MI300X:4`. ## Fine-tuning > If you're planning multi-node AMD training, validate cluster networking first with the [NCCL/RCCL tests](../clusters/nccl-rccl-tests.md) example. === "TRL" Below is an example of LoRA fine-tuning Llama 3.1 8B using [TRL](https://rocm.docs.amd.com/en/latest/how-to/llm-fine-tuning-optimization/single-gpu-fine-tuning-and-inference.html) and the [`mlabonne/guanaco-llama2-1k`](https://huggingface.co/datasets/mlabonne/guanaco-llama2-1k) dataset.
```yaml type: task name: trl-amd-llama31-train # Using Runpod's ROCm Docker image image: runpod/pytorch:2.1.2-py3.10-rocm6.1-ubuntu22.04 # Required environment variables env: - HF_TOKEN # Mount files files: - train.py # Commands of the task commands: - export PATH=/opt/conda/envs/py_3.10/bin:$PATH - git clone https://github.com/ROCm/bitsandbytes - cd bitsandbytes - git checkout rocm_enabled - pip install -r requirements-dev.txt - cmake -DBNB_ROCM_ARCH="gfx942" -DCOMPUTE_BACKEND=hip -S . - make - pip install . - pip install trl - pip install peft - pip install transformers datasets huggingface-hub scipy - cd .. - python train.py # Uncomment to leverage spot instances #spot_policy: auto resources: gpu: MI300X disk: 150GB ```
=== "Axolotl" Below is an example of fine-tuning Llama 3.1 8B using [Axolotl](https://rocm.blogs.amd.com/artificial-intelligence/axolotl/README.html) and the [tatsu-lab/alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) dataset.
```yaml type: task # The name is optional, if not specified, generated randomly name: axolotl-amd-llama31-train # Using Runpod's ROCm Docker image image: runpod/pytorch:2.1.2-py3.10-rocm6.0.2-ubuntu22.04 # Required environment variables env: - HF_TOKEN - WANDB_API_KEY - WANDB_PROJECT - WANDB_NAME=axolotl-amd-llama31-train - HUB_MODEL_ID # Commands of the task commands: - export PATH=/opt/conda/envs/py_3.10/bin:$PATH - pip uninstall torch torchvision torchaudio -y - python3 -m pip install --pre torch==2.3.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0/ - git clone https://github.com/OpenAccess-AI-Collective/axolotl - cd axolotl - git checkout d4f6c65 - pip install -e . # Latest pynvml is not compatible with axolotl commit d4f6c65, so we need to fall back to version 11.5.3 - pip uninstall pynvml -y - pip install pynvml==11.5.3 - cd .. - wget https://dstack-binaries.s3.amazonaws.com/flash_attn-2.0.4-cp310-cp310-linux_x86_64.whl - pip install flash_attn-2.0.4-cp310-cp310-linux_x86_64.whl - wget https://dstack-binaries.s3.amazonaws.com/xformers-0.0.26-cp310-cp310-linux_x86_64.whl - pip install xformers-0.0.26-cp310-cp310-linux_x86_64.whl - git clone --recurse https://github.com/ROCm/bitsandbytes - cd bitsandbytes - git checkout rocm_enabled - pip install -r requirements-dev.txt - cmake -DBNB_ROCM_ARCH="gfx942" -DCOMPUTE_BACKEND=hip -S . - make - pip install . - cd .. - accelerate launch -m axolotl.cli.train -- axolotl/examples/llama-3/fft-8b.yaml --wandb-project "$WANDB_PROJECT" --wandb-name "$WANDB_NAME" --hub-model-id "$HUB_MODEL_ID" resources: gpu: MI300X disk: 150GB ```
Note, to support ROCm, we need to checkout to commit `d4f6c65`. This commit eliminates the need to manually modify the Axolotl source code to make xformers compatible with ROCm, as described in the [xformers workaround](https://docs.axolotl.ai/docs/amd_hpc.html#apply-xformers-workaround). This installation approach is also followed for building Axolotl ROCm docker image. [(See Dockerfile)](https://github.com/ROCm/rocm-blogs/blob/release/blogs/artificial-intelligence/axolotl/src/Dockerfile.rocm). > To speed up installation of `flash-attention` and `xformers`, we use pre-built binaries uploaded to S3. ## Running a configuration Once a configuration is ready, save it to a `.dstack.yml` file. If your configuration references environment variables such as `HF_TOKEN` or `WANDB_API_KEY`, export them first. Then run `dstack apply -f `, and `dstack` will automatically provision the cloud resources and run the configuration.
```shell $ dstack apply -f ```
## What's next? 1. Browse the dedicated [SGLang](../inference/sglang.md) and [vLLM](../inference/vllm.md) examples, plus [Axolotl](https://github.com/ROCm/rocm-blogs/tree/release/blogs/artificial-intelligence/axolotl), [TRL](https://rocm.docs.amd.com/en/latest/how-to/llm-fine-tuning-optimization/fine-tuning-and-inference.html), and [ROCm Bitsandbytes](https://github.com/ROCm/bitsandbytes) 2. For multi-node training, run [NCCL/RCCL tests](../clusters/nccl-rccl-tests.md) to validate AMD cluster networking. 3. Check [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), and [services](../../concepts/services.md). # docs/examples/accelerators/tpu.md --- title: TPU description: Deploying and fine-tuning models on Google Cloud TPUs using Optimum TPU and vLLM --- # TPU If you've configured the `gcp` backend in `dstack`, you can run dev environments, tasks, and services on [TPUs](https://cloud.google.com/tpu/docs/intro-to-tpu). Choose a TPU instance by specifying the TPU version and the number of cores (e.g. `v5litepod-8`) in the `gpu` property under `resources`, or request TPUs by specifying `tpu` as `vendor` ([see examples](../../guides/protips.md#gpu)). Below are a few examples on using TPUs for deployment and fine-tuning. !!! info "Multi-host TPUs" Currently, `dstack` supports only single-host TPUs, which means that the maximum supported number of cores is `8` (e.g. `v2-8`, `v3-8`, `v5litepod-8`, `v5p-8`, `v6e-8`). Multi-host TPU support is on the roadmap. !!! info "TPU storage" By default, each TPU VM contains a 100GB boot disk and its size cannot be changed. If you need more storage, attach additional disks using [Volumes](../../concepts/volumes.md). ## Deployment Many serving frameworks including vLLM and TGI have TPU support. Here's an example of a [service](../../concepts/services.md) that deploys Llama 3.1 8B using [Optimum TPU](https://github.com/huggingface/optimum-tpu) and [vLLM](https://github.com/vllm-project/vllm). === "Optimum TPU"
```yaml type: service name: llama31-service-optimum-tpu image: dstackai/optimum-tpu:llama31 env: - HF_TOKEN - MODEL_ID=meta-llama/Meta-Llama-3.1-8B-Instruct - MAX_TOTAL_TOKENS=4096 - MAX_BATCH_PREFILL_TOKENS=4095 commands: - text-generation-launcher --port 8000 port: 8000 # Register the model model: meta-llama/Meta-Llama-3.1-8B-Instruct resources: gpu: v5litepod-4 ```
Note that for Optimum TPU `MAX_INPUT_TOKEN` is set to 4095 by default. We must also set `MAX_BATCH_PREFILL_TOKENS` to 4095. ??? info "Docker image" The official Docker image `huggingface/optimum-tpu:latest` doesn’t support Llama 3.1-8B. We’ve created a custom image with the fix: `dstackai/optimum-tpu:llama31`. Once the [pull request](https://github.com/huggingface/optimum-tpu/pull/92) is merged, the official Docker image can be used. === "vLLM"
```yaml type: service name: llama31-service-vllm-tpu env: - MODEL_ID=meta-llama/Meta-Llama-3.1-8B-Instruct - HF_TOKEN - DATE=20240828 - TORCH_VERSION=2.5.0 - VLLM_TARGET_DEVICE=tpu - MAX_MODEL_LEN=4096 commands: - pip install https://storage.googleapis.com/pytorch-xla-releases/wheels/tpuvm/torch-${TORCH_VERSION}.dev${DATE}-cp311-cp311-linux_x86_64.whl - pip3 install https://storage.googleapis.com/pytorch-xla-releases/wheels/tpuvm/torch_xla-${TORCH_VERSION}.dev${DATE}-cp311-cp311-linux_x86_64.whl - pip install torch_xla[tpu] -f https://storage.googleapis.com/libtpu-releases/index.html - pip install torch_xla[pallas] -f https://storage.googleapis.com/jax-releases/jax_nightly_releases.html -f https://storage.googleapis.com/jax-releases/jaxlib_nightly_releases.html - git clone https://github.com/vllm-project/vllm.git - cd vllm - pip install -r requirements-tpu.txt - apt-get install -y libopenblas-base libopenmpi-dev libomp-dev - python setup.py develop - vllm serve $MODEL_ID --tensor-parallel-size 4 --max-model-len $MAX_MODEL_LEN --port 8000 port: 8000 # Register the model model: meta-llama/Meta-Llama-3.1-8B-Instruct # Uncomment to leverage spot instances #spot_policy: auto resources: gpu: v5litepod-4 ```
Note, when using Llama 3.1 8B with a `v5litepod` which has 16GB memory per core, we must limit the context size to 4096 tokens to fit the memory. ### Memory requirements Below are the approximate memory requirements for serving LLMs with the minimal required TPU configuration: | Model size | bfloat16 | TPU | int8 | TPU | |------------|----------|--------------|-------|----------------| | **8B** | 16GB | v5litepod-4 | 8GB | v5litepod-4 | | **70B** | 140GB | v5litepod-16 | 70GB | v5litepod-16 | | **405B** | 810GB | v5litepod-64 | 405GB | v5litepod-64 | Note, `v5litepod` is optimized for serving transformer-based models. Each core is equipped with 16GB of memory. ### Supported frameworks | Framework | Quantization | Note | |-----------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **TGI** | bfloat16 | To deploy with TGI, Optimum TPU must be used. | | **vLLM** | int8, bfloat16 | int8 quantization still requires the same memory because the weights are first moved to the TPU in bfloat16, and then converted to int8. See the [pull request](https://github.com/vllm-project/vllm/pull/7005) for more details. | ### Running a configuration Once the configuration is ready, run `dstack apply -f `, and `dstack` will automatically provision the cloud resources and run the configuration. ## Fine-tuning with Optimum TPU Below is an example of fine-tuning Llama 3.1 8B using [Optimum TPU](https://github.com/huggingface/optimum-tpu) and the [`Abirate/english_quotes`](https://huggingface.co/datasets/Abirate/english_quotes) dataset.
```yaml type: task name: optimum-tpu-llama-train python: "3.11" env: - HF_TOKEN files: - train.py - config.yaml commands: - git clone -b add_llama_31_support https://github.com/dstackai/optimum-tpu.git - mkdir -p optimum-tpu/examples/custom/ - cp train.py optimum-tpu/examples/custom/train.py - cp config.yaml optimum-tpu/examples/custom/config.yaml - cd optimum-tpu - pip install -e . -f https://storage.googleapis.com/libtpu-releases/index.html - pip install datasets evaluate - pip install accelerate -U - pip install peft - python examples/custom/train.py examples/custom/config.yaml resources: gpu: v5litepod-8 ```
[//]: # (### Fine-Tuning with TRL) [//]: # (Use the example `examples/single-node-training/optimum-tpu/gemma/train.dstack.yml` to Finetune `Gemma-2B` model using `trl` with `dstack` and `optimum-tpu`. ) ### Memory requirements Below are the approximate memory requirements for fine-tuning LLMs with the minimal required TPU configuration: | Model size | LoRA | TPU | |------------|-------|--------------| | **8B** | 16GB | v5litepod-8 | | **70B** | 160GB | v5litepod-16 | | **405B** | 950GB | v5litepod-64 | Note, `v5litepod` is optimized for fine-tuning transformer-based models. Each core is equipped with 16GB of memory. ### Supported frameworks | Framework | Quantization | Note | |-----------------|--------------|---------------------------------------------------------------------------------------------------| | **TRL** | bfloat16 | To fine-tune using TRL, Optimum TPU is recommended. TRL doesn't support Llama 3.1 out of the box. | | **Pytorch XLA** | bfloat16 | | ## What's next? 1. Browse [Optimum TPU](https://github.com/huggingface/optimum-tpu), [Optimum TPU TGI](https://github.com/huggingface/optimum-tpu/tree/main/text-generation-inference) and [vLLM](https://docs.vllm.ai/en/latest/getting_started/tpu-installation.html). 2. Check [dev environments](../../concepts/dev-environments.md), [tasks](../../concepts/tasks.md), [services](../../concepts/services.md), and [fleets](../../concepts/fleets.md). # docs/examples/accelerators/tenstorrent.md --- title: Tenstorrent description: Running dev environments, tasks, and services on Tenstorrent Wormhole accelerators --- # Tenstorrent `dstack` supports running dev environments, tasks, and services on Tenstorrent [Wormwhole](https://tenstorrent.com/en/hardware/wormhole) accelerators via SSH fleets. ??? info "SSH fleets"
```yaml type: fleet name: tt-fleet ssh_config: user: root identity_file: ~/.ssh/id_rsa # Configure any number of hosts with n150 or n300 PCEe boards hosts: - 192.168.2.108 ```
> Hosts should be pre-installed with [Tenstorrent software](https://docs.tenstorrent.com/getting-started/README.html#software-installation). This should include the drivers, `tt-smi`, and HugePages. To apply the fleet configuration, run:
```bash $ dstack apply -f tt-fleet.dstack.yml FLEET RESOURCES PRICE STATUS CREATED tt-fleet cpu=12 mem=32GB disk=243GB n150:12GB $0 idle 18 sec ago ```
For more details on fleet configuration, refer to [SSH fleets](../../concepts/fleets.md#ssh-fleets). ## Services Here's an example of a service that deploys [`Llama-3.2-1B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-1B) using [Tenstorrent Inference Service](https://github.com/tenstorrent/tt-inference-server).
```yaml type: service name: tt-inference-server env: - HF_TOKEN - HF_MODEL_REPO_ID=meta-llama/Llama-3.2-1B-Instruct image: ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-20.04-amd64:0.0.4-v0.56.0-rc47-e2e0002ac7dc commands: - | . ${PYTHON_ENV_DIR}/bin/activate pip install "huggingface_hub[cli]" export LLAMA_DIR="/data/models--$(echo "$HF_MODEL_REPO_ID" | sed 's/\//--/g')/" huggingface-cli download $HF_MODEL_REPO_ID --local-dir $LLAMA_DIR python /home/container_app_user/app/src/run_vllm_api_server.py port: 7000 model: meta-llama/Llama-3.2-1B-Instruct # Cache downloaded model volumes: - /mnt/data/tt-inference-server/data:/data resources: gpu: n150:1 ```
Go ahead and run configuration using `dstack apply`:
```bash $ dstack apply -f service.dstack.yml ```
Once the service is up, it will be available via the service endpoint at `/proxy/services///`.
```shell $ curl http://127.0.0.1:3000/proxy/services/main/tt-inference-server/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ "model": "meta-llama/Llama-3.2-1B-Instruct", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What is Deep Learning?" } ], "stream": true, "max_tokens": 512 }' ```
Additionally, the model is available via `dstack`'s control plane UI: ![](https://dstack.ai/static-assets/static-assets/images/dstack-tenstorrent-model-ui.png){ width=800 } When a [gateway](../../concepts/gateways.md) is configured, the service endpoint is available at `https://./`. > Services support many options, including authentication, auto-scaling policies, etc. To learn more, refer to [Services](../../concepts/services.md). ## Tasks Below is a task that simply runs `tt-smi -s`. Tasks can be used for training, fine-tuning, batch inference, or antything else.
```yaml type: task # The name is optional, if not specified, generated randomly name: tt-smi env: - HF_TOKEN # (Required) Use any image with TT drivers image: dstackai/tt-smi:latest # Use any commands commands: - tt-smi -s # Specify the number of accelerators, model, etc resources: gpu: n150:1 # Uncomment if you want to run on a cluster of nodes #nodes: 2 ```
> Tasks support many options, including multi-node configuration, max duration, etc. To learn more, refer to [Tasks](../../concepts/tasks.md). ## Dev environments Below is an example of a dev environment configuration. It can be used to provision a dev environemnt that can be accessed via your desktop IDE.
```yaml type: dev-environment # The name is optional, if not specified, generated randomly name: cursor # (Optional) List required env variables env: - HF_TOKEN image: dstackai/tt-smi:latest # Can be `vscode` or `cursor` ide: cursor resources: gpu: n150:1 ```
If you run it via `dstack apply`, it will output the URL to access it via your desktop IDE. ![](https://dstack.ai/static-assets/static-assets/images/dstack-tenstorrent-cursor.png){ width=800 } > Dev nevironments support many options, including inactivity and max duration, IDE configuration, etc. To learn more, refer to [Dev environments](../../concepts/tasks.md). ??? info "Feedback" Found a bug, or want to request a feature? File it in the [issue tracker](https://github.com/dstackai/dstack/issues), or share via [Discord](https://discord.gg/u8SmfwPpMd).