Dev environments¶
A dev environment lets you provision an instance and access it with your desktop IDE.
Apply a configuration¶
First, define a dev environment configuration as a YAML file in your project folder.
The filename must end with .dstack.yml (e.g. .dstack.yml or dev.dstack.yml are both acceptable).
type: dev-environment
# The name is optional, if not specified, generated randomly
name: vscode
python: "3.11"
# Uncomment to use a custom Docker image
#image: huggingface/trl-latest-gpu
ide: vscode
# Uncomment to leverage spot instances
#spot_policy: auto
resources:
gpu: 24GB
To run a dev environment, pass the configuration to dstack apply:
$ dstack apply -f examples/.dstack.yml
# BACKEND REGION RESOURCES SPOT PRICE
1 runpod CA-MTL-1 9xCPU, 48GB, A5000:24GB yes $0.11
2 runpod EU-SE-1 9xCPU, 43GB, A5000:24GB yes $0.11
3 gcp us-west4 4xCPU, 16GB, L4:24GB yes $0.214516
Submit the run vscode? [y/n]: y
Launching `vscode`...
---> 100%
To open in VS Code Desktop, use this link:
vscode://vscode-remote/ssh-remote+vscode/workflow
dstack apply automatically provisions an instance, uploads the contents of the repo (incl. your local uncommitted changes),
and sets up an IDE on the instance.
Windows
On Windows, dstack works both natively and inside WSL. But, for dev environments,
it's recommended not to use dstack apply inside WSL due to a VS Code issue .
To open the dev environment in your desktop IDE, use the link from the output
(such as vscode://vscode-remote/ssh-remote+fast-moth-1/workflow).
SSH
Alternatively, while the CLI is attached to the run, you can connect to the dev environment via SSH:
$ ssh vscode
Configuration options¶
Initialization¶
If you want to pre-configure the dev environment, specify the init
property with a list of commands to run at startup:
type: dev-environment
name: vscode
python: "3.11"
ide: vscode
init:
- pip install wandb
Resources¶
When you specify a resource value like cpu or memory,
you can either use an exact value (e.g. 24GB) or a
range (e.g. 24GB.., or 24GB..80GB, or ..80GB).
type: dev-environment
# The name is optional, if not specified, generated randomly
name: vscode
ide: vscode
resources:
# 16 or more x86_64 cores
cpu: 16..
# 200GB or more RAM
memory: 200GB..
# 4 GPUs from 40GB to 80GB
gpu: 40GB..80GB:4
# Shared memory (required by multi-gpu)
shm_size: 16GB
# Disk size
disk: 500GB
The cpu property lets you set the architecture (x86 or arm) and core count — e.g., x86:16 (16 x86 cores), arm:8.. (at least 8 ARM cores).
If not set, dstack infers it from the GPU or defaults to x86.
The gpu property lets you specify vendor, model, memory, and count — e.g., nvidia (one NVIDIA GPU), A100 (one A100), A10G,A100 (either), A100:80GB (one 80GB A100), A100:2 (two A100), 24GB..40GB:2 (two GPUs with 24–40GB), A100:40GB:2 (two 40GB A100s).
If vendor is omitted, dstack infers it from the model or defaults to nvidia.
Google Cloud TPU
To use TPUs, specify its architecture via the gpu property.
type: dev-environment
name: vscode
ide: vscode
resources:
gpu: v2-8
Currently, only 8 TPU cores can be specified, supporting single TPU device workloads. Multi-TPU support is coming soon.
Shared memory
If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure
shm_size, e.g. set it to 16GB.
If you’re unsure which offers (hardware configurations) are available from the configured backends, use the
dstack offercommand to list them.
Docker¶
Default image¶
If you don't specify image, dstack uses its base Docker image pre-configured with
uv, python, pip, essential CUDA drivers, and NCCL tests (under /opt/nccl-tests/build).
Set the python property to pre-install a specific version of Python.
type: dev-environment
name: vscode
python: 3.12
ide: vscode
NVCC¶
By default, the base Docker image doesn’t include nvcc, which is required for building custom CUDA kernels.
If you need nvcc, set the nvcc property to true.
type: dev-environment
name: vscode
python: 3.12
nvcc: true
ide: vscode
init:
- uv pip install flash_attn --no-build-isolation
Custom image¶
If you want, you can specify your own Docker image via image.
type: dev-environment
name: vscode
image: huggingface/trl-latest-gpu
ide: vscode
Docker in Docker¶
Set docker to true to enable the docker CLI in your dev environment, e.g., to run or build Docker images, or use Docker Compose.
type: dev-environment
name: vscode
docker: true
ide: vscode
init:
- docker run --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi
Cannot be used with python or image. Not supported on runpod, vastai, or kubernetes.
Privileged mode¶
To enable privileged mode, set privileged to true.
Not supported with runpod, vastai, and kubernetes.
Private registry¶
Use the registry_auth property to provide credentials for a private Docker registry.
type: dev-environment
name: vscode
env:
- NGC_API_KEY
image: nvcr.io/nim/deepseek-ai/deepseek-r1-distill-llama-8b
registry_auth:
username: $oauthtoken
password: ${{ env.NGC_API_KEY }}
ide: vscode
Environment variables¶
type: dev-environment
name: vscode
env:
- HF_TOKEN
- HF_HUB_ENABLE_HF_TRANSFER=1
ide: vscode
If you don't assign a value to an environment variable (see HF_TOKEN above),
dstack will require the value to be passed via the CLI or set in the current process.
System environment variables
The following environment variables are available in any run by default:
| Name | Description |
|---|---|
DSTACK_RUN_NAME |
The name of the run |
DSTACK_REPO_ID |
The ID of the repo |
DSTACK_GPUS_NUM |
The total number of GPUs in the run |
Retry policy¶
By default, if dstack can't find capacity or the instance is interrupted, the run will fail.
If you'd like dstack to automatically retry, configure the
retry property accordingly:
type: dev-environment
# The name is optional, if not specified, generated randomly
name: vscode
ide: vscode
retry:
# Retry on specific events
on_events: [no-capacity, error, interruption]
# Retry for up to 1 hour
duration: 1h
Inactivity duration¶
Set inactivity_duration
to automatically stop the dev environment after a configured period of inactivity.
type: dev-environment
name: vscode
ide: vscode
# Stop if inactive for 2 hours
inactivity_duration: 2h
The dev environment becomes inactive when you close the remote VS Code window,
close any ssh <run name> shells, and stop the dstack apply or dstack attach command.
If you go offline without stopping anything manually, the dev environment will also become inactive
within about 3 minutes.
If inactivity_duration is configured for your dev environment, you can see how long
it has been inactive in dstack ps --verbose (or -v).
$ dstack ps -v
NAME BACKEND RESOURCES PRICE STATUS SUBMITTED
vscode cudo 2xCPU, 8GB, $0.0286 running 8 mins ago
100.0GB (disk) (inactive for 2m 34s)
If you reattach to the dev environment using dstack attach,
the inactivity timer will be reset within a few seconds.
In-place update
As long as the configuration defines the name property, the value of inactivity_duration
can be changed for a running dev environment without a restart.
Just change the value in the configuration and run dstack apply again.
$ dstack apply -f .dstack.yml
Detected configuration changes that can be updated in-place: ['inactivity_duration']
Update the run? [y/n]:
inactivity_durationis not to be confused withidle_duration. The latter determines how soon the underlying cloud instance will be terminated after the dev environment is stopped.
Utilization policy¶
Sometimes it’s useful to track whether a dev environment is fully utilizing all GPUs. While you can check this with
dstack metrics, dstack also lets you set a policy to auto-terminate the run if any GPU is underutilized.
Below is an example of a dev environment that auto-terminate if any GPU stays below 10% utilization for 1 hour.
type: dev-environment
name: my-dev
python: 3.12
ide: cursor
resources:
gpu: H100:8
utilization_policy:
min_gpu_utilization: 10
time_window: 1h
Spot policy¶
By default, dstack uses on-demand instances. However, you can change that
via the spot_policy property. It accepts spot, on-demand, and auto.
Creation policy¶
By default, when you run dstack apply with a dev environment, task, or service,
if no idle instances from the available fleets meet the requirements, dstack creates a new fleet
using configured backends.
To ensure dstack apply doesn't create a new fleet but reuses an existing one,
pass -R (or --reuse) to dstack apply.
$ dstack apply -R -f examples/.dstack.yml
Or, set creation_policy to reuse in the run configuration.
Idle duration¶
If a fleet is created automatically, it stays idle for 5 minutes by default and can be reused within that time.
If the fleet is not reused within this period, it is automatically terminated.
To change the default idle duration, set
idle_duration in the run configuration (e.g., 0s, 1m, or off for
unlimited).
Fleets
For greater control over fleet provisioning, it is recommended to create fleets explicitly.
Reference
Dev environments support many more configuration options,
incl. backends,
regions,
max_price, and
max_duration,
among others.
Manage runs¶
dstack provides several commands to manage runs:
dstack ps– Lists all running jobs and their statuses. Use--watch(or-w) to monitor the live status of runs.dstack stop– Stops a run gracefully. Pass--abortor-xto stop it immediately without waiting for a graceful shutdown. By default, a run runs until you stop it or its lifetime exceeds the value ofmax_duration.dstack attach– By default,dstack applyruns in attached mode, establishing an SSH tunnel to the run, forwarding ports, and displaying real-time logs. If you detach from a run, use this command to reattach.dstack logs– Displays run logs. Pass--diagnoseor-dto view diagnostic logs, which can help troubleshoot failed runs.