Python API¶
The Python API enables running tasks, services, and managing runs programmatically.
Usage example¶
Below is a quick example of submitting a task for running and displaying its logs.
import sys
from dstack.api import Task, GPU, Client, Resources
client = Client.from_config()
task = Task(
name="my-awesome-run", # If not specified, a random name is assigned
image="ghcr.io/huggingface/text-generation-inference:latest",
env={"MODEL_ID": "TheBloke/Llama-2-13B-chat-GPTQ"},
commands=[
"text-generation-launcher --trust-remote-code --quantize gptq",
],
ports=["80"],
resources=Resources(gpu=GPU(memory="24GB")),
)
run = client.runs.apply_configuration(
configuration=task,
repo=None, # Specify to mount additional files
)
run.attach()
try:
for log in run.logs():
sys.stdout.buffer.write(log)
sys.stdout.buffer.flush()
except KeyboardInterrupt:
run.stop(abort=True)
finally:
run.detach()
NOTE:
- The
configuration
argument in theapply_configuration
method can be eitherdstack.api.Task
,dstack.api.Service
, ordstack.api.DevEnvironment
. - When you create
dstack.api.Task
,dstack.api.Service
, ordstack.api.DevEnvironment
, you can specify theimage
argument. Ifimage
isn't specified, the default image will be used. For a private Docker registry, ensure you also pass theregistry_auth
argument. - The
repo
argument in theapply_configuration
method allows the mounting of a local folder, a remote repo, or a programmatically created repo. In this case, thecommands
argument can refer to the files within this repo. - The
attach
method waits for the run to start and, fordstack.api.Task
sets up an SSH tunnel and forwards configuredports
tolocalhost
.
dstack.api
¶
dstack.api.Client
¶
High-level API client for interacting with the dstack
server
Attributes:
Name | Type | Description |
---|---|---|
project |
str
|
The project name. |
runs |
RunCollection
|
Operations with runs. |
repos |
RepoCollection
|
Operations with repositories. |
backends |
BackendCollection
|
Operations with backends. |
client |
APIClient
|
Low-level API client that supports all API endpoints. |
from_config(project_name=None, server_url=None, user_token=None, ssh_identity_file=None)
staticmethod
¶
Creates a Client using the default configuration from ~/.dstack/config.yml
if it exists.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
project_name
|
Optional[str]
|
The name of the project. required if |
None
|
server_url
|
Optional[str]
|
The dstack server URL (e.g. |
None
|
user_token
|
Optional[str]
|
The dstack user token. |
None
|
ssh_identity_file
|
Optional[PathLike]
|
The private SSH key path for SSH tunneling. |
None
|
Returns:
Type | Description |
---|---|
Client
|
A client instance. |
dstack.api.RunCollection
¶
Operations with runs.
apply_configuration(configuration, repo=None, profile=None, configuration_path=None, reserve_ports=True)
¶
Apply the run configuration. Use this method to apply configurations without getting a run plan first.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
configuration
|
Union[Task, Service, DevEnvironment]
|
The run configuration. |
required |
repo
|
Union[LocalRepo, RemoteRepo, VirtualRepo, None]
|
The repo to use for the run. Pass |
None
|
profile
|
Optional[Profile]
|
The profile to use for the run. |
None
|
configuration_path
|
Optional[str]
|
The path to the configuration file. Omit if the configuration is not loaded from a file. |
None
|
reserve_ports
|
bool
|
Reserve local ports before applying. Use if you'll attach to the run. |
True
|
Returns:
Type | Description |
---|---|
Run
|
Submitted run. |
apply_plan(run_plan, repo=None, reserve_ports=True)
¶
Apply the run plan.
Use this method to apply run plans returned by get_run_plan
.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
run_plan
|
RunPlan
|
The result of |
required |
repo
|
Union[LocalRepo, RemoteRepo, VirtualRepo, None]
|
The repo to use for the run. Should be the same repo that is passed to |
None
|
reserve_ports
|
bool
|
Reserve local ports before applying. Use if you'll attach to the run. |
True
|
Returns:
Type | Description |
---|---|
Run
|
Submitted run. |
get(run_name)
¶
Get run by run name.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
run_name
|
str
|
Run name. |
required |
Returns:
Type | Description |
---|---|
Optional[Run]
|
The run or |
get_run_plan(configuration, repo=None, profile=None, configuration_path=None)
¶
Get a run plan. Use this method to see the run plan before applying the cofiguration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
configuration
|
Union[Task, Service, DevEnvironment]
|
The run configuration. |
required |
repo
|
Union[LocalRepo, RemoteRepo, VirtualRepo, None]
|
The repo to use for the run. Pass |
None
|
profile
|
Optional[Profile]
|
The profile to use for the run. |
None
|
configuration_path
|
Optional[str]
|
The path to the configuration file. Omit if the configuration is not loaded from a file. |
None
|
Returns:
Type | Description |
---|---|
RunPlan
|
Run plan. |
list(all=False)
¶
List runs.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
all
|
bool
|
Show all runs (active and finished) if |
False
|
Returns:
Type | Description |
---|---|
List[Run]
|
List of runs. |
dstack.api.RepoCollection
¶
Operations with repos
init(repo, git_identity_file=None, oauth_token=None)
¶
Initializes the repo and configures its credentials in the project. Must be invoked before mounting the repo to a run.
Example:
repo=RemoteRepo.from_url(
repo_url="https://github.com/dstackai/dstack-examples",
repo_branch="main",
)
client.repos.init(repo)
By default, it uses the default Git credentials configured on the machine.
You can override these credentials via the git_identity_file
or oauth_token
arguments of the init
method.
Once the repo is initialized, you can pass the repo object to the run:
run = client.runs.apply_configuration(
configuration=...,
repo=repo,
)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo
|
Repo
|
The repo to initialize. |
required |
git_identity_file
|
Optional[PathLike]
|
The private SSH key path for accessing the remote repo. |
None
|
oauth_token
|
Optional[str]
|
The GitHub OAuth token to access the remote repo. |
None
|
is_initialized(repo)
¶
Checks if the remote repo is initialized in the project
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo
|
Repo
|
The repo to check. |
required |
Returns:
Type | Description |
---|---|
bool
|
Whether the repo is initialized or not. |
load(repo_dir, local=False, init=False, git_identity_file=None, oauth_token=None)
¶
Loads the repo from the local directory using global config
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo_dir
|
PathLike
|
Repo root directory. |
required |
local
|
bool
|
Do not try to load |
False
|
init
|
bool
|
Initialize the repo if it's not initialized. |
False
|
git_identity_file
|
Optional[PathLike]
|
Path to an SSH private key to access the remote repo. |
None
|
oauth_token
|
Optional[str]
|
GitHub OAuth token to access the remote repo. |
None
|
Raises:
Type | Description |
---|---|
ConfigurationError
|
If the repo is not initialized and |
Returns:
Name | Type | Description |
---|---|---|
repo |
Union[RemoteRepo, LocalRepo]
|
Initialized repo. |
dstack.api.Task
¶
nodes
- (Optional) Number of nodes. Defaults to 1
.¶
name
- (Optional) The run name. If not specified, a random name is generated.¶
image
- (Optional) The name of the Docker image to run.¶
user
- (Optional) The user inside the container, user_name_or_id[:group_name_or_id]
(e.g., ubuntu
, 1000:1000
). Defaults to the default image
user.¶
privileged
- (Optional) Run the container in privileged mode.¶
entrypoint
- (Optional) The Docker entrypoint.¶
working_dir
- (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow
) and should be inside it. Defaults to "."
.¶
registry_auth
- (Optional) Credentials for pulling a private Docker image.¶
python
- (Optional) The major version of Python. Mutually exclusive with image
.¶
nvcc
- (Optional) Use image with NVIDIA CUDA Compiler (NVCC) included. Mutually exclusive with image
.¶
single_branch
- (Optional) Whether to clone and track only the current branch or all remote branches. Relevant only when using remote Git repos. Defaults to false
for dev environments and to true
for tasks and services.¶
env
- (Optional) The mapping or the list of environment variables.¶
resources
- (Optional) The resources requirements to run the configuration.¶
volumes
- (Optional) The volumes mount points.¶
ports
- (Optional) Port numbers/mapping to expose.¶
commands
- (Optional) The bash commands to run.¶
backends
- (Optional) The backends to consider for provisioning (e.g., [aws, gcp]
).¶
regions
- (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]
).¶
availability_zones
- (Optional) The availability zones to consider for provisioning (e.g., [eu-west-1a, us-west4-a]
).¶
instance_types
- (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]
).¶
reservation
- (Optional) The existing reservation to use for instance provisioning. Supports AWS Capacity Reservations and Capacity Blocks.¶
spot_policy
- (Optional) The policy for provisioning spot or on-demand instances: spot
, on-demand
, or auto
. Defaults to on-demand
.¶
retry
- (Optional) The policy for resubmitting the run. Defaults to false
.¶
max_duration
- (Optional) The maximum duration of a run (e.g., 2h
, 1d
, etc). After it elapses, the run is automatically stopped. Use off
for unlimited duration. Defaults to off
.¶
stop_duration
- (Optional) The maximum duration of a run graceful stopping. After it elapses, the run is automatically forced stopped. This includes force detaching volumes used by the run. Use off
for unlimited duration. Defaults to 5m
.¶
max_price
- (Optional) The maximum instance price per hour, in dollars.¶
creation_policy
- (Optional) The policy for using instances from fleets. Defaults to reuse-or-create
.¶
idle_duration
- (Optional) Time to wait before terminating idle instances. Defaults to 5m
for runs and 3d
for fleets. Use off
for unlimited duration.¶
utilization_policy
- (Optional) Run termination policy based on utilization.¶
dstack.api.Service
¶
port
- The port, that application listens on or the mapping.¶
gateway
- (Optional) The name of the gateway. Specify boolean false
to run without a gateway. Omit to run with the default gateway.¶
strip_prefix
- (Optional) Strip the /proxy/services/<project name>/<run name>/
path prefix when forwarding requests to the service. Only takes effect when running the service without a gateway. Defaults to True
.¶
model
- (Optional) Mapping of the model for the OpenAI-compatible endpoint provided by dstack
. Can be a full model format definition or just a model name. If it's a name, the service is expected to expose an OpenAI-compatible API at the /v1
path.¶
https
- (Optional) Enable HTTPS if running with a gateway. Defaults to True
.¶
auth
- (Optional) Enable the authorization. Defaults to True
.¶
replicas
- (Optional) The number of replicas. Can be a number (e.g. 2
) or a range (0..4
or 1..8
). If it's a range, the scaling
property is required. Defaults to 1
.¶
scaling
- (Optional) The auto-scaling rules. Required if replicas
is set to a range.¶
name
- (Optional) The run name. If not specified, a random name is generated.¶
image
- (Optional) The name of the Docker image to run.¶
user
- (Optional) The user inside the container, user_name_or_id[:group_name_or_id]
(e.g., ubuntu
, 1000:1000
). Defaults to the default image
user.¶
privileged
- (Optional) Run the container in privileged mode.¶
entrypoint
- (Optional) The Docker entrypoint.¶
working_dir
- (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow
) and should be inside it. Defaults to "."
.¶
registry_auth
- (Optional) Credentials for pulling a private Docker image.¶
python
- (Optional) The major version of Python. Mutually exclusive with image
.¶
nvcc
- (Optional) Use image with NVIDIA CUDA Compiler (NVCC) included. Mutually exclusive with image
.¶
single_branch
- (Optional) Whether to clone and track only the current branch or all remote branches. Relevant only when using remote Git repos. Defaults to false
for dev environments and to true
for tasks and services.¶
env
- (Optional) The mapping or the list of environment variables.¶
resources
- (Optional) The resources requirements to run the configuration.¶
volumes
- (Optional) The volumes mount points.¶
commands
- (Optional) The bash commands to run.¶
backends
- (Optional) The backends to consider for provisioning (e.g., [aws, gcp]
).¶
regions
- (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]
).¶
availability_zones
- (Optional) The availability zones to consider for provisioning (e.g., [eu-west-1a, us-west4-a]
).¶
instance_types
- (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]
).¶
reservation
- (Optional) The existing reservation to use for instance provisioning. Supports AWS Capacity Reservations and Capacity Blocks.¶
spot_policy
- (Optional) The policy for provisioning spot or on-demand instances: spot
, on-demand
, or auto
. Defaults to on-demand
.¶
retry
- (Optional) The policy for resubmitting the run. Defaults to false
.¶
max_duration
- (Optional) The maximum duration of a run (e.g., 2h
, 1d
, etc). After it elapses, the run is automatically stopped. Use off
for unlimited duration. Defaults to off
.¶
stop_duration
- (Optional) The maximum duration of a run graceful stopping. After it elapses, the run is automatically forced stopped. This includes force detaching volumes used by the run. Use off
for unlimited duration. Defaults to 5m
.¶
max_price
- (Optional) The maximum instance price per hour, in dollars.¶
creation_policy
- (Optional) The policy for using instances from fleets. Defaults to reuse-or-create
.¶
idle_duration
- (Optional) Time to wait before terminating idle instances. Defaults to 5m
for runs and 3d
for fleets. Use off
for unlimited duration.¶
utilization_policy
- (Optional) Run termination policy based on utilization.¶
dstack.api.DevEnvironment
¶
ide
- The IDE to run. Supported values include vscode
and cursor
.¶
version
- (Optional) The version of the IDE.¶
init
- (Optional) The bash commands to run on startup.¶
inactivity_duration
- (Optional) The maximum amount of time the dev environment can be inactive (e.g., 2h
, 1d
, etc). After it elapses, the dev environment is automatically stopped. Inactivity is defined as the absence of SSH connections to the dev environment, including VS Code connections, ssh <run name>
shells, and attached dstack apply
or dstack attach
commands. Use off
for unlimited duration. Can be updated in-place. Defaults to off
.¶
name
- (Optional) The run name. If not specified, a random name is generated.¶
image
- (Optional) The name of the Docker image to run.¶
user
- (Optional) The user inside the container, user_name_or_id[:group_name_or_id]
(e.g., ubuntu
, 1000:1000
). Defaults to the default image
user.¶
privileged
- (Optional) Run the container in privileged mode.¶
entrypoint
- (Optional) The Docker entrypoint.¶
working_dir
- (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow
) and should be inside it. Defaults to "."
.¶
registry_auth
- (Optional) Credentials for pulling a private Docker image.¶
python
- (Optional) The major version of Python. Mutually exclusive with image
.¶
nvcc
- (Optional) Use image with NVIDIA CUDA Compiler (NVCC) included. Mutually exclusive with image
.¶
single_branch
- (Optional) Whether to clone and track only the current branch or all remote branches. Relevant only when using remote Git repos. Defaults to false
for dev environments and to true
for tasks and services.¶
env
- (Optional) The mapping or the list of environment variables.¶
resources
- (Optional) The resources requirements to run the configuration.¶
volumes
- (Optional) The volumes mount points.¶
ports
- (Optional) Port numbers/mapping to expose.¶
backends
- (Optional) The backends to consider for provisioning (e.g., [aws, gcp]
).¶
regions
- (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]
).¶
availability_zones
- (Optional) The availability zones to consider for provisioning (e.g., [eu-west-1a, us-west4-a]
).¶
instance_types
- (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]
).¶
reservation
- (Optional) The existing reservation to use for instance provisioning. Supports AWS Capacity Reservations and Capacity Blocks.¶
spot_policy
- (Optional) The policy for provisioning spot or on-demand instances: spot
, on-demand
, or auto
. Defaults to on-demand
.¶
retry
- (Optional) The policy for resubmitting the run. Defaults to false
.¶
max_duration
- (Optional) The maximum duration of a run (e.g., 2h
, 1d
, etc). After it elapses, the run is automatically stopped. Use off
for unlimited duration. Defaults to off
.¶
stop_duration
- (Optional) The maximum duration of a run graceful stopping. After it elapses, the run is automatically forced stopped. This includes force detaching volumes used by the run. Use off
for unlimited duration. Defaults to 5m
.¶
max_price
- (Optional) The maximum instance price per hour, in dollars.¶
creation_policy
- (Optional) The policy for using instances from fleets. Defaults to reuse-or-create
.¶
idle_duration
- (Optional) Time to wait before terminating idle instances. Defaults to 5m
for runs and 3d
for fleets. Use off
for unlimited duration.¶
utilization_policy
- (Optional) Run termination policy based on utilization.¶
dstack.api.Run
¶
Attributes:
Name | Type | Description |
---|---|---|
name |
str
|
run name |
ports |
Optional[Dict[int, int]]
|
ports mapping, if run is attached |
backend |
Optional[BackendType]
|
backend type |
status |
RunStatus
|
run status |
hostname |
str
|
instance hostname |
attach(ssh_identity_file=None, bind_address=None, ports_overrides=None, replica_num=0, job_num=0)
¶
Establish an SSH tunnel to the instance and update SSH config
Parameters:
Name | Type | Description | Default |
---|---|---|---|
ssh_identity_file
|
Optional[PathLike]
|
SSH keypair to access instances. |
None
|
Raises:
Type | Description |
---|---|
PortUsedError
|
If ports are in use or the run is attached by another process. |
detach()
¶
Stop the SSH tunnel to the instance and update SSH config
logs(start_time=None, diagnose=False, replica_num=0, job_num=0)
¶
Iterate through run's log messages.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
start_time
|
Optional[datetime]
|
Minimal log timestamp. |
None
|
diagnose
|
bool
|
Return runner logs if |
False
|
Yields:
Type | Description |
---|---|
Iterable[bytes]
|
Log messages. |
refresh()
¶
Get up-to-date run info.
stop(abort=False)
¶
Terminate the instance and detach.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
abort
|
bool
|
Gracefully stop the run if |
False
|
dstack.api.Resources
¶
cpu
- (Optional) The number of CPU cores. Defaults to 2..
.¶
memory
- (Optional) The RAM size (e.g., 8GB
). Defaults to 8GB..
.¶
shm_size
- (Optional) The size of shared memory (e.g., 8GB
). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.¶
gpu
- (Optional) The GPU requirements.¶
disk
- (Optional) The disk resources.¶
dstack.api.GPU
¶
vendor
- (Optional) The vendor of the GPU/accelerator, one of: nvidia
, amd
, google
(alias: tpu
), intel
.¶
name
- (Optional) The name of the GPU (e.g., A100
or H100
).¶
count
- (Optional) The number of GPUs. Defaults to 1
.¶
memory
- (Optional) The RAM size (e.g., 16GB
). Can be set to a range (e.g. 16GB..
, or 16GB..80GB
).¶
total_memory
- (Optional) The total RAM size (e.g., 32GB
). Can be set to a range (e.g. 16GB..
, or 16GB..80GB
).¶
compute_capability
- (Optional) The minimum compute capability of the GPU (e.g., 7.5
).¶
dstack.api.Disk
¶
size
- Disk size.¶
dstack.api.LocalRepo
¶
Creates an instance of a local repo from a local path.
Example:
run = client.runs.apply_configuration(
configuration=...,
repo=LocalRepo.from_dir("."), # Mount the current folder to the run
)
from_dir(repo_dir)
staticmethod
¶
Creates an instance of a local repo from a local path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo_dir
|
PathLike
|
The path to a local folder. |
required |
Returns:
Type | Description |
---|---|
LocalRepo
|
A local repo instance. |
dstack.api.RemoteRepo
¶
Creates an instance of a remote Git repo for mounting to a submitted run.
Using a locally checked-out remote Git repo:
repo=RemoteRepo.from_dir(repo_dir=".")
Using a remote Git repo by a URL:
repo=RemoteRepo.from_url(
repo_url="https://github.com/dstackai/dstack-examples",
repo_branch="main"
)
Initialize the repo before mounting it.
client.repos.init(repo)
By default, it uses the default Git credentials configured on the machine.
You can override these credentials via the git_identity_file
or oauth_token
arguments of the init
method.
Finally, you can pass the repo object to the run:
run = client.runs.apply_configuration(
configuration=...,
repo=repo,
)
from_dir(repo_dir)
staticmethod
¶
Creates an instance of a remote repo from a local path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo_dir
|
PathLike
|
The path to a local folder. |
required |
Returns:
Type | Description |
---|---|
RemoteRepo
|
A remote repo instance. |
from_url(repo_url, repo_branch=None, repo_hash=None)
staticmethod
¶
Creates an instance of a remote repo from a URL.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo_url
|
str
|
The URL of a remote Git repo. |
required |
repo_branch
|
Optional[str]
|
The name of the remote branch. Must be specified if |
None
|
repo_hash
|
Optional[str]
|
The hash of the revision. Must be specified if |
None
|
Returns:
Type | Description |
---|---|
RemoteRepo
|
A remote repo instance. |
dstack.api.VirtualRepo
¶
Allows mounting a repo created programmatically.
Example:
virtual_repo = VirtualRepo(repo_id="some-unique-repo-id")
virtual_repo.add_file_from_package(package=some_package, path="requirements.txt")
virtual_repo.add_file_from_package(package=some_package, path="train.py")
run = client.runs.apply_configuration(
configuration=...,
repo=virtual_repo,
)
Attributes:
Name | Type | Description |
---|---|---|
repo_id |
A unique identifier of the repo |
add_file(path, content)
¶
Adds a given file to the repo.
Attributes:
Name | Type | Description |
---|---|---|
path |
str
|
The path inside the repo to add the file. |
content |
bytes
|
The contents of the file. |
add_file_from_package(package, path)
¶
Includes a file from a given package to the repo.
Attributes:
Name | Type | Description |
---|---|---|
package |
Union[ModuleType, str]
|
A package to include the file from. |
path |
str
|
The path to the file to include to the repo. Must be relative to the package directory. |
dstack.api.RegistryAuth
¶
username
- The username.¶
password
- The password or access token.¶
dstack.api.Scaling
¶
metric
- The target metric to track. Currently, the only supported value is rps
(meaning requests per second).¶
target
- The target value of the metric. The number of replicas is calculated based on this number and automatically adjusts (scales up or down) as this metric changes.¶
scale_up_delay
- (Optional) The delay in seconds before scaling up. Defaults to 300
.¶
scale_down_delay
- (Optional) The delay in seconds before scaling down. Defaults to 600
.¶
dstack.api.BackendType
¶
Attributes:
Name | Type | Description |
---|---|---|
AWS |
BackendType
|
Amazon Web Services |
AZURE |
BackendType
|
Microsoft Azure |
CUDO |
BackendType
|
Cudo |
DSTACK |
BackendType
|
dstack Sky |
GCP |
BackendType
|
Google Cloud Platform |
DATACRUNCH |
BackendType
|
DataCrunch |
KUBERNETES |
BackendType
|
Kubernetes |
LAMBDA |
BackendType
|
Lambda Cloud |
OCI |
BackendType
|
Oracle Cloud Infrastructure |
RUNPOD |
BackendType
|
Runpod Cloud |
TENSORDOCK |
BackendType
|
TensorDock Marketplace |
VASTAI |
BackendType
|
Vast.ai Marketplace |
VULTR |
BackendType
|
Vultr |