Skip to content

Python API

The Python API enables running tasks, services, and managing runs programmatically.

Usage example

Below is a quick example of submitting a task for running and displaying its logs.

import sys

from dstack.api import Task, GPU, Client, Resources

client = Client.from_config()

task = Task(
    image="ghcr.io/huggingface/text-generation-inference:latest",
    env={"MODEL_ID": "TheBloke/Llama-2-13B-chat-GPTQ"},
    commands=[
        "text-generation-launcher --trust-remote-code --quantize gptq",
    ],
    ports=["80"],
    resources=Resources(gpu=GPU(memory="24GB")),
)

run = client.runs.submit(
    run_name="my-awesome-run",  # If not specified, a random name is assigned 
    configuration=task,
    repo=None, # Specify to mount additional files
)

run.attach()

try:
    for log in run.logs():
        sys.stdout.buffer.write(log)
        sys.stdout.buffer.flush()
except KeyboardInterrupt:
    run.stop(abort=True)
finally:
    run.detach()

NOTE:

  1. The configuration argument in the submit method can be either dstack.api.Task or dstack.api.Service.
  2. If you create dstack.api.Task or dstack.api.Service, you may specify the image argument. If image isn't specified, the default image will be used. For a private Docker registry, ensure you also pass the registry_auth argument.
  3. The repo argument in the submit method allows the mounting of a local folder, a remote repo, or a programmatically created repo. In this case, the commands argument can refer to the files within this repo.
  4. The attach method waits for the run to start and, for dstack.api.Task sets up an SSH tunnel and forwards configured ports to localhost.

dstack.api

dstack.api.Client

High-level API client for interacting with dstack server

Attributes:

Name Type Description
runs RunCollection

Operations with runs.
repos RepoCollection

Operations with repositories.
backends BackendCollection

Operations with backends.

from_config(project_name=None, server_url=None, user_token=None, ssh_identity_file=None) staticmethod

Creates a Client using the default configuration from ~/.dstack/config.yml if it exists.

Parameters:

Name Type Description Default
project_name Optional[str]

The name of the project, required if server_url and user_token are specified

 None
server_url Optional[str]

The dstack server URL (e.g. http://localhost:3000/ or https://sky.dstack.ai)

 None
user_token Optional[str]

The dstack user token

 None
ssh_identity_file Optional[PathLike]

The private SSH key path for SSH tunneling

 None

Returns:

Type Description
Client

A client instance

dstack.api.RunCollection

Operations with runs

get(run_name)

Get run by run name

Parameters:

Name Type Description Default
run_name str

run name

 required

Returns:

Type Description
Optional[Run]

The run or None if not found

list(all=False)

List runs

Parameters:

Name Type Description Default
all bool

show all runs (active and finished) if True

 False

Returns:

Type Description
List[Run]

list of runs

submit(configuration, configuration_path=None, repo=None, backends=None, regions=None, instance_types=None, resources=None, spot_policy=None, retry_policy=None, max_duration=None, max_price=None, working_dir=None, run_name=None, reserve_ports=True)

Submit a run

Parameters:

Name Type Description Default
configuration Union[Task, Service]

A run configuration.

 required
configuration_path Optional[str]

The path to the configuration file, relative to the root directory of the repo.

 None
repo Union[LocalRepo, RemoteRepo, VirtualRepo]

A repo to mount to the run.

 None
backends Optional[List[BackendType]]

A list of allowed backend for provisioning.

 None
regions Optional[List[str]]

A list of cloud regions for provisioning.

 None
resources Optional[ResourcesSpec]

The requirements to run the configuration. Overrides the configuration's resources.

 None
spot_policy Optional[SpotPolicy]

A spot policy for provisioning.

 None
retry_policy RetryPolicy

A retry policy.

 None
max_duration Optional[Union[int, str]]

The max instance running duration in seconds.

 None
max_price Optional[float]

The max instance price in dollars per hour for provisioning.

 None
working_dir Optional[str]

A working directory relative to the repo root directory

 None
run_name Optional[str]

A desired name of the run. Must be unique in the project. If not specified, a random name is assigned.

 None
reserve_ports bool

Whether local ports should be reserved in advance.

 True

Returns:

Type Description
Run

submitted run

dstack.api.RepoCollection

Operations with repos

init(repo, git_identity_file=None, oauth_token=None)

Initializes the repo and configures its credentials in the project. Must be invoked before mounting the repo to a run.

Example:

repo=RemoteRepo.from_url(
    repo_url="https://github.com/dstackai/dstack-examples",
    repo_branch="main",
)
client.repos.init(repo)

By default, it uses the default Git credentials configured on the machine. You can override these credentials via the git_identity_file or oauth_token arguments of the init method.

Once the repo is initialized, you can pass the repo object to the run:

run = client.runs.submit(
    configuration=...,
    repo=repo,
)

Parameters:

Name Type Description Default
repo Repo

The repo to initialize.

 required
git_identity_file Optional[PathLike]

The private SSH key path for accessing the remote repo.

 None
oauth_token Optional[str]

The GitHub OAuth token to access the remote repo.

 None

dstack.api.Task

nodes - (Optional) Number of nodes. Defaults to 1.

name - (Optional) The run name.

image - (Optional) The name of the Docker image to run.

privileged - (Optional) Run the container in privileged mode.

entrypoint - (Optional) The Docker entrypoint.

working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to "." .

home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.

registry_auth - (Optional) Credentials for pulling a private Docker image.

python - (Optional) The major version of Python. Mutually exclusive with image.

nvcc - (Optional) Use image with NVIDIA CUDA Compiler (NVCC) included. Mutually exclusive with image.

env - (Optional) The mapping or the list of environment variables.

setup - (Optional) The bash commands to run on the boot.

resources - (Optional) The resources requirements to run the configuration.

volumes - (Optional) The volumes mount points.

ports - (Optional) Port numbers/mapping to expose.

commands - (Optional) The bash commands to run.

backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).

regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).

instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).

spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto. Defaults to on-demand.

retry - (Optional) The policy for resubmitting the run. Defaults to false.

retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.

max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.

max_price - (Optional) The maximum instance price per hour, in dollars.

pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.

instance_name - (Optional) The name of the instance.

creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.

termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.

termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.

dstack.api.Service

port - The port, that application listens on or the mapping.

gateway - (Optional) The name of the gateway. Specify boolean false to run without a gateway. Omit to run with the default gateway.

model - (Optional) Mapping of the model for the OpenAI-compatible endpoint provided by dstack. Can be a full model format definition or just a model name. If it's a name, the service is expected to expose an OpenAI-compatible API at the /v1 path.

https - (Optional) Enable HTTPS if running with a gateway. Defaults to True.

auth - (Optional) Enable the authorization. Defaults to True.

replicas - (Optional) The number of replicas. Can be a number (e.g. 2) or a range (0..4 or 1..8). If it's a range, the scaling property is required. Defaults to 1.

scaling - (Optional) The auto-scaling rules. Required if replicas is set to a range.

name - (Optional) The run name.

image - (Optional) The name of the Docker image to run.

privileged - (Optional) Run the container in privileged mode.

entrypoint - (Optional) The Docker entrypoint.

working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to "." .

home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.

registry_auth - (Optional) Credentials for pulling a private Docker image.

python - (Optional) The major version of Python. Mutually exclusive with image.

nvcc - (Optional) Use image with NVIDIA CUDA Compiler (NVCC) included. Mutually exclusive with image.

env - (Optional) The mapping or the list of environment variables.

setup - (Optional) The bash commands to run on the boot.

resources - (Optional) The resources requirements to run the configuration.

volumes - (Optional) The volumes mount points.

commands - (Optional) The bash commands to run.

backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).

regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).

instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).

spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto. Defaults to on-demand.

retry - (Optional) The policy for resubmitting the run. Defaults to false.

retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.

max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.

max_price - (Optional) The maximum instance price per hour, in dollars.

pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.

instance_name - (Optional) The name of the instance.

creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.

termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.

termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.

dstack.api.Run

Attributes:

Name Type Description
name str

run name
ports Optional[Dict[int, int]]

ports mapping, if run is attached
backend Optional[BackendType]

backend type
status RunStatus

run status
hostname str

instance hostname

attach(ssh_identity_file=None, bind_address=None, ports_overrides=None, replica_num=0, job_num=0)

Establish an SSH tunnel to the instance and update SSH config

Parameters:

Name Type Description Default
ssh_identity_file Optional[PathLike]

SSH keypair to access instances

 None

Raises:

Type Description
PortUsedError

If ports are in use or the run is attached by another process.

detach()

Stop the SSH tunnel to the instance and update SSH config

logs(start_time=None, diagnose=False, replica_num=0, job_num=0)

Iterate through run's log messages

Parameters:

Name Type Description Default
start_time Optional[datetime]

minimal log timestamp

 None
diagnose bool

return runner logs if True

 False

Yields:

Type Description
Iterable[bytes]

log messages

refresh()

Get up-to-date run info

stop(abort=False)

Terminate the instance and detach

Parameters:

Name Type Description Default
abort bool

gracefully stop the run if False

 False

dstack.api.Resources

cpu - (Optional) The number of CPU cores. Defaults to 2...

memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...

shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.

gpu - (Optional) The GPU requirements.

disk - (Optional) The disk resources.

dstack.api.GPU

vendor - (Optional) The vendor of the GPU/accelerator, one of: nvidia, amd, google (alias: tpu).

name - (Optional) The name of the GPU (e.g., A100 or H100).

count - (Optional) The number of GPUs. Defaults to 1.

memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).

total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).

compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).

dstack.api.Disk

size - Disk size.

dstack.api.LocalRepo

Creates an instance of a local repo from a local path.

Example:

run = client.runs.submit(
    configuration=...,
    repo=LocalRepo.from_dir("."), # Mount the current folder to the run
)

from_dir(repo_dir) staticmethod

Creates an instance of a local repo from a local path.

Parameters:

Name Type Description Default
repo_dir PathLike

The path to a local folder

 required

Returns:

Type Description
LocalRepo

A local repo instance

dstack.api.RemoteRepo

Creates an instance of a remote Git repo for mounting to a submitted run.

Using a locally checked-out remote Git repo:

repo=RemoteRepo.from_dir(repo_dir=".")

Using a remote Git repo by a URL:

repo=RemoteRepo.from_url(
    repo_url="https://github.com/dstackai/dstack-examples",
    repo_branch="main"
)

Initialize the repo before mounting it.

client.repos.init(repo)

By default, it uses the default Git credentials configured on the machine. You can override these credentials via the git_identity_file or oauth_token arguments of the init method.

Finally, you can pass the repo object to the run:

run = client.runs.submit(
    configuration=...,
    repo=repo,
)

from_dir(repo_dir) staticmethod

Creates an instance of a remote repo from a local path.

Parameters:

Name Type Description Default
repo_dir PathLike

The path to a local folder

 required

Returns:

Type Description
RemoteRepo

A remote repo instance

from_url(repo_url, repo_branch=None, repo_hash=None) staticmethod

Creates an instance of a remote repo from a URL.

Parameters:

Name Type Description Default
repo_url str

The URL of a remote Git repo

 required
repo_branch Optional[str]

The name of the remote branch. Must be specified if hash is not specified.

 None
repo_hash Optional[str]

The hash of the revision. Must be specified if branch is not specified.

 None

Returns:

Type Description
RemoteRepo

A remote repo instance

dstack.api.VirtualRepo

Allows mounting a repo created programmatically.

Example:

virtual_repo = VirtualRepo(repo_id="some-unique-repo-id")
virtual_repo.add_file_from_package(package=some_package, path="requirements.txt")
virtual_repo.add_file_from_package(package=some_package, path="train.py")

run = client.runs.submit(
    configuration=...,
    repo=virtual_repo,
)

Attributes:

Name Type Description
repo_id

A unique identifier of the repo

add_file(path, content)

Adds a given file to the repo.

Attributes:

Name Type Description
path str

The path inside the repo to add the file.
content bytes

The contents of the file.

add_file_from_package(package, path)

Includes a file from a given package to the repo.

Attributes:

Name Type Description
package Union[ModuleType, str]

A package to include the file from.
path str

The path to the file to include to the repo. Must be relative to the package directory.

dstack.api.RegistryAuth

username - The username.

password - The password or access token.

dstack.api.Scaling

metric - The target metric to track. Currently, the only supported value is rps (meaning requests per second).

target - The target value of the metric. The number of replicas is calculated based on this number and automatically adjusts (scales up or down) as this metric changes.

scale_up_delay - (Optional) The delay in seconds before scaling up. Defaults to 300.

scale_down_delay - (Optional) The delay in seconds before scaling down. Defaults to 600.

dstack.api.BackendType

Attributes:

Name Type Description
AWS BackendType

Amazon Web Services
AZURE BackendType

Microsoft Azure
CUDO BackendType

Cudo
DSTACK BackendType

dstack Sky
GCP BackendType

Google Cloud Platform
DATACRUNCH BackendType

DataCrunch
KUBERNETES BackendType

Kubernetes
LAMBDA BackendType

Lambda Cloud
RUNPOD BackendType

Runpod Cloud
TENSORDOCK BackendType

TensorDock Marketplace
VASTAI BackendType

Vast.ai Marketplace