Python API¶
The Python API enables running tasks, services, and managing runs programmatically.
Usage example¶
Below is a quick example of submitting a task for running and displaying its logs.
import sys
from dstack.api import Task, GPU, Client, Resources
client = Client.from_config()
task = Task(
image="ghcr.io/huggingface/text-generation-inference:latest",
env={"MODEL_ID": "TheBloke/Llama-2-13B-chat-GPTQ"},
commands=[
"text-generation-launcher --trust-remote-code --quantize gptq",
],
ports=["80"],
resources=Resources(gpu=GPU(memory="24GB")),
)
run = client.runs.submit(
run_name="my-awesome-run", # If not specified, a random name is assigned
configuration=task,
repo=None, # Specify to mount additional files
)
run.attach()
try:
for log in run.logs():
sys.stdout.buffer.write(log)
sys.stdout.buffer.flush()
except KeyboardInterrupt:
run.stop(abort=True)
finally:
run.detach()
NOTE:
- The
configuration
argument in thesubmit
method can be eitherdstack.api.Task
ordstack.api.Service
. - If you create
dstack.api.Task
ordstack.api.Service
, you may specify theimage
argument. Ifimage
isn't specified, the default image will be used. For a private Docker registry, ensure you also pass theregistry_auth
argument. - The
repo
argument in thesubmit
method allows the mounting of a local folder, a remote repo, or a programmatically created repo. In this case, thecommands
argument can refer to the files within this repo. - The
attach
method waits for the run to start and, fordstack.api.Task
sets up an SSH tunnel and forwards configuredports
tolocalhost
.
dstack.api
¶
dstack.api.Client
¶
High-level API client for interacting with dstack server
Attributes:
Name | Type | Description |
---|---|---|
runs |
RunCollection
|
Operations with runs. |
repos |
RepoCollection
|
Operations with repositories. |
backends |
BackendCollection
|
Operations with backends. |
from_config(project_name=None, server_url=None, user_token=None, ssh_identity_file=None)
staticmethod
¶
Creates a Client using the default configuration from ~/.dstack/config.yml
if it exists.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
project_name |
Optional[str]
|
The name of the project, required if |
None
|
server_url |
Optional[str]
|
The dstack server URL (e.g. |
None
|
user_token |
Optional[str]
|
The dstack user token |
None
|
ssh_identity_file |
Optional[PathLike]
|
The private SSH key path for SSH tunneling |
None
|
Returns:
Type | Description |
---|---|
Client
|
A client instance |
dstack.api.RunCollection
¶
Operations with runs
get(run_name)
¶
Get run by run name
Parameters:
Name | Type | Description | Default |
---|---|---|---|
run_name |
str
|
run name |
required |
Returns:
Type | Description |
---|---|
Optional[Run]
|
The run or |
list(all=False)
¶
List runs
Parameters:
Name | Type | Description | Default |
---|---|---|---|
all |
bool
|
show all runs (active and finished) if |
False
|
Returns:
Type | Description |
---|---|
List[Run]
|
list of runs |
submit(configuration, configuration_path=None, repo=None, backends=None, regions=None, instance_types=None, resources=None, spot_policy=None, retry_policy=None, max_duration=None, max_price=None, working_dir=None, run_name=None, reserve_ports=True)
¶
Submit a run
Parameters:
Name | Type | Description | Default |
---|---|---|---|
configuration |
Union[Task, Service]
|
A run configuration. |
required |
configuration_path |
Optional[str]
|
The path to the configuration file, relative to the root directory of the repo. |
None
|
repo |
Union[LocalRepo, RemoteRepo, VirtualRepo]
|
A repo to mount to the run. |
None
|
backends |
Optional[List[BackendType]]
|
A list of allowed backend for provisioning. |
None
|
regions |
Optional[List[str]]
|
A list of cloud regions for provisioning. |
None
|
resources |
Optional[ResourcesSpec]
|
The requirements to run the configuration. Overrides the configuration's resources. |
None
|
spot_policy |
Optional[SpotPolicy]
|
A spot policy for provisioning. |
None
|
retry_policy |
RetryPolicy
|
A retry policy. |
None
|
max_duration |
Optional[Union[int, str]]
|
The max instance running duration in seconds. |
None
|
max_price |
Optional[float]
|
The max instance price in dollars per hour for provisioning. |
None
|
working_dir |
Optional[str]
|
A working directory relative to the repo root directory |
None
|
run_name |
Optional[str]
|
A desired name of the run. Must be unique in the project. If not specified, a random name is assigned. |
None
|
reserve_ports |
bool
|
Whether local ports should be reserved in advance. |
True
|
Returns:
Type | Description |
---|---|
Run
|
submitted run |
dstack.api.RepoCollection
¶
Operations with repos
init(repo, git_identity_file=None, oauth_token=None)
¶
Initializes the repo and configures its credentials in the project. Must be invoked before mounting the repo to a run.
Example:
repo=RemoteRepo.from_url(
repo_url="https://github.com/dstackai/dstack-examples",
repo_branch="main",
)
client.repos.init(repo)
By default, it uses the default Git credentials configured on the machine.
You can override these credentials via the git_identity_file
or oauth_token
arguments of the init
method.
Once the repo is initialized, you can pass the repo object to the run:
run = client.runs.submit(
configuration=...,
repo=repo,
)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo |
Repo
|
The repo to initialize. |
required |
git_identity_file |
Optional[PathLike]
|
The private SSH key path for accessing the remote repo. |
None
|
oauth_token |
Optional[str]
|
The GitHub OAuth token to access the remote repo. |
None
|
dstack.api.Task
¶
Attributes:
Name | Type | Description |
---|---|---|
commands |
List[str]
|
The bash commands to run |
ports |
List[PortMapping]
|
Port numbers/mapping to expose |
env |
Dict[str, str]
|
The mapping or the list of environment variables |
image |
Optional[str]
|
The name of the Docker image to run |
python |
Optional[str]
|
The major version of Python |
entrypoint |
Optional[str]
|
The Docker entrypoint |
registry_auth |
Optional[RegistryAuth]
|
Credentials for pulling a private Docker image |
home_dir |
str
|
The absolute path to the home directory inside the container. Defaults to |
resources |
Optional[ResourcesSpec]
|
The requirements to run the configuration. |
dstack.api.Service
¶
Attributes:
Name | Type | Description |
---|---|---|
commands |
List[str]
|
The bash commands to run |
port |
PortMapping
|
The port, that application listens to or the mapping |
env |
Dict[str, str]
|
The mapping or the list of environment variables |
image |
Optional[str]
|
The name of the Docker image to run |
python |
Optional[str]
|
The major version of Python |
entrypoint |
Optional[str]
|
The Docker entrypoint |
registry_auth |
Optional[RegistryAuth]
|
Credentials for pulling a private Docker image |
home_dir |
str
|
The absolute path to the home directory inside the container. Defaults to |
resources |
Optional[ResourcesSpec]
|
The requirements to run the configuration. |
model |
Optional[ModelMapping]
|
Mapping of the model for the OpenAI-compatible endpoint. |
auth |
bool
|
Enable the authorization. Defaults to |
replicas |
Range[int]
|
The range of the number of replicas. Defaults to |
scaling |
Range[int]
|
Optional[ScalingSpec]: The auto-scaling configuration. |
dstack.api.Run
¶
Attributes:
Name | Type | Description |
---|---|---|
name |
str
|
run name |
ports |
Optional[Dict[int, int]]
|
ports mapping, if run is attached |
backend |
Optional[BackendType]
|
backend type |
status |
RunStatus
|
run status |
hostname |
str
|
instance hostname |
attach(ssh_identity_file=None)
¶
Establish an SSH tunnel to the instance and update SSH config
Parameters:
Name | Type | Description | Default |
---|---|---|---|
ssh_identity_file |
Optional[PathLike]
|
SSH keypair to access instances |
None
|
Raises:
Type | Description |
---|---|
PortUsedError
|
If ports are in use or the run is attached by another process. |
detach()
¶
Stop the SSH tunnel to the instance and update SSH config
logs(start_time=None, diagnose=False, replica_num=0, job_num=0)
¶
Iterate through run's log messages
Parameters:
Name | Type | Description | Default |
---|---|---|---|
start_time |
Optional[datetime]
|
minimal log timestamp |
None
|
diagnose |
bool
|
return runner logs if |
False
|
Yields:
Type | Description |
---|---|
Iterable[bytes]
|
log messages |
refresh()
¶
Get up-to-date run info
stop(abort=False)
¶
Terminate the instance and detach
Parameters:
Name | Type | Description | Default |
---|---|---|---|
abort |
bool
|
gracefully stop the run if |
False
|
dstack.api.Resources
¶
Creates required resources specification.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
cpu |
Optional[Range[int]]
|
The number of CPUs |
DEFAULT_CPU_COUNT
|
memory |
Optional[Range[Memory]]
|
The size of RAM memory (e.g., |
DEFAULT_MEMORY_SIZE
|
gpu |
Optional[GPUSpec]
|
The GPU spec |
None
|
shm_size |
Optional[Range[Memory]]
|
The of shared memory (e.g., |
None
|
disk |
Optional[DiskSpec]
|
The disk spec |
None
|
Returns:
Type | Description |
---|---|
ResourcesSpec
|
resources specification |
dstack.api.GPU
¶
Creates GPU specification.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name |
Optional[List[str]]
|
The name of the GPU (e.g., |
None
|
count |
Optional[Range[int]]
|
The number of GPUs |
DEFAULT_GPU_COUNT
|
memory |
Optional[Range[Memory]]
|
The size of a single GPU memory (e.g., |
None
|
total_memory |
Optional[Range[Memory]]
|
The total size of all GPUs memory (e.g., |
None
|
compute_capability |
Optional[float]
|
The minimum compute capability of the GPU (e.g., |
None
|
Returns:
Type | Description |
---|---|
GPUSpec
|
GPU specification |
dstack.api.Disk
¶
Creates disk specification.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
size |
Range[Memory]
|
The size of the disk (e.g., |
required |
Returns:
Type | Description |
---|---|
DiskSpec
|
disk specification |
dstack.api.LocalRepo
¶
Creates an instance of a local repo from a local path.
Example:
run = client.runs.submit(
configuration=...,
repo=LocalRepo.from_dir("."), # Mount the current folder to the run
)
from_dir(repo_dir)
staticmethod
¶
Creates an instance of a local repo from a local path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo_dir |
PathLike
|
The path to a local folder |
required |
Returns:
Type | Description |
---|---|
LocalRepo
|
A local repo instance |
dstack.api.RemoteRepo
¶
Creates an instance of a remote Git repo for mounting to a submitted run.
Using a locally checked-out remote Git repo:
repo=RemoteRepo.from_dir(repo_dir=".")
Using a remote Git repo by a URL:
repo=RemoteRepo.from_url(
repo_url="https://github.com/dstackai/dstack-examples",
repo_branch="main"
)
Initialize the repo before mounting it.
client.repos.init(repo)
By default, it uses the default Git credentials configured on the machine.
You can override these credentials via the git_identity_file
or oauth_token
arguments of the init
method.
Finally, you can pass the repo object to the run:
run = client.runs.submit(
configuration=...,
repo=repo,
)
from_dir(repo_dir)
staticmethod
¶
Creates an instance of a remote repo from a local path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo_dir |
PathLike
|
The path to a local folder |
required |
Returns:
Type | Description |
---|---|
RemoteRepo
|
A remote repo instance |
from_url(repo_url, repo_branch=None, repo_hash=None)
staticmethod
¶
Creates an instance of a remote repo from a URL.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
repo_url |
str
|
The URL of a remote Git repo |
required |
repo_branch |
Optional[str]
|
The name of the remote branch. Must be specified if |
None
|
repo_hash |
Optional[str]
|
The hash of the revision. Must be specified if |
None
|
Returns:
Type | Description |
---|---|
RemoteRepo
|
A remote repo instance |
dstack.api.VirtualRepo
¶
Allows mounting a repo created programmatically.
Example:
virtual_repo = VirtualRepo(repo_id="some-unique-repo-id")
virtual_repo.add_file_from_package(package=some_package, path="requirements.txt")
virtual_repo.add_file_from_package(package=some_package, path="train.py")
run = client.runs.submit(
configuration=...,
repo=virtual_repo,
)
Attributes:
Name | Type | Description |
---|---|---|
repo_id |
A unique identifier of the repo |
add_file(path, content)
¶
Adds a given file to the repo.
Attributes:
Name | Type | Description |
---|---|---|
path |
str
|
The path inside the repo to add the file. |
content |
bytes
|
The contents of the file. |
add_file_from_package(package, path)
¶
Includes a file from a given package to the repo.
Attributes:
Name | Type | Description |
---|---|---|
package |
Union[ModuleType, str]
|
A package to include the file from. |
path |
str
|
The path to the file to include to the repo. Must be relative to the package directory. |
dstack.api.RegistryAuth
¶
Credentials for pulling a private Docker image.
Attributes:
Name | Type | Description |
---|---|---|
username |
str
|
The username |
password |
str
|
The password or access token |
dstack.api.BackendType
¶
Attributes:
Name | Type | Description |
---|---|---|
AWS |
BackendType
|
Amazon Web Services |
AZURE |
BackendType
|
Microsoft Azure |
CUDO |
BackendType
|
Cudo |
DSTACK |
BackendType
|
dstack Sky |
GCP |
BackendType
|
Google Cloud Platform |
DATACRUNCH |
BackendType
|
DataCrunch |
KUBERNETES |
BackendType
|
Kubernetes |
LAMBDA |
BackendType
|
Lambda Cloud |
RUNPOD |
BackendType
|
Runpod Cloud |
TENSORDOCK |
BackendType
|
TensorDock Marketplace |
VASTAI |
BackendType
|
Vast.ai Marketplace |