endpoint¶
The endpoint configuration type describes a model request and the constraints
used to create or apply an endpoint preset.
Root reference¶
backends - (Optional) list["amddevcloud" | "aws" | "azure" | "cloudrift" | "crusoe" | "cudo" | "datacrunch" | "digitalocean" | "dstack" | "gcp" | "hotaisle" | "jarvislabs" | "kubernetes" | "lambda" | "remote" | "nebius" | "oci" | "runpod" | "tensordock" | "vastai" | "verda" | "vultr" | "slurm"] The backends to consider for provisioning (e.g., [aws, gcp]).¶
regions - (Optional) list[str] The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).¶
availability_zones - (Optional) list[str] The availability zones to consider for provisioning (e.g., [eu-west-1a, us-west4-a]).¶
instance_types - (Optional) list[str] The cloud-specific instance types to consider for provisioning (e.g., [g6e.24xlarge, n1-standard-4]).¶
reservation - (Optional) str The existing reservation to use for instance provisioning. Supports AWS Capacity Reservations, AWS Capacity Blocks, and GCP reservations.¶
spot_policy - (Optional) "auto" | "on-demand" | "spot" The policy for provisioning spot or on-demand instances: spot, on-demand, auto. Defaults to on-demand.¶
retry - (Optional) bool | object The policy for resubmitting the run. Defaults to false.¶
max_duration - (Optional) int | str | "off" The maximum duration of a run (e.g., 2h, 1d, etc) in a running state, excluding provisioning and pulling. After it elapses, the run is automatically stopped. Use off for unlimited duration. Defaults to off.¶
stop_duration - (Optional) int | str | "off" The maximum duration of a run graceful stopping. After it elapses, the run is automatically forced stopped. This includes force detaching volumes used by the run. Use off for unlimited duration. Defaults to 5m.¶
max_price - (Optional) float The maximum instance price per hour, in dollars.¶
creation_policy - (Optional) "reuse" | "reuse-or-create" The policy for using instances from fleets: reuse, reuse-or-create. Defaults to reuse-or-create.¶
idle_duration - (Optional) int | str Time to wait before terminating idle instances. When the run reuses an existing fleet instance, the fleet's idle_duration applies. When the run provisions a new instance, the shorter of the fleet's and run's values is used. Defaults to 5m for runs and 3d for fleets. Use off for unlimited duration. Only applied for VM-based backends.¶
utilization_policy - (Optional) object Run termination policy based on utilization.¶
startup_order - (Optional) "any" | "master-first" | "workers-first" The order in which master and workers jobs are started: any, master-first, workers-first. Defaults to any.¶
stop_criteria - (Optional) "all-done" | "master-done" The criteria determining when a multi-node run should be considered finished: all-done, master-done. Defaults to all-done.¶
schedule - (Optional) object The schedule for starting the run at specified time.¶
fleets - (Optional) list[str | object] The fleets considered for reuse. For fleets owned by the current project, specify fleet names. For imported fleets, specify <project name>/<fleet name>.¶
instances - (Optional) list The specific fleet instances to consider for reuse. Each value can be an instance name string, or an object with name, hostname, or fleet and instance. When set, the run is only placed on matching existing instances..¶
tags - (Optional) dict The custom tags to associate with the resource. The tags are also propagated to the underlying backend resources. If there is a conflict with backend-level tags, does not override them.¶
backend_options - (Optional) list[object] Backend-specific options, applied only to offers from that backend.¶
type - (Required) "endpoint" The configuration type. Must be endpoint.¶
name - (Optional) str The endpoint name. Required unless passed with --name.¶
model - (Required) str | object The model to serve. Use a string or repo for an exact repo/path, or base to allow compatible model variants..¶
context_length - (Optional) int The minimum required context length.¶
preset - (Optional) str The preset ID to use when applying the endpoint.¶
gateway - (Optional) bool | str | object The name of the gateway. Specify boolean false to run without a gateway. Specify boolean true to run with the default gateway. Omit to run with the default gateway if there is one, or without a gateway otherwise.¶
env - (Optional) list[str] | dict The mapping or the list of environment variables.¶
model¶
Allows the creation agent to select a compatible model variant.
base - (Required) str The base model for which the agent may select a compatible variant.¶
retry¶
on_events - (Optional) list["no-capacity" | "interruption" | "error"] The list of events that should be handled with retry. Supported events are no-capacity, interruption, error. Omit to retry on all events.¶
duration - (Optional) int | str The maximum period of retrying the run, e.g., 4h or 1d. The period is calculated as a run age for no-capacity event and as a time passed since the last interruption and error for interruption and error events..¶
utilization_policy¶
min_gpu_utilization - (Required) int Minimum required GPU utilization, percent. If any GPU has utilization below specified value during the whole time window, the run is terminated.¶
time_window - (Required) int | str The time window of metric samples taking into account to measure utilization (e.g., 30m, 1h). Minimum is 5m.¶
schedule¶
cron - (Required) str | list[str] A cron expression or a list of cron expressions specifying the UTC time when the run needs to be started.¶
instances[n]¶
When instances is set, the run is placed only on matching existing fleet instances.
Short syntax
The short syntax for instances is an instance name string.
my-fleet-1, same as{name: my-fleet-1}
backend_options¶
Backend-specific options that only take effect for offers of the respective backend.