VastAI.create_endpoint

Create a new serverless endpoint.

Signature

VastAI.create_endpoint(
    min_load: Optional[float] = 0.0,
    min_cold_load: Optional[float] = 0.0,
    target_util: Optional[float] = 0.9,
    cold_mult: Optional[float] = 2.5,
    cold_workers: Optional[int] = 5,
    max_workers: Optional[int] = 20,
    endpoint_name: Optional[str] = None,
    max_queue_time: Optional[float] = None,
    target_queue_time: Optional[float] = None,
    inactivity_timeout: Optional[int] = None
) -> dict

Parameters

min_load · Optional[float] · default: "0.0"

minimum floor load in perf units/s (token/s for LLms)

min_cold_load · Optional[float] · default: "0.0"

minimum floor load in perf units/s (token/s for LLms), but allow handling with cold workers

target_util · Optional[float] · default: "0.9"

target capacity utilization (fraction, max 1.0, default 0.9)

cold_mult · Optional[float] · default: "2.5"

cold/stopped instance capacity target as multiple of hot capacity target (default 2.5)

cold_workers · Optional[int] · default: "5"

min number of workers to keep ‘cold’ when you have no load (default 5)

max_workers · Optional[int] · default: "20"

max number of workers your endpoint group can have (default 20)

endpoint_name · Optional[str]

deployment endpoint name (allows multiple autoscale groups to share same deployment endpoint)

max_queue_time · Optional[float]

maximum seconds requests may be queued on each worker (default 30.0)

target_queue_time · Optional[float]

target seconds for the queue to be cleared (default 10.0)

inactivity_timeout · Optional[int]

seconds of no traffic before the endpoint can scale to zero active workers

Returns

dict

Example

from vastai import VastAI

client = VastAI(api_key="YOUR_API_KEY")
result = client.create_endpoint()
print(result)