# Vast.ai Instance Management Guide

Complete reference for searching, creating, configuring, connecting to, and managing GPU instances on Vast.ai.

## Table of Contents
- [Authentication](#authentication)
- [Searching for Instances](#searching-for-instances)
- [Creating Instances](#creating-instances)
- [SSH Keys](#ssh-keys)
- [Connecting to Instances](#connecting-to-instances)
- [Docker Images & Configuration](#docker-images--configuration)
- [Templates](#templates)
- [Instance Lifecycle](#instance-lifecycle)
- [Data Transfer](#data-transfer)
- [Instance Portal & Monitoring](#instance-portal--monitoring)
- [REST API Reference](#rest-api-reference)
- [Python Examples](#python-examples)

---

## Authentication

### API Key Setup

Get your API key from https://cloud.vast.ai/account/ and configure it:

```bash
# Install CLI
pip install --upgrade vastai

# Set API key (saves to ~/.config/vastai/vast_api_key)
vastai set api-key <your_api_key>
```

### REST API Authentication

All API requests require a Bearer token:

```bash
Authorization: Bearer <your_api_key>
```

**Base URL:** `https://console.vast.ai/api/v0/`

---

## Searching for Instances

### CLI Search

```bash
# Basic search
vastai search offers

# Search with filters
vastai search offers 'reliability > 0.99 num_gpus>=4 gpu_name=RTX_4090'

# Search with ordering
vastai search offers 'gpu_ram >= 24' -o 'dph_total'

# On-demand vs interruptible (bid) pricing
vastai search offers --type on-demand
vastai search offers --type bid
vastai search offers --type reserved

# Output as JSON
vastai search offers 'num_gpus=1' --raw
```

### Search Query Syntax

```
query = comparison comparison...
comparison = field op value
op = <, <=, ==, !=, >=, >, in, notin
```

**Note:** Replace spaces in string values with underscores (e.g., `RTX_4090` not `RTX 4090`)

### Available Search Fields

| Field | Type | Description |
|-------|------|-------------|
| `gpu_name` | string | GPU model (RTX_4090, A100, H100, etc.) |
| `num_gpus` | int | Number of GPUs |
| `gpu_ram` | float | GPU VRAM in GB |
| `cpu_cores` | int | Virtual CPU count |
| `cpu_ram` | float | System RAM in GB |
| `disk_space` | float | Disk storage in GB |
| `reliability` | float | Machine reliability score (0-1) |
| `dph` | float | $/hour rental cost |
| `dlperf` | float | Deep learning performance score |
| `inet_down` | float | Download speed in Mb/s |
| `inet_up` | float | Upload speed in Mb/s |
| `cuda_vers` | float | Max supported CUDA version |
| `compute_cap` | int | CUDA compute capability × 100 |
| `verified` | bool | Machine verification status |
| `datacenter` | bool | Datacenter-only offers |
| `rentable` | bool | Currently available |
| `rented` | bool | Include already-rented (for duplicates) |
| `geolocation` | string | Two-letter country code |
| `static_ip` | bool | Stable IP address |
| `direct_port_count` | int | Open ports on host router |
| `pcie_bw` | float | PCIe bandwidth (CPU to GPU) |
| `disk_bw` | float | Disk read bandwidth MB/s |
| `duration` | float | Max rental duration in days |

### REST API Search

**POST** `/bundles/`

```bash
curl -X POST https://console.vast.ai/api/v0/bundles/ \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "limit": 100,
    "type": "on-demand",
    "verified": {"eq": true},
    "rentable": {"eq": true},
    "num_gpus": {"gte": 1},
    "gpu_name": {"in": ["RTX_4090", "RTX_3090"]},
    "reliability": {"gt": 0.95},
    "gpu_ram": {"gte": 24000}
  }'
```

**Filter Operators:**

| Operator | Meaning | Example |
|----------|---------|---------|
| `eq` | Equal | `{"eq": true}` |
| `neq` | Not equal | `{"neq": false}` |
| `gt` | Greater than | `{"gt": 0.99}` |
| `lt` | Less than | `{"lt": 1.0}` |
| `gte` | Greater/equal | `{"gte": 4}` |
| `lte` | Less/equal | `{"lte": 8}` |
| `in` | In list | `{"in": ["RTX_4090", "A100"]}` |
| `nin` | Not in list | `{"nin": ["CN", "RU"]}` |

**Response:**

```json
{
  "offers": [
    {
      "id": 12345678,
      "gpu_name": "RTX 4090",
      "num_gpus": 1,
      "gpu_ram": 24576,
      "cpu_cores": 16,
      "cpu_ram": 64000,
      "disk_space": 100,
      "dph_total": 0.35,
      "reliability": 0.9987,
      "geolocation": "US",
      "inet_down": 500,
      "inet_up": 500,
      "cuda_max_good": 12.4,
      "rentable": true
    }
  ]
}
```

---

## Creating Instances

### CLI Instance Creation

```bash
# Basic creation
vastai create instance <offer_id> --image pytorch/pytorch --disk 40

# With SSH access
vastai create instance <offer_id> --image pytorch/pytorch --disk 50 --ssh --direct

# With Jupyter
vastai create instance <offer_id> --image pytorch/pytorch --disk 50 --jupyter --direct

# With JupyterLab
vastai create instance <offer_id> --image pytorch/pytorch --jupyter --jupyter-lab --direct

# With environment variables
vastai create instance <offer_id> --image pytorch/pytorch \
  --env '-e MY_VAR=value -e API_KEY=secret -p 8080:8080'

# With onstart script
vastai create instance <offer_id> --image pytorch/pytorch \
  --onstart-cmd 'pip install transformers && python setup.py'

# With custom label
vastai create instance <offer_id> --image pytorch/pytorch --label "training-job-001"

# Private Docker registry
vastai create instance <offer_id> \
  --image myrepo/myimage:latest \
  --login '-u myuser -p mytoken docker.io'

# Interruptible (bid) instance with price
vastai create instance <offer_id> --image pytorch/pytorch --price 0.25

# Full example
vastai create instance 12345678 \
  --image pytorch/pytorch:2.0.1-cuda11.8-cudnn8-runtime \
  --disk 100 \
  --ssh --direct \
  --env '-e WANDB_API_KEY=xxx -e HF_TOKEN=xxx -p 8080:8080 -p 6006:6006' \
  --onstart-cmd 'pip install wandb transformers accelerate' \
  --label "llm-training"
```

### Create Instance Options

| Option | Description |
|--------|-------------|
| `--image IMAGE` | Docker image (required) |
| `--disk DISK` | Disk space in GB |
| `--ssh` | Launch as SSH instance |
| `--jupyter` | Launch as Jupyter instance |
| `--direct` | Use direct connections (faster) |
| `--jupyter-lab` | Use JupyterLab instead of Notebook |
| `--jupyter-dir DIR` | Jupyter launch directory |
| `--price PRICE` | Bid price $/hour (for interruptible) |
| `--label LABEL` | Instance label |
| `--env ENV` | Environment vars and port mappings |
| `--onstart SCRIPT` | Onstart script filename |
| `--onstart-cmd CMD` | Onstart commands as string |
| `--login LOGIN` | Docker registry credentials |
| `--entrypoint CMD` | Override container entrypoint |
| `--args ARGS` | Arguments passed to entrypoint |

### REST API Instance Creation

**PUT** `/asks/{offer_id}/`

```bash
curl -X PUT "https://console.vast.ai/api/v0/asks/$OFFER_ID/" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": "pytorch/pytorch:2.0.1-cuda11.8-cudnn8-runtime",
    "disk": 100,
    "runtype": "ssh_direct",
    "label": "my-training-job",
    "env": {
      "-e WANDB_KEY": "xxx",
      "-p 8080": "8080"
    },
    "onstart": "pip install transformers"
  }'
```

**Request Body Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `image` | string | Docker image (required) |
| `disk` | number | Disk space in GB |
| `runtype` | enum | `ssh`, `jupyter`, `args`, `ssh_direct`, `jupyter_direct` |
| `target_state` | enum | `running` or `stopped` |
| `price` | number | Bid price (0.001-128 $/hr) |
| `label` | string | Instance label (max 1024 chars) |
| `env` | object | Environment variables |
| `onstart` | string | Startup commands |
| `use_jupyter_lab` | bool | Use JupyterLab |
| `jupyter_dir` | string | Jupyter directory |
| `template_id` | int | Use existing template |

**Response:**

```json
{
  "success": true,
  "new_contract": 7835610
}
```

---

## SSH Keys

### Generate SSH Key

```bash
# Generate Ed25519 key (recommended)
ssh-keygen -t ed25519 -C "your_email@example.com"

# Generate RSA key
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
```

### Add SSH Key to Account

**CLI:**

```bash
# Add key from file
vastai create ssh-key "$(cat ~/.ssh/id_ed25519.pub)"

# Or via API key flag
vastai create ssh-key --api-key YOUR_API_KEY
```

**REST API - POST** `/ssh/`

```bash
curl -X POST https://console.vast.ai/api/v0/ssh/ \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "ssh_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... user@host"
  }'
```

**Response:**

```json
{
  "success": true,
  "key": {
    "id": 123,
    "user_id": 456,
    "public_key": "ssh-ed25519 AAAAC3...",
    "created_at": "2024-01-15T12:00:00Z"
  }
}
```

**Important:** New keys only apply to instances created AFTER adding the key.

### List SSH Keys

**CLI:**

```bash
vastai show ssh-keys
```

**REST API - GET** `/ssh/`

```bash
curl https://console.vast.ai/api/v0/ssh/ \
  -H "Authorization: Bearer $API_KEY"
```

**Response:**

```json
[
  {
    "id": 123,
    "user_id": 456,
    "key": "ssh-ed25519 AAAAC3...",
    "created_at": "2024-01-15T12:00:00Z",
    "deleted_at": null
  }
]
```

### Delete SSH Key

**CLI:**

```bash
vastai delete ssh-key <key_id>
```

**REST API - DELETE** `/ssh/{id}/`

```bash
curl -X DELETE "https://console.vast.ai/api/v0/ssh/$KEY_ID/" \
  -H "Authorization: Bearer $API_KEY"
```

### Attach SSH Key to Existing Instance

**CLI:**

```bash
vastai attach ssh <instance_id>
```

---

## Connecting to Instances

### SSH Connection

```bash
# Get SSH URL
vastai ssh-url <instance_id>

# Connect directly
ssh $(vastai ssh-url <instance_id>)

# Manual connection format
ssh -p <port> root@<host>

# Example
ssh -p 20544 root@142.214.185.187
```

### SSH with Port Forwarding

```bash
# Single port
ssh -p 20544 root@142.214.185.187 -L 8080:localhost:8080

# Multiple ports
ssh -p 20544 root@142.214.185.187 \
  -L 8080:localhost:8080 \
  -L 6006:localhost:6006 \
  -L 8888:localhost:8888
```

### File Transfer

**SCP (note uppercase -P):**

```bash
# Upload file
scp -P <ssh_port> local_file.txt root@<host>:/workspace/

# Download file
scp -P <ssh_port> root@<host>:/workspace/output.txt ./

# Upload directory
scp -P <ssh_port> -r ./my_folder root@<host>:/workspace/
```

**SFTP:**

```bash
sftp -P <ssh_port> root@<host>
```

**Rsync:**

```bash
rsync -avz -e "ssh -p <ssh_port>" ./local_dir/ root@<host>:/workspace/
```

### Tmux (Default Session)

By default, you're placed in a tmux session. Key commands:

| Command | Action |
|---------|--------|
| `Ctrl+B, C` | New window |
| `Ctrl+B, N` | Next window |
| `Ctrl+B, P` | Previous window |
| `Ctrl+B, D` | Detach session |
| `tmux attach` | Reattach session |

**Disable tmux:** `touch ~/.no_auto_tmux`

### Troubleshooting SSH

```bash
# Verbose debugging
ssh -vv -p <port> root@<host>

# Fix permissions
chmod 600 ~/.ssh/id_ed25519
chmod 700 ~/.ssh

# Clear known hosts if host key changed
ssh-keygen -R "[<host>]:<port>"
```

---

## Docker Images & Configuration

### Image Selection

Use any public Docker image or private registry images:

```bash
# Public images
--image pytorch/pytorch:2.0.1-cuda11.8-cudnn8-runtime
--image nvidia/cuda:12.1-devel-ubuntu22.04
--image huggingface/transformers-pytorch-gpu
--image tensorflow/tensorflow:latest-gpu

# Private registry
--image myregistry.io/myimage:tag \
--login '-u username -p password myregistry.io'
```

### Environment Variables

**Via CLI:**

```bash
--env '-e VAR1=value1 -e VAR2=value2 -e SECRET=xxx'
```

**Common environment variables:**

| Variable | Description |
|----------|-------------|
| `WANDB_API_KEY` | Weights & Biases API key |
| `HF_TOKEN` | Hugging Face token |
| `OPENAI_API_KEY` | OpenAI API key |
| `AWS_ACCESS_KEY_ID` | AWS credentials |
| `AWS_SECRET_ACCESS_KEY` | AWS credentials |

### Port Mapping

```bash
# Map ports
--env '-p 8080:8080 -p 6006:6006 -p 5000:5000'

# Format: -p external:internal
# Ports 70000+ enable identity mapping (external = internal)
```

### Predefined Environment Variables

These are automatically set by Vast.ai:

| Variable | Description |
|----------|-------------|
| `CONTAINER_ID` | Instance ID |
| `CONTAINER_API_KEY` | API key for CLI use inside container |
| `GPU_COUNT` | Number of GPUs |
| `PUBLIC_IPADDR` | Public IP address |
| `SSH_PUBLIC_KEY` | Your SSH public key |
| `VAST_TCP_PORT_X` | Mapped external port for internal port X |

### Onstart Scripts

Commands run after container starts:

```bash
# Inline command
--onstart-cmd 'pip install -r requirements.txt && python setup.py'

# From file
--onstart setup.sh
```

**Example onstart script:**

```bash
#!/bin/bash
# Make environment variables available in SSH sessions
env | grep _ >> /etc/environment

# Install dependencies
pip install torch transformers accelerate
pip install wandb tensorboard

# Clone repo
git clone https://github.com/myrepo/myproject.git /workspace/project

# Start services
tensorboard --logdir=/workspace/logs --port=6006 &
```

**Note:** For SSH/Jupyter modes, export vars to `/etc/environment` for them to be available in SSH sessions.

### Resource Allocation

| Resource | Allocation |
|----------|------------|
| **GPU** | Exclusive per instance |
| **CPU** | Proportional to GPU fraction, with burst |
| **RAM** | Proportional to GPU fraction |
| **Disk** | Fixed at creation, cannot change |
| **Network** | Shared IP, random external ports |

---

## Templates

Templates save instance configurations for reuse.

### Create Template

**CLI:**

```bash
vastai create template "my-template" pytorch/pytorch:latest \
  --env '-e VAR=value' \
  --onstart 'pip install transformers'
```

**REST API - POST** `/template/`

```bash
curl -X POST https://console.vast.ai/api/v0/template/ \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "llm-training-template",
    "image": "pytorch/pytorch:2.0.1-cuda11.8-cudnn8-runtime",
    "tag": "latest",
    "runtype": "ssh",
    "ssh_direct": true,
    "env": {"-e WANDB_KEY": "xxx"},
    "onstart": "pip install transformers accelerate",
    "recommended_disk_space": 100
  }'
```

**Template Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Template name (required) |
| `image` | string | Docker image (required) |
| `tag` | string | Image tag |
| `runtype` | enum | `ssh`, `jupyter`, `args` |
| `ssh_direct` | bool | Direct SSH connection |
| `jup_direct` | bool | Direct Jupyter connection |
| `env` | object | Environment variables |
| `onstart` | string | Startup commands |
| `use_jupyter_lab` | bool | Use JupyterLab |
| `jupyter_dir` | string | Jupyter directory |
| `docker_login_repo` | string | Private registry URL |
| `docker_login_user` | string | Registry username |
| `docker_login_pass` | string | Registry password |
| `recommended_disk_space` | number | Recommended disk GB |

**Response:**

```json
{
  "success": true,
  "msg": "Template Created Successfully! New Template ID: 123456",
  "template": {
    "name": "llm-training-template",
    "id": 123456,
    "hash_id": "abc123def"
  }
}
```

### Search Templates

**CLI:**

```bash
vastai search templates
```

**REST API - GET** `/template/`

```bash
curl "https://console.vast.ai/api/v0/template/?query=pytorch" \
  -H "Authorization: Bearer $API_KEY"
```

### Use Template to Create Instance

```bash
# Via CLI (using template hash)
vastai create instance <offer_id> --template_hash <hash_id>

# Via API
curl -X PUT "https://console.vast.ai/api/v0/asks/$OFFER_ID/" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"template_id": 123456}'
```

---

## Instance Lifecycle

### List Instances

**CLI:**

```bash
# Table format
vastai show instances

# JSON format
vastai show instances --raw

# IDs only
vastai show instances -q
```

**REST API - GET** `/instances/`

```bash
curl https://console.vast.ai/api/v0/instances/ \
  -H "Authorization: Bearer $API_KEY"
```

**Response:**

```json
{
  "instances_found": 3,
  "instances": [
    {
      "id": 7835610,
      "actual_status": "running",
      "intended_status": "running",
      "cur_state": "running",
      "label": "training-job",
      "image_uuid": "pytorch/pytorch:latest",
      "gpu_name": "RTX 4090",
      "num_gpus": 1,
      "gpu_util": 85.5,
      "cpu_util": 45.2,
      "dph_total": 0.35,
      "ssh_host": "ssh4.vast.ai",
      "ssh_port": 20544,
      "public_ipaddr": "142.214.185.187",
      "jupyter_token": "abc123..."
    }
  ]
}
```

### Show Single Instance

**CLI:**

```bash
vastai show instance <id>
vastai show instance <id> --raw
```

**REST API - GET** `/instances/{id}/`

```bash
curl "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY"
```

### Instance Status Values

| Status | Description |
|--------|-------------|
| `running` | Instance is active |
| `loading` | Container/image loading |
| `exited` | Container stopped |
| `stopped` | Manually stopped (preserves data) |
| `created` | Created but not started |
| `offline` | Host machine offline |
| `scheduling` | Starting up |

### Stop Instance

Preserves data, stops billing for compute (storage still billed).

**CLI:**

```bash
vastai stop instance <id>

# Multiple instances
vastai stop instances <id1> <id2> <id3>

# All instances
vastai stop instances $(vastai show instances -q)
```

**REST API - PUT** `/instances/{id}/`

```bash
curl -X PUT "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"state": "stopped"}'
```

### Start Instance

Restarts a stopped instance (subject to resource availability).

**CLI:**

```bash
vastai start instance <id>

# Multiple instances
vastai start instances <id1> <id2> <id3>
```

**REST API - PUT** `/instances/{id}/`

```bash
curl -X PUT "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"state": "running"}'
```

### Reboot Instance

Stops and restarts container, preserves GPU priority.

**CLI:**

```bash
vastai reboot instance <id>
```

**REST API - PUT** `/instances/reboot/{id}/`

```bash
curl -X PUT "https://console.vast.ai/api/v0/instances/reboot/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY"
```

### Recycle Instance

Destroys and recreates container from freshly pulled image. Preserves GPU priority but loses container state.

**CLI:**

```bash
vastai recycle instance <id>
```

**REST API - PUT** `/instances/recycle/{id}/`

```bash
curl -X PUT "https://console.vast.ai/api/v0/instances/recycle/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY"
```

### Destroy Instance

**Permanently deletes instance and all data. Irreversible!**

**CLI:**

```bash
vastai destroy instance <id>

# Multiple instances
vastai destroy instances <id1> <id2> <id3>
```

**REST API - DELETE** `/instances/{id}/`

```bash
curl -X DELETE "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY"
```

### Label Instance

**CLI:**

```bash
vastai label instance <id> "my-new-label"
```

**REST API - PUT** `/instances/{id}/`

```bash
curl -X PUT "https://console.vast.ai/api/v0/instances/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"label": "my-new-label"}'
```

### Change Bid Price

For interruptible instances only.

**CLI:**

```bash
vastai change bid <id> --price 0.25
```

**REST API - PUT** `/instances/bid_price/{id}/`

```bash
curl -X PUT "https://console.vast.ai/api/v0/instances/bid_price/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"client_id": "me", "price": 0.25}'
```

---

## Data Transfer

### Vast.ai Copy Command

```bash
# Instance to instance
vastai copy <src_id>:/workspace/data <dst_id>:/workspace/

# Local to instance
vastai copy local:./data C.<instance_id>:/workspace/

# Instance to local
vastai copy C.<instance_id>:/workspace/output local:./results/

# Cloud to instance (requires cloud connection setup)
vastai copy s3.101:/bucket/data C.<instance_id>:/workspace/
vastai copy drive:/folder C.<instance_id>:/workspace/
```

### Execute Remote Commands (on stopped instances)

```bash
# List files
vastai execute <id> 'ls -la /workspace'

# Check disk usage
vastai execute <id> 'du -d1 -h /workspace'

# Remove files
vastai execute <id> 'rm -rf /workspace/temp'
```

**REST API - PUT** `/instances/command/{id}/`

```bash
curl -X PUT "https://console.vast.ai/api/v0/instances/command/$INSTANCE_ID/" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"command": "ls -la /workspace"}'
```

**Response:**

```json
{
  "success": true,
  "writeable_path": "/workspace",
  "result_url": "https://s3.amazonaws.com/vast.ai/instance_logs/abc123.log",
  "msg": "Command is executing, wait a few seconds and then view the result_url"
}
```

**Available commands:** `ls`, `rm`, `du`

### Cloud Sync

First, connect cloud provider at https://cloud.vast.ai/account/

```bash
vastai cloud_copy \
  --src /workspace/output \
  --dst s3://mybucket/results \
  --instance <instance_id> \
  --connection <connection_id> \
  --transfer "Instance To Cloud"
```

---

## Instance Portal & Monitoring

### Instance Portal

Instances with Vast.ai images include a web portal accessible via the instance URL. Features:

- **Launch Applications:** Buttons to open running services
- **Cloudflare Tunnels:** Secure HTTPS access via `*.trycloudflare.com`
- **Port Management:** Create tunnels to ports opened after launch
- **Logs Viewer:** Live stream from `/var/log/portal/`

### View Logs

**CLI:**

```bash
# Default (last 1000 lines)
vastai logs <id>

# More lines
vastai logs <id> --tail 5000
```

**REST API - PUT** `/instances/request_logs/{id}`

```bash
curl -X PUT "https://console.vast.ai/api/v0/instances/request_logs/$INSTANCE_ID" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "tail": "1000",
    "filter": "error"
  }'
```

**Request Body:**

| Parameter | Description |
|-----------|-------------|
| `tail` | Number of lines from end |
| `filter` | Grep pattern to filter logs |
| `daemon_logs` | `"true"` for system logs instead of container |

**Response:**

```json
{
  "success": true,
  "result_url": "https://s3.amazonaws.com/vast.ai/instance_logs/xyz.log"
}
```

### Health Monitoring

Monitor instance health via the API response fields:

| Field | Description |
|-------|-------------|
| `actual_status` | Current container state |
| `gpu_util` | GPU utilization % |
| `cpu_util` | CPU utilization % |
| `reliability2` | Machine reliability score |
| `status_msg` | Human-readable status |

**Example health check script:**

```python
import requests

API_KEY = "your_key"
INSTANCE_ID = 12345

response = requests.get(
    f"https://console.vast.ai/api/v0/instances/{INSTANCE_ID}/",
    headers={"Authorization": f"Bearer {API_KEY}"}
)

instance = response.json()["instances"]
print(f"Status: {instance['actual_status']}")
print(f"GPU Util: {instance.get('gpu_util', 'N/A')}%")
print(f"CPU Util: {instance.get('cpu_util', 'N/A')}%")
```

---

## REST API Reference

### Base URL

```
https://console.vast.ai/api/v0/
```

### Endpoints Summary

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/bundles/` | Search offers |
| PUT | `/asks/{id}/` | Create instance |
| GET | `/instances/` | List instances |
| GET | `/instances/{id}/` | Show instance |
| PUT | `/instances/{id}/` | Update instance (start/stop/label) |
| DELETE | `/instances/{id}/` | Destroy instance |
| PUT | `/instances/reboot/{id}/` | Reboot instance |
| PUT | `/instances/recycle/{id}/` | Recycle instance |
| PUT | `/instances/bid_price/{id}/` | Change bid |
| PUT | `/instances/command/{id}/` | Execute command |
| PUT | `/instances/request_logs/{id}` | Get logs |
| GET | `/ssh/` | List SSH keys |
| POST | `/ssh/` | Add SSH key |
| DELETE | `/ssh/{id}/` | Delete SSH key |
| GET | `/template/` | List templates |
| POST | `/template/` | Create template |

### Error Codes

| Code | Description |
|------|-------------|
| 200 | Success |
| 400 | Bad request / Invalid parameters |
| 401 | Unauthorized / Invalid API key |
| 403 | Forbidden / Insufficient permissions |
| 404 | Not found |
| 410 | Gone / Resource unavailable |
| 429 | Rate limited |

---

## Python Examples

### Complete Workflow Example

```python
import requests
import time

API_KEY = "your_api_key"
BASE_URL = "https://console.vast.ai/api/v0"
HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

# 1. Search for offers
def search_offers(gpu_name="RTX_4090", min_gpu_ram=24, max_price=0.50):
    response = requests.post(
        f"{BASE_URL}/bundles/",
        headers=HEADERS,
        json={
            "limit": 50,
            "type": "on-demand",
            "verified": {"eq": True},
            "rentable": {"eq": True},
            "rented": {"eq": False},
            "gpu_name": {"eq": gpu_name},
            "gpu_ram": {"gte": min_gpu_ram * 1024},
            "dph_total": {"lte": max_price},
            "reliability": {"gt": 0.95},
            "order": [["dph_total", "asc"]]
        }
    )
    return response.json().get("offers", [])

# 2. Create instance
def create_instance(offer_id, image, disk=50, label="my-instance"):
    response = requests.put(
        f"{BASE_URL}/asks/{offer_id}/",
        headers=HEADERS,
        json={
            "image": image,
            "disk": disk,
            "runtype": "ssh_direct",
            "label": label,
            "onstart": "pip install torch transformers"
        }
    )
    return response.json()

# 3. Wait for instance to be ready
def wait_for_instance(instance_id, timeout=300):
    start_time = time.time()
    while time.time() - start_time < timeout:
        response = requests.get(
            f"{BASE_URL}/instances/{instance_id}/",
            headers=HEADERS
        )
        instance = response.json().get("instances", {})
        status = instance.get("actual_status")

        if status == "running":
            return instance
        elif status in ["exited", "offline"]:
            raise Exception(f"Instance failed: {status}")

        print(f"Status: {status}, waiting...")
        time.sleep(10)

    raise TimeoutError("Instance did not start in time")

# 4. Get SSH connection info
def get_ssh_info(instance_id):
    response = requests.get(
        f"{BASE_URL}/instances/{instance_id}/",
        headers=HEADERS
    )
    instance = response.json().get("instances", {})
    return {
        "host": instance.get("ssh_host"),
        "port": instance.get("ssh_port"),
        "command": f"ssh -p {instance.get('ssh_port')} root@{instance.get('ssh_host')}"
    }

# 5. Stop instance
def stop_instance(instance_id):
    response = requests.put(
        f"{BASE_URL}/instances/{instance_id}/",
        headers=HEADERS,
        json={"state": "stopped"}
    )
    return response.json()

# 6. Destroy instance
def destroy_instance(instance_id):
    response = requests.delete(
        f"{BASE_URL}/instances/{instance_id}/",
        headers=HEADERS
    )
    return response.json()

# Example usage
if __name__ == "__main__":
    # Find cheapest RTX 4090
    offers = search_offers("RTX_4090", min_gpu_ram=24, max_price=0.50)

    if not offers:
        print("No offers found")
        exit(1)

    best_offer = offers[0]
    print(f"Found offer: {best_offer['id']} at ${best_offer['dph_total']}/hr")

    # Create instance
    result = create_instance(
        best_offer['id'],
        image="pytorch/pytorch:2.0.1-cuda11.8-cudnn8-runtime",
        disk=100,
        label="training-run-001"
    )

    instance_id = result.get("new_contract")
    print(f"Created instance: {instance_id}")

    # Wait for ready
    instance = wait_for_instance(instance_id)

    # Get SSH info
    ssh_info = get_ssh_info(instance_id)
    print(f"SSH Command: {ssh_info['command']}")

    # ... do your work ...

    # Cleanup
    # destroy_instance(instance_id)
```

### SSH Key Management

```python
def add_ssh_key(public_key):
    response = requests.post(
        f"{BASE_URL}/ssh/",
        headers=HEADERS,
        json={"ssh_key": public_key}
    )
    return response.json()

def list_ssh_keys():
    response = requests.get(
        f"{BASE_URL}/ssh/",
        headers=HEADERS
    )
    return response.json()

def delete_ssh_key(key_id):
    response = requests.delete(
        f"{BASE_URL}/ssh/{key_id}/",
        headers=HEADERS
    )
    return response.json()

# Add key from file
with open(os.path.expanduser("~/.ssh/id_ed25519.pub")) as f:
    add_ssh_key(f.read().strip())
```

### Monitoring Multiple Instances

```python
def list_all_instances():
    response = requests.get(
        f"{BASE_URL}/instances/",
        headers=HEADERS
    )
    return response.json().get("instances", [])

def monitor_instances():
    instances = list_all_instances()

    for inst in instances:
        print(f"\nInstance {inst['id']} ({inst.get('label', 'no-label')})")
        print(f"  Status: {inst['actual_status']}")
        print(f"  GPU: {inst['gpu_name']} x{inst['num_gpus']}")
        print(f"  GPU Util: {inst.get('gpu_util', 'N/A')}%")
        print(f"  Cost: ${inst['dph_total']}/hr")

        if inst['actual_status'] == 'running':
            print(f"  SSH: ssh -p {inst['ssh_port']} root@{inst['ssh_host']}")
```

---

## Quick Reference

### Common CLI Commands

```bash
# Search
vastai search offers 'gpu_name=RTX_4090 num_gpus=1 reliability>0.95'

# Create
vastai create instance <offer_id> --image pytorch/pytorch --disk 50 --ssh --direct

# Connect
ssh $(vastai ssh-url <id>)

# Manage
vastai show instances
vastai show instance <id>
vastai stop instance <id>
vastai start instance <id>
vastai destroy instance <id>

# Logs
vastai logs <id>

# Files (on stopped instances)
vastai execute <id> 'ls -la /workspace'
vastai copy C.<id>:/workspace/file local:./file
```

### Instance Types

| Type | Description | Use Case |
|------|-------------|----------|
| On-demand | Fixed price, guaranteed availability | Production workloads |
| Interruptible (bid) | Variable price, can be preempted | Cost-sensitive batch jobs |
| Reserved | Prepaid discount | Long-running workloads |

### Run Types

| Type | Description |
|------|-------------|
| `ssh` | SSH access via proxy |
| `ssh_direct` | Direct SSH (faster) |
| `jupyter` | Jupyter Notebook via proxy |
| `jupyter_direct` | Direct Jupyter (faster) |
| `args` | Custom entrypoint args |