# Troubleshooting

> What the messages in the console, the Fleet Manager and your applications mean, and how to fix the problems behind them.

## Installing a device

If the installation stops or fails, it will print an exit code naming the problem that occured.

### Installer exit codes

| Code | Meaning | What to do |
| --- | --- | --- |
| 10 | The installer isn't running with admin rights, the OS isn't supported, or (Windows) it couldn't restrict access to `%ProgramData%\Cascadia` | Windows: run PowerShell as administrator. Ubuntu: run with `sudo`. Only 64-bit Ubuntu 22.04 or later is supported. |
| 11 | The enrollment credential or CA address is missing or malformed | Copy the install block from **Devices → Add device** again and paste the whole thing. |
| 12 | The credential is expired or already used, the organization is suspended or gone, or the network failed | For an expired or used credential, create a new installer in **Devices → Add device** and run it. For a suspended organization, contact your Cascadia representative. For a network error, run it again. |
| 13 | Downloading or verifying the release failed | Usually a temporary network problem; run it again. If it keeps happening, contact your Cascadia representative. |

### Resuming an installation after the device is enrolled

Once the machine has been enrolled, the credential will expire.

To finish the install, get a new installer for the same machine:

1. In the console, open **Devices → Add device** and create a new installer, the same way you did the first time. See [Add a device](/enterprise/devices/add-a-device/).
2. On the machine that stopped, run the new installer or paste its terminal command.

The machine keeps the identity it enrolled with, so it skips enrollment and finishes the remaining steps.

### The device never appears in the console

If the console says **Nothing has enrolled yet**:

- **Windows:** check the install window for a message.
- **Ubuntu:** open a terminal and run `journalctl -u cascadia-installer`. Or, attempt an [installation from the terminal](/enterprise/devices/add-a-device/#install-from-a-terminal).

## A device is offline or not reachable

Each device checks in with the console every minute or so. 

Its status on the Devices page will show one of the following:

| Status | Meaning |
| --- | --- |
| **online** | Checked in within the last two minutes |
| **stale** | Last checked in between two and ten minutes ago. Often a brief network drop or a restart in progress. |
| **offline** | Hasn't checked in for more than ten minutes |
| **revoked** | Removed from the fleet. It can't serve or rejoin without enrolling again. |

### Stale or offline

1. Check that the machine is switched on, awake and connected to the network. A laptop that has left the office goes offline until it's back on a network that can reach the console.
2. If it's online elsewhere but not in the console, check the device's log (see the start of this page).
3. When it's back online, select **Restart** on its row if it still misbehaves. Only active devices can be restarted, and a restart queued for an offline device expires after an hour.

The console warns you with **Devices offline** when a device has been silent for more than 24 hours, and **Fleet offline** when every device is.

### Online, but other devices can't reach it

The device checks in with the console, but other devices and applications can't connect to it.

- **Windows:** the installer's firewall rules only apply on **Private** and **Domain** networks. If the device is on a network marked **Public**, change the network profile.
- Check that the ports in [Requirements](/enterprise/getting-started/requirements/#network) are open in any firewall between your devices.
- Devices on different networks or behind NAT need a [relay](/enterprise/devices/configure-a-relay/).

## Relays

If a relay shows **not reachable**, see [Configure a relay](/enterprise/devices/configure-a-relay/#if-it-shows-not-reachable).

Relays can't be restarted from the console. To restart one, run `sudo systemctl restart cascadia-relay` on the relay host.

### When you add a relay in the console

| Message | What to do |
| --- | --- |
| This organization is already at its limit of active relays | Revoke a relay you no longer need, or select **Re-enrol** on an existing relay's row to move or upgrade it. |
| That public address is not valid | Enter a hostname or IP address without a port, for example `relay.example.com` or `203.0.113.10`. |
| That public address points at a loopback, link-local, metadata or other non-routable target | Use a public IP address or a public hostname that your devices can reach. |

### When you run the relay installer

| Message | What to do |
| --- | --- |
| This credential re-enrols one existing relay, and this host does not hold its key | You selected **Re-enrol**, which only works on the relay's original host. Run the installer on that host, or add a new relay instead. |
| This organization is already at its limit of active relays | Another relay enrolled after you created this installer. Revoke a relay you no longer need, then create a new installer. |

## The Fleet Manager won't connect

These messages appear when you connect the Fleet Manager to your account. See [Install the Fleet Manager](/enterprise/fleet-manager/install/#connect-it-to-your-account) for the full steps.

| Message | What to do |
| --- | --- |
| Use an https:// console URL | Enter the console address starting with `https://`. A LAN address such as `http://192.168.x.x` won't work. |
| Couldn't reach the console at that URL | Check the address and your internet connection. If you're on a VPN, check that it allows the console. |
| This machine's clock is too far from the console's | Turn on automatic time sync on the computer running the Fleet Manager, then try again. |
| This key isn't allowed to manage the fleet | In **Settings → Fleet manager keys**, approve the key again with **Fleet manager** ticked. |
| This key has expired / The console no longer knows this key | Approve the Fleet Manager's key again in **Settings → Fleet manager keys** and paste the new code. |
| A newer code was approved for this key | Paste the latest code from **Settings → Fleet manager keys**. |
| This code was approved for a different fleet manager | Copy the key from step 1 in the Fleet Manager again, and approve that one. |
| The console doesn't recognize this organization | The key is for a different organization or console. Check the console URL. |

If the Fleet Manager shows **Fleet data is out of date** or **Couldn't refresh**, it can't currently reach the console. It keeps showing the last data it fetched and retries on its own.

### The Fleet Manager couldn't update itself

If an in-app update fails, the Fleet Manager keeps running the version you had. Most failures, such as a download that didn't match its checksum, a folder the Fleet Manager can't write to, or an admin prompt that was dismissed, are fixed by downloading the latest installer from the console's **Overview** page and running it yourself.

## A deployment didn't finish

When some devices don't take a model, the Fleet Manager's **Activity** feed marks the job **Failed** or **Partly failed** and gives a reason for each device.

| Reason | What to do |
| --- | --- |
| The device is offline / The device couldn't be reached | Bring the device back online and deploy again. An offline device can take new models when it reconnects. |
| The device never picked up the command before it expired | The device was offline or busy for too long. Check it's online, then deploy again. |
| The device stayed busy past the deadline | Something on the device is stuck. Restart it, then deploy again. |
| The download failed | Check the device's internet connection, then deploy again. |
| The download failed its integrity check | Usually a corrupted download. Deploy again. If it keeps happening, contact your Cascadia representative. |
| The device refused: check its model directory | The device couldn't write the model to disk. Check it has free space, then deploy again. |
| The device couldn't check its free disk space | Restart the device, then deploy again. |
| The device rejected the request's signature | Reconnect the Fleet Manager to your account so its key is current. |
| The device hit an unexpected error | Restart the device. If it happens again, contact your Cascadia representative. |

If a device is short of space, [free up disk space](/enterprise/fleet-manager/deploy-a-model/#free-up-disk-space) by deleting models it no longer serves.

### A model is degraded

A model is **degraded** when one of its pipelines is missing a stage, usually because a device serving that stage went offline. Requests go to the copies that still serve. Bring the missing device back online, or deploy the model to other devices.

### Nothing to pull

**No pre-exported shards published for this model yet** means Cascadia hasn't published this model in a form devices can download. Choose another model from the catalog.

## An update didn't apply

Devices update themselves. If the Activity feed says **The device couldn't apply the update** or **The revert couldn't be verified**, restart the device and roll the release out again. See [Update your fleet](/enterprise/fleet-manager/updates/) to find devices that are behind.

## Applications get errors

The gateway returns errors in the same format as the OpenAI API, so the OpenAI SDKs raise them as exceptions. The `code` field says what went wrong. For errors partway through a response, see [Handle errors](/enterprise/use/connect-an-application/#handle-errors).

| HTTP status | `code` | What to do |
| --- | --- | --- |
| 401 | `missing_token`, `malformed_token` | Send the token in an `Authorization: Bearer <token>` header. |
| 401 | `invalid_token`, `token_invalid` | The token isn't one the fleet recognizes. Check you copied the whole token, or create a new one. |
| 401 | `token_expired` | Create a new token in **Settings → API tokens**. |
| 401 | `token_revoked` | The token was revoked. Create a new one under a different label. |
| 400 | `malformed_body` | The request body isn't valid. Check it against the OpenAI chat completions format. |
| 404 | `model_not_found` | No device serves that model. Check the model name, or deploy it. |
| 503 | `at_capacity`, `backend_busy` | Every device serving the model is busy. Retry after the `Retry-After` header. |
| 503 | `backend_unavailable`, `no_alternative_peer` | No device serving the model is available. Check that devices serving it are online. |
| 503 | `maintenance_mode` | The device is temporarily not taking requests. Retry after the `Retry-After` header. |

### Creating a token

| Message | What to do |
| --- | --- |
| That label has been revoked | Revoking a token revokes its label for good. Choose a different label. |
| A token already uses that label | Give the new token a label of its own. |

## Signing in and team access

| Message | What to do |
| --- | --- |
| Your session has expired | Sign in again. |
| Your role does not allow this action | Ask an owner or admin to do it, or to change your role. See [Team and roles](/enterprise/use/team-and-roles/). |
| Your email address is not confirmed yet | Open the confirmation link in the email we sent, then sign in again. |
| This invitation is no longer valid | Ask an admin in your organization to invite you again. |
| An organization needs at least one owner | Make someone else an owner before you change or remove the last one. |
| This organization is suspended | Contact your Cascadia representative to restore service. |

## Get help

If you can't fix a problem here, contact your Cascadia representative. To help them, include:

- The names of the affected devices and their status in the console
- The message you saw, or the reason in the Activity feed
- The installer's exit code, if the problem is with an install
- The device's recent log (see the start of this page)
- What you expected to happen

Remove private prompts and other sensitive data from logs before you share them.
