Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -552,7 +552,8 @@
{
"group": "VM Install",
"pages": [
"enterprise/vm-install/admin-console-configuration"
"enterprise/vm-install/admin-console-configuration",
"enterprise/vm-install/log-collection"
]
},
{
Expand Down
7 changes: 7 additions & 0 deletions enterprise/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
VM, you can use standard Kubernetes commands to inspect OHE. For Helm
deployments, use your existing Kubernetes access to run the same commands.

Most OHE workloads run in the `openhands` namespace. The Replicated Admin

Check warning on line 12 in enterprise/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

enterprise/troubleshooting.mdx#L12

Did you really mean 'namespace'?
Console runs in `kotsadm`, and ingress runs in `traefik`.

## Start With a Support Bundle
Expand Down Expand Up @@ -90,7 +90,7 @@

<Warning>
Keep your investigation read-only. Do not change Kubernetes resources unless
directed by OpenHands Support. Ad hoc `kubectl` changes can be overwritten

Check warning on line 93 in enterprise/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

enterprise/troubleshooting.mdx#L93

Did you really mean 'hoc'?
during a deployment or upgrade and may leave the installation in an
inconsistent state.
</Warning>
Expand Down Expand Up @@ -163,6 +163,10 @@
Use `--previous` after a container restarts. Use `-c` to select a specific
container, including an init container such as `migrate-db`.

On a Replicated VM, these logs are also written to files on the VM. See
[Log Collection](/enterprise/vm-install/log-collection) to send them to your own
observability platform.

### Choose the Right Component

Pod names may include a release prefix and generated suffix. Match the
Expand All @@ -176,7 +180,7 @@
| `runtime-...` | A particular conversation's sandbox. |
| `litellm` | Model-provider requests and authentication. |
| `keycloak` | Login, SSO, and authentication. |
| `kotsadm` namespace | Replicated Admin Console problems. |

Check warning on line 183 in enterprise/troubleshooting.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

enterprise/troubleshooting.mdx#L183

Did you really mean 'namespace'?

### Temporarily Enable Debug Logging

Expand Down Expand Up @@ -204,4 +208,7 @@
<Card title="Resource Limits" icon="gauge-high" href="/enterprise/k8s-install/resource-limits">
Diagnose and tune CPU, memory, replicas, and storage.
</Card>
<Card title="Log Collection" icon="file-lines" href="/enterprise/vm-install/log-collection">
Send VM installation logs to your own observability platform.
</Card>
</CardGroup>
102 changes: 102 additions & 0 deletions enterprise/vm-install/log-collection.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
title: Log Collection
description: Send logs from an OpenHands Enterprise VM installation to your own observability platform.
icon: file-lines
---

An OpenHands Enterprise VM installation writes the output of every service to log files on the VM. To
bring those logs into your observability platform, install your platform's log agent on the VM and
point it at those files.

For one-off diagnostics, collect a support bundle instead. See
[Troubleshooting](/enterprise/troubleshooting).

## Where the Logs Are

Application logs live under `/var/log/pods`. Each path is built from the namespace, the pod, and the

Check warning on line 16 in enterprise/vm-install/log-collection.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

enterprise/vm-install/log-collection.mdx#L16

Did you really mean 'namespace'?
container:

```
/var/log/pods/<namespace>_<pod>_<pod-id>/<container>/<restart-count>.log
```

For example:

```
/var/log/pods/openhands_openhands-cbdbd996b-r54j8_30f64156-29b8-4b64-b663-cf5b4c697b64/openhands/17.log
```

The VM installation writes to files ending in `.log`. It rotates a file once it grows large,
appending a timestamp to the name and compressing it, for example `16.log.20260824-235907.gz`. A
pattern ending in `*.log` therefore collects current output and skips the rotated copies.

`/var/log/containers` holds a symlink to every one of those files, carrying the same details in the
file name rather than in the directories:

```
/var/log/containers/<pod>_<namespace>_<container>-<container-id>.log
```

Log agents with built-in Kubernetes support read that directory, because they can take the pod and
container names straight from the file name.

| Location | Contains |
|---|---|
| `/var/log/pods/` | Output from OpenHands, its supporting services, and sandboxes. |
| The systemd journal | Cluster and operating system logs. |

Check warning on line 46 in enterprise/vm-install/log-collection.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

enterprise/vm-install/log-collection.mdx#L46

Did you really mean 'systemd'?
| `/var/log/embedded-cluster/` | Installer output, written during installation and upgrades. |

The application log files are readable only by `root`.

<Note>
The VM keeps only recent output, roughly 50 MB per service, and the log files for a sandbox are
deleted when its conversation is cleaned up. Run your log agent continuously and set your
retention period in your observability platform.
</Note>

## Collect the Logs

<Steps>
<Step title="Install your log agent">
Install the Linux log agent for your observability platform on the VM, following your vendor's
instructions. Run it as `root` so that it can read the log files.
</Step>
<Step title="Tail the application logs">
Configure a file input for `/var/log/pods/*/*/*.log`, or `/var/log/containers/*.log` if your
log agent reads the symlinks.

Every line begins with a timestamp and the output stream:

```
2026-08-25T13:12:11.300228843Z stdout F {"message": "GET /health 200", "severity": "INFO"}
```

Enable your log agent's parser for this format, called `cri` in Fluent Bit, so that the
timestamp and the message arrive as separate fields. The message itself is JSON.
</Step>
<Step title="Collect the journal">
Enable your log agent's journald input to pick up cluster and operating system logs.

Check warning on line 78 in enterprise/vm-install/log-collection.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

enterprise/vm-install/log-collection.mdx#L78

Did you really mean 'journald'?
</Step>
<Step title="Confirm the logs arrive">
Print a recent line on the VM, then search for it in your observability platform:

```bash
sudo sh -c 'tail -n 1 /var/log/pods/openhands_openhands-*/openhands/*.log'
```
</Step>
<Step title="Repeat on every VM">
A VM only holds the logs for the services that run on it. Repeat these steps on each VM in the
installation, including any VM that runs sandboxes.
</Step>
</Steps>

## Related Guides

<CardGroup cols={2}>
<Card title="Troubleshooting" icon="life-ring" href="/enterprise/troubleshooting">
Collect a support bundle and inspect workloads.
</Card>
<Card title="Admin Console Configuration" icon="sliders" href="/enterprise/vm-install/admin-console-configuration">
Configure a Replicated VM installation.
</Card>
</CardGroup>
Loading