Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 78 additions & 4 deletions docs/developer-guide/100-system-administration.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ This document summarises the database performance, storage footprint, and system

Use the following docker command to access the database on the docker host:
```shell
docker exec -it openremote-postgresql-1 psql -U postgres
docker exec -it or-postgresql-1 psql -U postgres -d openremote
```

### Datapoint retention
Expand Down Expand Up @@ -202,6 +202,9 @@ Here's a summary of critical PostgreSQL memory settings:
| **`effective_cache_size`** | Query Planner Hint | Estimates how much RAM the Linux OS has left over for caching files. | **Does not allocate RAM.** Helps the query planner decide the fastest way to scan compressed Hypercore blocks. |

**The Golden Rule:** `shared_buffers` (Fixed) + `maintenance_work_mem` (Spiky) + `work_mem` (Highly Spiky) must **never** exceed the **Total Container RAM**.
Note that `maintenance_work_mem` can be allocated by each autovacuum worker (see `autovacuum_max_workers`, default `3`) and `work_mem` by each sort or hash operation of each active query, so a single query can use several multiples of `work_mem`.
A more realistic worst case is therefore `shared_buffers` + (`autovacuum_max_workers` x `maintenance_work_mem`) + (active connections x `work_mem` x a few operations per query).
The manager and Keycloak connection pools can hold dozens of connections, so keep `work_mem` modest.

To keep the instance healthy and prevent Docker crashes, you can follow these guidelines:

Expand All @@ -212,9 +215,10 @@ To keep the instance healthy and prevent Docker crashes, you can follow these gu
5. Set `maintenance_work_mem` (0.125 x P) = 250MB (Round up to a clean number 256MB)
6. Set `work_mem` (1% x P) = 16MB (or larger if there are few concurrent connections)
7. Set `effective_cache_size` (P - `shared_buffers`) = 1.5GB
8. Actively monitor `pg_hot_update_percent` metric to attain a >90% HOT update percentage; tweak the asset table `fillfactor` to achieve this (default is set to: 90% as asset table is not significantly large)
8. Actively monitor `pg_hot_update_percent` metric to attain a >90% HOT update percentage; tweak the asset table `fillfactor` to achieve this (OpenRemote sets the asset table `fillfactor` to 90% in a DB migration; the PostgreSQL default is 100%). If the HOT update percentage is already above 90%, `fillfactor` does not need further tuning.

Apply this config in docker compose yaml as follows (use environment variables for convenience:
Apply this config in docker compose yaml as follows (use environment variables for convenience).
The default values shown are the minimal values used in `profile/deploy.yml` and are only suitable for small instances; override them with the environment variables for larger instances, e.g. for the 2GB example above set `OR_DB_RAM=2G`, `OR_DB_SHM=1gb`, `OR_DB_SHARED_BUFFERS=512MB`, `OR_DB_CACHE_SIZE=1536MB`, `OR_DB_MAINTENANCE_WORK_MEM=256MB` and `OR_DB_WORK_MEM=16MB`:
```yaml
services:
postgresql:
Expand All @@ -239,14 +243,84 @@ services:
```

#### Some simple guidelines
* Docker's `shm_size` must be sized appropriately. A simple rule of thumb is to take PostgreSQL `shared_buffers` and add a margin (10% or a flat 256MB); the `shared_buffers` must fit inside this size (**NOTE: Docker sets this very low by default and this will cause problems for any reasonable size DB**)
* Docker's `shm_size` must be sized appropriately. A simple rule of thumb is to take PostgreSQL `shared_buffers` and add a margin of 10-20%; the `shared_buffers` must fit inside this size (**NOTE: Docker sets this very low by default and this will cause problems for any reasonable size DB**)
* **Do not** run `timescaledb-tune` automatically on container startup. It may miscalculate limits if Docker's cgroup namespaces are restricted.
* **Best Practice:** Run `timescaledb-tune` with explicit CPU and memory values to avoid known issues with the script not being able to figure out docker aware values itself `timescaledb-tune --memory="2GB" --cpus="2" --dry-run`, the `dry-run` setting ensures no changes are made to the configuration file and instead hardcode the tuned `shared_buffers`, `effective_cache_size`, `maintenance_work_mem` and `work_mem` directly into the `docker-compose.yml` command args.
* **The 25% Rule:** The largest uncompressed chunk should never exceed 25% of the total Docker container RAM limit.
* For a 2GB container limit, the `pg_datapoint_largest_uncompressed_chunk` metric must stay below **512 MB**.
* If chunks grow too large, the background compression worker will exhaust RAM during the sorting phase and trigger the Linux OOM killer. If this occurs, reduce the hypertable's `chunk_time_interval` via `set_chunk_time_interval()`.
* The asset table is constantly being updated (whenever an attribute event is written to the attributes JSONB column) this can cause excessive autovacuum executions and can be a significant bottleneck for large asset tables due to the way row updates are handled by PostgreSQL; HOT updates is an optimisation PostgreSQL uses to minimise this overhead and the asset table `fillfactor` can be tweaked to optimise this (default: 90%)

#### Monitoring metrics
The [Query Exporter configuration file](https://github.com/openremote/core/blob/main/deployment/query-exporter/config.yaml) exports PostgreSQL metrics for Prometheus.
Most values are point-in-time samples, so some fluctuate with normal database activity:

| Metric | Healthy value | Notes |
| :------------------------------------------------------- | :----------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pg_cache_hit_percentage` | >95% | Lower values mean data is regularly read from disk instead of RAM. |
| `pg_connections_stuck` | `0` | Connections idle in a transaction for more than 1 minute. These prevent autovacuum from removing dead rows and cause table bloat. |
| `pg_hot_update_percent` | >90% for the `asset` table | See step 8 above. |
| `pg_dead_tuple_percent` | Below the autovacuum trigger (about 20%) | Rises and falls with each autovacuum run on frequently updated tables such as `asset`. |
| `pg_table_bloat_count` | `0` sustained | Tables over 5MB with dead rows above 30% of live rows. Busy tables can briefly cross this threshold between autovacuum runs, so occasional `1` values are expected. |
| `pg_asset_raw_data_size` | A few MB per few thousand assets | Asset table heap size. Growth without a matching increase in the number of assets indicates bloat, see [Asset table bloat](#asset-table-bloat). |
| `pg_datapoint_largest_uncompressed_chunk` | Below 25% of the container RAM limit | See the 25% rule above. |

Alert on sustained values rather than single samples, e.g. `min_over_time(pg_table_bloat_count[30m]) > 0` or an alert rule with `for: 30m`.

#### Asset table bloat
The asset table is updated on every attribute event, so autovacuum runs on it frequently.
If autovacuum cannot remove dead rows for a period of time, e.g. because a connection was left idle in a transaction, the table grows and PostgreSQL does not return the space to the operating system after it has been cleaned up.
Every subsequent autovacuum then has to scan the larger table, which makes each run slower.
A high HOT update percentage does not prevent this.

Compare the size of the asset table heap with its number of rows:
```sql
SELECT count(*) AS assets,
pg_size_pretty(pg_relation_size('openremote.asset')) AS heap,
pg_size_pretty(pg_total_relation_size('openremote.asset')) AS total
FROM openremote.asset;
```

PostgreSQL compresses or moves large rows (over about 2kB) to the TOAST table, so the heap should hold at most a few kB per asset.
A heap that is many times larger than that is bloated.

Before compacting the table, check that nothing is holding back cleanup:
```sql
SELECT pid, application_name, client_addr, now() - xact_start AS xact_age, now() - state_change AS idle_for, backend_xmin, left(query, 200) AS last_query
FROM pg_stat_activity
WHERE state = 'idle in transaction';
```

Terminate any long-lived idle transactions with `SELECT pg_terminate_backend(<PID>);`.
To prevent them, PostgreSQL can end sessions that stay idle in a transaction for longer than a timeout.
The `openremote/postgresql` image sets this timeout using the `POSTGRES_IDLE_IN_TRANSACTION_SESSION_TIMEOUT` environment variable on the `postgresql` service (default `10min`, set to `0` to disable), which `profile/deploy.yml` passes through:
```yaml
services:
postgresql:
environment:
POSTGRES_IDLE_IN_TRANSACTION_SESSION_TIMEOUT: ${POSTGRES_IDLE_IN_TRANSACTION_SESSION_TIMEOUT:-10min}
```

Older images ignore this environment variable; for those, set the timeout on the database instead:
```sql
ALTER DATABASE openremote SET idle_in_transaction_session_timeout = '10min';
```
This only applies to new connections, so restart the manager and Keycloak containers to apply it to their connection pools.
The setting is not included in a `pg_dump` of the `openremote` database, so it must be set again after a restore.

Compact the table with `VACUUM FULL`:
```shell
docker exec or-postgresql-1 psql -U postgres -d openremote -c "VACUUM (FULL, VERBOSE, ANALYZE) openremote.asset;"
```

:::warning

`VACUUM FULL` rewrites the table and holds an exclusive lock on it until it completes, which blocks all asset reads and writes. For an asset table of a few thousand rows this takes well under a second, but run it during a quiet period.

:::

The `VERBOSE` output reports how many pages the table occupied before compaction and how many dead row versions could not be removed; if dead row versions cannot be removed, a transaction is still holding back cleanup.

### Backup/Restore
* Create backup: `docker exec or-postgresql-1 pg_dump -Fc openremote -f /tmp/db.bak`
* Optional: Exclude datapoint records from the backup using the following command: `docker exec or-postgresql-1 pg_dump -Fc openremote -f /tmp/db.bak --exclude-table-data='_timescaledb_internal._hyper_*'`
Expand Down