Skip to content

Document asset table bloat and PostgreSQL monitoring metrics - #211

Open
richturner wants to merge 2 commits into
mainfrom
enhancement/db_maintenance_monitoring
Open

richturner wants to merge 2 commits into
mainfrom
enhancement/db_maintenance_monitoring

Conversation

@richturner

@richturner richturner commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Description

Updates the Database section of docs/developer-guide/100-system-administration.md (Next docs) based on an investigation into a production instance where pg_table_bloat_count kept flipping between 0 and 1. The cause was a bloated asset table heap: 177 MB for about 2,900 rows, reduced to 3.7 MB with VACUUM FULL.

Companion to openremote/core#3599, which adds the pg_asset_raw_data_size / pg_asset_disk_size metrics and changes pg_connections_stuck to count only connections idle in transaction for more than 1 minute. This PR documents those metrics, so merge it after or together with the core PR.

Changes

  • Monitoring metrics: new table of the key query-exporter metrics with healthy values and how to read them. Explains that some metrics rise and fall with normal autovacuum activity, and recommends alerting on sustained values.
  • Asset table bloat: new section covering how to detect it (heap size vs row count), finding and preventing idle-in-transaction connections (idle_in_transaction_session_timeout), and compacting with VACUUM FULL, with a lock warning and how to read the VERBOSE output.
  • Golden rule: clarifies that maintenance_work_mem is used per autovacuum worker and work_mem per sort or hash step, and gives a more realistic worst-case estimate.
  • Fillfactor step: the 90% default comes from OpenRemote's DB migration (PostgreSQL's default is 100%), and no further tuning is needed once HOT updates are above 90%.
  • Compose example: notes that the defaults are the minimal values from profile/deploy.yml and lists the environment variable overrides for the 2 GB worked example.
  • Consistency: uses one shm_size margin (10-20%) throughout, and the psql access command now uses or-postgresql-1 like the rest of the page.

Backport

The golden rule, fillfactor and shm_size corrections also apply to the released 1.34.0 docs in versioned_docs/, which this PR does not change.

Dependency

The asset table bloat section recommends the new POSTGRES_IDLE_IN_TRANSACTION_SESSION_TIMEOUT environment variable, which needs openremote/postgresql#37 (image) and openremote/core#3600 (profile/deploy.yml). Keep this PR open until those are merged. It still documents ALTER DATABASE as the fallback for older images.

- Add a monitoring metrics table with healthy values and alerting advice
- Add an asset table bloat section covering detection, idle in
  transaction connections and compaction with VACUUM FULL
- Clarify the memory golden rule, fillfactor default and compose defaults
- Make the shm_size margin and DB container name consistent
@richturner
richturner force-pushed the enhancement/db_maintenance_monitoring branch from 73602c2 to 80cd4ed Compare November 20, 2026 11:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant