Skip to content

Reduce query-exporter false positives and add asset table size metrics - #3599

Open
richturner wants to merge 2 commits into
mainfrom
enhancement/query_exporter_asset_monitoring
Open

richturner wants to merge 2 commits into
mainfrom
enhancement/query_exporter_asset_monitoring

Conversation

@richturner

@richturner richturner commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Description

Three improvements to deployment/query-exporter/config.yaml that cut false positive alarms and make real table bloat visible.

They came out of investigating an instance where pg_table_bloat_count kept flipping between 0 and 1. The cause was a heavily bloated openremote.asset heap: 177 MB (22,613 pages) for 2,882 rows. After a VACUUM FULL it was 3.7 MB (468 pages). The table was being autovacuumed about once a minute, and each run had to scan the bloated heap. That let the dead/live tuple ratio pass the 30% threshold between runs, so some 5-minute scrapes caught it above the line and some didn't.

  • pg_connections_stuck now counts only connections that have been idle in transaction for more than 1 minute. The manager's pooled JDBC connections are briefly in that state between statements of every short transaction, so the old query regularly reported 1 with nothing actually stuck. A non-zero value is now actionable. A long-lived idle transaction is the usual cause of table bloat, because it stops vacuum removing dead rows.
  • pg_dead_tuple_percent now reports 0 for tables with fewer than 10,000 dead tuples. Autovacuum only runs once dead tuples exceed 50 + 20% of live tuples, so small tables naturally sit at a high percentage. On another instance the DeadTuple alarm kept firing for dashboard, a table with a handful of rows, while asset was healthy at 96% HOT updates. The metric now only moves for tables where dead space actually matters, and the table_name series stay in place so the CloudWatch Metrics Insights alarm and dashboard keep working.
  • New asset_table_size query (hourly) with pg_asset_raw_data_size (heap only) and pg_asset_disk_size (heap + TOAST + indexes), in MB. A healthy asset heap stays small, so the bloat count query's 5 MB size filter can skip it entirely. These metrics make heap growth visible regardless: growth without a matching increase in asset count indicates bloat.

Documentation changes covering asset table bloat and the monitoring metrics: openremote/documentation#211.

Related: openremote/postgresql#37 and #3600 add a default idle_in_transaction_session_timeout (10min), which stops the idle in transaction sessions this metric reports from causing table bloat.

Checklist

  • 1. Acceptance criteria of the linked issue(s) are met
  • 2. Tests are written and all tests pass
  • 3. Changes are manually tested by you and the reviewer
  • 4. Documentation is written or updated

- pg_connections_stuck now only counts connections idle in transaction
  for more than 1 minute, so brief idle periods between statements of
  short pooled transactions are no longer reported as stuck.
- Add pg_asset_raw_data_size and pg_asset_disk_size to track asset
  table heap and total size, so heap bloat is visible even when the
  table is below the size filter of the bloat count query.
Autovacuum only runs once dead tuples exceed 50 + 20% of live tuples, so small
tables such as dashboard naturally sit at a high dead tuple percentage and
tripped the DeadTuple alarm without any real bloat. Report 0 for tables with
fewer than 10,000 dead tuples so the metric only reflects tables where dead
space matters.
@richturner richturner changed the title Improve query-exporter stuck connection and asset table size metrics Reduce query-exporter false positives and add asset table size metrics Nov 20, 2026
@sonarqubecloud

Copy link
Copy Markdown

@richturner

Copy link
Copy Markdown
Member Author

Deployed this to ***sten production

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant