Repository navigation
Reduce query-exporter false positives and add asset table size metrics - #3599
Open
richturner wants to merge 2 commits into
Open
richturner wants to merge 2 commits into
richturner wants to merge 2 commits into
Conversation
- pg_connections_stuck now only counts connections idle in transaction for more than 1 minute, so brief idle periods between statements of short pooled transactions are no longer reported as stuck. - Add pg_asset_raw_data_size and pg_asset_disk_size to track asset table heap and total size, so heap bloat is visible even when the table is below the size filter of the bloat count query.
richturner
force-pushed
the
enhancement/query_exporter_asset_monitoring
branch
from
November 20, 2026 11:47
e83a0ea to
1b9e6ff
Compare
Autovacuum only runs once dead tuples exceed 50 + 20% of live tuples, so small tables such as dashboard naturally sit at a high dead tuple percentage and tripped the DeadTuple alarm without any real bloat. Report 0 for tables with fewer than 10,000 dead tuples so the metric only reflects tables where dead space matters.
|
4 tasks
Member
Author
|
Deployed this to ***sten production |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Description
Three improvements to
deployment/query-exporter/config.yamlthat cut false positive alarms and make real table bloat visible.They came out of investigating an instance where
pg_table_bloat_countkept flipping between 0 and 1. The cause was a heavily bloatedopenremote.assetheap: 177 MB (22,613 pages) for 2,882 rows. After aVACUUM FULLit was 3.7 MB (468 pages). The table was being autovacuumed about once a minute, and each run had to scan the bloated heap. That let the dead/live tuple ratio pass the 30% threshold between runs, so some 5-minute scrapes caught it above the line and some didn't.pg_connections_stucknow counts only connections that have beenidle in transactionfor more than 1 minute. The manager's pooled JDBC connections are briefly in that state between statements of every short transaction, so the old query regularly reported1with nothing actually stuck. A non-zero value is now actionable. A long-lived idle transaction is the usual cause of table bloat, because it stops vacuum removing dead rows.pg_dead_tuple_percentnow reports 0 for tables with fewer than 10,000 dead tuples. Autovacuum only runs once dead tuples exceed 50 + 20% of live tuples, so small tables naturally sit at a high percentage. On another instance theDeadTuplealarm kept firing fordashboard, a table with a handful of rows, whileassetwas healthy at 96% HOT updates. The metric now only moves for tables where dead space actually matters, and thetable_nameseries stay in place so the CloudWatch Metrics Insights alarm and dashboard keep working.asset_table_sizequery (hourly) withpg_asset_raw_data_size(heap only) andpg_asset_disk_size(heap + TOAST + indexes), in MB. A healthy asset heap stays small, so the bloat count query's 5 MB size filter can skip it entirely. These metrics make heap growth visible regardless: growth without a matching increase in asset count indicates bloat.Documentation changes covering asset table bloat and the monitoring metrics: openremote/documentation#211.
Related: openremote/postgresql#37 and #3600 add a default
idle_in_transaction_session_timeout(10min), which stops the idle in transaction sessions this metric reports from causing table bloat.Checklist