perf(dashboard): optimize financial log queries (load 5-15s -> <500ms)#3725
Open
TaprootFreak wants to merge 7 commits into
Open
perf(dashboard): optimize financial log queries (load 5-15s -> <500ms)#3725TaprootFreak wants to merge 7 commits into
TaprootFreak wants to merge 7 commits into
Conversation
The log table grows by ~525k rows/year (FinancialDataLog cron writes every minute). Dashboard endpoints filter on (system, subsystem, severity) and range-scan on created — without an index this is a full scan. The new IDX_7765c3f5f663a0c6d250d28255 on (subsystem, severity, created) covers getLatestFinancialLog, getFinancialLogs and getFinancialChangesLogs, plus the maxEntity lookup the cron runs every minute.
The dashboard chart endpoint reads only 5 aggregate numbers per row but JSON.parse'd the full nvarchar(MAX) message column for every entry. With ~1440-4320 rows per request this dominates the response time. Add dedicated columns (totalBalanceChf, plusBalanceChf, minusBalanceChf, btcPriceChf, balancesByTypeJson) on the log entity. The FinancialDataLog cron now mirrors these aggregates when writing each row. The service uses them as a fast path and falls back to JSON.parse for legacy rows written before this migration. All columns are nullable so the migration is non-breaking — existing rows continue to work via the JSON fallback.
Without dailySample, getFinancialLogs(from) returned every row in the range — 4320 rows for a 3-day window. Add a getSampleIntervalMinutes helper that buckets per 5 minutes for ranges between 26h and 1 week while leaving the 24h live view at full per-minute resolution. The bucketing uses MAX(id) GROUP BY DATEADD/DATEDIFF on MSSQL, mirroring the existing dailySample pattern. Beyond 1 week we keep full resolution and let callers opt into dailySample as before, so behaviour for the existing daily and 24h cases is unchanged.
Add the standard express compression middleware so dashboard JSON payloads (financial log can be a few hundred kilobytes) are gzipped on the wire. The middleware short-circuits when the client does not send Accept-Encoding: gzip, so non-browser clients are unaffected. Service-layer caching for the dashboard endpoints landed alongside the denormalised columns in the previous commit (30s TTL via the existing AsyncCache); these two changes together let repeated dashboard refreshes hit a warm in-memory cache and ship a small compressed payload.
5 tasks
…regation The MAX(id) GROUP BY subquery in getSampledFinancialLogs aggregated over the entire log table (~525k rows) because only the outer query filtered by created >= :from. SQL Server may push the predicate down, but this is not guaranteed. Apply the same filter inside the subquery so the aggregation runs only over the requested time window.
…setter Apply the canonical JSON column pattern (priceSteps/priceStepsObject, indicators/indicatorCodes): keep the raw nvarchar column on the entity but expose a typed getter/setter so producers and consumers never touch the JSON string directly. The DTO now accepts a typed object, and LogService.create destructures it and invokes the setter — mirroring MrosService.create. Migration #1779221847925 already created the DB column, so no schema change is needed.
Standalone utility functions belong in *.util.ts files per the repo convention. Keep the function's behaviour and JSDoc unchanged; only the location and imports move.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Financial Overview dashboard page took 5-15s to load because the backend re-parsed the entire
nvarchar(MAX)messageJSON for every row in an un-indexedlogtable (~525k rows/year, cron writes every minute), shipped the full uncompressed payload, and had no caching layer.This PR addresses all four hot spots without changing endpoint contracts.
Changes
Fix #1 - Composite index (
migration/1779221816705-AddLogFinancialIndex.js)IDX_7765c3f5f663a0c6d250d28255 (subsystem, severity, created)onloggetLatestFinancialLog,getFinancialLogs,getFinancialChangesLogsand the per-minutemaxEntitylookup from the cronnpm run migrationruns see no driftFix #2 - Denormalised aggregate columns (
migration/1779221847925-AddLogFinancialAggregates.js)log:totalBalanceChf,plusBalanceChf,minusBalanceChf,btcPriceChf,balancesByTypeJsonlog-job.service.tspopulates them when writing eachFinancialDataLogrowdashboard-financial.service.tsmapLogToEntry()uses these as a fast path and onlyJSON.parses the nvarchar(MAX)messagefor legacy rows pre-migrationgetLatestBalance()still parses the full JSON (it needs the per-asset breakdown for the live cards) but is a single row, not thousandsFix #3 - DB-side sampling for sub-week ranges (
log.repository.ts)getSampleIntervalMinutes(from, dailySample)helper:nullfor 24h (full per-minute resolution, ~1440 rows) and beyond 1 week,5for 3-day/1-week ranges (cuts 4320 rows to ~864)getFinancialLogs/getFinancialChangesLogsshare a singlegetSampledFinancialLogsimplMAX(id) GROUP BY DATEADD(MINUTE, (DATEDIFF(MINUTE, 0, created) / N) * N, 0)to mirror the existingdailySamplepatternFix #4 - gzip compression + service-layer caching (
main.ts+dashboard-financial.service.ts)app.use(compression())so JSON payloads are gzippedAsyncCacheongetLatestBalance,getLatestFinancialChanges,getFinancialLog,getFinancialChanges(cron writes every minute, so a 30s TTL never serves data older than the next tick on average)fromanddailySamplequery params; no shared state across requests with different parametersTest plan
npm run lintnpm run format:checknpm run buildnpm test— 947 passed (was 938), 0 failed. 9 new tests inlog.repository.spec.ts(5) anddashboard-financial.service.spec.ts(4) cover the bucketing helper, the fast-path/legacy-fallback inmapLogToEntry, and the cache TTL + cache-key behaviournpm run migration:runskipped locally — no MSSQL instance available; both migrations are plainALTER TABLE/CREATE INDEXand will run on the DEV deployMigration Notes
JSON.parse(message)fallback in the service