[Ocean572]

Incremental Counters for Counts, Pending, and Unfinished (Schema v6) — #194

Overview

This PR implements #193: incremental SQLite counters for counts(), pendingCount(), and unfinishedCount().

Previously, these operations repeatedly aggregated data from the files and segments tables. As the index grew, these queries became increasingly expensive, particularly for large media libraries.

This implementation maintains derived counter tables as files and segments are inserted, updated, or deleted. Count queries can then read the relevant counter rows instead of repeatedly scanning and aggregating the underlying index.

The implementation introduces:

• Schema v6, including a derived files.bucket column.
• A counters table tracking file counts by directory, kind, and processing state.
• A segcounters table tracking segment counts by directory and vector status.
• Eight SQLite triggers that maintain counters automatically during database writes.
• A state-first covering index to accelerate counter lookups.
• Transactional migration from schema v5 to v6, including backfill and consistency verification.
• Counter-based implementations of counts(), pendingCount(), and unfinishedCount().
• Aggregation-based reference implementations retained as test oracles.
• 25 new counter lifecycle and regression tests.

All performance figures in this PR are measured results, collected on an Apple M1 running macOS 26.5.1 with system SQLite 3.51.0. Comparisons use the PR #191 implementation as the baseline, running on the same machine with the same data.

Stacking and Branch Information

This PR is stacked on [PR #191] (#191), which provides the folder-scoping machinery used by the new counter queries.

Consequently, this branch also contains PR #191's three commits:

• b17d59a
• 174c409
• 8da015b

Only the final three commits introduce the changes for this PR. Once PR #191 is merged, the remaining diff will contain only the incremental counter implementation.

Implementation

1. Schema v6 and Incremental Counter Maintenance

The migration introduces two derived counter tables.

counters(bucket, kind, state)

Tracks file counts grouped by:

• Directory bucket.
• File kind.
• Processing state.

segcounters(bucket, hasvec)

Tracks segment counts grouped by:

• Directory bucket.
• Whether a vector is present.

The migration also adds files.bucket, a derived directory value computed from each file's path. This allows counter queries to operate on directory ranges without repeatedly deriving directory information from file paths.

Eight SQLite triggers maintain these counters automatically when the underlying files and segments change.

The implementation also adds a state-first covering index:

counters_state (state, bucket, kind, n, leftsum)

This index allows SQLite to locate counter rows by processing state and directory range while retrieving the required counter values directly from the index where possible.

2. Transactional Migration

The v5-to-v6 migration runs inside a single SQLite transaction.

The migration follows this sequence:

1. Add the derived files.bucket column.
2. Backfill bucket values for existing files.
3. Create the counter tables and indexes.
4. Install the eight maintenance triggers.
5. Populate the counter tables from the existing files and segments.
6. Verify the counters against live aggregation of the underlying tables.
7. Set the schema version to v6.

The schema version is stamped last, after the counter backfill and consistency check have succeeded.

This ensures that an interrupted or failed migration rolls back to the original v5 database rather than exposing partially populated counters as a valid v6 database.

Migration steps are guarded with IF NOT EXISTS and column-presence checks using pragma_table_info, allowing safe migration retries.

3. Counter-Based Reads and Folder Scoping

The implementations of counts(), pendingCount(), and unfinishedCount() now read from the counter tables instead of repeatedly aggregating the files and segments tables.

When a folder scope is active, the query uses the temporary scopeb table introduced by PR #191.

Each scope range drives an indexed lookup against counters_state:

state = ?
AND bucket > ?
AND bucket < ?

This allows SQLite to seek directly into the relevant counter ranges.

The scoped counter query therefore scales with the number of relevant directory ranges and matching counter rows, rather than requiring aggregation across millions of file and segment records.

The original aggregation implementations remain available as reference methods:

• fullCounts()
• fullPendingCount()
• fullUnfinishedCount()

These serve as correctness oracles in the test suite, allowing the optimized results to be compared against the original calculations.

Commits

The implementation is divided into three commits:

Commit	Description
c8f0527	Schema v6, migration, triggers, and covering index
5718d15	Counter-based reads, aggregation oracles, and scoped queries
2e2d50c	25 counter lifecycle and regression tests

Each commit compiles independently. The first retains the original aggregation-based reads, the second introduces the optimized reads and reference implementations, and the third adds the counter-specific test coverage.

Migration Performance and Safety

All migration measurements below were collected using the production migration code.

Database size	Migration time	Peak WAL size	Peak RSS	Counter rows
1.2M files	5.8–6.7 s	93 MB	322 MB	7,200
5M files	34.3 s	418 MB	986 MB	500,000

Additional verification:

• At 1.2M files, the post-migration row-for-row consistency check completed in 0.99 seconds.
• At 5M files, the consistency check completed in 4.98 seconds.
• At 1.2M files, the counter tables contained 7,200 rows, with SUM(n) = 1,208,000, matching the total files and live segments.
• At 5M files, the counter tables contained 500,000 rows.
• Reopening an existing v6 database short-circuits the migration. This took 0.8 ms at 1.2M files.
• Creating an empty database still follows the migration chain and reaches v6 successfully.

Interrupted Migration Test

Migration recovery was tested by killing an actual DigUp CLI child process with SIGKILL during migration of a disposable v5 database.

The test verified that:

1. The interrupted migration rolls back to an intact v5 database.
2. No partially populated v6 counter state is exposed.
3. The next database open successfully completes the migration to v6.

The regression test is:

interruptedMigrationRollsBackToV5OnSIGKILL

Read and Write Performance

All results below compare the incremental counter implementation against the PR #191 baseline on the same Apple M1 and equivalent datasets.

Read Performance

Operation	1.2M files	5M files / 500k folders
Scoped counts (1,200 folders)	3.4 ms vs. 620 ms (~182× faster)	920 ms vs. 6.11 s (~6.6× faster)
Narrow 100-folder scope	0.3 ms vs. 451 ms	12.4 ms vs. 2.53 s (~204× faster)
Unscoped counts (nil scope)	2.6 ms vs. 72 ms	196 ms vs. 294 ms
setScope	31 ms vs. 31 ms	13.7 s vs. 14.1 s

The largest improvements occur when queries operate on a relatively narrow folder scope. These queries can read a small number of relevant counter rows instead of aggregating the entire index.

Unscoped counts() also improves, although its advantage becomes smaller as the number of directory buckets grows.

Write Performance

Maintaining derived counters introduces additional work during database writes. The following figures show the measured percentage increase in operation time compared with the PR #191 baseline.

Operation	1.2M files	5M files
Insert	+73%	+9%
State transition	+140%	+53%
Segment add	+72%	+3%
Segment delete	+47%	+7%
File delete	+68%	+5%
resetAll	+286%	+353%
Requeue-heavy churn	+135% to +1,187%	State-dependent

The largest costs are associated with operations that update many counter rows, particularly resetAll and requeue-heavy workloads.

These are intentional tradeoffs: the implementation performs additional work when the index changes to avoid repeatedly performing expensive aggregations during normal reads.

Counter Consistency Under Churn

The benchmark harness verified that counter results matched the aggregation-based oracle at every tested scale (equal=yes).

A separate 30,000-operation mixed-write workload also maintained row-for-row consistency.

Some counter rows become zero-valued as files and segments are deleted or move between states. These rows are correctly ignored by counter reads.

Approximately 23–53% of counter rows became zero-valued in the tested churn scenarios. The public pruneZeroCounterRows() operation is available to remove these unused rows during housekeeping.

At 5M files, peak RSS was identical between the two implementations: 1,101 MB.

Tests

The full test suite passed on a clean rebuild:

154/154 tests passing across 18 suites.

The test run included 25 new CounterTests and the existing DigUpKit and LlamaRuntime suites.

Coverage includes:

• Schema v5-to-v6 migration and counter backfill.
• Migration version stamping and consistency verification.
• Malformed legacy partial metadata.
• Interrupted migration and SIGKILL recovery.
• Multiple database connections operating on the same WAL-backed index.
• Randomized file and segment lifecycle operations compared against the aggregation oracle.
• Counter updates during inserts, state transitions, moves, requeues, and deletes.
• Cascade deletion behavior in both supported deletion orders.
• Counter consistency and recovery through rebuildCounters().

Important Invariant Tests

Two additional tests explicitly pin important implementation invariants.

Segment ownership and kind immutability

Production code does not reparent an existing segment or change its kind in place. The counter implementation relies on this invariant because segment ownership and kind changes do not have dedicated counter-maintenance triggers.

The test segmentOwnershipAndKindAreNeverUpdated verifies that no production update statement violates this assumption.

Trigger firing boundaries

The counter triggers rely on specific update shapes to ensure that each file update adjusts counters exactly once.

The test guardsFireOnceAndTheForbiddenShapeIsHealedByRebuild verifies that:

• Supported production update shapes fire the expected maintenance triggers.
• A forbidden combination of state and bucket changes can cause counter drift.
• rebuildCounters() restores consistency after such a violation.

These invariants are documented in the production code. Future changes to file or segment update paths must preserve them or update the counter-maintenance logic accordingly.

SQLite 3.54 Compatibility

The vendored SQLite 3.54.0 amalgamation was tested against the system SQLite 3.51.0 implementation.

The compatibility harness used the production package compilation flags, including:

SQLITE_DQS=0

It exercised:

• Production schema creation.
• The complete v5-to-v6 migration in one transaction.
• All eight maintenance triggers.
• File insertion.
• Segment insertion, including fp16 vector blobs.
• Completion across all three processing states.
• File moves.
• Requeue operations.
• markStopped.
• File deletion and cascade deletion.
• resetAll.
• Folder scope ranges, including U+0001 and U+10FFFF boundaries.
• Scoped and unscoped counter reads.
• Counter pruning and rebuilding.
• Final counter consistency verification.

The SQL was captured from the production IndexStore implementation at OP_Init, with bound values expanded, and replayed against both SQLite versions.

Results:

• Zero SQL prepare or step errors on either engine.
• Byte-identical output, except for the SQLite version.
• Final row-for-row counter consistency check returned zero discrepancies on both engines.

Compatibility Test Limitations

This was a SQL-level compatibility test, not a complete application-level test of both SQLite versions.

The following were not exercised against SQLite 3.54:

• Swift Database wrapper behavior.
• WAL and prepared-statement cache semantics.
• Memory mapping and busy-timeout behavior.
• Large-scale performance.

Additional platform limitations:

• Windows has not been tested.
• The Linux process path has not been tested.
• The compatibility harness used a 63-file dataset.

The results establish SQL-level compatibility, not full cross-platform validation.

Known Limitations and Disclosed Tradeoffs

• Write overhead: resetAll and requeue-heavy workloads are the most expensive operations because of the additional counter maintenance. The measured overhead is documented above.
• Unscoped counts: counts() with no folder scope scales with the number of counter rows. At 500,000 folders, it was approximately 1.5× the aggregation cost on state-uniform data. It was never slower than the baseline in the measured tests.
• Zero-valued counter rows: These can accumulate during normal indexing and deletion activity. pruneZeroCounterRows() is available for housekeeping.
• Segment invariants: Existing segment ownership and kind are assumed to remain immutable. This requirement is documented and covered by regression tests.
• Trigger update shapes: Future production updates must preserve the documented trigger firing boundaries. rebuildCounters() provides a recovery mechanism if counter consistency is lost.
• U+10FFFF path-range limitation: The upper range bound is inherited unchanged from the pathRanges implementation in PR Keep the scoped queries alive past a thousand folders #191. A folder name beginning with U+10FFFF can escape the scope in both the existing path query and the new counter query. This behavior is identical in both implementations and is not introduced by this PR. Such names are representable on ext4 and NTFS, but not APFS.
• Migration resources: The one-time migration of a 5M-file index took 34.3 seconds, with a 418 MB peak WAL and approximately 986 MB peak RSS.
• CI: Fork PR workflows require maintainer approval before they run. The local test suite passing does not imply that CI has passed.

Summary

This PR replaces repeated aggregation of the files and segments tables with incrementally maintained SQLite counters.

Combined with the folder-scoping improvements from PR #191, this allows DigUp to calculate indexing and processing counts much more efficiently for large media libraries.

The implementation preserves the original aggregation queries as correctness oracles, verifies counter consistency during migration, and provides explicit rebuild and pruning operations for maintenance.

The measured results show substantial read-performance improvements, particularly for scoped queries, with additional write overhead that is documented and measured.

The implementation passed a clean 154-test suite, migration interruption testing, counter consistency verification, and SQL-level compatibility testing against SQLite 3.54.