Skip to content

perf(agentos): batch chunked_sqlite index saves into transactions - #2017

Merged
eersnington merged 2 commits into
stack/fix-agentos-chunked-sqlite-metadata-bind-limit-vwrkspxtfrom
stack/perf-agentos-chunked-sqlite-index-one-transaction-xnprllkk
Sep 30, 2026
Merged

eersnington merged 2 commits into
stack/fix-agentos-chunked-sqlite-metadata-bind-limit-vwrkspxtfrom
stack/perf-agentos-chunked-sqlite-index-one-transaction-xnprllkk

Conversation

@eersnington

@eersnington eersnington commented Sep 30, 2026 •

Copy link
Copy Markdown
Member
  • The index is saved after every filesystem change. Each save was 2N + 4 separate SQL requests for N pieces.
  • The same statements are now sent in transactions of up to 32 statements. VM SQLite requests are JSON and a sidecar frame is at most 16 MiB, so each request stays under about 8 MiB. An index under 2 MiB (about 6,000 files) is saved in one transaction.
  • The head update and one DELETE of the other generations go in the last transaction. A load reads the previous index until that transaction commits.
  • Adds a test: after a save whose reply is lost and a save that fails, the index still loads.
Step Before After
Next generation SELECT MAX(generation) same
Write pieces one INSERT request per piece up to 32 statements per transaction
Move the head separate UPSERT request in the last transaction
Remove old pieces SELECT + one DELETE per piece, after the save one DELETE in the last transaction
let mut statements = pieces.map(/* INSERT piece */).chain([
    /* UPSERT head -> generation */,
    /* DELETE FROM agentos_fs_metadata_chunks WHERE namespace = ? AND generation <> ? */,
]);
loop {
    let batch = statements
        .by_ref()
        .take(METADATA_SAVE_STATEMENTS_PER_TRANSACTION)
        .collect::<Vec<_>>();
    if batch.is_empty() {
        return Ok(());
    }
    database.transaction(batch).await.map_err(actor_sql_error)?;
}

Code: crates/native-sidecar/src/plugins/chunked_sqlite.rs


This is part 3 of 4 in a stack:

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 1 high-severity finding

Reviewed commit 5d11641.

Comment thread crates/native-sidecar/src/plugins/chunked_sqlite.rs Outdated
@eersnington
eersnington force-pushed the stack/perf-agentos-chunked-sqlite-index-one-transaction-xnprllkk branch from 5d11641 to 2aac4f9 Compare September 30, 2026 18:45

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 1 high-severity finding

Reviewed commit 2aac4f9.

Comment thread crates/native-sidecar/src/plugins/chunked_sqlite.rs Outdated
@eersnington
eersnington force-pushed the stack/perf-agentos-chunked-sqlite-index-one-transaction-xnprllkk branch from 2aac4f9 to 9f59435 Compare September 30, 2026 19:11

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues found

Reviewed commit 9f59435.

@eersnington
eersnington force-pushed the stack/perf-agentos-chunked-sqlite-index-one-transaction-xnprllkk branch from 9f59435 to 140f2f0 Compare September 30, 2026 19:12
@eersnington eersnington changed the title perf(agentos): save the chunked_sqlite index in one transaction perf(agentos): batch chunked_sqlite index saves into transactions Sep 30, 2026
@eersnington
eersnington merged commit 0d99114 into stack/fix-agentos-chunked-sqlite-metadata-bind-limit-vwrkspxt Sep 30, 2026
8 checks passed
@eersnington
eersnington deleted the stack/perf-agentos-chunked-sqlite-index-one-transaction-xnprllkk branch September 30, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant