Skip to content

Adding cache schema version - #1204

Merged
cdolfi merged 12 commits into
devfrom
cache_schema_version
Oct 12, 2026
Merged

cdolfi merged 12 commits into
devfrom
cache_schema_version

Conversation

@EngCaioFonseca

@EngCaioFonseca EngCaioFonseca commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request Change Description

This solves #1198 and adds the cache schema versioning as well as lay the groundwork for the follow up steps or making the cache persistent.

The cache schema is now versioned with Alembic. It runs on startup: a fresh cache gets stamped at the latest version, and an existing cache gets any missing migrations applied automatically. Base tables are still created with the raw CREATE blocks in db_init.py, so adding a new table works exactly like before. Alembic only handles changes to existing tables.

A few things added for the cache long-term:

  • build the repo_id indexes without locking the tables (CREATE INDEX CONCURRENTLY)
  • only flush the Redis broker when the cache actually changed instead of on every startup, and stopped it from also wiping user sessions
  • build the DB connection strings with psycopg2's make_dsn so passwords with special characters don't break them

This is also groundwork for making the cache persistent instead of rebuilt on every boot.

Generative AI disclosure

Please select one option:

  • This contribution was NOT assisted or created by Generative AI tools.
  • This contribution was assisted or created by Generative AI tools.

If AI tools were used, please provide details below:
- What tools were used? Claude - Opus 4.8; GPT - 5.6 Sol
- How were these tools used? Draft, review iteration.
- Did you review these outputs before submitting this PR? Yes.

Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
@EngCaioFonseca EngCaioFonseca self-assigned this Aug 25, 2026
@EngCaioFonseca EngCaioFonseca moved this from Backlog to In Progress in Aspen Project Board Aug 25, 2026
@EngCaioFonseca EngCaioFonseca linked an issue Aug 25, 2026 that may be closed by this pull request
@MoralCode

Copy link
Copy Markdown
Contributor

#1189 made changes to the schema, so this may need to change to account for that

Comment thread 8Knot/cache_manager/db_init.py Outdated
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
@EngCaioFonseca
EngCaioFonseca marked this pull request as ready for review August 30, 2026 23:51
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
@EngCaioFonseca EngCaioFonseca moved this from In Progress to Testing | Waiting on PR response | Blocked in Aspen Project Board Sep 14, 2026
Comment thread 8Knot/cache_manager/db_init.py
Comment thread 8Knot/cache_manager/migrations/env.py Outdated
Comment thread 8Knot/cache_manager/migrations/versions/0001_add_labels_to_issues_query.py Outdated
Comment thread 8Knot/cache_manager/migrations/versions/0001_add_labels_to_issues_query.py Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the change in this file to unify our connection strings! this will be a great place to pivot to storing the cache db somewhere more persistent/in the same postgres instance

Comment thread 8Knot/cache_manager/db_init.py Outdated
Comment thread 8Knot/cache_manager/db_init.py Outdated
Comment thread 8Knot/cache_manager/db_init.py Outdated
Comment on lines +640 to +641
# Reconcile indexes after migrations so they reflect the final schema.
_ensure_repo_id_indexes()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can version the indexes alongside the schema if we were using sqlalchemy for our data models

Comment thread 8Knot/cache_manager/migrations/env.py Outdated
Comment thread 8Knot/cache_manager/migrations/env.py
Replace the historical labels migration with a data-preserving baseline and generate numeric Alembic revisions. Resolve migration paths relative to the config and use snapshot-free advisory-lock polling. Add PostgreSQL regression checks for baseline adoption, future upgrades, revision generation, and lock cleanup.

Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
@cdolfi

cdolfi commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator

@EngCaioFonseca @MoralCode LMK when this PR is ready to be tested

Comment thread 8Knot/cache_manager/migrations/versions/1_initial_cache_schema.py Outdated
Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>

@cdolfi cdolfi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something in this pr is making Contribution File Heatmap endlessly load, tested dev and did not see the same behavior

Signed-off-by: Caio Fonseca <engcaiofonseca@protonmail.com>
@EngCaioFonseca

Copy link
Copy Markdown
Contributor Author

Something in this pr is making Contribution File Heatmap endlessly load, tested dev and did not see the same behavior

Resolved. It was necessary to ensure that the app and workers wait for the db-init to finish successfully and no redis flush before this.

@cdolfi cdolfi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested and lgtm

@cdolfi
cdolfi dismissed MoralCode’s stale review October 12, 2026 14:17

verbal approval

@cdolfi
cdolfi merged commit 9a52b08 into dev Oct 12, 2026
7 checks passed
@github-project-automation github-project-automation Bot moved this from Testing | Waiting on PR response | Blocked to Sprint 55 in Aspen Project Board Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Version 8Knot cache schema

3 participants