Skip to content

Always disable sync in SQLite cache#21184

Merged
ilevkivskyi merged 1 commit intopython:masterfrom
ilevkivskyi:always-sync-off
Apr 8, 2026
Merged

Always disable sync in SQLite cache#21184
ilevkivskyi merged 1 commit intopython:masterfrom
ilevkivskyi:always-sync-off

Conversation

@ilevkivskyi
Copy link
Copy Markdown
Member

Fixes #21176

This seems to resolve all the HDD performance issues. Although this makes cache less durable, this is fine, as FS is not perfect in this sense either (and it doesn't really need to be that durable).

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2026

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

# This is a bit unfortunate (as we may get corrupt cache after e.g. Ctrl + C),
# but without this flag, commits are *very* slow, especially when using HDDs,
# see https://www.sqlite.org/faq.html#q19 for details.
db.execute("PRAGMA synchronous=OFF")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we could probably get away with having synchronous enabled on macOS, since they all have had SSDs for a long time now, but since we'll need this for parallel checking, it makes more sense to keep things consistent.

@ilevkivskyi ilevkivskyi merged commit c8f4074 into python:master Apr 8, 2026
24 checks passed
@ilevkivskyi ilevkivskyi deleted the always-sync-off branch April 8, 2026 09:46
@cburroughs
Copy link
Copy Markdown
Contributor

this makes cache less durable

This disables fsync, correct? So this is not just "your recent writes might be missing", but "your file might be corrupted". (I'm not saying this is a bad tradeoff for a cache, but it isn't obvious to me from the linked issues how mypy would handle corrupt sqlite files.)

@ilevkivskyi
Copy link
Copy Markdown
Member Author

This disables fsync, correct?

I don't know the details, but probably yes.

how mypy would handle corrupt sqlite files

It will likely just crash on startup. If this will become a problem we can detect this condition and manually delete the cache.

@JukkaL
Copy link
Copy Markdown
Collaborator

JukkaL commented Apr 8, 2026

I agree that implementing automatic recovery from a corrupt cache could be worth it. However, I believe disabling fsync mostly impacts situations where power is lost or the system is hard rebooted -- just killing mypy shouldn't be a problem?

@cburroughs
Copy link
Copy Markdown
Contributor

just killing mypy shouldn't be a problem?

That matches my expectation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Very bad performance with SQLite cache (on a Linux machine with HDD)

3 participants