Skip to content

Take check_upload_history checksums from the upload checksum cache - #950

Open
caglarpir wants to merge 2 commits into
mapillary:mainfrom
caglarpir:check-history-checksum-cache
Open

caglarpir wants to merge 2 commits into
mapillary:mainfrom
caglarpir:check-history-checksum-cache

Conversation

@caglarpir

@caglarpir caglarpir commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Desktop Uploader runs check_upload_history before upload, and both read every file in full to compute its checksum, so a folder of dashcam videos on a memory card was read twice per session, minutes each time.

check_upload_history uses upload's checksum cache (second commit). It takes checksums from the cache and caches those it computes, so a file is read once for its checksum; a later check, or the upload, looks it up.

  • As in upload, the cache is used only when the upload history is, and a cache that can't be opened or written gives a warning and is not used.
  • The check writes no upload history, only checksum_cache. Its stdout is unchanged; the checksum progress goes to stderr, as in upload. The integration test that asserted the history folder is not created now asserts it holds only the checksum cache. The README and the command reference no longer say that the check changes nothing.

The cache key adds the change time and file ID (first commit). The check needs it to avoid false positives, and it fixes upload since #943. The key was path, size and modification time. Editing the GPS of an image with exiftool -P keeps both when the image was copied from a camera card (whole-second modification times), so the checksum of the old image was taken and the edited image was skipped as already uploaded.

  • Key: path, size, modification time, change time and file ID, as git compares. The system sets the change time on every change, and it can't be set back; a replaced file gets a new file ID. On Windows st_ctime is the creation time, so there a replaced file is caught on NTFS (exiftool replaces by default), not one edited in place. Not tested on FAT or exFAT (memory cards): Windows makes their file IDs from the position of the file in its folder, which the replacing file can take.
  • A file that changes while it is read is not cached.
  • Entries cached with the old key are not matched: each file is read once more, and they expire.

Checked by hand with 936 GoPro images, a BlackVue video, a GoPro MAX .360 and a CAMM video (process, then check_upload_history):

  • stdout byte-identical to main; a second check took 0.8 s instead of 4.7 s (files on a warm SSD).
  • upload --dry_run after the check computed no checksums; the check after the upload gave the same output as main's.
  • 20 images uploaded, one edited with exiftool -P and uploaded, then edited again (same size and modification time): main's upload skipped it as already uploaded, and the check with the cache but the old key reported it as uploaded. With both commits, the check reports it as not uploaded and upload uploads it.

Tests: cache hits read nothing; size, modification time, in-place edit (not on Windows) and replacement are misses; a file changed while read is not cached; an edited image is uploaded again; for the check: a second run reads nothing and gives the same results, upload after a check reads nothing, unusable caches, no cache without history, stdout of the command.

upload caches the checksum of each file, keyed by its path, size and
modification time, and looks up the upload history by it. Editing an
image can keep both: exiftool -P, when it rewrites the GPS values of an
image copied from a camera card (whose modification time has whole
seconds), keeps the size and the modification time to the nanosecond.
The cached checksum of the image before the edit was then taken, so the
edited image matched the upload history of the old one and was skipped
as already uploaded.

Key the cache by the change time and the file ID as well, as git does.
The system sets the change time on every change of a file, and it can
not be set back; a file replaced by another gets a new file ID. On
Windows, st_ctime is the creation time, so there only the file ID tells
an edited file: on NTFS, one replaced by the editor (as exiftool does by
default), not one edited in place. FAT and exFAT, as on memory cards,
store no change time, and Windows makes their file IDs from the position
of the file in its folder, which the replacing file can take: there an
edit that keeps the size and modification time may not be told.

A file that changes while it is read is not cached, as its checksum
may be of the contents before or during the change.

Entries cached before this change are not found by the new keys; they
are read again once and expire.

The command reference lists the new key.
check_upload_history computed the checksum of every file by reading it
in full, and kept none, so a check of a folder of dashcam videos on a
memory card took minutes, and the upload that followed read every file
again to compute the same checksums.

Take the checksums from the checksum cache that upload uses, and cache
those computed, as upload does: a file is read once for its checksum,
by the check, and a later check or the upload looks it up. As in
upload, the cache is used only when the upload history is, and a file
is taken as unchanged while its path, size, modification time, change
time and file ID are. A cache that can not be opened or written is
warned about and not used, as in upload.

The check still writes no upload history; only the checksum cache. Its
output is unchanged, and the progress of the checksums goes to stderr,
as in upload.

The README and the command reference said that the check changes
nothing: they now say that it writes the checksum cache.
@caglarpir
caglarpir force-pushed the check-history-checksum-cache branch from 3365220 to 9e04495 Compare November 16, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant