Repository navigation
Conversation
Before it uploads images, upload deletes the expired entries of its file handle cache. Each entry was deleted in its own transaction, so SQLite synced the database to disk once per entry. With full syncs (PRAGMA fullfsync on macOS) that took about 5 ms per entry, so 100,000 expired entries took about 8 minutes before upload uploaded or skipped anything. Entries expire 2 days after their image is uploaded, so the first upload after a large batch paid for all of them. Delete them in one transaction: 200,000 expired entries now take under a second. Also clear the cache once per upload, not twice: ImageSequenceUploader cleared it, then created a CachedImageUploader, which cleared it again.
Before it uploads a sequence or a video, or skips it as already uploaded, upload computes the checksum of every file to look it up in the upload history. That reads every file in full, and showed nothing: for a large sequence on a slow drive, upload looked hung after "==> Uploading...", and an already uploaded sequence was skipped only after a long silence. Show a progress bar while it reads the files, as for the upload that follows, e.g. "Checksumming IMAGE (1/3)". Its speed shows how fast the drive is read.
upload looks up the upload history by the checksums of the files, so it read every file in full before it could skip an already uploaded sequence or video: a run that skips 10,000 images or a long video read all of them first. Cache the checksum of each file in the upload history folder, keyed by its path, size and modification time, as rsync's quick check is. A file whose key is unchanged is not read again: 10,000 cached files are looked up in about 0.1 s. Entries expire after 30 days, which only bounds the size of the cache. The cache is used only when the upload history is, and is written in one transaction per sequence. process does not compute checksums, as mapillary#725 removed that from it to keep it fast.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #941 and #942 (this branch merges both): review the last commit.
uploadlooks up the upload history by the checksums of the files, so it read every file in full before it could skip an already uploaded sequence or video. A run that skips 10,000 images, or a long video, read all of it first.This caches each file's checksum in the upload history folder (
checksum_cache), keyed by its path, size and modification time, like rsync's quick check:processstill doesn't compute checksums: #725 removed that from it to keep it fast.Example,
upload --dry_runtwice with history enabled. The second run skips everything at once, with no checksum bars:Tests: unit tests for
KeyValueStore.set_many(one transaction),PersistentCache.get_many/set_many, and_update_md5sumswith a cache (unchanged file taken from the cache without reading it; changed size read again). Also checked by hand with images and a GoPro video,processthenupload --dry_runtwice.