Repository navigation
Conversation
Before it uploads images, upload deletes the expired entries of its file handle cache. Each entry was deleted in its own transaction, so SQLite synced the database to disk once per entry. With full syncs (PRAGMA fullfsync on macOS) that took about 5 ms per entry, so 100,000 expired entries took about 8 minutes before upload uploaded or skipped anything. Entries expire 2 days after their image is uploaded, so the first upload after a large batch paid for all of them. Delete them in one transaction: 200,000 expired entries now take under a second. Also clear the cache once per upload, not twice: ImageSequenceUploader cleared it, then created a CachedImageUploader, which cleared it again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before it uploads images,
uploaddeletes the expired entries of its file handle cache (entries expire 2 days after their image is uploaded). Each entry was deleted in its own transaction, so SQLite synced the database to disk once per entry. With full syncs (PRAGMA fullfsyncon macOS), that is about 5 ms per entry: 100,000 expired entries took about 8 minutes, anduploadshowed nothing after==> Uploading...until it uploaded or skipped the first sequence. The first upload after a large batch paid for all of it.KeyValueStore.delete_matching()reads the entries and deletes the expired ones in oneBEGIN IMMEDIATEtransaction: 200,000 expired entries now take under a second.ImageSequenceUploadercleared it, then created aCachedImageUploader, which cleared it again.Tests: unit tests for
delete_matching(deletes in one transaction; deletes nothing if the predicate fails), and the existing cache tests on Python 3.9, 3.11 and 3.13, which differ in howsqlite3handles transactions.