Skip to content

Show progress while upload computes checksums - #942

Open
caglarpir wants to merge 1 commit into
mapillary:mainfrom
caglarpir:show-pre-upload-progress
Open

caglarpir wants to merge 1 commit into
mapillary:mainfrom
caglarpir:show-pre-upload-progress

Conversation

@caglarpir

Copy link
Copy Markdown
Contributor

Before it uploads a sequence or a video, or skips it as already uploaded, upload computes the checksum of every file to look it up in the upload history. That reads every file in full and showed nothing, so for a large sequence on a slow drive upload looked hung after ==> Uploading..., and an already uploaded sequence was skipped only after a long silence.

This shows a progress bar while it reads the files, like the upload bar that follows:

==> Uploading...
Checksumming IMAGE (1/3): 100%|██████████| 2.94M/2.94M [00:00<00:00, 745MB/s]
UploadedAlready: Skipping sequence_0, already uploaded a moment ago (2026-10-28 15:36:42)
Checksumming GOPRO max-360mode.mp4 (1/1): 100%|██████████| 4.08M/4.08M [00:00<00:00, 465MB/s]

Its speed also shows how fast the drive reads. As with the other bars, --verbose hides it. Checksums that the description file already gives are not computed again, and get no bar.

Tests: unit tests for the new md5sum_fp progress callback and for _update_md5sums. Also checked by hand: process then upload --dry_run twice, of images and of a GoPro video.

Before it uploads a sequence or a video, or skips it as already
uploaded, upload computes the checksum of every file to look it up in
the upload history. That reads every file in full, and showed nothing:
for a large sequence on a slow drive, upload looked hung after
"==> Uploading...", and an already uploaded sequence was skipped only
after a long silence.

Show a progress bar while it reads the files, as for the upload that
follows, e.g. "Checksumming IMAGE (1/3)". Its speed shows how fast the
drive is read.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant