Skip to content

feat(gpui): audio scrub while dragging the playhead - #2845

Open
YoavLevavi wants to merge 3 commits into
CapSoftware:mainfrom
YoavLevavi:feat/gpui-audio-scrub
Open

YoavLevavi wants to merge 3 commits into
CapSoftware:mainfrom
YoavLevavi:feat/gpui-audio-scrub

Conversation

@YoavLevavi

@YoavLevavi YoavLevavi commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Audio scrubbing: with playback stopped, dragging on the ruler plays a short slice of the mix under the playhead, like Premiere and CapCut.

  • EditorInstance::audio_snippet(time, secs) (cap-editor) renders the slice with the regular AudioRenderer (recording audio plus music tracks) and fades its edges so back-to-back slices don't click.
  • scrub_audio.rs plays each slice on a small cpal output stream owned by its own thread, so the playback pipeline is untouched. A new slice replaces what is left of the previous one, and the stream pauses after 2s idle.
  • One request in flight at a time; the newest pointer position waits and plays next.

Testing: cargo test --bin cap-gpui, cargo check -p cap-editor.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

The callback-memory and Tokio-worker findings are fixed, but the existing comment-policy requirement still needs to be satisfied before merging.

Findings

  1. P2 Replaced slices can click ▶
  2. P2 Comments repeat the code ▶
Fix with agent prompt
### Issue 1
crates/editor/src/audio_output.rs:1588-1589
Each slice lasts 90 ms, but a waiting slice can replace it after roughly 60 ms. The callback discards the old slice before its fade-out, jumping from a full-volume sample to the new slice's faded-in start. This can click during a drag. Fade or crossfade at replacement instead of relying only on the buffer's ending fade.

### Issue 2
apps/desktop-gpui/src/store.rs:undefined-1099
The new comment above `EDITOR_PREFERENCES` only restates the constant's purpose. `AGENTS.md` requires comments to explain non-obvious context and explicitly bans narration of the code. Remove this comment and the similar narration above `audio_scrub_enabled` and `SCRUB_SLICE_SECS`; the pattern also appears elsewhere in the additions. This repository requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds audio previews while dragging the ruler with playback stopped.

  • Dragging the ruler plays short slices of the project mix.

Reviews (2) · Last reviewed commit: "Hand scrub slices through preallocated r..." · Reviewed by Greptile

With playback stopped, each ruler move plays a 90ms slice of the mix under
the playhead (recording audio plus music) on a small output stream of its
own. EditorInstance::audio_snippet renders the slice with the regular
AudioRenderer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@YoavLevavi

Copy link
Copy Markdown
Contributor Author

@greptileai

@vercel

vercel Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

@YoavLevavi is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

@richiemcilroy

Copy link
Copy Markdown
Member

Thanks @YoavLevavi, audio scrubbing is something people ask for, and rendering the slice with the normal AudioRenderer (apps/desktop-gpui/src/editor_window.rs scrub_audio_at, crates/editor/src/editor_instance.rs audio_snippet) and keeping one request in flight is a good approach. The output side needs work before we can ship it on all three platforms. apps/desktop-gpui/src/scrub_audio.rs builds its own cpal stream in a static PLAYER (line 24) that lives for the whole process, even after the editor closes. It also skips what crates/editor/src/audio_output.rs ensure_stream (around line 1437) already handles: 1) it rebuilds when the default device changes or the stream fails. Here, once the stream exists, tx.send keeps succeeding after headphones are unplugged, so scrub audio goes quiet for the rest of the session. 2) it supports I16/I32/U8/F64 devices. Here a non-F32 device returns Err from build_stream (line 103), so every slice (about every 150 ms while dragging) spawns a new thread, opens the device and logs a warning, and spawn_player blocks a tokio worker in recv_timeout(2s) (line 90) while holding the PLAYER lock. 3) it only idle-pauses on built-in macOS output (default_output_supports_idle_pause), while this pauses and resumes on every device (line 80), including Bluetooth. Could you play the slices through the existing AudioOutput (or share its device and format handling), tie the stream to the editor so it is dropped on close, and cache a failed open so it is not retried on every slice? Separately, this makes every ruler drag audible with no way to turn it off, so we would like a setting or toggle for it. We will confirm that part on our side.

- AudioOutput gains play_snippet: slices go to the shared output thread,
  which uses ensure_stream (rebuilt on device change or stream failure,
  every sample format, the usual idle-pause policy), converts them to the
  device's rate, channels and sample type there, and hands them to the
  stream callback. Played-out buffers go back to the output thread, so the
  callback neither allocates nor frees. Playback always takes precedence.
  A failed open drops slices for 2s instead of retrying per slice.
- EditorInstance::scrub_audio renders and plays on its own AudioOutput, so
  the stream lives and dies with the editor; the process-wide player is gone.
- The timeline toolbar has a speaker toggle (saved as
  editor_preferences.audioScrub, on by default) to turn scrub audio off.
- Merges origin/main (keeps its scrub_to_time repaint path).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@YoavLevavi

Copy link
Copy Markdown
Contributor Author

Thanks @richiemcilroy, reworked in 78cb060 (also merged main):

  • Shared output. Slices now go through the editor's AudioOutput (play_snippet) instead of a separate player. The output thread calls ensure_stream, so it rebuilds on device change or stream failure, supports every sample format and keeps the usual idle-pause policy. It converts each slice to the device's rate, channels and sample type, then hands it to the stream callback. Played-out buffers go back to the output thread, so the callback neither allocates nor frees, and playback always takes precedence over a slice.
  • Tied to the editor. EditorInstance::scrub_audio plays on its own AudioOutput, so the stream lives and dies with the editor. scrub_audio.rs and the static PLAYER are gone.
  • Failed opens. A failed open drops slices for 2s instead of reopening the device on every slice.
  • Toggle. A speaker toggle in the timeline toolbar turns scrub audio off. It's saved as editor_preferences.audioScrub and is on by default. Happy to move it into Settings if you'd prefer.
  • Tests. snippets_are_resampled_and_mapped_to_the_device_layout covers rate, mono, 6-channel and i16 conversion.

@greptileai review

Comment thread crates/editor/src/audio_output.rs Outdated
Comment on lines +1588 to +1589
if let Some((spent, _)) = snippet.replace((next, 0)) {
let _ = spent_tx.send(spent);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Replaced slices can click

Each slice lasts 90 ms, but a waiting slice can replace it after roughly 60 ms. The callback discards the old slice before its fade-out, jumping from a full-volume sample to the new slice's faded-in start. This can click during a drag. Fade or crossfade at replacement instead of relying only on the buffer's ending fade.

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/editor/src/audio_output.rs
Line: 1588-1589

Comment:
**Replaced slices can click**

Each slice lasts 90 ms, but a waiting slice can replace it after roughly 60 ms. The callback discards the old slice before its fade-out, jumping from a full-volume sample to the new slice's faded-in start. This can click during a drag. Fade or crossfade at replacement instead of relying only on the buffer's ending fade.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Comment thread crates/editor/src/audio_output.rs Outdated
Comment on lines +1578 to +1579
let (snippet_tx, snippet_rx) = std_mpsc::channel::<Vec<T>>();
let (spent_tx, spent_rx) = std_mpsc::channel::<Vec<T>>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Audio callback allocates memory

Returning buffers through std_mpsc::channel does not keep memory work out of the audio callback. This unbounded queue can allocate blocks during spent_tx.send, and receiving from snippet_rx can free queue blocks. Memory contention can then delay audio delivery. Use a preallocated queue or buffer exchange in both directions.

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/editor/src/audio_output.rs
Line: 1578-1579

Comment:
**Audio callback allocates memory**

Returning buffers through `std_mpsc::channel` does not keep memory work out of the audio callback. This unbounded queue can allocate blocks during `spent_tx.send`, and receiving from `snippet_rx` can free queue blocks. Memory contention can then delay audio delivery. Use a preallocated queue or buffer exchange in both directions.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Comment thread crates/editor/src/editor_instance.rs Outdated
Comment on lines +881 to +885
let mut renderer = crate::AudioRenderer::new(segments).with_music(music);
renderer.set_playhead(time.max(0.0), &project);
let frames =
(secs.clamp(0.01, 0.5) * f64::from(crate::AudioRenderer::SAMPLE_RATE)) as usize;
let Some((_, mut samples)) = renderer.render_frame_raw(frames, &project) else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Scrubbing holds Tokio workers

audio_snippet calls render_frame_raw directly inside the Tokio task. For clips with changed speed, that call builds and runs audio filters synchronously. The fresh renderer repeats this setup for every slice, which can hold a Tokio worker and delay other editor tasks during a drag. Move setup and rendering into spawn_blocking or a dedicated worker, as regular playback does.

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/editor/src/editor_instance.rs
Line: 881-885

Comment:
**Scrubbing holds Tokio workers**

`audio_snippet` calls `render_frame_raw` directly inside the Tokio task. For clips with changed speed, that call builds and runs audio filters synchronously. The fresh renderer repeats this setup for every slice, which can hold a Tokio worker and delay other editor tasks during a drag. Move setup and rendering into `spawn_blocking` or a dedicated worker, as regular playback does.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

section.insert("isolation".into(), serde_json::to_value(isolation).unwrap());
set_store_value(AUDIO_ENHANCEMENT, Value::Object(section))
}
/// Editor preferences that belong to the user rather than a project.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Comments repeat the code

The new comment above EDITOR_PREFERENCES only restates the constant's purpose. AGENTS.md requires comments to explain non-obvious context and explicitly bans narration of the code. Remove this comment and the similar narration above audio_scrub_enabled and SCRUB_SLICE_SECS; the pattern also appears elsewhere in the additions. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/store.rs
Line: 1099

Comment:
**Comments repeat the code**

The new comment above `EDITOR_PREFERENCES` only restates the constant's purpose. `AGENTS.md` requires comments to explain non-obvious context and explicitly bans narration of the code. Remove this comment and the similar narration above `audio_scrub_enabled` and `SCRUB_SLICE_SECS`; the pattern also appears elsewhere in the additions. This repository requirement must be satisfied before merging.

**Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@YoavLevavi

Copy link
Copy Markdown
Contributor Author

Both fixed in a637871:

  • Callback memory. Slices now travel through two preallocated ringbuf::HeapRbs instead of std_mpsc channels. The callback only takes a new slice when the spent ring has room, and a finished slice it can't return yet stays in place (silent) until there's space, so the callback never allocates or frees. a_full_spent_ring_keeps_the_slice_instead_of_freeing_it covers the full-ring case.
  • Tokio workers. audio_snippet builds and renders the slice inside spawn_blocking, so speed-changed clips' filter setup no longer runs on an async worker.

@greptileai review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants