Skip to content

feat: collect the OTLP data exported by the benchmarks #589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lvaroqui wants to merge 4 commits into from
Open

feat: collect the OTLP data exported by the benchmarks #589

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
69e2ea1
feat(runner-shared): add a versioned file format for benchmark telemetry
lvaroqui Oct 20, 2026
361b84e
feat: add a telemetry runner mode on a generic analysis executor
lvaroqui Oct 20, 2026
f78db2b
test: run the divan telemetry example end to end in telemetry mode
lvaroqui Oct 20, 2026
c40417d
fix(otel): drop the telemetry output of a previous run in the profile…
lvaroqui Oct 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 11 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,11 @@ reqwest-retry = "0.9"
serde = { workspace = true }
serde_json = { workspace = true, features = ["preserve_order"] }
url = "2.5.8"
hyper = { version = "1.9", features = ["server", "http1"] }
hyper-util = { version = "0.1.20", features = ["tokio"] }
http-body-util = "0.1.3"
sha256 = "1.6"
tokio = { version = "1", features = ["macros", "rt", "process"] }
tokio = { version = "1", features = ["fs", "macros", "net", "rt", "sync", "process"] }
tokio-tar = { package = "astral-tokio-tar", version = "0.6.2" }
tokio-util = "0.7.18"
crc-fast = { version = "1.9.0", default-features = false, features = ["std"] }
Expand Down Expand Up @@ -79,6 +82,13 @@ which = "8.0.2"
crc32fast = "1.5.0"
samply = { path = "crates/samply-codspeed/samply" }
bytes = "1"
opentelemetry-proto = { version = "0.33", default-features = false, features = [
"gen-tonic-messages",
"trace",
"logs",
"with-serde",
] }
prost = "0.14"

# Memory profiling (memtrack) and the capability handling around it are Linux-only.
[target.'cfg(target_os = "linux")'.dependencies]
Expand Down
2 changes: 1 addition & 1 deletion crates/exec-harness/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ pub fn execute_benchmarks(

match measurement_mode {
MeasurementMode::Walltime => walltime::perform(commands),
MeasurementMode::Memory => analysis::perform(commands),
MeasurementMode::Memory | MeasurementMode::Telemetry => analysis::perform(commands),

@greptile-apps greptile-apps Bot Oct 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Plain telemetry commands fail

Plain Exec targets still use the downloaded exec-harness version 1.4.1. AnalysisExecutor passes CODSPEED_RUNNER_MODE=telemetry, but that released harness only accepts the old modes, so it exits before running the command. Adding support in this workspace does not update the downloaded binary.

Release the updated harness and bump its version and installer pin.

Knowledge Base Used:

Prompt To Fix With AI 
This is a comment left during a code review.
Path: crates/exec-harness/src/lib.rs
Line: 72

Comment:
**Plain telemetry commands fail**

Plain `Exec` targets still use the downloaded `exec-harness` version `1.4.1`. `AnalysisExecutor` passes `CODSPEED_RUNNER_MODE=telemetry`, but that released harness only accepts the old modes, so it exits before running the command. Adding support in this workspace does not update the downloaded binary.

Release the updated harness and bump its version and installer pin.

**Knowledge Base Used:**
- [Benchmark execution engine](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/benchmark-execution.md)
- [Execution harness and instrumentation bindings](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/execution-harness-and-instrumentation.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

MeasurementMode::Simulation => analysis::perform_with_valgrind(commands),
}
}
4 changes: 3 additions & 1 deletion crates/runner-shared/benches/memtrack_reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ fn sizes() -> &'static [usize] {
.ok()
.and_then(|mode| MeasurementMode::from_str(&mode, true).ok());
match mode {
Some(MeasurementMode::Simulation | MeasurementMode::Memory) => ANALYSIS_MODE_SIZES,
Some(
MeasurementMode::Simulation | MeasurementMode::Memory | MeasurementMode::Telemetry,
) => ANALYSIS_MODE_SIZES,
Some(MeasurementMode::Walltime) | None => WALLTIME_MODE_SIZES,
}
}
Expand Down
1 change: 1 addition & 0 deletions crates/runner-shared/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,6 @@ pub mod metadata;
pub mod module_symbols;
pub mod perf_event;
pub mod runtime_env;
pub mod telemetry;
pub mod unwind_data;
pub mod walltime_results;
2 changes: 2 additions & 0 deletions crates/runner-shared/src/measurement_mode.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ pub enum MeasurementMode {
Memory,
#[value(alias = "instrumentation")]
Simulation,
Telemetry,
}

impl MeasurementMode {
Expand All @@ -19,6 +20,7 @@ impl MeasurementMode {
Self::Simulation => "instrumentation",
Self::Walltime => "walltime",
Self::Memory => "memory",
Self::Telemetry => "telemetry",
}
}
}
Loading
Loading