Skip to content

v0.9.32: ec2 integration, nextjs minor version bump, logs perf improvements, memory pressure reduced - #9320

Merged
waleedlatif1 merged 30 commits into
mainfrom
staging
Oct 23, 2026
Merged

waleedlatif1 merged 30 commits into
mainfrom
staging

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

waleedlatif1 and others added 21 commits October 22, 2026 10:43
…nd Scheduling Compared (#9299)

* feat(library): OpenClaw vs Claude Cowork: Setup, Pricing, Security, and Scheduling Compared

* Pi Babysit: address PR #9299 feedback

---------

Co-authored-by: Sim Pi Agent <[email protected]>
…ework Should You Choose? (#9300)

* feat(library): CrewAI vs LangGraph vs AutoGen: Which Multi-Agent Framework Should You Choose?

* Pi Babysit: address PR #9300 feedback

---------

Co-authored-by: Sim Pi Agent <[email protected]>
* improvement(ci): reuse E2E builds and rebalance integration runners

* improvement(ci): use free integration runners and shard browser viewports

* improvement(ci): share production builds across HTTP and Electron suites

* improvement(ci): select affected task groups with Turborepo

* improvement(ci): scope type checks and right-size hosted runners

* improvement(ci): partition macOS desktop smoke coverage

* fix(ci): retain integration coverage for shared UI and harness changes

* fix(ci): preserve landing dependencies and favor integration latency

* docs(ci): clarify paired provider benchmark count

* ci: retain free GitHub integration runners

* docs(ci): clarify push and manual check selection

* fix(ci): preserve hosted SCIM entitlement coverage
* feat(ec2): add Amazon EC2 integration

* feat(ec2): add network ACL, endpoint, peering, AMI, key pair, ingress rule, and EBS delete operations

- Add describe_network_acls, describe_vpc_endpoints, describe_vpc_peering_connections,
  describe_images, describe_key_pairs, authorize/revoke_security_group_ingress,
  delete_volume, and delete_snapshot
- Regenerate block successor names (fixes the lint job's repo audit)
- Merge the two tag tables into one tags field so fixed tags work in the Agent editor
- Keep a __proto__ tag key by building records with Object.fromEntries
- Pass [{Key, Value}] tag arrays through the block instead of dropping them
- Use a filter example each describe operation actually accepts
- Report revoked: false when a revoke matched no rule; mark the ingress target clauses core

* fix(ec2): generate filters with the json-array wand
…9305)

Trace secret projection is CPU-bound and its awaits settle as microtasks,
which never let the event loop service I/O. A logs read with trace spans
projects a page of traces concurrently, so large traces chained
multi-second synchronous blocks back to back and starved every other
request on the same server.

- Check a process-wide 10ms slice at every unit of projection work (span,
  tool call, provider segment, content field) and yield a macrotask once it
  expires. The slice is handed to one waiting projection per event-loop
  turn, so a turn stays bounded whatever the concurrency.
- Run the whole-trace content invariant per span in projection pre-order
  against one shared budget, so an over-budget trace fails at the span that
  crosses the budget instead of after projecting the whole trace and
  discarding it. Output is unchanged.
The logs query agent-CLI engine reads only traceSpans from each per-run detail read (GET /api/v2/logs/{runId}), so it now sends includeWorkflowState=false and the route no longer loads, sanitizes and serializes the saved workflow snapshot. The chunked Promise.all batches are replaced with the existing mapWithConcurrency sliding window (at most 5 reads in flight per query), so one slow read no longer stalls the next chunk. Rows stay in run order and output is unchanged.
…ad no output (#9308)

readWorkflowExecutionStatus returns the column-only response with error/finalOutput/blockOutputs null unless the run failed, a completed run was polled with includeOutput, or selectedOutputs were requested - the only three readers of the materialized execution data. Other polls skip the archive fetch, parse, and display projection; response key order is unchanged. Skipped polls no longer emit the archive-unavailable, provenance-fault, or write-time-projected-span warnings, and a projection exception on such a poll no longer turns it into a 500. Adds regression tests for skipped polls (fail fast via a macrotask sentinel) and for selected outputs on a running run, including the truncated-row rejection.
…aching file-type's ASF loop, and patch transitive advisories (#9306)

* fix(deps): update Next.js to 16.4.1, stop untrusted Office uploads reaching file-type's ASF loop, and patch transitive advisories

* fix(deps): exempt the exact Electron pin from the release-age gate, like Next.js

* fix(deps): carry the Next.js Vary-merging patch to 16.4.1

* test(file-parsers): prove the ZIP guard without running the stalling payload in the test worker
Logs-list pages that return execution data (includeTraceSpans/includeFinalOutput, not withheld by permissions) are capped at MATERIALIZE_CONCURRENCY (20) rows before the SQL read and at a 64MB stored-archive budget (MAX_DURABLE_LARGE_VALUE_BYTES), first row always admitted. The cursor is minted from the last returned row, so walks never skip or repeat. Applies to v1 and v2; callers may receive pages shorter than limit and should follow nextCursor. v2 limit description, OpenAPI doc, and sim-cli generated client updated.
…coercion (#9311)

- PDF extraction yields a macrotask once 10ms have passed, via one shared time-sliced check used in the page loop, the per-page text-stream read loop, and between and within page-assembly passes (replaces the fixed sleep(0) every 32 pages). pdf.js's fake worker resolves messages as microtasks, so these loops never let timers or I/O run before.
- Table CSV append/replace coerces rows in slices of 2,000 with a yield between slices; output and cellsRejected unchanged.
- Row limit is checked right after parsing for append and replace, before coercion and before claiming the job slot: an over-limit replace now fails with 'Replace would exceed table row limit…' (HTTP 400), and an over-limit file reports the limit before a concurrent-import conflict.
…9310)

- Stream each CSV export row as bounded pieces (toCsvLinePieces, 256K chars) with a cooperative yield between them, so escaping and encoding a multi-megabyte trace no longer holds the event loop; output bytes are identical.
- Pieces never split a real surrogate pair; lone surrogates stay lone; non-finite or sub-1 piece sizes are guarded.
- Extract the process-wide CPU-slice turnstile from trace-secret-projection.ts into lib/core/utils/event-loop.ts, shared by both callers.
* ci: balance desktop smoke shards by observed file durations

* test: verify balancing guards and document duration evidence

* style: use the script alias for shard planning

* fix: select Playwright file suites rather than test declaration locations

* test: prove desktop discovery preserves imported and unmeasured cases
- Spreadsheet buffers (xlsx/xls/xlsm/xlsb/xltx/ods) through parseBuffer parse on a one-worker worker_threads pool running the same XlsxParser, taking synchronous XLSX.read off the event loop (max event-loop gap 1.1-1.9s -> 8-14ms on 16-23 MB workbooks, identical output hashes)
- Input is copied into a fresh ArrayBuffer and transferred; queued jobs hold only the caller's reference; idle workers terminate after 30s; Bun ignores resourceLimits, so the bound is concurrency
- Abort drops a queued job or terminates the running worker and rejects with signal.reason; FileParserError/ZipBombError/ArchiveIntegrityError rebuild with the same class, code and message
- Missing bundle, load failure, sync or async start failure, or a mid-parse crash disables the pool once (one warn) and parses in-process, draining every queued job
- Enabled only on the production app server (not dev/tests/Trigger.dev); bundle built by build:xlsx-worker into apps/sim/dist (declared Turbo output), shipped via Dockerfile COPY and an outputFileTracingIncludes entry
- xlsx-worker-pool.test.ts builds the real bundle and covers parity, buffer integrity, fallbacks, and cancellation
…e-serializing every subtree (#9316)

Each compacted node now carries its JSON byte size, so a parent sums its children (keys, separators, child bytes) instead of re-stringifying its whole subtree, and compactSubflowResults sums its aggregate from per-result sizes. A node serializes for real only when it spills or when a toJSON (its own or a child's, which receives the key) or a reject-mode __proto__ key makes the sum unsound. The walk stays synchronous until something spills or block logs compact; children start in the same order and a sync throw still observes pending siblings. Output identical to the previous serializer on a 60,000-tree differential harness; 2-17x faster on Bun across benched shapes, loop exits about 4-5x. Adds byte-exact spill-boundary, toJSON, and subflow-aggregate regression tests.
…dlers (#9312)

- EventLoopTelemetry INFO 'Event loop snapshot' every 60s (lag p50/p99/max/mean, stalls, cpuPct, inFlight, inFlightPeak, windowMs; elu only where real, inFlightDropped when non-zero) and WARN 'Event loop stalled' when a 250ms probe fires >=1s late (max 10 per window) with inFlightDuringStall as concurrency context
- withRouteHandler registers each handler in an in-flight registry kept on globalThis so instrumentation and route module instances share it; entries keyed by per-call token, capped at 4,096 with oldest-first eviction
- Stall window starts at the probe's due time with a 50ms edge tolerance at both ends
- TSDoc documents Bun vs Node lag semantics, the ~1.0-1.25s stall floor, and what is not attributed (streamed bodies, page renders, background loops)
…ting it as missing (#9314)

- Secret projection sets `contentWithheld: true` on a span when it omits any of its content (input/output/thinking/errorMessage, tool-call or provider-segment content) and on every span of a structural-only fallback; marker only, no content restored
- v2 `LogTraceSpan` contract documents the optional marker (additive); regenerated openapi-v2-logs.json
- `logs query` reports a withheld note and a `contentWithheld` count instead of "path not found on span"; under --where, withheld runs are kept (with --field still resolved) instead of silently filtered out
…field was omitted (#9318)

- The projection omits a content field whole, so logs query treats a path as withheld only when the span is marked and the content field it reads through (input/output/thinking/errorMessage, tool-call or provider-segment content) is absent; a key missing inside a present field is path not found and fails --where again
- A direct path through a withheld field (e.g. thinking) stays unknown instead of falling back to a same-named key under output
- Runs whose --where could not be checked get whereUnchecked: true and a separate whereUnchecked summary count; contentWithheld counts only withheld --field rows
- Usage text and the v2 contentWithheld description state whole-field semantics (openapi regenerated)
…eep logs off the anonymous one (#9317)

* fix(telemetry): send browser events to the configured collector and keep logs off the anonymous one

The browser telemetry route read only TELEMETRY_ENDPOINT, so a deployment exporting to its own
collector through OTEL_EXPORTER_OTLP_ENDPOINT still posted browser events to the anonymous
default, ignored NEXT_TELEMETRY_DISABLED, and built spans without trace or span ids under the
retired instrumentationLibrarySpans field. It now resolves the endpoint and headers exactly like
the server export, honors the disable flag, and emits valid OTLP. Server logs and metrics are no
longer exported to the anonymous collector, which receives traces only.

* fix(telemetry): never send collector credentials to the anonymous endpoint, match it by host

OTEL_EXPORTER_OTLP_HEADERS belong to the operator's collector, so they are dropped when the
endpoint resolves to the anonymous one; that check now compares hosts, so any spelling of the
anonymous URL gets traces only. The browser relay falls back to telemetry.config.ts like the
server export, and the traces URL is built on the parsed path, keeping query strings and
accepting a trailing slash.

* fix(telemetry): never attach operator collector credentials to browser events

/api/telemetry relays unauthenticated input, so sending it with OTEL_EXPORTER_OTLP_HEADERS let
anyone write spans into the operator's authenticated collector. Browser events now go only to
TELEMETRY_ENDPOINT or the configured anonymous collector, without credentials; the server's own
export keeps using the operator's collector.

* fix(telemetry): post browser events to TELEMETRY_ENDPOINT as-is and document routing per side
@waleedlatif1
waleedlatif1 requested a review from a team as a code owner October 22, 2026 22:51
@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Oct 23, 2026 2:36am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 148 files

Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/file-parsers/pdf-parser.ts Outdated
Comment thread apps/sim/lib/internal/ec2/schema.ts
Comment thread apps/sim/lib/file-parsers/pdf-parser.ts Outdated
Comment thread scripts/duration-shards.ts
Comment thread apps/sim/app/api/logs/export/route.ts
Comment thread bunfig.toml
Comment thread apps/sim/tools/ec2/start_instances.ts Outdated
Comment thread apps/sim/lib/monitoring/event-loop-telemetry.ts
Comment thread apps/sim/app/api/telemetry/route.ts
Comment thread .github/scripts/http-e2e.sh
@greptile-apps

greptile-apps Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[High impact] The largest reach is a new integration that can change AWS resources using workflow-supplied credentials.

The PR appears safe to merge, with non-blocking improvements needed for spreadsheet queue limits and acceptance-build packaging.

Findings

  1. P2 Waiting uploads have no limit ▶
  2. P2 Acceptance skips the spreadsheet worker ▶

Summary

Adds Amazon EC2 with 34 actions and four comparison articles. Updates Next.js and several transitive dependencies.

  • Sim workflows can inspect and manage Amazon EC2 resources.
  • Log queries show when trace content is withheld and load less data at once.
  • Large parsing and payload work gives the server chances to serve other requests.
  • Operators can see when event-loop delays and slow requests affect the server.
  • Browser events go to the configured collector while logs stay off the anonymous collector.
  • CI picks affected checks and balances desktop smoke tests by measured run time.
  • Next.js moves to 16.4.1 and Office uploads get an early file check.
  • The library adds four guides comparing popular AI and automation tools.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Spreadsheet buffer] --> B{Worker bundle available?}
  B -->|Yes| C[Queue upload buffer]
  C --> D[Single parse worker]
  B -->|No| E[Parse on main thread]
  F[App build] --> G[dist/xlsx-worker.mjs]
  F --> H[.next]
  H --> I[Shared acceptance artifact]
  I --> B
Loading

Reviews (1) · Last reviewed commit: "fix(telemetry): send browser events to t..." · Reviewed by Greptile

Comment thread apps/sim/lib/file-parsers/xlsx-worker-pool.ts
Comment thread .github/workflows/checks.yml Outdated
…evels (#9319)

The bottom-up compaction from #9316 walks a tree on one synchronous stack until something spills, so deep trees (and every rejectLargeValues tree past ~1,000 levels) overflow where the old awaiting walk compacted them. Every 256 nesting levels the walk now resumes from a microtask; later siblings wait for the resumed subtree to finish walking, so objects are first seen in the same order and the output and stored values stay identical. Adds regression tests for 10,000-level plain and 8,000-level Date-per-level trees and for a shared object first reached inside a resumed subtree.
…ield inside whole-document passes (#9323)

- PDF parsing yields through the shared process-wide turnstile instead of a per-parse slicer
- suppressFurniture and the line passes yield between pages instead of only between passes
- Text-flow reachability is precomputed linearly, replacing a per-band-line walk that was quadratic on tall pages
- The e2e build artifact includes dist/xlsx-worker.mjs so acceptance runs use the spreadsheet worker
…ild caches (#9328)

* improvement(ci): refresh integration shard balance for GitHub runners

* improvement(ci): run database checks alongside application shards

* improvement(ci): omit unused source-map processing from E2E builds

* fix(tests): await observable document lock contention

* improvement(ci): preserve builds across developer documentation edits

* fix(tests): use the fixture lock-observation deadline explicitly
…im knowledge import` (#9327)

* feat(knowledge): import knowledge base bundles through the API and `sim knowledge import`

- v2 knowledge imports resource (create, part URLs, complete, get, cancel) over a new knowledge_import upload purpose
- background runner recreates the base, tags, and documents from an exported bundle; reuses vectors when the embedding model matches and queues re-processing otherwise
- knowledge_base_import job table with a stale-import janitor sweep
- `sim knowledge import <bundle.simkb.zip>` plus `knowledge imports get/cancel`, and an import guide
- shared dispatchBackgroundTask helper for Trigger.dev-or-in-process task dispatch

* fix(knowledge): validate bundles before writing, close import cancel and cleanup gaps, and fix CI

- read every chunk entry before the base is created, reject reversed offsets, and length-check vectors before decoding
- confirm the import is still running before creating the base, and remove it if a cancel wins
- record the import inside upload completion, so a refusal or crash cannot strand the bundle; re-read on a concurrent completion
- serialize progress writes, keep the source character count, cap rebuilt text, and recheck uploads when issuing part URLs
- janitor fails in-process documents a dead worker left processing, and deletes bundles when pruning settled imports
- shrink the unused-exports baseline, count the new janitor passes in its test, and fix the integration test's Blob typing

* fix(knowledge): re-dispatch an import whose first dispatch was lost, and remove a base whose tag setup fails

- the base takes the import's id and a worker claims the import before working, so a second dispatch exits untouched
- a retried completion dispatches again an import no worker has claimed
- a worker that finds its import settled before any claim deletes the unused bundle
- tag-definition setup that fails removes the still-empty base

* fix(knowledge): keep a failed dispatch from stopping a live import, audit each import once, and close review gaps

- a dispatch never settles the import itself; a retry or the stale-import janitor does
- projected audit entries take an idempotency key, recorded through recordAuditOnce, so a retried completion audits the import exactly once
- a cancel that races completion stops the record the completion wrote
- refuse empty rebuilt chunk-only documents; knowledge_base_import.user_id references the user

* fix(knowledge): keep import records through account deletion, and recheck uploads on re-dispatch

- knowledge_base_import.user_id is attribution only, so the janitor's cleanup record outlives its importer's account
- re-dispatching an unclaimed import repeats the knowledge.upload assertion

* fix(knowledge): audit an import as started, and drop the mock-only dispatch test

- the completion records knowledge_base.import_started; the base exists only once the runner creates it
- background-task.test.ts only checked forwarded mock arguments; the delete-async route test and the integration suite exercise the helper's real paths
@waleedlatif1
waleedlatif1 merged commit df3db64 into main Oct 23, 2026
146 checks passed

This branch was successfully deployed

1 active deployment
Preview — ff98c865 Deployed Oct 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant