Repository navigation
feat(desktop): local CopilotKit Intelligence from the app alone, in the CopilotKit visual system - #832
Draft
dustywusty wants to merge 65 commits into
Draft
feat(desktop): local CopilotKit Intelligence from the app alone, in the CopilotKit visual system#832dustywusty wants to merge 65 commits into
dustywusty wants to merge 65 commits into
Conversation
…nded native boundary local_eval.rs runs a SHA-256-pinned Intelligence CLI with fixed arguments only (local status|start|stop --stack ID --json, local list --json, local setup --defer-model --json), no shell, an environment allowlist, an owner-only output file capped at 1 MiB, per-action timeouts, per-attempt cancellation and process-group cleanup, one call at a time. It normalizes the CLI's version-1 JSON to allowlisted facts; readiness and licence stay unknown because v1 cannot state them. Release builds run no CLI; debug builds accept a developer pin by path and digest. local_eval_state.rs records the local evaluation checkpoint (stack, created or shared, CLI digest, phase). preparation.rs shares its atomic checkpoint write. localEvalCli.ts is the window's typed client.
…p and Quit Stop and Quit ask the CLI to stop the local stack after OpenBot's own cleanup and Compose down, only when OpenBot created it. In-flight calls are cancelled and new ones refused while closing; Quit allows the CLI two minutes and no longer waits behind a Stop. Intelligence failures are reported apart from OpenBot's and never mark OpenBot for recovery; a failed stop at Quit is kept for the next launch. Adds the Tauri commands for status/start/stop, list, setup and cancel, and exposes the checkpoint as AlreadyConfigured.localEval.
…ure split Connect offers "Use Intelligence on this computer" only when a pinned CLI answers. The panel shows independent Docker Desktop, Intelligence, OpenBot and provider rows, runs setup for this installation, and keeps Start disabled because the CLI does not report readiness or give OpenBot a credential. A Stop that fails only for Intelligence marks OpenBot stopped and shows the Intelligence failure separately; a failed stop at Quit is shown on the next launch without holding auto-start.
…uests The v2 requirements artifact (decisions, architecture, coding sequence, progress and release gates), the original r21 artifact unchanged, the implementation handoff, and the contract requests for the Intelligence CLI owners.
…utomatic step 4 Local checkpoint for the demo; not for review as-is.
…t step for local Local checkpoint for the demo; not for review as-is.
… and automatic test Local checkpoint for the demo; not for review as-is.
…temporary model-field debug line The debug line logs field names and token length only, never values. Remove before review.
…minal Local checkpoint for the demo; not for review as-is.
…e status for slow actions Local checkpoint for the demo; not for review as-is.
…o menu-bar status text The menu-bar title is gone (tray-icon never clears it on macOS). An Intelligence row at the bottom of the app sidebar opens the local dashboard through a two-command plugin granted only to the owned loopback app origin. After the first sign-in of a Desktop run, Open sends the stored dashboard address straight to the browser with no CLI call. Debug builds hash the CLI pin with an optimised sha2 (4.86 s to 0.29 s).
…n Intelligence; question box with presets The test question went straight to the harness, which keeps its thread in memory, so Intelligence never saw it and View in Intelligence opened a dashboard with no conversation. A picked Bot is now asked through the owned server's runtime: the server mints the thread, the runtime records the run in Intelligence, and the desktop reads the thread's events until RUN_FINISHED or RUN_ERROR. The local test step is a question box again: a question already in it, presets one click away (strawberry r's, Die Hard, Pop-Tarts, 17 x 23), your own typed question, and Ask again after an answer. Nothing is asked until Ask.
…d build time Several worktrees build this app on one machine and the window looks the same from each, so which build is running was a guess. build.rs records the commit (marked -dirty for uncommitted changes), the branch and the build time, rerunning on source and git changes; About OpenBot shows them.
… that cycles through the rest The preset chips crowded the screen. The box starts on one question at random; Another question beside it steps through the others in order and back round; typing your own still works.
…ilotKit hosted, assuming neither Install with a small 'hosted instead' link assumed local. The first screen now offers both as equal choices with nothing selected, and Continue waits for one.
…thin its screen 900x640 was too short for Connect your AI once a provider's sign-in opened under its row: the page scrolled and the step label slid under the title bar. The window opens at 1000x800, brought within the monitor's work area on launch, and the setup page asks the shell to grow it whenever a step is taller than the window. fit_setup_window only grows, never past the work area, and leaves a maximized or full-screen window alone; past that the page scrolls as before. The page gets no core:window permission.
…ndow The root size goes from 14px to 16px; everything else is in rem, so type, column width and spacing scale together. The answer card's fixed px sizes become rem so it scales with them.
…n only Upstream's self-hosted sign-in sets signingIn too and says its own line; both showed at once.
… shared or toolkit manifest #784 added shared and toolkit to the packages whose manifests preparation reads, but the latest release, v0.2.0, predates it: shared is plain source and there is no toolkit. A desktop built from main failed every new install with 'shared/package.json: No such file or directory'. Those two are now read only when the release has them; the four every release has stay required.
…to choose Sam's call: one fewer decision before anything is installed. With a CLI the first screen is Install, which sets up OpenBot and Intelligence on this computer; the hosted choice and the 'hosted instead' link are gone from it. The hosted way out after a failed local setup stays.
After a Claude code is accepted, the ChatGPT redirect lands, or a Gemini/xAI OAuth completes, the screen continues as Continue would. Checked once right after the sign-in, so a field filled in later never moves the screen on from under somebody typing; a saved sign-in and a typed key still wait for Continue.
…hen it is not; never Podman in its place With Docker Desktop installed but quit, setup found no answering engine and installed Podman behind an administrator prompt: a second engine on a machine that already had Docker, and one local Intelligence refuses. On macOS and Windows the engine is now Docker Desktop. Installed and stopped, it is opened and waited for. Missing, Docker's own installer (4.96.3, pinned to the digests Docker publishes) is downloaded with progress, installed without accepting Docker's terms, and opened so the person accepts them in Docker's window. A Podman somebody already has is still used; a stopped docker with no Docker Desktop (OrbStack, Colima) is left for its owner to start. Linux is unchanged.
…hoice moved on by itself
The local sequence starts OpenBot a tick after the go-ahead and read the model from the last render.
A Continue click has rendered by then; a sign-in that finished and continued from an effect may not
have, so with Intelligence already ready Start ran with model null ('invalid type: null, expected
struct ChosenModel'). The choice is now carried in a ref set when it is made, and the sequence reads
that. The Claude plan test again signs in only after Intelligence is ready.
…ing on '✓ Signed in to <provider>.' appears beside Continue, then the screen continues by itself after CONFIRMED_FOR_MS (1 s). Choosing another provider row in that second cancels the move.
…ad of repeating it The tick was a second 'Signed in to <provider>.' line under the first, so for one second the screen (and a screen reader) said it twice, and tests that look for the sentence failed whenever they looked inside that second. The tick now goes in front of the sentence already there.
…d CLI, CopilotKit sign-in and the agreement in the window A customer opens OpenBot and presses Install; nothing is done in Terminal. - A build made with OPENBOT_BUNDLED_CLI=1 carries the Intelligence CLI in its resources (Node and the CLI, tauri.bundled-cli.conf.json) and runs it pinned to the digests build.rs recorded, in a release build too. Other builds are unchanged. - When setup needs a CopilotKit account, the window offers Sign in to CopilotKit, runs 'login --json' (the CLI opens the approval page), and setup carries on by itself. - When setup needs the evaluation agreement, the window reads it with 'onboard local-terms --json' and shows the organization, a link to read it, and Accept. The click is the person's acceptance (product decision, 2026-10-22) and setup runs again with --accept-eula-version and that version; nothing accepts it without the click. The 'run this in Terminal' footnotes are gone. - The CLI gets Docker's own directory on its PATH: opened from Finder the app has the system's minimal PATH, which does not reach Docker Desktop's /usr/local/bin.
… longer than 30 seconds deployment::get read the body with bytes(), which the blocking client bounds by its 30-second default timeout and reports as 'error decoding response body' when that runs out. Any download taking longer than 30 seconds failed however steadily it arrived: Bun's 30 MB on a connection under about 1 MB/s, the Compose and Podman downloads, and the release lookup. Present since get() was written (#392). The body is now read as a stream, where the timeout bounds each read, so a download is given up only after 30 seconds with nothing arriving. A slow-server test fails with the old code's exact message and passes now.
…step of their own in a local first run Sign-in and Accept sat on the AI provider screen, under the Intelligence status line. A local first run is now four steps: install OpenBot, then CopilotKit (sign in, then read and accept the evaluation agreement), then choose the AI while Intelligence sets up behind it, then test. Nothing licensed starts before the account step finishes, and setup runs with the version accepted there. Somebody already signed in with the current agreement accepted passes the step without a click. The provider screen no longer offers sign-in; the Getting ready panel keeps it as a fallback for a session that ends later. step_viewed telemetry gains 'account'.
…ot's CLI calls like its own Quit does Quitting OpenBot from the Dock or with an AppleScript quit left a running 'local setup' behind, orphaned with its sign-in worker and an image pull; the next setup was refused as local_command_busy. Two causes. - Tauri on macOS answers a quit macOS starts only in applicationWillTerminate:, which ends the event loop at once, so RunEvent::ExitRequested, where OpenBot's Quit cleanup lives (CLI calls, containers, the Intelligence stack it made), never ran. The app delegate now also answers applicationShouldTerminate:, cancelling macOS's termination and asking for OpenBot's own Quit, whose cleanup then exits. Tauri leaves with stop: and process::exit, so it is not asked again. - close_for_quit only asked calls to cancel. The CLI forwards SIGTERM to its children instead of exiting, so its tree went only at SIGKILL after the grace, and an app that exited sooner left it running. close_for_quit now waits, up to 10 s, until every CLI process tree it started is gone. A CLI that ignores SIGTERM is tested: the test fails without the wait. The menu's Quit and its Cmd-Q already went through the cleanup and are unchanged.
…and Intelligence sets up alongside the install The order is now: CopilotKit (sign in, accept the agreement), Installing OpenBot, Connect your AI, then Getting ready and Test. The account needs nothing installed, so it comes first. Intelligence starts once the install reports the container engine ready (it runs on the same Docker, which the install starts or installs), not after the whole install, and its status shows on the install screen. Connect your AI follows the install because a plan sign-in runs in the image the install brings; Intelligence carries on behind it. set_up_local_eval creates the installation folder, which may not exist yet when it starts.
The six colour fields were discs under filter: blur(103px), which WebKit repaints whenever anything above changes; the spinners and elapsed counters change every frame and second, and the window felt laggy. They are now radial gradients with the same centres and soft edges. Buttons faded their hover colour over 150ms and had no pressed state, so nothing moved at the moment of a click. Pressed is now instant and the fades are 80ms.
… load Importing the .svg as a module worked in Vite but Bun's test runner has no loader for it, so App.test.tsx failed to load and 133 tests silently did not run. Served as a static file at /copilotkit-logo.svg instead.
The CopilotKit theme defines light only; the brand's verified dark context is near-black #010507 with the reversed wordmark. Dark builds on that as an implementation default: faint white glass, the verified grey scale for text, a light filled button, lilac and mint ambient fields, and the CopilotKit logotype with a white wordmark. Brand marks on provider rows keep their white ground in both themes. The macOS title bar takes the matching ground and follows a theme change while setup shows.
The setup window pins its own appearance to dark (SETUP_THEME), and the page follows the window through prefers-color-scheme, so both stylesheets stay and one constant chooses: None follows the system, Light pins light. The window starts dark; OpenBot gets the system's appearance back.
SETUP_THEME is None, so setup is light or dark with the system and both stylesheets stay. The window gets its setup ground for that appearance in setup, before the first page, instead of a fixed colour in the config.
…s you The install and AI provider screens showed Intelligence's progress as a second running line, which competed with the screen's own task and fitted awkwardly; its full progress has its own step. Those screens now show nothing while it works, and a card the moment it stops: lilac with the action when it waits on the person (sign in, accept), the failure dot with Retry when it failed.
"Step 1 of 4" over a screen that is itself two steps (sign in, then the agreement) read as a miscount. Setup now shows its sections by name, Account · Install · AI · Start for a local first run and Bot · Install · AI · Start for a hosted one, the current one in full ink and marked aria-current=step.
By the time it shows, OpenBot is up; "Test OpenBot" made the question sound like a check the person had to pass. It is now "Try OpenBot": OpenBot is up, ask your Bot something.
The shadcn tokens now carry the CopilotKit theme the desktop setup window uses: lavender-gray ground, white cards, near-black actions, a white-edged frosted sidebar over the theme's ambient colour fields, lilac focus, and brand chart colours; dark is the near-black set from setup. Plus Jakarta Sans replaces Inter and Spline Sans Mono is the mono face, both bundled with pinned @fontsource-variable packages. Theme only: no component is edited.
…ar was Over OpenBot served from this machine the window's title bar becomes an overlay with its title hidden, so the sidebar runs to the top edge with the window buttons over its corner and the name appears once. The desktop marks the page and grants that origin only window dragging, and the sidebar's header is the drag handle; the app's styles make room above the header (or above the main column with the sidebar collapsed). Setup keeps its transparent bar; any other page gets the system's.
OpenBot Status… replaces Stop OpenBot in the menu bar and opens its own window, so OpenBot stays on screen. It shows OpenBot (its host processes' resident memory and its containers'), the local Intelligence stack (its containers' memory and the CLI's readiness) and the container engine, with Restart and Stop for OpenBot, Restart, Stop and Start for Intelligence, where it is installed, the build, and Copy diagnostics. Memory is Docker's per-container figure, read every few seconds while the window shows. Intelligence's readiness is asked only on opening, Refresh and after an action: the main window's own Intelligence calls take the same lock, and polling it would make them fail as busy. Restart OpenBot ends its host processes and lets the existing supervisor start them again, so its containers and database keep running.
Laid out as Mac menu bar apps have theirs: "OpenBot is running" with the system's status dot, then Open OpenBot, the Intelligence dashboard and OpenBot Status… with thin geometric icons, then About, then Quit OpenBot. The application menu keeps the plain Mac order without icons. macOS does not draw these icons as templates, so each is drawn in the light and the dark menu's ink (icons/menu/draw.py, standard library only) and the menus are rebuilt when the appearance changes, and after a start or a stop so the status line is true. Menu rebuilding is skipped under the mock runtime, whose main-thread work runs on a worker where AppKit panics.
Built with the macOS 27 SDK, every image in the menu bar menu was hidden, the system's status dot and SF Symbols included, also in a bundled test app. macOS 27 added NSMenuItem.preferredImageVisibility, whose default lets AppKit hide images; each item now asks for Visible where the selector exists. Older SDK builds (Docker's) never saw the change.
Running / Not running, Show Window, Status…, About, Quit. The icon the menu opens from already says whose it is, the name is a per-build brand (OpenBot, Soar, a company's own), and a verb in front of some of those names reads badly: never "Open Soar".
About is the system's predefined item, which takes no icon through the menu API, so its image is set natively alongside the macOS 27 visibility opt-in, from the same light and dark pair as the other icons.
The frosted sidebar against the lavender ground already separates the two; the 1px border on top read as a bright line down the window in light.
This reverts commit 457a90594936b7246737dc278b15642e7d05d0d2.
With the Mac in dark and OpenBot set to light in its Settings, the window kept the dark appearance and macOS drew its dark-window outline, a black line, around a light window. OpenBot's page now sets the window's theme to its own whenever its dark class changes (granted only set_theme beside dragging); the desktop no longer resets it on that page. The menu bar's icon ink reads the system's appearance directly, since the main window no longer follows it. Also: the overlay title bar is used only when the installed OpenBot's styles make room for the window buttons. Releases before that change, v0.2.0 among them, keep the system's title bar, so the buttons never sit on the sidebar's name.
…Bot's theme This computer is the Status window's content inside OpenBot: what runs here and the memory each part takes, Restart and Stop for OpenBot, Start, Stop and Restart for local Intelligence, Docker, where it is installed, the build, and Copy diagnostics. Built from PageShell rows, read through a queryOptions factory and written through mutationOptions in lib/desktop. Shown only inside OpenBot Desktop; the menu's Status… opens it when the installed OpenBot has the page and falls back to the Status window. The desktop reaches the page through a new inline plugin, openbot-desktop, granted only to the owned app's origin. Its appearance command also fixes the black outline: the window plugin's set_theme, granted to that origin, was refused (the window stayed dark three seconds after load), so the page's light or dark now goes through this plugin instead.
Running meant 'this session of the desktop app holds OpenBot's processes', so an OpenBot left running by a session that ended without Quit (here, dev rebuilds restarting the app; for real, a crash) showed as Stopped with no actions while it was serving the very page. Running now means OpenBot answers on its own address; openbotHeld says whether this session holds the processes, and Restart, which is its supervisor bringing them back, shows only then. Stop works either way: it stops everything under the install.
It logged which credential fields arrived (never their values) for one demo's diagnosis and was marked to be removed after.
… thinned, memory at row size At the 15px root tuned for Inter's tall lowercase, Plus Jakarta Sans read a size down: row titles at 13px and memory figures at 11px. The root is 16px, the rem Tailwind documents. In the dark theme the root's grayscale antialiasing turned off macOS's thickening of light-on-dark text, so small text read thin; dark keeps the system's smoothing. This computer's memory figures are at the row's own size. Comments and the layout skill that quoted 15px figures say 16px, and keep the measure-don't-assume rule.
…cked Regular and medium are 450 and 550 (a variable font, so neither tips into bold): at 400 on the dark ground small text read thin. Tailwind's tight and tighter tracking, tuned for Inter, ran this wider face's letters together in titles and previews; they are -0.01em and -0.02em. Both set once in the theme, so the 22 places that use them follow.
… under the window buttons Hidden, the sidebar's toggle sat under the window buttons: the rule that moves the main column below them aimed at the primitive's SidebarInset, which no shell uses. It now moves the shell's own <main>, and the desktop marks that strip as the window's drag handle only while the sidebar is hidden. The sidebar's edge resizes it: drag to change the width, click to hide it as before (a press counts as a drag once it moves 3px). Each shell remembers its own width. It never goes narrower than its footer's widest button needs, measured from what is drawn when the drag starts, nor under 220px, nor wider than 480px, and it follows the pointer without easing while dragged.
The page was blank for about two seconds on arrival: every status read ran docker stats, which samples for two seconds before it reports. The page now reads twice: without memory, listing the containers with docker ps in milliseconds so every row draws at once, and with it, filling the figures in and repeating while the page shows. Totals appear only once memory is measured, since a total of the processes alone would be a smaller number that then jumps. No placeholder, as elsewhere in the app: a cell with nothing measured yet is empty.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
OpenBot Desktop sets up CopilotKit Intelligence on this computer next to OpenBot, from the app alone, and the setup window and OpenBot itself take on the CopilotKit visual system. 61 commits; squash on merge.
Local Intelligence, from the app alone
OPENBOT_BUNDLED_CLI=1) behind a bounded native boundary. Every call is checksum-pinned, enumerated and cancellable, and only failure categories reach the window.Setup window
Desktop chrome
preferredImageVisibility).openbot-desktopplugin granted only to the owned app's origin. The Status window is the fallback.App (
app/)lib/desktop/queries.tsandmutations.ts), shown only inside Desktop.Fixes along the way
Depends on
app/: the desktop downloads v0.2.0, which has none of the app-side changes, and degrades gracefully without them.Not in this PR
dustywusty/local-eval-windows, untested).Tests
worker_exhaustion_does_not_adopt_healthy_survivorswas skipped locally only because a running OpenBot held port 3010; it passes with the port free.tscand build clean.bun test: 1556 pass and 51 fail. The same 51 fail onmain(compared test by test); none are new.