feat(knowledge): import knowledge base bundles through the API and `sim knowledge import` by waleedlatif1 · Pull Request #9327 · simstudioai/sim

waleedlatif1 · 2026-10-23T01:11:15Z

Summary

Import is the other half of KB export: a .simkb.zip bundle becomes a new knowledge base with its description, chunking config, tag definitions, tag values, enabled flags, and original files
New v2 resource knowledge/imports (create → part URLs → complete, plus get and cancel) over a new knowledge_import upload purpose; the bundle is uploaded straight to storage like other large uploads
A background runner (Trigger.dev task, in-process fallback) reads the bundle from a temp file with yauzl and validates the whole manifest and every entry before writing anything; it then creates the base and imports documents two at a time
When the bundle's vectors match this deployment's embedding model, documents are indexed directly from the bundle's chunks and vectors (no re-embedding, nothing billed). Otherwise documents are processed from their original files exactly like uploads. Done through a narrow preparedIndex seam in processDocumentAsync, so the claim, provenance, index swap, completion, and billing paths stay canonical
A document the bundle carries only as chunks is stored as a plain-text file of its chunk text, so a later Retry still works
knowledge_base_import job table (migration 0436, additive) with a guarded running → completed | failed | canceled state machine; the stale-execution janitor fails imports whose worker died and deletes their bundle
Gated by knowledge.create on every step plus knowledge.upload at create and complete; one knowledge_base.imported audit event per import, naming the new base
CLI: sim knowledge import <bundle.simkb.zip> [--name] [--no-wait], sim knowledge imports get/cancel; tables import now shares the same poller
Docs: the export page is now "Export and import", with the CLI/API flow, vector reuse rules, and limits
New dispatchBackgroundTask helper (Trigger.dev in the resolved region, or detached in-process) replaces the copy-pasted dispatch blocks in table import/export/backfill/delete and the sandbox image build

Type of Change

New feature

Testing

bundle-import.integration.ts against real Postgres + local storage: real export → import round trip (reused vectors, tags, enabled flags, no embedding calls, upload guard claimed, bundle deleted), model-mismatch routing, bad archive, name conflict, create → complete twice → background run → single audit row, stale-import janitor. Mutation-checked: disabling vector reuse, chunk enabled propagation, the upload-guard claim, or the janitor sweep each turns it red
import-archive.test.ts (refusal modes for untrusted bundles + round trip), background-task.test.ts, knowledge-import.test.ts (CLI), plus existing table import/export/backfill/delete, sandbox image registry, upload-session, storage-sweep, CLI contract, and copilot route inventory tests
Migration applied with db:migrate on a fresh pgvector database; replay is a no-op; check:migrations passes
Regenerated and checked OpenAPI, CLI API, MCP operations, v2 route table, CLI docs, and docs manifest; ran check:openapi, check:spec-example-ids, check:permission-group-enforcement, check:api-validation:strict, check:boundaries, check:test-patterns, and the other source audits individually. Full type-check and check:audits run in CI

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…im knowledge import` - v2 knowledge imports resource (create, part URLs, complete, get, cancel) over a new knowledge_import upload purpose - background runner recreates the base, tags, and documents from an exported bundle; reuses vectors when the embedding model matches and queues re-processing otherwise - knowledge_base_import job table with a stale-import janitor sweep - `sim knowledge import <bundle.simkb.zip>` plus `knowledge imports get/cancel`, and an import guide - shared dispatchBackgroundTask helper for Trigger.dev-or-in-process task dispatch

vercel · 2026-10-23T01:11:19Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 23, 2026 2:24am UTC

cubic-dev-ai

2 issues found across 67 files

Confidence score: 3/5

imports/route.ts exposes cross-tenant authorization failures, letting callers distinguish inaccessible workspace or import scopes. Use concealImportAuthorization while preserving the intended authorization behavior.
upload-session/service.ts can leave a completed knowledge_import upload orphaned if the process fails before durable import creation. Add cleanup or recovery for completed sessions.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/app/api/v2/knowledge/imports/route.ts">

<violation number="1" location="apps/sim/app/api/v2/knowledge/imports/route.ts:16">
P2: This create endpoint exposes cross-tenant authorization failures instead of concealing them. Use `concealImportAuthorization` so callers cannot distinguish an inaccessible workspace/import scope, while retaining the usage-limit rendering.</violation>
</file>

<file name="apps/sim/lib/uploads/upload-session/service.ts">

<violation number="1" location="apps/sim/lib/uploads/upload-session/service.ts:229">
P2: A completed knowledge-import upload can become an orphaned object if the process fails between upload completion and durable import creation. Make cleanup or recovery recognize completed `knowledge_import` sessions without a corresponding import record and delete or resume their source object; otherwise reads remain `uploading` and storage is retained indefinitely.</violation>
</file>

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-23T01:18:13Z

Confidence Score: 5/5

[High impact] The changes since the previous review appear safe to merge.

Summary

Adds knowledge-base bundle imports through the API and sim knowledge import.

Knowledge imports create a new base from an uploaded bundle.
Imported bundles keep their knowledge-base settings, tags, and documents.
The CLI and API clients can start, follow, and cancel bundle imports.
Table and sandbox jobs share one background-task dispatcher.

No new actionable issues were found in the changes since the previous review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[CLI or API caller] --> B[Create import upload]
  B --> C[Upload bundle to storage]
  C --> D[Complete upload and save import record]
  D --> E[Dispatch worker]
  D --> F[Record import-started event once]
  E --> G[Claim import and validate bundle]
  G --> H[Create knowledge base]
  H --> I{Compatible vectors?}
  I -->|Yes| J[Index saved chunks and vectors]
  I -->|No| K[Process files or chunk text]
  J --> L[Save outcome and delete bundle]
  K --> L

_{Reviews (6) · Last reviewed commit: "fix(knowledge): audit an import as start..." · Reviewed by Greptile}

…and cleanup gaps, and fix CI - read every chunk entry before the base is created, reject reversed offsets, and length-check vectors before decoding - confirm the import is still running before creating the base, and remove it if a cancel wins - record the import inside upload completion, so a refusal or crash cannot strand the bundle; re-read on a concurrent completion - serialize progress writes, keep the source character count, cap rebuilt text, and recheck uploads when issuing part URLs - janitor fails in-process documents a dead worker left processing, and deletes bundles when pruning settled imports - shrink the unused-exports baseline, count the new janitor passes in its test, and fix the integration test's Blob typing

waleedlatif1 · 2026-10-23T01:31:40Z

@greptile

waleedlatif1 · 2026-10-23T01:31:41Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-23T01:31:47Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 69 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

…and remove a base whose tag setup fails - the base takes the import's id and a worker claims the import before working, so a second dispatch exits untouched - a retried completion dispatches again an import no worker has claimed - a worker that finds its import settled before any claim deletes the unused bundle - tag-definition setup that fails removes the still-empty base

waleedlatif1 · 2026-10-23T01:44:13Z

@greptile

waleedlatif1 · 2026-10-23T01:44:15Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-23T01:44:20Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 69 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

…udit each import once, and close review gaps - a dispatch never settles the import itself; a retry or the stale-import janitor does - projected audit entries take an idempotency key, recorded through recordAuditOnce, so a retried completion audits the import exactly once - a cancel that races completion stops the record the completion wrote - refuse empty rebuilt chunk-only documents; knowledge_base_import.user_id references the user

waleedlatif1 · 2026-10-23T01:59:04Z

@greptile

waleedlatif1 · 2026-10-23T01:59:06Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-23T01:59:13Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 70 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

…heck uploads on re-dispatch - knowledge_base_import.user_id is attribution only, so the janitor's cleanup record outlives its importer's account - re-dispatching an unclaimed import repeats the knowledge.upload assertion

waleedlatif1 · 2026-10-23T02:09:50Z

@greptile

waleedlatif1 · 2026-10-23T02:09:51Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-23T02:09:57Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 70 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

…spatch test - the completion records knowledge_base.import_started; the base exists only once the runner creates it - background-task.test.ts only checked forwarded mock arguments; the delete-async route test and the integration suite exercise the helper's real paths

waleedlatif1 · 2026-10-23T02:20:18Z

@greptile

waleedlatif1 · 2026-10-23T02:20:19Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-23T02:20:25Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 69 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 requested a review from a team as a code owner October 23, 2026 01:11

vercel Bot deployed to Preview October 23, 2026 01:15

View deployment