Skip to content

Native: return rerank_oversample and fused_pq from VectorIndex.get_metadata when the engine has them - #89

Merged
tae898 merged 1 commit into
mainfrom
native/vector-metadata-rerank-fused-pq
Oct 26, 2026
Merged

tae898 merged 1 commit into
mainfrom
native/vector-metadata-rerank-fused-pq

Conversation

@tae898

@tae898 tae898 commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

What

VectorIndex.get_metadata() on the native build now returns rerank_oversample and fused_pq when the engine the image links has them (ArcadeDB 26.11.1, ArcadeData/arcadedb#10545). The JVM wrapper already returns them. This is backlog item N-05.

How

The native metadata op in NativeExtras.java lists a fixed set of fields. It now also reads rerankOversample and fusedPQ from LSMVectorIndex.toJSON(), which writes every knob the linked engine's LSMVectorIndexMetadata has. The glue names no field that 26.10.2 lacks, so it compiles and links against both engines (checked: mvn compile with -Darcadedb.version=26.10.2 and with 26.11.1-SNAPSHOT built from upstream main 9b7cd9a5eb, both clean). On 26.10.2 the two keys stay out, which _OPTIONAL_METADATA_KEYS in schema_objects.py already expects. No Python change is needed.

encoding, neighborOverflowFactor and alphaDiversityRelaxation are not added: the JVM get_metadata() does not return them either (the tests that read them go through the Java metadata object and stay NOTYET on native).

Why now

tests/test_vector_product_quantization.py probes the engine (has_pq_rerank) and asserts meta["rerank_oversample"] and meta["fused_pq"] once the engine accepts rerankOversample. On a 26.11.1 image without this change those assertions fail with a KeyError. On 26.10.2 CI nothing changes.

Test plan

  • mvn compile of the glue against 26.10.2 and against 26.11.1-SNAPSHOT (no errors, no warnings)
  • scripts/check_no_embedded_name.py --root .
  • Full native parity against an image linked with 26.11.1-SNAPSHOT (in flight on the laptop, result added as a comment)
  • CI on 26.10.2 (starts when the draft is marked ready)

🤖 Generated with Claude Code

…dex metadata when the engine has them

The native vector-index metadata op listed a fixed set of fields, so VectorIndex.get_metadata() never carried rerank_oversample and fused_pq on an image linked against ArcadeDB 26.11.1 (ArcadeData/arcadedb#10545), although the JVM wrapper returns them. The glue reads them from the index's own definition JSON (LSMVectorIndex.toJSON writes every knob the linked engine's metadata has), so it still compiles and links against 26.10.2, where the keys stay out, which the Python side already expects (_OPTIONAL_METADATA_KEYS).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@tae898

tae898 commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Parity of the native wheel built from this branch against engine 26.11.1-SNAPSHOT (upstream main 9b7cd9a5eb), full suite, wheel installed, one process per test file:

  • 1449 passed, 1 failed, 0 error, 0 notyet, 124 na_jvm, 0 skipped, 0 timeouts. The 26.10.2 reference is 1441 passed, 0 failed.
  • tests/test_vector_product_quantization.py passes in full, so the metadata keys this PR adds (rerank_oversample, fused_pq) are returned on 26.11.1.
  • The one failure is tests/test_python_function_interrupt.py::test_sigint_during_a_query_that_calls_python_is_a_keyboard_interrupt (assert interrupted >= 1, got 0: the eight 300k-row queries each ran to the end and no SIGINT surfaced). The Python glue of this wheel and of the 26.10.2-engine wheel is byte-identical, so the difference is in the engine or in the compiled Java glue. Whether it is a flake or a real behaviour change is not decided yet: a 20 + 20 interleaved rerun on both wheels and a thread diagnostic are queued on the laptop. This PR stays a draft until that is settled.
  • The first wheel build stopped at the notices check (THIRD_PARTY_NOTICES.md is out of date because six runtime jars moved version); the parity wheel was built with --skip-notices-check. Notices are regenerated on release day.

Details: .notes/MIGRATION.md, section "26.11.1 engine bump".

@tae898

tae898 commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Parity on engine 26.11.1-SNAPSHOT (laptop run): 1449 passed, 1 failed, 124 NA-JVM, 0 errors, 0 NOTYET, 0 timeouts (the 26.10.2 reference is 1441 / 0). The one failure, test_python_function_interrupt.py::test_sigint_during_a_query_that_calls_python_is_a_keyboard_interrupt, is not caused by the engine: on the same laptop, 20 interleaved runs per engine fail 20 of 20 on BOTH 26.10.2 and 26.11.1-SNAPSHOT, and a diagnostic shows every Python function call on the main thread on both engines and the query interrupted as designed once a SIGINT handler is installed explicitly. It passes in CI. The laptop-only cause is being traced separately; it does not block this PR.

@tae898
tae898 marked this pull request as ready for review October 26, 2026 07:30
@tae898
tae898 merged commit 43d9bca into main Oct 26, 2026
11 checks passed
@tae898
tae898 deleted the native/vector-metadata-rerank-fused-pq branch October 26, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant