Fix PyLucene sidecar packaging and add smoke tests by nvzm123 · Pull Request #174 · NVIDIA/cuvs-lucene

nvzm123 · 2026-07-08T22:50:37Z

Summary

This PR fixes the PyLucene sidecar packaging and expands the PyLucene smoke coverage into a pytest-backed CPU/GPU end-to-end suite.

The sidecar jar now keeps PyLucene responsible for Lucene classes while cuvs-lucene contributes only its own classes and service providers. The smoke wrapper validates that layout before running the Python tests, so packaging regressions fail early with a clear message.

The expanded PyLucene suite covers the main index/search paths we expect users to exercise:

GPU CAGRA index build and CAGRA search
GPU CAGRA index build converted to HNSW search
CPU HNSW fallback
one-layer and three-layer CAGRA-to-HNSW configurations
single-vector edge cases
1-segment indexes, 10-segment indexes, 10 segments force-merged to 1, and 100 segments force-merged to 10
missing-vector documents, deletions, filtered KNN, vector metadata, expected index files, and force-merge topology

The expanded suite prints the path exercised by each case, such as path=gpu-cagra, path=gpu-hnsw, or path=cpu-hnsw-fallback, so the test output makes it clear whether each case used the intended CPU or GPU path.

Notable Changes

Adds a pytest entrypoint for the PyLucene smoke suite.
Adds --cases, --rows, --dims, and --topk options to ci/test_pylucene_smoke.sh.
Adds named one-layer and three-layer CAGRA-to-HNSW codecs for explicit PyLucene coverage.
Adds lightweight writer-path telemetry used by the smoke suite to assert which indexing path was used.
Tightens Lucene delegate-codec selection so cuvs-lucene only delegates to supported Lucene codec versions.
Documents the expected PyLucene/cuvs-lucene/cuvs-java classpath layout.

Running the Suite

From a PyLucene environment on a GPU machine, the standard full run is:

./ci/test_pylucene_smoke.sh --gpu-e2e --cases=all

That command builds the cuvs-lucene sidecar jar when needed, locates the base cuvs-java jar, validates the sidecar contents, sets the required environment variables, and then runs the pytest suite.

If the sidecar jar has already been built, the same suite can be run without rebuilding:

./ci/test_pylucene_smoke.sh --gpu-e2e --cases=all --no-build

With --no-build, the wrapper expects an existing sidecar jar under target/ or via CUVS_LUCENE_PYLUCENE_JAR. It does not build a missing jar.

After setup, the wrapper invokes:

python -m pytest -q -s examples/test_pylucene_smoke.py

pytest must be installed in the active PyLucene Python environment.

Validation

Validated on a g5.2xlarge EC2 instance with NVIDIA A10G:

built cuvs-lucene against cuVS 26.08
built/installed PyLucene 10.0.0
ran ./ci/test_pylucene_smoke.sh --gpu-e2e --cases=all --no-build
passed the expanded 20-case PyLucene matrix covering GPU CAGRA, GPU HNSW, forced CPU HNSW fallback, CAGRA-to-HNSW one-layer and three-layer cases, and segment/force-merge cases

Also checked locally:

bash -n ci/test_pylucene_smoke.sh
Python syntax compilation for the PyLucene smoke files
git diff --check

Local Maven validation was not run in this Codex workspace because mvn is not installed here.

copy-pr-bot · 2026-07-08T22:50:41Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

and finish updating cuvs version

Convert the PyLucene smoke runner to a pytest-backed case matrix covering GPU CAGRA search, CAGRA-built HNSW, and forced CPU HNSW fallback across segment and force-merge topologies. Add named one-layer and three-layer CAGRA-to-HNSW codecs plus writer-path telemetry so the e2e suite can assert which CPU/GPU path was exercised. Tighten sidecar and Lucene delegate-codec validation to avoid misleading expected probe warnings while still failing on unsupported delegate codecs.

imotov · 2026-07-17T00:14:52Z

  @Override
  public KnnVectorsWriter fieldsWriter(SegmentWriteState state) throws IOException {
    var flatWriter = FLAT_VECTORS_FORMAT.fieldsWriter(state);
+    System.setProperty(


These System.setProperty methods make sense in the context of this PR as a way to communicate between the Java and Python layers. However, they introduce unnecessary overhead for most other applications. Moreover, this approach would not work reliably in a multithreaded environment, since the same property is reused across requests. I think it would be better to separate the telemetry concerns and extract them into a dedicated method that is invoked only when telemetry data is actually needed.

Good point! Removed the JVM-global system properties from the indexing path. Telemetry is now computed on demand from the vectors format only when the PyLucene test requests it.

cjnolet · 2026-07-17T14:15:33Z

+To run the PyLucene pytest smoke suite against a local PyLucene environment:
+
+```sh
+./ci/test_pylucene_smoke.sh


I like that we have ci scripts for this , but a user should not have to invoke ci scripts in order to run tests. Rather, we should document how to run these pytests without the need to call scripts inside the CI directory while also providing the scripts in the CI directory for GitHub actions to run automatically.

@cjnolet I moved the user-facing logic to test_pylucene.sh and kept a thin wrapper under ci/. The current GitHub Actions workflows do not invoke it yet. Did you intend for this PR to add a PyLucene Actions job as well, or is providing the CI entry-point sufficient for now?

cjnolet · 2026-07-17T14:16:37Z

+./ci/test_pylucene_smoke.sh --gpu-e2e
+```
+
+The expanded suite runs the `gpu-basic`, `gpu-segments`, `cpu-hnsw`, and


Great description here. How to build and run tests should really be in a separate build and install guide. I think this is okay for now, especially since we are moving cuVS-Lucene to cuVS, but it's something to consider.

makes sense

cjnolet · 2026-07-17T14:24:46Z

+            .withCagraGraphBuildAlgo(CagraGraphBuildAlgo.NN_DESCENT)
+            .withGraphDegree(CAGRA_GRAPH_DEGREE)
+            .withIntermediateGraphDegree(CAGRA_INTERMEDIATE_GRAPH_DEGREE)
+            .withHNSWLayer(1)


Wasn't this an issue that you resolved? Why layer 1?

Yes, that issue was resolved. Layer 1 is intentional test coverage, paired with the three-layer case -- that said, I am not sure that these two particular tests will provide much utility

cjnolet and others added 3 commits May 13, 2026 16:23

Add PyLucene integration support

1c0f648

Fix benchmark version marker

e27a53b

Fix PyLucene sidecar packaging and update cuVS version

3cab384

nvzm123 requested review from a team as code owners July 8, 2026 22:50

nvzm123 requested a review from msarahan July 8, 2026 22:50

Merge remote-tracking branch 'upstream/main' into pr-147

6e3cf20

and finish updating cuvs version

nvzm123 force-pushed the pr-147 branch from 0b028cf to 6e3cf20 Compare July 8, 2026 23:50

nvzm123 marked this pull request as draft July 9, 2026 19:58

cjnolet mentioned this pull request Jul 15, 2026

Add PyLucene integration support #147

Closed

nvzm123 changed the title ~~Fix PyLucene sidecar packaging and add smoke test~~ Fix PyLucene sidecar packaging and add smoke tests Jul 16, 2026

Merge branch 'main' into pr-147

5ade7af

nvzm123 marked this pull request as ready for review July 16, 2026 20:19

imotov reviewed Jul 17, 2026

View reviewed changes

cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Jul 17, 2026

cjnolet assigned nvzm123 Jul 17, 2026

cjnolet added this to Unstructured Data Processing Jul 17, 2026

cjnolet moved this to In Progress in Unstructured Data Processing Jul 17, 2026

cjnolet requested changes Jul 17, 2026

View reviewed changes

nvzm123 added 6 commits July 21, 2026 03:44

Make writer telemetry on-demand

692c3db

Avoid duplicate binary format initialization

0e6bb43

Cache binary quantized vector formats

eeff876

Expose writer diagnostics through format descriptions

b76e2f5

Use Lucene 102 binary vector formats

bf23e24

Use the standard jar for PyLucene

f7a8c66

nvzm123 added 3 commits July 21, 2026 03:44

Add a public PyLucene test entrypoint

7084157

Move PyLucene tests under examples Python

d13184d

Clarify binary format version handling

df836cc

nvzm123 requested review from cjnolet and imotov July 22, 2026 03:04

imotov approved these changes Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix PyLucene sidecar packaging and add smoke tests#174

Fix PyLucene sidecar packaging and add smoke tests#174
nvzm123 wants to merge 15 commits into
NVIDIA:mainfrom
nvzm123:pr-147

nvzm123 commented Jul 8, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 8, 2026

Uh oh!

imotov Jul 17, 2026

Uh oh!

nvzm123 Jul 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cjnolet Jul 17, 2026

Uh oh!

nvzm123 Jul 21, 2026

Uh oh!

cjnolet Jul 17, 2026

Uh oh!

nvzm123 Jul 21, 2026

Uh oh!

Uh oh!

Uh oh!

cjnolet Jul 17, 2026

Uh oh!

nvzm123 Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

nvzm123 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notable Changes

Running the Suite

Validation

Uh oh!

copy-pr-bot Bot commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nvzm123 commented Jul 8, 2026 •

edited

Loading