CONTRIBUTING: document testing and commit/PR organization

CLAUDE.md

@@ -0,0 +1,28 @@
+# Repository instructions
+
+Vector Store is a Rust service that provides vector search for ScyllaDB.
+
+Before making changes, follow the project's contributor and coding guidelines:
+
+- **[CONTRIBUTING.md](CONTRIBUTING.md)** — pre-review checklist, commit and PR
+  organization (subject format, patch structure, Jira references), static checks,
+  how to run the tests (unit/integration, the validator harness, and the example
+  docker-compose stacks), CI expectations, and the OpenAPI workflow.
+- **[docs/rust_instructions.md](docs/rust_instructions.md)** — Rust coding
+  conventions and best practices used throughout the codebase.
+
+## Quick reference
+
+- Format and lint before committing, matching CI (warnings are errors):
+  ```sh
+  cargo fmt --all --check
+  cargo clippy --workspace --all-targets -- -Dwarnings
+  ```
+- Run unit and integration tests: `cargo test --workspace`
+  (`--workspace` is needed because `default-members` is only `crates/vector-store`).
+- Run the end-to-end validator harness: see the Testing section of
+  [CONTRIBUTING.md](CONTRIBUTING.md).
+- Do not hand-edit `api/openapi.json`; regenerate it with `cargo openapi`.
+- Organize commits and PRs per the Commit and PR Organization section of
+  [CONTRIBUTING.md](CONTRIBUTING.md) (subject format `module: changes`, small
+  self-contained patches, and `Fixes:`/`Refs: VECTOR-<n>` references).

CONTRIBUTING.md

@@ -8,14 +8,27 @@ Before submitting a pull request (PR), ensure your contribution meets these requ
 
 - **Single Change:** PR should consist of a single business change. If implementation requires multiple logical changes, split them into separate PRs. Each PR can contain multiple commits, but they should all relate to the same single logical change.
 - **Change Quality:** Each PR should introduce one logically coherent change. PR and commit messages must clearly describe what is being changed and why.
-- **Issue Closing Reference** If PR fixes/closes an issue, add a `Fixes: #XYZ` line at the end of PR. If you need you can also add it to the commit message.
-- **Issue References:** For related issues or discussions that are not directly fixed by the PR, add a `Reference: #XYZ` line at the end of PR message. If you need you can also add it to the commit message.
+- **Issue Closing Reference** If a PR fixes/closes a Jira issue, add a `Fixes: VECTOR-XYZ` line at the end of the PR. If you need you can also add it to the commit message.
+- **Issue References:** For related issues or discussions that are not directly fixed by the PR, add a `Refs: VECTOR-XYZ` line at the end of the PR message. If you need you can also add it to the commit message.
 - **Testing:** All new features and bug fixes must include appropriate tests.
 - **PR Checks:** Every PR must pass all CI checks, including compilation, test execution, formatting, and static code analysis.
 - **PR Description:** Clearly explain the motivation and reasoning behind your changes in the PR description.
 
 If you cannot meet any of these requirements, explain why in your PR description. Maintainers may grant exceptions if justified.
 
+## Commit and PR Organization
+
+Organize your changes into one or more small, self-contained commits. These conventions are inspired by the ScyllaDB [patch organization guidelines](https://github.com/scylladb/scylladb/blob/master/docs/dev/review-checklist.md), adapted to our workflow — note in particular that a Vector Store PR should introduce exactly one change (plus any adjustments it requires), as described in the Pre-Review Checklist above:
+
+- **Small patches:** Keep each patch small and focused — a patch should do one thing and be self-contained.
+- **Individually correct:** Every patch should compile and pass tests on its own. Order patches so the tree is always correct — for example, never place a regression test before the fix that makes it pass, or use a symbol before the patch that introduces it.
+- **Descriptive commit log:** The subject line has the form `module: changes`, where `module` is the directory/component the change applies to. List multiple modules joined with `,`, and use the `tree:` prefix for tree-wide changes. The prefix may be omitted for files in the root directory. Modules commonly used in this repo include `vector-store`, `validator`, `scripts`, `README`, and `chore`, but these are examples rather than a closed set — use whichever module best describes the change. Make the summary specific: prefer `sstables: close file-descriptor on error` over a generic `sstables: fix the bug`. Try to keep commit-log lines within ~72 characters (the subject may be longer when unavoidable).
+- **Self-contained description:** Write for a non-expert — include enough context to understand the change without being an expert on the matter, and do not refer to out-of-band discussions (e.g. "as agreed on the daily call").
+- **Motivation:** Every patch must explain *why*, not just *what*. A good litmus test: if the inverse patch ("make X not do Y") would be just as compelling, the description is not explaining enough.
+- **PR motivation:** Each PR must include an overall motivation. For a multi-patch series put it in the cover letter (PR description); for a single patch, the patch description is enough.
+
+Reference the Jira issue at the end of the commit/PR message: `Fixes: VECTOR-<n>` if the change closes the issue, or `Refs: VECTOR-<n>` if it is only related.
+
 ## Review and Merging
 
 After submitting a compliant PR, it will be reviewed by one or more maintainers. Once approved, a maintainer will merge your contribution.
@@ -58,13 +71,120 @@ Static checks run automatically in CI. All warnings are treated as errors (`RUST
 
 ## Testing
 
-The project includes unit and integration tests to ensure correctness.
+The project includes unit tests, integration tests, and an end-to-end validator harness.
+
+### Unit and integration tests
 
 **Run all tests with verbose output:**
 ```sh
-cargo test --verbose
+cargo test --workspace --verbose
+```
+`--workspace` is required to cover every crate — the workspace's `default-members`
+is only `crates/vector-store`, so a plain `cargo test` skips the other crates.
+
+### Validator harness (end-to-end tests)
+
+The validator (`crates/validator`) is an end-to-end test suite that spins up its
+own ScyllaDB + Vector Store cluster and drives real CQL / Alternator workloads
+against it. This is the suite CI runs (see `.github/workflows/validator.yml`).
+
+Because the harness launches actual `scylla` and `vector-store` processes, it
+needs two release binaries plus a ScyllaDB image.
+
+**1. Build the binaries with the release toolchain:**
+```sh
+./scripts/run-with-release-toolchain cargo build --release --bin vector-store
+./scripts/run-with-release-toolchain cargo build --release --bin vector-search-validator -p vector-search-validator
+```
+
+Build with `run-with-release-toolchain` rather than a plain host `cargo build`.
+The script builds inside the pinned toolchain image and writes to
+`target/<arch>/release/`, producing binaries whose glibc is compatible with the
+ScyllaDB container that runs them. Host-built binaries may link a newer glibc and
+fail to start inside the image.
+
+`<arch>` is the `TARGETARCH` the script builds for; it defaults to `amd64`
+(so the path is `target/amd64/release/`). It must match the architecture of the
+ScyllaDB image you run the harness against — on an arm64 host running an arm64
+image, set it explicitly for both builds:
+```sh
+TARGETARCH=arm64 ./scripts/run-with-release-toolchain cargo build --release --bin vector-store
+TARGETARCH=arm64 ./scripts/run-with-release-toolchain cargo build --release --bin vector-search-validator -p vector-search-validator
+```
+
+**2. Run the harness against a ScyllaDB image:**
+```sh
+./scripts/run-validator-with-scylla-docker \
+    <scylla-image> \
+    target/<arch>/release/vector-store \
+    target/<arch>/release/vector-search-validator \
+    [filters...] --verbose
 ```
 
+- `<scylla-image>` is a ScyllaDB docker tag, e.g. `scylladb/scylla-nightly:latest`.
+  CI runs against `scylla-nightly`; some tests exercise features that are only
+  available there.
+- Filters select test cases using `<group>::<test>` syntax. Partial matches work,
+  and wrapping either side in double quotes forces an exact match. Omit filters to
+  run everything. For example, `cdc_direct::` runs every group whose name contains
+  `cdc_direct`; to match that group exactly, keep the double quotes intact through
+  the shell with single quotes: `'"cdc_direct"::'`.
+
+**List available test cases:**
+```sh
+./target/<arch>/release/vector-search-validator list
+```
+
+**Using a locally built ScyllaDB:**
+
+The docker image is convenient, but not required. To run the harness against a
+locally built `scylla` binary instead of an image, use
+`run-validator-with-scylla-unshare`:
+```sh
+./scripts/run-validator-with-scylla-unshare \
+    <path/to/scylla> \
+    target/<arch>/release/vector-store \
+    target/<arch>/release/vector-search-validator \
+    --scylla-default-conf <path/to/scylla.yaml> \
+    [filters...] --verbose
+```
+The script runs the harness as root inside a private network namespace (via
+`sudo unshare`), setting up the loopback addresses and DNS resolution it needs, so
+you do not have to configure any networking or grant extra privileges yourself
+(it will prompt for `sudo`). Point `--scylla-default-conf` at a ScyllaDB config
+file — the ScyllaDB source tree ships one at `conf/scylla.yaml`. Running against a
+local ScyllaDB build is especially useful when developing a feature that spans
+both ScyllaDB and Vector Store, where you want to test your own ScyllaDB changes
+rather than a published image.
+
+### Manual testing with the example docker-compose stacks
+
+For quick manual or functional testing — not a replacement for the automated
+harness — the `docs/examples/docker/` directory ships ready-to-run single-node
+stacks that pair ScyllaDB with Vector Store:
+
+- `docker-compose.yml` — CQL / native vector API (ScyllaDB on `9042`, Vector Store on `6080`).
+- `docker-compose-alternator.yml` — Alternator (DynamoDB-compatible) API on `8000`, with CQL still on `9042`.
+
+**Bring up the CQL stack and exercise it with the quick-start schema:**
+```sh
+docker compose -f docs/examples/docker/docker-compose.yml up -d --wait
+cqlsh -f docs/examples/quick-start.cql   # create a vector table + index and run an ANN query
+```
+`--wait` blocks until the ScyllaDB health check passes, so `cqlsh` does not race
+the database's startup.
+
+**Tear it down:**
+```sh
+docker compose -f docs/examples/docker/docker-compose.yml down
+```
+
+> These are single-node stacks intended for local development and exploration
+> only. They are not for production, and they do not replace the validator
+> harness that runs in CI. Create keyspaces with a replication factor of 1 on a
+> single node (note that `quick-start.cql` uses `replication_factor: 3`, which a
+> single node cannot fully satisfy).
+
 ## Continuous Integration (CI)
 
 We use GitHub Actions for CI, configured in `rust.yml`.