Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
099c5dd
chore(docs): Add more guidance in AGENTS.md for accessing hyperd and …
StefanSteiner Jul 8, 2026
efc0e0d
docs: add design spec for native key-value store (M1 API + M2 MCP)
StefanSteiner Jul 8, 2026
07dcfa8
docs: add guidelines-compliance + Harness review sections to KV spec
StefanSteiner Jul 8, 2026
2cf8c7b
docs: fold DEVELOPMENT.md facts into KV spec (param facade, lifetimes…
StefanSteiner Jul 8, 2026
98cd06f
docs(kv): add M1 (Rust API) implementation plan
StefanSteiner Jul 9, 2026
5ca98e3
docs(kv): revise M1 plan from adversarial review
StefanSteiner Jul 9, 2026
94948e2
feat(kv): add Error::Serialization variant for get_as/set_as
StefanSteiner Jul 9, 2026
18d735c
feat(kv): add KV name validator and shared constants
StefanSteiner Jul 9, 2026
c8a4ffa
feat(kv): add KvStore scaffolding, Connection::kv_store + kv_list_stores
StefanSteiner Jul 9, 2026
219734c
feat(kv): add get/set upsert and serde get_as/set_as
StefanSteiner Jul 9, 2026
7a90549
feat(kv): add delete/exists/size/keys/clear
StefanSteiner Jul 9, 2026
45f7def
feat(kv): add transactional pop and set_batch
StefanSteiner Jul 9, 2026
f7080bb
test(kv): add compile-fail lifetime doc test for KvStore
StefanSteiner Jul 9, 2026
c32a308
feat(kv): add AsyncKvStore async twin
StefanSteiner Jul 9, 2026
f62ebb7
feat(kv): add KV write benchmark (single vs batched commit)
StefanSteiner Jul 9, 2026
52dcf61
docs(kv): document KV store in README, CHANGELOG, DEVELOPMENT
StefanSteiner Jul 9, 2026
eb758e9
docs(pool): fix broken intra-doc links in pool module docs
StefanSteiner Jul 9, 2026
7672928
test(kv): broaden invalid-key and store-isolation coverage
StefanSteiner Jul 9, 2026
e3d5aa1
docs(kv): list Kv types in crate docs and mirror bind comment
StefanSteiner Jul 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -419,3 +419,7 @@ All commit messages **must** follow the format `<type>(<scope>): <subject>` —
When reviewing existing code or fixing bugs, flag and convert any narrowing `as` casts you encounter, even if they aren't the proximate cause of the bug — they're a latent-corruption vector and cheap to fix in the same change.

8. **Update `CHANGELOG.md` for user-visible crate-level changes.** When a PR adds, changes, or removes any public API surface in a publishable crate (`hyperdb-api`, `hyperdb-api-core`, `hyperdb-api-node`, `hyperdb-api-salesforce`, `hyperdb-bootstrap`, `hyperdb-mcp`, `sea-query-hyperdb`), append a bullet to the `## [Unreleased]` section of that crate's `CHANGELOG.md` under the appropriate [Keep a Changelog](https://keepachangelog.com/) heading (`### Added`, `### Changed`, `### Deprecated`, `### Removed`, `### Fixed`, `### Security`). Internal refactors that don't change the public API surface do not require a changelog entry. The `## [Unreleased]` section is promoted to a dated `## [X.Y.Z] - YYYY-MM-DD` section by the maintainer at release time. See [CONTRIBUTING.md](CONTRIBUTING.md#authoring-changes-every-contributor) for the full policy.

9. **Never invent `hyperd` flags or engine parameters.** Obtain `hyperd` via `make download-hyperd` (it bootstraps the release pinned in `hyperdb-bootstrap/hyperd-version.toml`) and start servers through the documented path — `HyperProcess::new()` in tests, the Makefile targets, or `HYPERD_PATH` as described above. If you think a startup flag or parameter is needed, confirm it against `hyperd --help`, an existing script, or this file **before** relying on it. Fabricated `hyperd` parameters silently fail against the real binary — they have previously made tests hang while appearing to "run."

10. **Never report a test/build as passing without seeing real output.** Check exit codes. If a command produces no output for ~30s, treat it as **hanging/failed**, not passing, and say so explicitly. A green claim backed by no captured output is a defect, not a result — tests here start a real `hyperd` subprocess (`HyperProcess::drop()` stops it), so a misconfigured server hangs rather than erroring cleanly.
1,876 changes: 1,876 additions & 0 deletions docs/superpowers/plans/2026-07-08-kv-store-m1-api.md

Large diffs are not rendered by default.

467 changes: 467 additions & 0 deletions docs/superpowers/specs/2026-07-08-kv-store-design.md

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions hyperdb-api/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

## [Unreleased]

### Added

- Key-value store API: `Connection::kv_store` / `AsyncConnection::kv_store` returning
`KvStore` / `AsyncKvStore` handles over a fixed `_hyperdb_kv_store` table, with
`get`/`set`/`get_as`/`set_as`/`delete`/`exists`/`size`/`keys`/`pop`/`clear`/`set_batch`,
plus `kv_list_stores`. Adds the `Error::Serialization` variant.

## [0.1.1] - 2026-05-13

### Added
Expand Down
4 changes: 4 additions & 0 deletions hyperdb-api/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -131,3 +131,7 @@ path = "benches/async_parallel_benchmark.rs"
[[example]]
name = "benchmark_suite"
path = "benches/benchmark_suite.rs"

[[example]]
name = "kv_benchmark"
path = "benches/kv_benchmark.rs"
47 changes: 40 additions & 7 deletions hyperdb-api/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,13 +277,46 @@ cargo kani -p hyperdb-api

### Parameterized Queries

Hyper does not support PG's native extended query protocol. `query_params()`
and `command_params()` substitute `$1`/`$2` with safely escaped SQL literals
via `ToSqlParam::to_sql_literal()`, providing the same injection protection.
The `$1` syntax ensures forward-compatibility when Hyper adds server-side
prepared statements. `hyperdb-api-core::client` already implements the full extended
query protocol, so adding `Connection::prepare()` is wiring, not protocol work.
See rustdoc on `Connection::query_params()` in `connection.rs`.
`query_params()` and `command_params()` use the PostgreSQL extended query
protocol (Parse/Bind/Execute): each `$N` placeholder is bound to a
binary-encoded parameter via `ToSqlParam::sql_oid()` + `encode_param()`, routed
through `Connection::prepare_typed()`. Parameters are never interpolated into
the SQL text, so there is no injection surface. gRPC transport does not support
prepared statements and returns `Error::FeatureNotSupported`. See rustdoc on
`Connection::query_params()` in `connection.rs`.

### Key-Value Store

`KvStore` / `AsyncKvStore` (see `kv_store.rs` / `async_kv_store.rs`) are
string-native key-value handles over a single fixed backing table,
`_hyperdb_kv_store(store_name, key, value)`, namespaced by `store_name`. Design
points:

- **No `PRIMARY KEY`.** Hyper rejects one at `CREATE TABLE` (`0A000: Index
support is disabled`, empirically confirmed). Per-`(store_name, key)`
uniqueness is an application-side invariant, not an engine constraint. All DDL
flows through one `kv_create_table_sql()` helper so the sync/async twins can't
diverge.
- **Upsert = UPDATE-then-conditional-INSERT.** Hyper has no `ON CONFLICT`/`MERGE`.
`set` issues an `UPDATE`; if it affects 0 rows, a conditional
`INSERT ... SELECT ... WHERE NOT EXISTS` runs. hyperd serializes statements on
a connection, so this cannot double-insert. Mirrors the `_table_catalog` idiom.
- **`pop`/`set_batch` are transactional** via `begin_transaction_raw` /
`commit_raw` / `rollback_raw` (the `&self` escape hatch; the RAII
`Transaction` guard needs `&mut self`, incompatible with `KvStore`'s shared
`&'conn Connection` borrow). Rollback on error is best-effort and preserves
the original error, matching the RAII guard's commit-failure semantics.
- **`table_ref` seam.** `KvStore` stores a `table_ref`; the default constructor
uses the bare table name, and a crate-internal `with_target()` accepts a
qualified reference. `table_ref` is a trusted, construction-time value
interpolated into SQL; `store_name`/`key`/`value` are always bound `$N`
params. The seam is reserved for the MCP milestone (routing into an attached
database) so the M1 public API needs no later change.
- **Batching amortizes commits, not statements.** `set_batch` wraps N upserts in
one transaction. Because each upsert is still 2 prepared-statement
round-trips, the `kv_benchmark` example shows batched throughput on par with
single-commit (~1×) on localhost TCP — `set_batch`'s real value is
all-or-nothing atomicity, not raw speed.

### Callback Connection (Process Lifecycle)

Expand Down
37 changes: 37 additions & 0 deletions hyperdb-api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Hyper database files (`.hyper`) without any C library dependencies.
- 22-24M rows/sec inserts, 18M rows/sec queries (100M row benchmark)
- Streaming by default — constant memory for billion-row results
- Both sync (`Connection`) and async (`AsyncConnection`) APIs
- Built-in string-native key-value store (`KvStore` / `AsyncKvStore`)
- No feature flags on `hyperdb-api` — everything is always available (opt-in
compile-time SQL validation lives behind a feature on the separate
`hyperdb-api-derive` crate)
Expand Down Expand Up @@ -390,6 +391,42 @@ txn.commit()?; // both inserts are committed; drop without commit() auto-rolls-

See [docs/TRANSACTIONS.md](../docs/TRANSACTIONS.md) for full details.

## Key-Value Store

`Connection::kv_store` (and `AsyncConnection::kv_store`) opens a `KvStore`
handle — a string-native key-value store backed by a single fixed table,
namespaced by store name. Values are plain strings, or any `serde`-serializable
type via `set_as`/`get_as`.

```rust
use hyperdb_api::{Connection, CreateMode, HyperProcess, Result};

fn main() -> Result<()> {
let hyper = HyperProcess::new(None, None)?;
let conn = Connection::new(&hyper, "app.hyper", CreateMode::CreateIfNotExists)?;

let kv = conn.kv_store("settings")?;
kv.set("theme", "dark")?;
assert_eq!(kv.get("theme")?, Some("dark".to_string()));

// Typed values via serde (stored as JSON).
kv.set_as("retries", &3u32)?;
assert_eq!(kv.get_as::<u32>("retries")?, Some(3));

// Write many entries atomically in one transaction.
kv.set_batch(&[("host", "localhost"), ("port", "7483")])?;

// Other operations: exists, delete, size, keys, clear, and pop
// (remove-and-return the lowest-ordered key).
assert!(kv.exists("host")?);
let _ = conn.kv_list_stores()?; // names of all non-empty stores
Ok(())
}
```

Store names and keys must be non-empty, at most 512 bytes, and contain only
ASCII `A-Z a-z 0-9 _ . -`; anything else returns `Error::InvalidName`.

## Connection Features

### Query Cancellation
Expand Down
94 changes: 94 additions & 0 deletions hyperdb-api/benches/kv_benchmark.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
// Copyright (c) 2026, Salesforce, Inc. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 OR MIT

//! Key-value store write benchmark.
//!
//! Compares two write strategies against a real `hyperd`:
//! - single-commit-per-set: one `KvStore::set` per key (implicit commit)
//! - batched: `KvStore::set_batch` of `BATCH` keys per transaction
//!
//! Run with:
//! cargo run -p hyperdb-api --release --example kv_benchmark # default 50k keys
//! cargo run -p hyperdb-api --release --example kv_benchmark 200000 # 200k keys

// Throughput math converts a `usize` key count to `f64`; the resulting
// precision loss is irrelevant for a keys/sec figure. `allow` (not `expect`)
// because this is the only pedantic cast lint that fires here — an `expect`
// listing others would trip `unfulfilled_lint_expectations` under `-D warnings`.
#![allow(
clippy::cast_precision_loss,
reason = "benchmark throughput math needs usize -> f64; precision loss is cosmetic"
)]

use hyperdb_api::{Connection, CreateMode, HyperProcess, Result};
use std::env;
use std::time::Instant;

const DEFAULT_KEYS: usize = 50_000;
const BATCH: usize = 25; // within the requested 10-50 range

fn key_count() -> usize {
env::args()
.nth(1)
.and_then(|s| s.parse::<usize>().ok())
.unwrap_or(DEFAULT_KEYS)
}

fn throughput(label: &str, keys: usize, secs: f64) {
let per_sec = if secs > 0.0 { keys as f64 / secs } else { 0.0 };
println!(" {label:<28} {keys} keys in {secs:>7.3}s => {per_sec:>12.0} keys/sec");
}

fn bench_single(conn: &Connection, keys: usize) -> Result<f64> {
let kv = conn.kv_store("bench_single")?;
kv.clear()?;
let start = Instant::now();
for i in 0..keys {
kv.set(&format!("k{i}"), "value")?;
}
Ok(start.elapsed().as_secs_f64())
}

fn bench_batched(conn: &Connection, keys: usize) -> Result<f64> {
let kv = conn.kv_store("bench_batched")?;
kv.clear()?;
let start = Instant::now();
let mut i = 0;
while i < keys {
let end = (i + BATCH).min(keys);
// Own the strings, then borrow into the &[(&str, &str)] slice.
let owned: Vec<(String, String)> = (i..end)
.map(|n| (format!("k{n}"), "value".to_string()))
.collect();
let batch: Vec<(&str, &str)> = owned
.iter()
.map(|(k, v)| (k.as_str(), v.as_str()))
.collect();
kv.set_batch(&batch)?;
i = end;
}
Ok(start.elapsed().as_secs_f64())
}

fn main() -> Result<()> {
let keys = key_count();
println!("\n=== KV Store write benchmark ({keys} keys, batch size {BATCH}) ===");

let db_path = std::env::temp_dir().join("kv_benchmark.hyper");
let hyper = HyperProcess::new(None, None)?;
let conn = Connection::new(&hyper, &db_path, CreateMode::CreateAndReplace)?;

let single_secs = bench_single(&conn, keys)?;
throughput("single commit per set", keys, single_secs);

let batched_secs = bench_batched(&conn, keys)?;
throughput(&format!("batched ({BATCH}/txn)"), keys, batched_secs);

if batched_secs > 0.0 {
println!(
"\n speedup (batched vs single): {:.2}x",
single_secs / batched_secs
);
}
Ok(())
}
Loading
Loading