diff --git a/.gitignore b/.gitignore
index ab47174..7140dd1 100644
--- a/.gitignore
+++ b/.gitignore
@@ -92,4 +92,7 @@ coverage/
jacoco.exec
# AI Tools
-.claude
\ No newline at end of file
+.claude
+
+# specification file
+SPEC.md
\ No newline at end of file
diff --git a/README.md b/README.md
index dfb0a92..cf11669 100644
--- a/README.md
+++ b/README.md
@@ -1,21 +1,22 @@
# Endee - Java Vector Database Client
-Endee is a Java client for a local vector database designed for maximum speed and efficiency. This package provides type-safe operations, modern Java features, and optimized code for rapid Approximate Nearest Neighbor (ANN) searches on vector data.
+Endee is a Java client for the Endee vector database, designed for maximum speed and efficiency. This package provides type-safe operations, modern Java features, and optimized code for rapid Approximate Nearest Neighbor (ANN) searches on vector data.
## Key Features
- **Type Safe**: Full compile-time type checking with builder patterns
- **Fast ANN Searches**: Efficient similarity searches on vector data
-- **Multiple Distance Metrics**: Support for cosine, L2, and inner product distance metrics
-- **Hybrid Indexes**: Support for dense vectors, sparse vectors, and hybrid (dense + sparse) searches
-- **Metadata Support**: Attach and search with metadata and filters
+- **Multiple Distance Metrics**: Cosine, L2, and inner product
+- **Hybrid Indexes**: Dense + sparse (BM25 or default) vector search
+- **Metadata & Filters**: Attach and query metadata with flexible filter operators
+- **Typed Exceptions**: Specific exception types per HTTP error code
- **High Performance**: HTTP/2, MessagePack serialization, and DEFLATE compression
-- **Modern Java**: Requires Java 17+, uses modern APIs
+- **Modern Java**: Java 17+, uses modern APIs
## Requirements
- Java 17 or higher
-- Endee Local server running (see [Quick Start](https://docs.endee.io/quick-start))
+- Endee server running (see [Quick Start](https://docs.endee.io/quick-start))
## Installation
@@ -25,636 +26,609 @@ Endee is a Java client for a local vector database designed for maximum speed an
io.endee
endee-java-client
- 0.1.1
+ 1.0.0
```
### Gradle
```groovy
-implementation 'io.endee:endee-java-client:0.1.1'
+implementation 'io.endee:endee-java-client:1.0.0'
```
## Quick Start
### Initialize the Client
-The Endee client connects to your local server (defaults to `http://127.0.0.1:8080/api/v1`):
-
```java
import io.endee.client.Endee;
import io.endee.client.Index;
import io.endee.client.types.*;
-// Connect to local Endee server (defaults to localhost:8080)
+// Local server (defaults to http://127.0.0.1:8080/api/v1)
Endee client = new Endee();
-```
-
-**Using Authentication?** If your server has `NDD_AUTH_TOKEN` set, pass the token when initializing:
-
-```java
-// With Auth Token
-Endee client = new Endee("auth-token");
-```
-
-### Setting a Custom Base URL
-If your server runs on a different port, use `setBaseUrl()`:
+// With an auth token
+Endee client = new Endee("account:password");
-```java
-Endee client = new Endee();
+// With a region (connects to https://{region}.endee.io/api/v1)
+Endee client = new Endee("account:password:us-east-1");
-// Set custom base URL for non-default port
+// Custom base URL
client.setBaseUrl("http://0.0.0.0:8081/api/v1");
```
+---
+
+## Index Management
+
### Create a Dense Index
```java
-import io.endee.client.types.CreateIndexOptions;
-import io.endee.client.types.Precision;
-import io.endee.client.types.SpaceType;
-
CreateIndexOptions options = CreateIndexOptions.builder("my_vectors", 384)
.spaceType(SpaceType.COSINE)
- .precision(Precision.INT16)
+ .precision(Precision.INT8)
+ .m(16)
+ .efCon(128)
.build();
client.createIndex(options);
```
-**Dense Index Parameters:**
+**Parameters:**
-| Parameter | Description | Default |
-| ----------- | ---------------------------------------------------------------------------- | -------- |
-| `name` | Unique name for your index (alphanumeric + underscore, max 48 chars) | Required |
-| `dimension` | Vector dimensionality (must match your embedding model's output, max 10,000) | Required |
-| `spaceType` | Distance metric - `COSINE`, `L2`, or `IP` (inner product) | `COSINE` |
-| `m` | Graph connectivity - higher values increase recall but use more memory | 16 |
-| `efCon` | Construction-time parameter - higher values improve index quality | 128 |
-| `precision` | Quantization precision | `INT16` |
+| Parameter | Description | Default | Constraints |
+|-------------|-----------------------------------------------------------------------|----------|----------------------|
+| `name` | Unique index name (alphanumeric + underscore) | required | max 48 chars |
+| `dimension` | Vector dimensionality (must match your embedding model) | required | 2 – 8,000 |
+| `spaceType` | Distance metric — `COSINE`, `L2`, `IP` | `COSINE` | — |
+| `m` | HNSW graph connectivity — higher = better recall, more memory | `16` | > 0 |
+| `efCon` | HNSW construction quality — higher = better index, slower build | `128` | > 0 |
+| `precision` | Quantization level | `INT8` | see Precision section |
### Create a Hybrid Index
-Hybrid indexes combine dense vector search with sparse vector search. Add the `sparseDimension` parameter:
+Hybrid indexes support both dense and sparse vectors. Set `sparseModel` to enable sparse search:
```java
+// Standard sparse search
CreateIndexOptions options = CreateIndexOptions.builder("hybrid_index", 384)
- .sparseDimension(30000) // Sparse vector dimension (vocabulary size)
.spaceType(SpaceType.COSINE)
- .precision(Precision.INT16)
+ .precision(Precision.INT8)
+ .sparseModel("default") // or "endee_bm25" for BM25 scoring
.build();
client.createIndex(options);
```
-### List and Access Indexes
+**`sparseModel` values:**
+
+| Value | Description |
+|----------------|-------------------------------------------------|
+| `"default"` | Standard sparse search without server-side IDF |
+| `"endee_bm25"` | BM25 scoring with server-side IDF |
+| `null` | Dense-only index (omit `sparseModel` entirely) |
+
+### List, Get, and Delete Indexes
```java
-// List all indexes (returns JSON string)
+// List all indexes (returns raw JSON string)
String indexes = client.listIndexes();
-// Get reference to an existing index
+// Get a reference to an existing index
Index index = client.getIndex("my_vectors");
-// Delete an index
+// Delete an index (irreversible)
client.deleteIndex("my_vectors");
```
+---
+
## Upserting Vectors
-The `index.upsert()` method adds or updates vectors in an existing index.
+### Dense Vectors
```java
-import io.endee.client.types.VectorItem;
-import java.util.List;
-import java.util.Map;
-
Index index = client.getIndex("my_index");
List vectors = List.of(
VectorItem.builder("vec1", new double[] {0.1, 0.2, 0.3 /* ... */})
- .meta(Map.of("title", "First document", "group" , 10)) // meta : {"title" : "First Document", "label" : 10 }
- .filter(Map.of("category", "tech", "group" , 10)) // filter : {"category" : "tech" , "group" : 10}
+ .meta(Map.of("title", "First document", "score", 95))
+ .filter(Map.of("category", "tech", "group", 1))
.build(),
- VectorItem.builder("vec2", new double[] {0.3, 0.4, 0.5 /* ... */})
- .meta(Map.of("title", "Second document"))
- .filter(Map.of("category", "science"))
+ VectorItem.builder("vec2", new double[] {0.4, 0.5, 0.6 /* ... */})
+ .meta(Map.of("title", "Second document", "score", 80))
+ .filter(Map.of("category", "science", "group", 2))
+ .build()
+);
+
+index.upsert(vectors);
+```
+
+### Hybrid Vectors
+
+For hybrid indexes, every upserted vector must supply both sparse fields:
+
+```java
+List vectors = List.of(
+ VectorItem.builder("doc1", new double[] {0.1, 0.2 /* ... */})
+ .sparseIndices(new int[] {10, 50, 200}) // non-zero term positions
+ .sparseValues(new double[] {0.8, 0.5, 0.3}) // weight for each position
+ .meta(Map.of("title", "Document 1"))
+ .filter(Map.of("category", "tech"))
.build()
);
index.upsert(vectors);
```
-**VectorItem Fields:**
+**`VectorItem` fields:**
-| Field | Required | Description |
-| -------- | -------- | --------------------------------------------------- |
-| `id` | Yes | Unique identifier for the vector (non-empty string) |
-| `vector` | Yes | Array of doubles representing the embedding |
-| `meta` | No | Arbitrary metadata map |
-| `filter` | No | Key-value pairs for filtering during queries |
+| Field | Required | Description |
+|-----------------|---------------|--------------------------------------------------------------|
+| `id` | Yes | Unique non-empty string identifier |
+| `vector` | Yes | Dense embedding (length must equal index `dimension`) |
+| `meta` | No | Arbitrary metadata `Map` — stored compressed, not filterable |
+| `filter` | No | Key-value pairs used for filtered queries |
+| `sparseIndices` | Hybrid only | Non-zero term positions in the sparse vector |
+| `sparseValues` | Hybrid only | Weight for each sparse index (same length as `sparseIndices`)|
**Limits:**
+- 1 – 1,000 vectors per `upsert` call
+- IDs must be unique within a batch
+- Vector values must be finite (no `NaN` or `Inf`)
-- Maximum 1,000 vectors per upsert call
-- Vector dimension must match index dimension
-- IDs must be unique within a single upsert batch
+---
-## Querying the Index
+## Querying
-The `index.query()` method performs a similarity search.
+### Basic Dense Query
```java
-import io.endee.client.types.QueryOptions;
-import io.endee.client.types.QueryResult;
-
List results = index.query(
QueryOptions.builder()
.vector(new double[] {0.15, 0.25 /* ... */})
.topK(5)
- .ef(128)
- .includeVectors(true)
.build()
);
for (QueryResult item : results) {
System.out.println("ID: " + item.getId());
System.out.println("Similarity: " + item.getSimilarity());
- System.out.println("Distance: " + item.getDistance());
+ System.out.println("Distance: " + item.getDistance()); // 1 - similarity
System.out.println("Meta: " + item.getMeta());
+ System.out.println("Vector: " + Arrays.toString(item.getVector())); // empty unless includeVectors=true
}
```
-**Query Parameters:**
-
-| Parameter | Description | Default | Max |
-| -------------------------------- | ------------------------------------------------------- | ------- | --------- |
-| `vector` | Query vector (must match index dimension) | Required | - |
-| `topK` | Number of results to return | 10 | 512 |
-| `ef` | Search quality parameter - higher values improve recall | 128 | 1024 |
-| `includeVectors` | Include vector data in results | false | - |
-| `prefilterCardinalityThreshold` | Switch to postfiltering when estimated matches exceed this value | 10,000 | 1,000,000 |
-| `filterBoostPercentage` | Bias results toward filter matches (0 = disabled) | 0 | 100 |
+### Filtered Query
-## Filtered Querying
-
-Use the `filter` parameter to restrict results. All filters are combined with **logical AND**.
+All filter conditions are combined with **logical AND**:
```java
List results = index.query(
QueryOptions.builder()
.vector(new double[] {0.15, 0.25 /* ... */})
- .topK(5)
+ .topK(10)
.filter(List.of(
Map.of("category", Map.of("$eq", "tech")),
- Map.of("score", Map.of("$range", List.of(80, 100)))
+ Map.of("score", Map.of("$range", List.of(80, 100)))
))
.build()
);
```
-### Filtering Operators
-
-| Operator | Description | Example |
-| -------- | ------------------------- | ---------------------------------------------------- |
-| `$eq` | Exact match | `Map.of("status", Map.of("$eq", "published"))` |
-| `$in` | Match any in list | `Map.of("tags", Map.of("$in", List.of("ai", "ml")))` |
-| `$range` | Numeric range (inclusive) | `Map.of("score", Map.of("$range", List.of(70, 95)))` |
+**Filter operators:**
-> **Note:** The `$range` operator supports values within **[0 - 999]**. Normalize larger values before upserting.
+| Operator | Description | Example |
+|-----------|---------------------------|-------------------------------------------------------|
+| `$eq` | Exact match | `Map.of("status", Map.of("$eq", "published"))` |
+| `$in` | Match any value in list | `Map.of("tags", Map.of("$in", List.of("ai", "ml")))` |
+| `$range` | Numeric range (inclusive) | `Map.of("score", Map.of("$range", List.of(70, 95)))` |
-### Filter Params
+> `$range` supports integer values in **[0, 999]**. Normalize larger values before upserting.
-Use `prefilterCardinalityThreshold` and `filterBoostPercentage` to fine-tune how filtering interacts with the ANN search:
+### Hybrid Query
```java
List results = index.query(
QueryOptions.builder()
- .vector(new double[] {0.15, 0.25 /* ... */})
+ .vector(new double[] {0.15, 0.25 /* ... */}) // dense component
+ .sparseIndices(new int[] {10, 100, 300}) // sparse query positions
+ .sparseValues(new double[] {0.7, 0.5, 0.4}) // sparse query weights
.topK(5)
- .filter(List.of(
- Map.of("category", Map.of("$eq", "tech"))
- ))
- .prefilterCardinalityThreshold(50_000) // Use postfilter when >50k vectors match
- .filterBoostPercentage(20) // Bias 20% toward filter-matching vectors
+ .denseRrfWeight(0.7) // weight for the dense component in RRF fusion (0.0–1.0)
+ .rrfRankConstant(60) // RRF rank constant (default 60)
.build()
);
```
-| Parameter | Description | Default | Range |
-| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------- | ----------------- |
-| `prefilterCardinalityThreshold` | When the estimated number of vectors matching the filter exceeds this value, postfiltering is used instead of prefiltering. | 10,000 | 1,000–1,000,000 |
-| `filterBoostPercentage` | Percentage by which filter-matching vectors are boosted during scoring. Set to `0` to disable. Higher values favor filtered results. | 0 | 0–100 |
-
-## Hybrid Search
+You can also query with only dense (`vector`) or only sparse (`sparseIndices` + `sparseValues`).
-### Upserting Hybrid Vectors
-
-Provide both dense vectors and sparse representations:
+### All Query Options
```java
-Index index = client.getIndex("hybrid_index");
-
-List vectors = List.of(
- VectorItem.builder("doc1", new double[] {0.1, 0.2 /* ... */})
- .sparseIndices(new int[] {10, 50, 200}) // Non-zero term positions
- .sparseValues(new double[] {0.8, 0.5, 0.3}) // Weights for each position
- .meta(Map.of("title", "Document 1"))
- .build(),
-
- VectorItem.builder("doc2", new double[] {0.3, 0.4 /* ... */})
- .sparseIndices(new int[] {15, 100, 500})
- .sparseValues(new double[] {0.9, 0.4, 0.6})
- .meta(Map.of("title", "Document 2"))
- .build()
-);
-
-index.upsert(vectors);
+QueryOptions.builder()
+ .vector(double[]) // dense query vector
+ .topK(int) // results to return (default: 10, max: 4,096)
+ .ef(int) // HNSW search depth (default: 128, max: 1,024)
+ .filter(List