Bad Vector Fill Behavior: Docs Say Only Bad Elements Are Replaced, But Code Replaces the Entire Vector

[oqoqo-bot]

Documentation Gap

Documentation claims [1.0, NaN, 3.0] becomes [1.0, 0.0, 3.0] (element-wise) but code replaces the ENTIRE vector with [0.0, 0.0, 0.0] (whole-vector replacement).

Description

The docs incorrectly describe on_bad_vectors='fill' as element-wise NaN replacement — the actual code replaces the ENTIRE vector.

• Docs claim [1.0, NaN, 3.0] becomes [1.0, 0.0, 3.0] with fill_value=0.0, but the code at table.py:3177-3181 replaces the entire vector with [0.0, 0.0, 0.0] — the is_bad flag is per-vector, not per-element
• Users lose ALL valid elements in partially-bad vectors without knowing it
• Zero fill vectors cause downstream issues: undefined cosine similarity (division by zero) and L2 results clustering near the origin

How to Validate

• Read the on_bad_vectors='fill' documentation and compare the stated behavior with the actual code at table.py line 3177-3181.
• Read the _handle_bad_vector_column function in table.py to verify that bad vectors are detected and replaced as whole vectors, not element-wise.

Affected Files

• python/python/lancedb/table.py
• python/python/lancedb/db.py
• docs/tables/consistency.mdx

---

Created by Oqoqo

[prrao87]

Based on a chat with @westonpace, we agree that this is a bug in the source code.  If we're replacing the whole vector we shouldn't even have the feature (we should just be dropping the bad vectors at that point). In such a failing scenario (where there is a bad vector that cannot be ingested) the user would be better served by us dropping the whole vector and point to the failing record ID, loudly calling it out and potentially spilling the bad vector IDs via a summary to stdout.