Round-trip binary primary key values in row URLs (#2419)#2839
Open
zbl1998-sdjn wants to merge 1 commit into
Open
Round-trip binary primary key values in row URLs (#2419)#2839zbl1998-sdjn wants to merge 1 commit into
zbl1998-sdjn wants to merge 1 commit into
Conversation
path_from_row_pks encoded a bytes (BLOB) primary key with str(bit), which isn't reversible, and the row lookup decoded every URL component to text, so a row with a binary primary key produced a 404. Encode bytes losslessly with a new tilde_encode_bytes, add tilde_decode_to_bytes, and decode BLOB primary key components back to bytes when looking up the row (non-UTF-8 values shown as hex). Adds regression tests.
There was a problem hiding this comment.
Pull request overview
This PR aims to fix row URL generation and resolution for tables whose primary key includes binary (BLOB) values, so those rows can be linked to and fetched without 404s (issue #2419).
Changes:
- Encode
bytesprimary key components losslessly inpath_from_row_pks()via newtilde_encode_bytes(). - Add
tilde_decode_to_bytes()and re-implementtilde_decode()in terms of it; update row resolution to bind BLOB PK components asbytes. - Add regression tests covering binary PK URL round-trips and row page/API fetches.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
datasette/utils/__init__.py |
Adds byte-safe tilde encoding/decoding and updates PK SQL parameter construction to support BLOB PKs. |
datasette/app.py |
Updates row resolution to use the new row_sql_params_pks() return shape and decoding behavior. |
tests/test_utils.py |
Adds a unit regression test for round-tripping binary PK components through URL encoding. |
tests/test_api_write.py |
Adds an end-to-end regression test ensuring binary-PK rows are reachable via HTML and JSON endpoints. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @documented | ||
| def tilde_decode(s: str) -> str: | ||
| "Decodes a tilde-encoded string, so ``~2Ffoo~2Fbar`` -> ``/foo/bar``" | ||
| return tilde_decode_to_bytes(s).decode("utf-8") |
Comment on lines
+1475
to
+1497
| # BLOB primary keys need their query parameter as bytes so binary values | ||
| # match; the returned pk_values stay as text for display / identity. | ||
| blob_pks = set() | ||
| if not use_rowid: | ||
| blob_pks = { | ||
| column.name | ||
| for column in await db.table_column_details(table) | ||
| if column.name in pks and (column.type or "").upper() == "BLOB" | ||
| } | ||
| pk_values = [] | ||
| param_values = [] | ||
| for i, component in enumerate(pks_string.split(",")): | ||
| if i < len(pks) and pks[i] in blob_pks: | ||
| raw = tilde_decode_to_bytes(component) | ||
| param_values.append(raw) | ||
| try: | ||
| pk_values.append(raw.decode("utf-8")) | ||
| except UnicodeDecodeError: | ||
| pk_values.append(raw.hex()) | ||
| else: | ||
| value = tilde_decode(component) | ||
| param_values.append(value) | ||
| pk_values.append(value) |
Comment on lines
+2726
to
+2743
| db = ds_write.get_database("data") | ||
| await db.execute_write( | ||
| "create table binary_pk (id integer, term blob, primary key (id, term))" | ||
| ) | ||
| terms = (b"0thei'", b"\xff\x00abc") | ||
| for term in terms: | ||
| await db.execute_write( | ||
| "insert into binary_pk (id, term) values (?, ?)", [3, term] | ||
| ) | ||
| for term in terms: | ||
| path = path_from_row_pks( | ||
| {"id": 3, "term": term}, ["id", "term"], use_rowid=False | ||
| ) | ||
| html = await ds_write.client.get(f"/data/binary_pk/{path}") | ||
| assert html.status_code == 200, html.text | ||
| js = await ds_write.client.get(f"/data/binary_pk/{path}.json?_shape=array") | ||
| assert js.status_code == 200, js.text | ||
| assert len(js.json()) == 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2419.
A row whose primary key is a binary (BLOB) value 404s — e.g.
/content/releases_fts_idx/3,b~270thei~27.Root cause
path_from_row_pksencodes each key withtilde_encode(str(bit)). For abytesvalue,str(bit)produces"b'...'", which can't round-trip back to the original bytes.row_sql_params_pks) decodes every URL component to text, so even a correctly-encoded binary key is queried as a string and never matches the BLOB column.Fix
path_from_row_pksnow encodesbytesvalues losslessly via a newtilde_encode_bytes(tilde-encodes the raw bytes instead ofstr(bytes)).tilde_decode_to_bytesdecodes a tilde string back to raw bytes;tilde_decodeis nowtilde_decode_to_bytes(...).decode("utf-8").row_sql_params_pkslooks up the column types and decodes BLOB primary-key components back tobytesfor the query. The returnedpk_valuesstay as text for display (UTF-8 where possible, else hex), so template/JSON output is unchanged for non-binary keys.Verification
test_utils.py) and an end-to-end test (test_api_write.py) that a binary-PK row returns 200 (HTML + JSON), for both UTF-8-decodable and non-UTF-8 bytes.pytest tests/test_utils.pypasses (227 passed; the 3 failures are pre-existing Windows-onlytemporary_docker_directorypermission errors, unrelated to this change) and the row/table/pk page tests pass.blackis clean on all changed files.I picked the schema-type-aware approach to keep the change small and avoid a URL marker — happy to adjust if you'd prefer a different scheme for representing binary primary keys in URLs.