feat: pure-client safetensors/parquet metadata preview (#27)#28
Merged
narugo1992 merged 12 commits intodev/narugo1992from Apr 24, 2026
Merged
feat: pure-client safetensors/parquet metadata preview (#27)#28narugo1992 merged 12 commits intodev/narugo1992from
narugo1992 merged 12 commits intodev/narugo1992from
Conversation
Implements issue #27 v4: file-level HF-compatible metadata preview computed entirely in the browser via HTTP Range reads against the existing /resolve/ 302 → presigned S3/MinIO URL. Zero new backend preview code, zero LRU, zero precomputation, zero new DB state. Backend (minimal CORS plumbing only): - main.py CORSMiddleware: add `expose_headers` so browsers can read Content-Range / X-Linked-* / X-Repo-Commit / ETag / Location off the final 206 response that follows the /resolve/ 302. - docker-compose.example.yml + scripts/dev/up_infra.sh: wire `MINIO_API_CORS_ALLOW_ORIGIN` so the SPA can cross-origin Range-read presigned targets. Configurable via `DEV_MINIO_CORS_ALLOW_ORIGIN`. - docs/development/local-dev.md: MinIO CORS section explaining the hard prerequisite + smoke-test probe + how to recreate the container. Frontend: - utils/safetensors.js (~190 LOC): pure-JS parser mirroring huggingface_hub.parse_safetensors_file_metadata byte-for-byte (speculative 100 KB first read, two-read fallback for fat headers, SAFETENSORS_MAX_HEADER_LENGTH guard). Exposes parseSafetensorsMetadata + summarizeSafetensors. - utils/parquet.js: thin wrapper over hyparquet's asyncBufferFromUrl + parquetMetadataAsync with mode:"cors" + credentials:"omit" so cookies never leak onto presigned URLs. Normalizes BigInt row counts. - components/repo/preview/FilePreviewDialog.vue: ElDialog with per-phase spinner text (range-head → parsing → done for safetensors, head → footer → parsing → done for parquet), dtype/row-group tables, and an explicit "CORS likely misconfigured" placeholder on failure. - RepoViewer.vue: HF-style chart-line-data icon next to .safetensors and .parquet rows; click opens the modal with the resolved /resolve/ URL for the current branch. Tests + fixtures: - test_files.py::test_resolve_get_302_exposes_cors_headers_for_browser_preview pins the `Access-Control-Expose-Headers` list against regressions. - test/kohaku-hub-ui/utils/test_safetensors.test.js: 6 cases covering the real-HF-format fixture, dtype summary, progress phases, fat-header fallback, oversized-header guard, and non-206 error paths. - test/kohaku-hub-ui/utils/test_parquet.test.js: footer parse + progress phase assertions. - test/kohaku-hub-ui/fixtures/previews/{tiny.safetensors,tiny.parquet}: byte-identical-to-HF fixtures produced by the real safetensors / pyarrow libs via scripts/dev/generate_preview_test_fixtures.py (committed so tests stay offline per AGENTS.md §5.2). Seed: - seed_demo_data.py: add two RemoteAsset entries for real HF-hosted small fixtures pinned by sha256, and wire them into visible paths (open-media-lab/vision-language-assistant-3b/fixtures/hf-tiny-random-bert.safetensors, open-media-lab/multimodal-benchmark-suite/fixtures/hf-no-robots-test.parquet) so the preview can be exercised against files that actually came off huggingface.co rather than purely local pyarrow/safetensors output. SEED_VERSION bumped to local-dev-demo-v4. Verified end-to-end against the dev stack: safetensors parser output on the seeded fixtures matches huggingface_hub.parse_safetensors_file_metadata byte-for-byte on the same file (100 tensors, 126,851 params, I64=512 / F32=126,339, metadata `{format: pt, ...}`). Browser preview modal renders both file kinds correctly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev/narugo1992 #28 +/- ##
==================================================
+ Coverage 94.14% 94.21% +0.06%
==================================================
Files 115 119 +4
Lines 14820 15994 +1174
Branches 761 995 +234
==================================================
+ Hits 13953 15069 +1116
- Misses 865 914 +49
- Partials 2 11 +9
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
verify_seed_data.py hardcoded EXPECTED_SEED_VERSION = "local-dev-demo-v3" but seed_demo_data.py was bumped to v4 in the preview PR, so the post-seed verifier would falsely fail with a version mismatch. Extract the constant to scripts/dev/seed_shared.py and import it from both sides so the two scripts always agree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
FastAPI/Pydantic v2 emits DeprecationWarning and schedules removal for the ``regex`` kwarg on ``Query``. Two routes still used the old name; swap to ``pattern`` to match the current idiom elsewhere in the repo and silence the warning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract the preview-icon helpers (getPreviewKind, canPreviewFile,
resolve-URL builder) from RepoViewer.vue into src/utils/file-preview.js
so they become directly unit-testable rather than buried in a Vue SFC,
then cover the remaining surface of the preview path:
- test/kohaku-hub-ui/utils/test_file_preview.test.js — 12 cases for
the extracted helpers: extension recognition, case-insensitive
suffix matching, directory rejection, bad-input defensiveness, URL
encoding per path segment and per branch (so refs/convert/parquet
survives), and guard-rail errors when required fields are missing.
- test/kohaku-hub-ui/utils/test_safetensors.test.js — 9 new cases
covering the guard-rails that real HF fixtures do not naturally
trigger: truncated <8-byte response, invalid UTF-8 header, null
/ non-object JSON root, truncated fat-header second read,
second-range HTTP error, error-class instance + .status surface,
defensive skipping of malformed tensor entries, missing shape →
[] fallback, and unknown-dtype byte-size accounting.
- test/kohaku-hub-ui/utils/test_parquet.test.js — 5 new cases
covering normalizeCount's null/bigint-in-range/bigint-out-of-range/
plain-number paths and summarizeParquetSchema's empty-tree and
shapeless-children fallbacks. normalizeCount is now exported from
parquet.js so the tests can poke it directly (real parquet footers
always yield BigInt, so the number-passthrough branch is not
reachable through hyparquet without an awkward stub).
- test/kohaku-hub-ui/components/test_file_preview_dialog.test.js —
14 cases mounting FilePreviewDialog with mocked parser modules.
Covers the loading-phase progress copy for both kinds (including
the fat-header copy and the "done" phase), safetensors and parquet
ready-state renders, the error-state + Retry path, the
CORS-likely-failure placeholder + its negative case, silent
AbortError swallowing, AbortController plumbing on visibility
change, re-request on URL change, the unsupported-kind error
view, and Close emits update:visible. Includes minimal ElTable /
ElTableColumn stubs that respect Element Plus's scoped-slot
contract because the shared ElementPlusStubs do not cover those.
Coverage post-patch:
file-preview.js ………… 100% / 100% / 100% / 100%
FilePreviewDialog.vue … 100% lines / ~81% branches / 100% funcs
parquet.js ……………… 100% lines / ~78% branches (gaps are
nullish-coalesce fallbacks on fields that
pyarrow always populates in real output)
safetensors.js ……… ~98% lines / ~95% branches (one guard on
Number.isFinite of a u64, unreachable
by construction)
Total frontend test count: 132 → 172 (+40 new).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author
Follow-up: exhaustive coverage for the new preview codePushed
What the new tests hit, concretely:
The last unreachable branches are genuinely dead by construction: Also carries two small housekeeping commits from earlier rounds:
Full branch now has 4 commits on top of |
Before this change, whenever a fallback source returned a non-retryable
4xx (typically 401 for a gated HuggingFace repo), ``try_fallback_resolve``
called ``should_retry_source`` which correctly said "do not retry"
but then returned ``None`` — dropping the status code, the upstream
body, and any chance a later source might serve the same artifact.
``with_repo_fallback`` interpreted ``None`` as a fallback miss and
re-raised the original local 404 (``RepoNotFound``), so the browser
saw a misleading "repository not found" for a file whose repo it had
just successfully listed via the fallback tree endpoint.
Reproduced live against ``animetimm/mobilenetv3_large_150d.dbv4-full``
(a gated model) while developing PR#28: tree browsing works, but
download + client-side safetensors preview both died with an unhelpful
404.
### Backend
* ``try_fallback_resolve`` now records every non-success source probe
(HEAD or GET) as a ``build_fallback_attempt`` dict and keeps iterating.
A 401 on source 1 no longer short-circuits the chain; a downstream
mirror that does not gate the same artifact (or that simply has a
file the first source lacks) still gets a chance to serve the
request. ``httpx.TimeoutException`` and other transport exceptions
are recorded the same way instead of being swallowed in an
``except``.
* ``build_aggregate_failure_response`` (new, in ``fallback/utils.py``)
combines per-source attempts into one ``JSONResponse``:
- HTTP status priority: ``401 > 403 > 404 > 502``. The rationale
is user-actionability — an auth failure is the most specific
next step ("attach a token"), 403 is next, a real "not found"
after that, and 5xx / timeout / network collapse to 502 Bad
Gateway.
- ``X-Error-Code`` aligns with ``huggingface_hub.utils._http.
hf_raise_for_status``: 401 → ``GatedRepo``, all-404 →
``EntryNotFound``. 403 and 502 get no specific code (HF's
generic path is correct there). This is deliberate: inventing
new codes would downgrade ``hf_hub_download`` on a gated repo
to ``RepositoryNotFoundError`` and lose the ``GatedRepoError``
exception users actually catch.
- Body keeps an HF-style ``{error, detail}`` shape plus a
``sources[]`` array listing every attempt with ``{name, url,
status|null, category, message}``, so a curl / CLI user can see
the full timeline at a glance.
* ``should_retry_source`` is no longer used for flow control (the
loop now records every failure regardless). Kept as-is to avoid
touching its other callers.
### Frontend
* ``SafetensorsFetchError`` carries ``errorCode`` / ``sources`` /
``detail`` parsed from the aggregated body + ``X-Error-*`` headers.
``fromResponse()`` does a defensive JSON parse so plain-text 4xx
bodies (e.g. raw HF "Access restricted ...") still produce a
reasonable message.
* ``FilePreviewDialog`` now renders four remediation-specific error
states based on the classification: ``gated`` ("Attach an access
token ..."), ``forbidden``, ``not-found``, ``upstream-unavailable``,
and a fallback generic. The existing CORS hint still fires for the
``TypeError`` / "Failed to fetch" failure mode. When the error
carries a ``sources[]`` list, it collapses into a ``<details>``
table for diagnostic use.
### Tests
Backend (``test/kohakuhub/api/fallback/``):
* 6 new cases in ``test_operations.py`` pinning the new contract:
single 401 → aggregate 401/``GatedRepo``; single 403 → 403; 401 +
next-source success (continues past 401!); mixed 401/404/503/timeout
across four sources (priority yields 401); all-404 → 404/
``EntryNotFound``; all-5xx/timeout → 502 with no code.
* Two pre-existing tests that pinned the old short-circuit behavior
(``stops_on_non_retryable_status_and_handles_timeouts``,
``returns_none_after_generic_source_failures``) have been rewritten
to match the new aggregating semantics.
* 3 new HF-interop tests in ``test_hf_hub_interop.py`` feed the
aggregated response into real ``huggingface_hub.utils._http.
hf_raise_for_status`` and assert: aggregate 401 raises
``GatedRepoError``, aggregate 404 raises ``EntryNotFoundError``,
aggregate 502 raises generic ``HfHubHTTPError`` (and *not* any of
the subclasses that would signal a wrong remediation). These pin
the HF-alignment invariant — a future refactor that drops
``X-Error-Code`` will fail these tests immediately.
Frontend (``test/kohaku-hub-ui/``):
* 2 new cases in ``test_safetensors.test.js`` for
``SafetensorsFetchError.fromResponse``: aggregated JSON body
populates ``errorCode`` / ``sources`` / ``detail`` from the body +
``X-Error-Code`` header; non-JSON bodies fall back to
``X-Error-Message`` header without crashing.
* 3 new cases in ``test_file_preview_dialog.test.js`` rendering the
three new error states (``gated``, ``not-found``,
``upstream-unavailable``) and asserting that gated-copy specifically
never cross-contaminates with the CORS copy.
### Verified live
Against the real gated repo through the patched local backend:
* ``HEAD /models/animetimm/mobilenetv3_large_150d.dbv4-full/resolve/
main/model.safetensors`` now returns 401 + ``X-Error-Code:
GatedRepo`` + aggregated JSON body (was: 404 + ``RepoNotFound``).
* ``huggingface_hub.HfApi.parse_safetensors_file_metadata`` against
the patched endpoint raises ``GatedRepoError`` with the upstream
message echoed into the exception text — the HF-native behavior
users already handle.
### Not in scope
Other fallback operations (``try_fallback_tree``, ``try_fallback_info``,
``try_fallback_paths_info``, avatars, user-repos aggregation) still use
the old "return None on any failure" pattern. They rarely surface
gated-vs-missing confusion in practice (the tree endpoint is public
for HF gated repos, which is why browsing worked while downloading
broke), so this change stays focused on resolve. Extending the
aggregation contract to the other operations is a reasonable
follow-up.
Builds on top of PR#28 (pure-client safetensors / parquet preview).
Target that branch so the UI error copy + the backend classification
land together.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 tasks
codecov/patch flagged 93.16% vs 94.12% target on the previous commit — the ~1% miss was the defensive branches in `build_fallback_attempt` and `build_aggregate_failure_response` that the integration tests in `test_operations.py` don't naturally exercise (odd status codes, the contract-violation fallback, the oversize-message cap, empty attempts list). Add 7 targeted unit tests in `test/kohakuhub/api/fallback/test_utils.py`: - `_categorize_status` maps each of 401 / 403 / 404 / 410 / 503 to the right category. - An unclassifiable status (418) lands in `CATEGORY_OTHER` so the aggregate shape stays consistent. - Contract-violation call (no response / timeout / network supplied) returns a safe default rather than throwing. - Oversized upstream bodies get truncated under the per-attempt cap. - `timeout=...` and `network=...` paths record the right categories and surface the exception message. - Empty-attempts aggregate is a well-formed 502 rather than a KeyError or a misleading 401. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
codecov/patch was still red at 93.16% after the previous unit-test fill. The residual miss was a defensive four-line `return None` after the source loop that cannot be reached by construction: the `if not sources: return None` short-circuit at the top of the function already handles the empty-sources case, and every branch of the loop body either returns on success or `attempts.append(...)`s before `continue`, so the attempts list is always non-empty once the loop exits. Replace the `if attempts:` guard + dead fallback with a direct call to `build_aggregate_failure_response(attempts)` and a comment recording the invariant, so a future change that adds an early `continue` without recording an attempt will fail its unit tests (the aggregated response will then claim "no fallback sources" with an empty sources list) rather than silently returning None again. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 3 new HF-compat tests (pattern_D_all_401/404/5xx_raises_...) feed `httpx.Response` objects directly into `huggingface_hub.utils._http.hf_raise_for_status`. That function migrated from `requests` to `httpx` in hf_hub 1.0: - hf_hub >= 1.0: catches `httpx.HTTPStatusError`, reads X-Error-Code, raises the right subclass (GatedRepoError / EntryNotFoundError / ...) - hf_hub < 1.0: catches `requests.HTTPError` only; our httpx.HTTPStatusError escapes the classification path entirely and the test sees a bare `httpx.HTTPStatusError` instead of the HF subclass it is pinning. The matrix cells that failed (python_version, hf_hub): (3.10, 0.20.3) / (3.10, 0.30.2) / (3.10, 0.36.2) / (3.11, 0.20.3) / (3.11, 0.30.2) / (3.11, 0.36.2) / ... Guard the 3 tests with a version-based `pytest.skipif` matching the existing pattern for `HAS_XET` higher up in the file (Xet was also a 1.0+ feature). The aggregate-response contract is still upheld on every cell — it's just that the HF-classification assertion is only observable on cells where hf_hub speaks httpx. The other 7 tests in this file remain matrix-wide green, including the pre-existing pattern A/B/C that use a different assertion surface that works across the version range. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Supersedes the `skipif(not _HF_SUPPORTS_HTTPX_CLASSIFICATION)` that
the previous commit added. Skipping the three pattern-D tests on
older cells left the HF-compat invariant unverified on exactly the
matrix cells that speak a different Response type — precisely the
cells most likely to silently drift.
The classification logic in `hf_raise_for_status` (X-Error-Code →
GatedRepoError / EntryNotFoundError / generic HfHubHTTPError) is
identical across 0.20.x / 0.30.x / 0.36.x / 1.0.x / 1.6.x / latest.
Only the Response TYPE it accepts differs (requests before 1.0, httpx
after). Build the right type per installed version and assert the
classification everywhere.
Changes:
- New helper `_to_hf_response(resp, request_url=...)` which rehydrates
the FastAPI aggregate response as either `httpx.Response` (hf_hub
>= 1.0) or `requests.Response` (< 1.0). Only the three pattern-D
tests use it; the existing A/B/C tests keep using `_to_httpx`
because they only read `.status_code` / `.headers`, which exist on
both shapes.
- New helper `_hf_error(name)` for looking up exception classes
across the different public-module layouts:
0.20.x → huggingface_hub.utils.<Name>
0.30.x+ → huggingface_hub.errors.<Name> (re-exported from utils)
`huggingface_hub` (top-level) does NOT export the error classes in
any version, which is why the original `from huggingface_hub.errors
import ...` failed on 0.20.3 with `ModuleNotFoundError`.
- Swap `from huggingface_hub.utils._http import hf_raise_for_status`
(private path, restructured in 1.0) to `from huggingface_hub.utils
import hf_raise_for_status` (stable public re-export, present in
every version).
Locally verified against:
hf_hub 0.20.3 (oldest matrix cell, requests branch) — 10/10 pass
hf_hub 1.11.0 (latest, httpx branch) — 10/10 pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small coverage-quality cleanups that together bring `safetensors.js`
to 100% / 100% / 100% / 100% (statements / branches / functions /
lines), which in turn closes the 0.02% project-coverage regression
codecov was flagging.
- Remove the unreachable `!Number.isFinite(headerLen) || headerLen < 0`
guard in `parseSafetensorsMetadata`. `DataView.getBigUint64` always
returns a non-negative BigInt in `[0, 2^64 - 1]`, which `Number()`
always maps to a finite non-negative float. No real input can hit
that branch; the `MAX_HEADER_LENGTH` check below is the only upper
bound that matters.
- Cover three previously untested branches in `SafetensorsFetchError.
fromResponse` and the tensor-parse fallback:
* tensor entry without `data_offsets` → parser defaults to [0, 0]
instead of crashing.
* aggregate error body with no `X-Error-Code` header → errorCode
still populated from `body.error`.
* aggregate error body whose `sources` is not an array → the
parser refuses to attach a non-list and leaves `sources=null`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix: classify fallback upstream errors (gated/not-found/unavailable) instead of collapsing to 404
Restructure the safetensors preview modal around the tensor-name
hierarchy instead of a flat tensor list.
utils/safetensors.js
- New `formatHumanReadable(n)` helper: compact "1.23B" / "126.85K"
formatting with trailing zeros stripped, used by the Total
parameters pill and the Parameters column when in human mode.
- New `buildTensorTree(tensors, total)` helper: splits each dotted
name into a nested tree, rolls `parameters` / `byteSize` /
`dtypeLabel` / `leafCount` up the tree, then
* collapses single-child chains. `a.b.c.d.weight` with no
siblings along the way becomes one row named
"a.b.c.d.weight" instead of five nested rows the user has to
click through. A fork only keeps its parent split.
* attaches `percent` relative to the IMMEDIATE parent's
parameter count (roots use the file total). So a leaf shows
its share of the block it lives in ("this head is 22% of
this layer"), not its share of the whole model (always tiny).
- Also drop the dead `!isFinite(headerLen) || <0` guard: a
`DataView.getBigUint64` return always sits in [0, 2^64-1] which
`Number()` always maps to a finite non-negative float, so the
branch was unreachable and the `MAX_HEADER_LENGTH` check above
is the only upper bound that matters.
FilePreviewDialog.vue
- Tensors table is now a tree table (`row-key="path"`, `tree-props:
{ children: "children" }`), default-expanded.
- Name column shows the collapsed segment for each row plus a
subtle `(N tensors)` hint on parents. Leaves have their `children`
stripped so Element Plus doesn't render a useless expand chevron
for them.
- `dtype`, `Shape`, `Parameters`, `Bytes` columns: no more
`min-width`; they size to content.
- `% of parent` column: a thin bar behind the numeric label, with
`rgba(100, 116, 139, <alpha>)` where alpha ramps continuously
from 0.08 at 0% to 0.72 at 100%. Low-mass tensors stay visibly
present; high-mass ones read as dark gray. Same RGB on both
themes (alpha composites against the table background either
way).
- Total parameters pill: renders human-readable by default, clicks
to toggle to the exact `toLocaleString()` form. Icon sits
BEFORE the label so it isn't clipped off the right edge of an
auto-sized cell.
- Parameters column header is clickable too — same `totalParamsFormat`
ref drives both the pill and every cell in the column, so they
flip together. Cells themselves are not click targets per the UX
brief (column-wide toggle, not per-cell).
- Bytes cells + Tensor bytes pill now carry a `title` tooltip with
the exact count (`"79,691,776 bytes"`) so a user eyeballing
"76.00 MB" can still copy the precise byte count.
Tests
- 5 new cases for `buildTensorTree`: chain-collapse of deep
single-chains, parent-relative percent math, 0-total guard,
malformed-entry tolerance, missing shape/data_offsets defaults.
- 3 new cases for `formatHumanReadable`: K/M/B/T buckets, trailing-
zero trim, negative + NaN + null guards.
- 2 new dialog cases: tree rendering (parent + leaves + percent),
Total parameters toggle flipping both the pill title and text.
- Updated the existing ElTable stub to walk `tree-props.children`
so leaf rows also show up in text-based assertions (real
ElTable hides collapsed subtrees, but the stub doesn't model
chevron state so rendering everything is the right fidelity
for text queries).
- Hardened the `summarizeSafetensors` mock to compute real totals
from the supplied header so tests that need distinguishable
human-vs-exact numbers aren't boxed into the default `total: 4`
fixed value.
Frontend suite: 162 → 189 passing (27 new).
Verified live against the seeded sharded model
`open-media-lab/vision-language-assistant-3b/model-00001-of-00003.safetensors`:
tree renders "language_model (2 tensors)" parent with two
chain-collapsed leaves (`embed_tokens.weight`,
`layers.0.mlp.down_proj.weight`), percents read 78.95% / 21.05%
summing to 100% of the parent, gradient visibly differentiates
them, bytes cell tooltip shows `"79,691,776 bytes"`, and toggling
either the pill or the column header flips both to
`"39,845,888"` in sync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #27 v4: pure-client HF-compatible metadata preview for
.safetensorsand.parquetfiles. The SPA gets a small chart-icon next to eligible rows (HF-style); clicking opens a modal that streams phase-by-phase status ("Fetching header Range (100 KB)…" → "Parsing header JSON…" → done) while it pulls ≤ 100 KB per file over HTTP Range. No backend preview code, no LRU, no precomputation, no new DB state, no worker.What changed
Backend — CORS plumbing only (zero preview logic):
src/kohakuhub/main.py:104-126— addexpose_headerslist so browsers can readContent-Range,X-Linked-*,X-Repo-Commit,ETag,Locationoff the final 206 response. Pinned bytest_resolve_get_302_exposes_cors_headers_for_browser_preview.docker-compose.example.yml+scripts/dev/up_infra.sh— wireMINIO_API_CORS_ALLOW_ORIGIN(default*for dev, configurable viaDEV_MINIO_CORS_ALLOW_ORIGIN). Hard prerequisite for the feature; documented indocs/development/local-dev.md.Frontend:
src/kohaku-hub-ui/src/utils/safetensors.js— pure-JS parser mirroringhuggingface_hub.parse_safetensors_file_metadatabyte-for-byte. Two Range reads at most (speculative 100 KB first, fat-header fallback).src/kohaku-hub-ui/src/utils/parquet.js— thin wrapper overhyparquet(~40 KB gzipped). CORS-safe request init so cookies never leak onto presigned URLs.src/kohaku-hub-ui/src/components/repo/preview/FilePreviewDialog.vue— ElDialog with loading spinner + per-phase text, dtype breakdown + tensor table (safetensors), schema + row-group table (parquet), graceful CORS-failure placeholder.src/kohaku-hub-ui/src/components/repo/RepoViewer.vue—i-carbon-chart-line-dataicon next to.safetensors/.parquetrows; opens the modal with the right/resolve/{rev}/{path}URL.@click.stopkeeps directory-navigation clicks unaffected.Tests + fixtures (all offline, per AGENTS.md §5.2):
test/kohakuhub/api/test_files.py::test_resolve_get_302_exposes_cors_headers_for_browser_preview— regression for the CORS contract.test/kohaku-hub-ui/utils/test_safetensors.test.js(6 cases) — real-fixture parse, dtype summary, phase ordering, fat-header two-read fallback, oversized-header guard, non-206 error.test/kohaku-hub-ui/utils/test_parquet.test.js(2 cases) — footer parse + phases.test/kohaku-hub-ui/fixtures/previews/{tiny.safetensors, tiny.parquet}— 2 KB + 2.7 KB fixtures produced via the realsafetensors.numpy.save+pyarrow.parquet.write_table(so wire format is byte-identical to what HF emits). Regeneratable viascripts/dev/generate_preview_test_fixtures.py.Seed (real HF-sourced fixtures):
scripts/dev/seed_demo_data.py— adds twoRemoteAssetentries (hf-tiny-random-bert.safetensors≈ 520 KB,hf-no-robots-test.parquet≈ 570 KB), pinned by sha256, wired into visible paths underopen-media-lab/vision-language-assistant-3b/fixtures/andopen-media-lab/multimodal-benchmark-suite/fixtures/.SEED_VERSION→local-dev-demo-v4so reseed triggers.Verification
All 8 new frontend unit tests + 2 new backend tests green.
Full frontend suite: 132 / 132 passing. Full backend
test_files.py: 7 / 7 passing.End-to-end against dev stack: parser output on the seeded real-HF file matches
huggingface_hub.HfApi(endpoint=<kohakuhub>).parse_safetensors_file_metadata(...)byte-for-byte:huggingface_hub(reference){I64: 512, F32: 126,339}{'I64': 512, 'F32': 126339}{format: pt, cls.predictions.decoder.bias: cls.predictions.bias}Real-browser Playwright drive: file list renders the preview icon; safetensors modal shows the right dtype/param/metadata breakdown; parquet modal shows schema + row-group data for the
no_robotstest split. Screenshots reviewed — HF-style affordance placement confirmed.CORS chain smoke test (backend OPTIONS + MinIO OPTIONS + final 206 with
Access-Control-Expose-Headers: … Content-Range, …) all green against the Vite origin (http://127.0.0.1:5173).Explicitly NOT in this PR (per #27 v4 scope)
?expand[]=safetensors,?expand[]=gguf, "total model size" pill) — if the UI ever wants this, the SPA should concurrently fetch shard headers itself, no backend work.datasets-server/*endpoints.Test plan
make test-backendrestricted totest_files.pypasses (7 / 7).npm testfull suite passes (132 / 132) with coverage../scripts/dev/up_infra.sh+./scripts/dev/run_backend.sh+npm run dev; open a repo with a seeded safetensors or parquet file; click the preview icon; confirm modal renders real data without download of the full file (verifiable via MinIO access logs).huggingface_hub.HfApi(endpoint=<kohakuhub>).parse_safetensors_file_metadata(repo_id=<seeded>, filename=fixtures/hf-tiny-random-bert.safetensors)output equals the in-browser parser output on the same file.Closes — partial scope for #27 (file-level preview only; repo-level aggregate explicitly deferred per v4 rewrite).
🤖 Generated with Claude Code