Include archive path in WebDataset TAR read errors#8345
Conversation
|
CI waiting on maintainer workflow approval ( |
|
related: this covers the WebDataset TAR ReadError path from #7280; broader |
|
I would still include the origin, in most cases it's a hf:// URLs that points to the file online which is useful for debugging |
|
Updated — errors now include both the archive path and the hf:// origin. |
Restore tracked_str origin (e.g. hf://…) in the re-raised ReadError so remote archive location remains visible for debugging, while still reporting the resolved local path.
1e9d19e to
b156f02
Compare
Strip userinfo and query from tracked origins before embedding them in exceptions so presigned URLs and basic-auth credentials don't leak into logs/tracebacks while keeping scheme/host/path for debugging. Co-authored-by: Cursor <cursoragent@cursor.com>
|
feel free to do it in |
|
Good idea — I'll move this into |
|
Went ahead and implemented this now rather than deferring to a follow-up (061468a): the origin sanitization now lives in |
Fixes #7280
What changed
WebDataset catches
tarfile.ReadErroraround TAR processing and re-raises with the resolved archive path; original remains on__cause__.Covers failures while advancing the archive iterator and while reading a member payload (feature inference and example generation).
Uses
str(tar_path)rather thanrepr(tar_path)sotracked_strorigins (presigned URLs/credentials) are not exposed in tracebacks.Before
After
Tests
f.read()ReadErrorpreserved as causetracked_strorigin with password/signature not exposedValidation
make style/git diff --check: passed