Skip to content

Include archive path in WebDataset TAR read errors#8345

Open
sankalpsthakur wants to merge 4 commits into
huggingface:mainfrom
sankalpsthakur:fix/webdataset-read-error-path-7280
Open

Include archive path in WebDataset TAR read errors#8345
sankalpsthakur wants to merge 4 commits into
huggingface:mainfrom
sankalpsthakur:fix/webdataset-read-error-path-7280

Conversation

@sankalpsthakur

@sankalpsthakur sankalpsthakur commented Jul 19, 2026

Copy link
Copy Markdown

Fixes #7280

What changed

WebDataset catches tarfile.ReadError around TAR processing and re-raises with the resolved archive path; original remains on __cause__.

Covers failures while advancing the archive iterator and while reading a member payload (feature inference and example generation).

Uses str(tar_path) rather than repr(tar_path) so tracked_str origins (presigned URLs/credentials) are not exposed in tracebacks.

Before

ReadError: empty file

After

ReadError: Failed to read TAR archive "/path/to/corrupted-shard-00042.tar": empty file

Tests

  • corrupt first shard (feature inference)
  • valid shard then corrupt shard (example generation)
  • truncated member during f.read()
  • original ReadError preserved as cause
  • Windows-safe path matching
  • tracked_str origin with password/signature not exposed

Validation

  • pytest (WebDataset / DownloadManager / StreamingDownloadManager): 46 passed, 10 optional skips
  • make style / git diff --check: passed

@sankalpsthakur

sankalpsthakur commented Jul 22, 2026

Copy link
Copy Markdown
Author

CI waiting on maintainer workflow approval (action_required). Local WebDataset ReadError-path tests are green.

@sankalpsthakur

Copy link
Copy Markdown
Author

related: this covers the WebDataset TAR ReadError path from #7280; broader _iter_from_urlpath remainder left for a follow-up.

@lhoestq

lhoestq commented Jul 22, 2026

Copy link
Copy Markdown
Member

I would still include the origin, in most cases it's a hf:// URLs that points to the file online which is useful for debugging

@sankalpsthakur

sankalpsthakur commented Jul 22, 2026

Copy link
Copy Markdown
Author

Updated — errors now include both the archive path and the hf:// origin.

Restore tracked_str origin (e.g. hf://…) in the re-raised ReadError
so remote archive location remains visible for debugging, while still
reporting the resolved local path.
@sankalpsthakur
sankalpsthakur force-pushed the fix/webdataset-read-error-path-7280 branch from 1e9d19e to b156f02 Compare July 22, 2026 19:03
Strip userinfo and query from tracked origins before embedding them in
exceptions so presigned URLs and basic-auth credentials don't leak into
logs/tracebacks while keeping scheme/host/path for debugging.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lhoestq

lhoestq commented Jul 23, 2026

Copy link
Copy Markdown
Member

feel free to do it in tracked_str.__repr__ directly insetad of having a custom logic just for webdataset, this way it will benefit other loaders as well

@sankalpsthakur

Copy link
Copy Markdown
Author

Good idea — I'll move this into tracked_str.__repr__ in a follow-up so other loaders benefit too.

@sankalpsthakur

Copy link
Copy Markdown
Author

Went ahead and implemented this now rather than deferring to a follow-up (061468a): the origin sanitization now lives in tracked_str.__repr__ (via a small _sanitize_origin helper), so any loader that reprs a tracked path gets the safe origin — userinfo and query strings are stripped while scheme/host/path (e.g. hf://…) are kept for debugging. WebDataset's except tarfile.ReadError block is back to a one-liner and its local sanitizer is gone. The existing WebDataset read-error tests cover the behaviour unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add filename in error message when ReadError or similar occur

2 participants