diff --git a/docs/v1/datasets.md b/docs/v1/datasets.md index 8f12f24..280af02 100644 --- a/docs/v1/datasets.md +++ b/docs/v1/datasets.md @@ -1,4 +1,4 @@ - + ## Curated picks @@ -20,6 +20,7 @@ | Dataset Short Name | Dataset Full Name | Dataset Description | Dataset Source (URL) | Data Kind | License | Row Count | Row Groups - Parquet | File Size - Parquet | File Size - Vortex | |--------------------|-------------------|---------------------|----------------------|-----------|---------|-----------|----------------------|---------------------|--------------------| | 120 years of Olympic history: athletes and results | 120 years of Olympic history: athletes and results | basic bio data on athletes and medal results from Athens 1896 to Rio 2016. This is a historical dataset on the modern Olympic Games, including all the Games from Athens 1896 to Rio 2016. I scraped this data from www.sports-reference.com in May 2018. The R code I used to scrape and wrangle the data is on GitHub. — adapted from the dataset's Kaggle description (heesoo37/120-years-of-olympic-history-athletes-and-results). | https://github.com/rgriff23/Olympic_history | Tabular (CSV) | CC0-1.0 | 271,116 | 1 | 4.4 MB | 5.6 MB | +| [⚠ Amazon Reviews 2023 (Subscription Boxes)](#scrape-advisory-amazon-reviews-2023-subscription-boxes) | Amazon Reviews 2023 — Subscription_Boxes user reviews (McAuley Lab, UCSD; Hou et al. 2024) | User reviews for the Subscription_Boxes category from the Amazon Reviews 2023 corpus — rating, title, text, asin, parent_asin, user_id, millisecond timestamp, helpful_vote, verified_purchase, and a nested images list. Smallest of 33 categories; the catalog's smoke-test slice of the 571M-review corpus. | https://huggingface.co/datasets/McAuley-Lab/Amazon-Reviews-2023 | Structured (JSON) | NoAssertion | 16,216 | 1 | 2.4 MB | 3.1 MB | | [⚠ C4 (en, validation)](#scrape-advisory-c4-en-validation) | AllenAI/C4 — Colossal Clean Crawled Corpus, English validation split | C4 (Raffel et al., JMLR 2020) — a heavily-filtered scrape of Common Crawl used to pretrain T5. This entry pulls only the 8-shard English validation split (~365k documents), enough for type coverage and as a smoke-test for the Common-Crawl scrape playbook. Flip allow_patterns to `en/c4-train.*.json.gz` to mirror the 327 GB English training set. | https://huggingface.co/datasets/allenai/c4 | Structured (JSON) | ODC-By-1.0 | 364,608 | — | 339.7 MB | 435.3 MB | | [⚠ CodeParrot Clean (validation)](#scrape-advisory-codeparrot-clean-valid) | codeparrot/codeparrot-clean — validation split | 61k Python source files scraped from MIT/BSD/Apache-licensed GitHub repos by the CodeParrot project. Validation split (142 MB raw .json.gz). Showcases code-corpus shape: string `content`, numeric quality metrics (line_mean, alpha_frac), boolean autogenerated flag, and per-row license attribution. | https://huggingface.co/datasets/codeparrot/codeparrot-clean-valid | Structured + Blobs | Apache-2.0 | 61,373 | — | 149.7 MB | 343.5 MB | | [⚠ FineMath (4+ quality subset)](#scrape-advisory-finemath-4plus) | HuggingFaceTB/finemath — quality≥4 math web pages | Math-focused subset of the fineweb pipeline, filtered to documents with an automated quality score ≥ 4 (the higher of the two quality tiers). 64 parquet shards. Schema mirrors fineweb (`text` + harvest metadata). | https://huggingface.co/datasets/HuggingFaceTB/finemath | Tabular (Parquet) | ODC-By-1.0 | 6,699,493 | 7 | 12,647.7 MB | 21,724.8 MB | @@ -273,6 +274,9 @@ These datasets aggregate or reference content whose underlying licenses have not been individually cleared. The aggregator's declared license (the License column above) governs only the metadata it ships, not the content it points at. Read each advisory before redistributing or building on top of one of these slugs. + +**Amazon Reviews 2023 (Subscription Boxes)** (`amazon-reviews-2023-subscription-boxes`) — Amazon-derived: built by crawling raw HTML from Amazon.com through September 2023 and parsing it to JSON. Amazon's Conditions of Use prohibit the data-mining that produced it and forbid derivative use / redistribution of product listings, descriptions, and prices; review text copyright sits with individual reviewers, not Amazon or the aggregator. No layer in the chain grants third-party redistribution rights. Build and use locally as a research convenience only — these artifacts must NEVER be uploaded to a shared mirror (publish refuses them by default on this advisory). + **C4 (en, validation)** (`c4-en-validation`) — C4 (Colossal Clean Crawled Corpus) is a heavily-filtered scrape of Common Crawl. Allen AI's CC-BY-4.0 license covers the harvest layer; the underlying web text remains subject to per-publisher copyright. Treat as research convenience, not a license-cleared corpus. diff --git a/docs/v1/handlers.md b/docs/v1/handlers.md index f6cb0f0..89d0a96 100644 --- a/docs/v1/handlers.md +++ b/docs/v1/handlers.md @@ -1,4 +1,4 @@ - + | Handler | Purpose | Streaming | Extra Deps | # Manifest Specs | Example Slugs | |---------|---------|-----------|------------|------------------|---------------| | `beijing_pm25_parse` | Parse UCI Beijing Multi-Site Air Quality (dataset 501). | no | — | 1 | `uci-beijing-multi-site-air-quality` | @@ -7,7 +7,7 @@ | `ghcn_daily_parse` | Parse NOAA GHCN-Daily .dly fixed-width records into a long-format parquet. | yes | — | 1 | `ghcn-daily` | | `glove_split` | Read a single-dimension GloVe text file and emit a | no | — | 3 | `glove-6b-100d`, `glove-6b-200d` (+1 more) | | `har_parse` | Parse UCI Human Activity Recognition Using Smartphones (dataset 240). | no | — | 1 | `uci-human-activity-recognition-using-smartphones` | -| `hf_concat_splits` | Concat HF parquet shards into one parquet, optionally injecting a | no | — | 52 | `ai2-arc`, `anthropic-hh-rlhf-helpful-base` (+50 more) | +| `hf_concat_splits` | Concat HF parquet shards into one parquet, optionally injecting a | no | — | 53 | `ai2-arc`, `amazon-reviews-2023-subscription-boxes` (+51 more) | | `identity` | Passthrough handler — single parsed table becomes the output. | no | — | 3 | `clickbench-hits`, `emotions-dataset-for-nlp` (+1 more) | | `jsonbench_variant_parse` | Parse ClickHouse JSONBench's Bluesky JSONL.gz dumps into a single parquet | yes | — | 1 | `jsonbench-bluesky-100m` | | `jsonl_as_string_parse` | Stream a JSONL[.gz] file into a parquet with a single `raw_json: string` | yes | — | 1 | `open-food-facts` | diff --git a/docs/v1/snapshot.json b/docs/v1/snapshot.json index d3e7eba..c87b85c 100644 --- a/docs/v1/snapshot.json +++ b/docs/v1/snapshot.json @@ -1,6 +1,6 @@ { "schema_version": 1, - "generated_at": "2026-05-27T18:14:16Z", + "generated_at": "2026-06-10T15:47:46Z", "note": "Auto-generated by scripts/pipeline/docs.py. Read by the TUI as a fallback when a local parquet isn't built. Regenerate after any build / schema change with `python -m scripts.pipeline.docs snapshot`.", "slugs": { "clickbench-hits": { @@ -117070,6 +117070,106 @@ "wide_row": false, "high_cardinality_present": true } + }, + "amazon-reviews-2023-subscription-boxes": { + "expected_rows": 16216, + "last_built_rows": 16216, + "last_built_row_groups": 1, + "parquet_bytes": 2567134, + "vortex_bytes": 3278148, + "parquet_sha256": "ce709109a3e505b3e26df331a54e15c09c92c20014f54a27afa2528730fb78be", + "vortex_sha256": "f68c40967bc938d8a8b03e5d9582738e84f03d7b7dd130ec3d4bcda917e10823", + "columns": [ + { + "name": "rating", + "type": "double", + "length": 5040, + "null_count": 0, + "min": 1.0, + "max": 5.0 + }, + { + "name": "title", + "type": "string", + "length": 215759, + "null_count": 0, + "min": "!", + "max": "\ud83e\udd73\ud83e\udd73\ud83e\udd73" + }, + { + "name": "text", + "type": "string", + "length": 1770531, + "null_count": 0, + "min": "", + "max": "\ud83e\udd1f\ud83d\udc4c\ud83d\ude0e" + }, + { + "name": "images", + "type": "list>", + "length": 98164, + "null_count": null, + "min": null, + "max": null + }, + { + "name": "asin", + "type": "string", + "length": 25015, + "null_count": 0, + "min": "B01M71IUZ7", + "max": "B0CBCVHKG7" + }, + { + "name": "parent_asin", + "type": "string", + "length": 20215, + "null_count": 0, + "min": "B01M71IUZ7", + "max": "B0BXVNMNV6" + }, + { + "name": "user_id", + "type": "string", + "length": 291260, + "null_count": 0, + "min": "AE224HI2GTJDDOBC3RN3O5SDBNOA", + "max": "AHZYSB4Z3JKMWDBSP7KXNXBLIJUA" + }, + { + "name": "timestamp", + "type": "uint64", + "length": 127663, + "null_count": 0, + "min": 1485302279000, + "max": 1693437206732 + }, + { + "name": "helpful_vote", + "type": "uint16", + "length": 8769, + "null_count": 0, + "min": 0, + "max": 985 + }, + { + "name": "verified_purchase", + "type": "bool", + "length": 1279, + "null_count": 0, + "min": false, + "max": true + } + ], + "size_bucket": "xs", + "shape_traits": { + "has_nested": true, + "has_timestamp": false, + "has_variant": false, + "string_heavy": false, + "wide_row": false, + "high_cardinality_present": null + } } } } diff --git a/scripts/pipeline/publish.py b/scripts/pipeline/publish.py index 62062e7..e1b40d5 100644 --- a/scripts/pipeline/publish.py +++ b/scripts/pipeline/publish.py @@ -26,6 +26,37 @@ class PublishMismatch(Exception): """On-disk artifact sha256 disagrees with the snapshot.""" +def scrape_advisory_slugs(manifest, slugs): + """Subset of `slugs` whose license carries a non-null `scrape_advisory`. + + These aggregate or reference content whose underlying licenses have not been + cleared for redistribution (public-web scrapes, Common Crawl derivatives, + Amazon-Conditions-of-Use-governed review corpora). `publish` refuses them by + default: building and using such artifacts locally is the customary research + posture, but uploading them to a shared mirror is the one act those terms + actually forbid. Tolerant of specs with no `license` block (test fixtures). + """ + by_slug = {d["slug"]: d for d in manifest.get("datasets", [])} + return [s for s in slugs + if (by_slug.get(s, {}).get("license") or {}).get("scrape_advisory")] + + +def no_redistribution_slugs(manifest, slugs): + """Subset of `slugs` whose license sets `redistribution_permitted` to False. + + An independent gate from `scrape_advisory_slugs`: the advisory flags a *gap* + between an aggregator's declared license and uncleared underlying content, + while this is the spec stating outright that the license does not grant + redistribution. A slug can trip either or both (the Amazon Reviews corpus + trips both); each gate has its own override, so clearing one never silently + clears the other. Tolerant of specs with no `license` block (test fixtures). + """ + by_slug = {d["slug"]: d for d in manifest.get("datasets", [])} + return [s for s in slugs + if (by_slug.get(s, {}).get("license") or {}).get("redistribution_permitted") + is False] + + def plan_uploads(slugs, snapshot, *, outputs_root: Path): """Return [(local_path, remote_key)] for present artifacts. @@ -95,6 +126,14 @@ def main(argv=None) -> int: ap.add_argument("--all", action="store_true") ap.add_argument("--mirror", required=True, help="fsspec base, e.g. s3://b/p") ap.add_argument("--dry-run", action="store_true") + ap.add_argument( + "--allow-scrape-advisory", action="store_true", + help="publish slugs whose license carries a scrape_advisory (refused by " + "default — their underlying content is not cleared for redistribution)") + ap.add_argument( + "--allow-no-redistribution", action="store_true", + help="publish slugs whose license sets redistribution_permitted=false " + "(refused by default — the license does not grant redistribution)") args = ap.parse_args(argv) if args.all and args.slugs: @@ -105,6 +144,33 @@ def main(argv=None) -> int: if not slugs: ap.error("pass slugs or --all") + # Default-block on two independent license gates: a mirror upload IS + # redistribution, and neither a scrape_advisory nor redistribution_permitted= + # false clears it. Each gate has its own --allow-* override, so clearing one + # never silently clears the other (a slug tripping both needs both flags). + # --all skips blocked slugs and keeps going; an explicit publish that leaves + # nothing to upload fails loudly so it doesn't read as a no-op success. + gates = ( + (scrape_advisory_slugs(manifest, slugs), args.allow_scrape_advisory, + "license carries a scrape_advisory — underlying content not cleared for " + "redistribution", "--allow-scrape-advisory"), + (no_redistribution_slugs(manifest, slugs), args.allow_no_redistribution, + "license sets redistribution_permitted=false", "--allow-no-redistribution"), + ) + blocked: set[str] = set() + for hit, allowed, reason, flag in gates: + if allowed: + continue + for s in sorted(hit): + print(f"refusing {s}: {reason} (pass {flag} to override)", file=sys.stderr) + blocked.update(hit) + if blocked: + slugs = [s for s in slugs if s not in blocked] + if not slugs: + print(f"refused all {len(blocked)} requested slug(s); nothing to publish", + file=sys.stderr) + return 1 + # Resolve the snapshot the same way the loader does (RAINCLOUD_SNAPSHOT -> # checkout -> wheel-packaged), so publish gates against the file the loader # will trust rather than a hardcoded REPO_ROOT path. diff --git a/sources.json b/sources.json index 0583594..a79cfb0 100644 --- a/sources.json +++ b/sources.json @@ -16531,6 +16531,68 @@ "nested-json", "prose" ] + }, + { + "slug": "amazon-reviews-2023-subscription-boxes", + "short_name": "Amazon Reviews 2023 (Subscription Boxes)", + "full_name": "Amazon Reviews 2023 \u2014 Subscription_Boxes user reviews (McAuley Lab, UCSD; Hou et al. 2024)", + "description": "User reviews for the Subscription_Boxes category from the Amazon Reviews 2023 corpus \u2014 rating, title, text, asin, parent_asin, user_id, millisecond timestamp, helpful_vote, verified_purchase, and a nested images list. Smallest of 33 categories; the catalog's smoke-test slice of the 571M-review corpus.", + "license": { + "spdx": "NoAssertion", + "source_url": "https://huggingface.co/datasets/McAuley-Lab/Amazon-Reviews-2023", + "redistribution_permitted": false, + "attribution_required": true, + "notes": "No upstream license of any kind. McAuley Lab requests citation of Hou et al. 2024 (arXiv:2403.03952). Research use only.", + "scrape_advisory": "Amazon-derived: built by crawling raw HTML from Amazon.com through September 2023 and parsing it to JSON. Amazon's Conditions of Use prohibit the data-mining that produced it and forbid derivative use / redistribution of product listings, descriptions, and prices; review text copyright sits with individual reviewers, not Amazon or the aggregator. No layer in the chain grants third-party redistribution rights. Build and use locally as a research convenience only \u2014 these artifacts must NEVER be uploaded to a shared mirror (publish refuses them by default on this advisory)." + }, + "fetch": { + "type": "huggingface", + "urls": [ + "hf://McAuley-Lab/Amazon-Reviews-2023" + ], + "auth": "huggingface", + "hf_allow_patterns": [ + "raw/review_categories/Subscription_Boxes.jsonl" + ], + "expected_bytes": null, + "expected_sha256": null, + "notes": "Fetches one category file (~9 MB raw JSONL). Sibling categories are added by swapping the allow-pattern + slug." + }, + "extract": { + "type": "passthrough", + "include": [ + "*.jsonl" + ], + "exclude": [], + "post": null + }, + "parse": { + "reader": "jsonl", + "options": {} + }, + "transform": { + "handler": "hf_concat_splits", + "params": { + "add_split_column": false + } + }, + "write": { + "output": "amazon-reviews-2023-subscription-boxes.parquet", + "compression": "zstd", + "row_group_size_rows": 1048576, + "statistics": true, + "page_index": false + }, + "expect": { + "rows": 16216, + "schema_hash": null, + "notes": "16,216 reviews; confirmed on first build.", + "row_stability": "static" + }, + "convert": { + "vortex": true, + "vortex_skip_reason": null + } } ] } diff --git a/tests/test_publish.py b/tests/test_publish.py index 0388550..abdbd08 100644 --- a/tests/test_publish.py +++ b/tests/test_publish.py @@ -117,6 +117,116 @@ def test_main_rejects_no_targets(tmp_path, monkeypatch): publish.main(["--mirror", f"file://{tmp_path}/m"]) +def _scrape_manifest(*slugs_with_advisory, extra=()): + """Manifest where the named slugs carry a license.scrape_advisory and any + `extra` slugs are clean (license present, advisory None).""" + datasets = [{"slug": s, "license": {"scrape_advisory": "scraped; do not mirror"}} + for s in slugs_with_advisory] + datasets += [{"slug": s, "license": {"scrape_advisory": None}} for s in extra] + return {"schema_version": 1, "datasets": datasets} + + +def test_scrape_advisory_slugs_selects_flagged(): + from scripts.pipeline import publish + m = _scrape_manifest("scraped-a", extra=["clean-b"]) + # A slug with no license block at all must not trip the filter. + m["datasets"].append({"slug": "bare"}) + assert publish.scrape_advisory_slugs(m, ["scraped-a", "clean-b", "bare"]) == \ + ["scraped-a"] + + +def test_main_refuses_explicit_scrape_slug(tmp_path, monkeypatch, capsys): + """An explicit `publish ` is blocked and fails loudly (rc=1), + never touching the mirror.""" + from scripts.pipeline import publish + monkeypatch.setattr(publish, "load_manifest", lambda: _scrape_manifest("scraped-a")) + rc = publish.main(["scraped-a", "--mirror", f"file://{tmp_path}/m"]) + assert rc == 1 + err = capsys.readouterr().err + assert "scrape_advisory" in err and "--allow-scrape-advisory" in err + + +def test_main_allow_flag_overrides_block(tmp_path, monkeypatch): + """--allow-scrape-advisory opts back in: the slug reaches plan_uploads (and, + absent any artifact on disk, publishes 0 with rc=0 rather than being refused).""" + from scripts.pipeline import publish + snap = tmp_path / "snapshot.json" + snap.write_text(json.dumps({"slugs": {}})) + monkeypatch.setenv("RAINCLOUD_SNAPSHOT", str(snap)) + monkeypatch.setattr(publish, "load_manifest", lambda: _scrape_manifest("scraped-a")) + monkeypatch.setattr(publish, "_outputs_root", lambda m=None: tmp_path / "outputs" / "v1") + rc = publish.main(["scraped-a", "--allow-scrape-advisory", + "--mirror", f"file://{tmp_path}/m"]) + assert rc == 0 + + +def test_main_all_skips_scrape_slug_publishes_rest(tmp_path, monkeypatch, capsys): + """--all drops the scrape-advisory slug with a notice and still publishes the + clean one.""" + from scripts.pipeline import publish + repo = tmp_path + art = repo / "outputs" / "v1" / "clean-b" / "parquet" / "clean-b.parquet" + art.parent.mkdir(parents=True); art.write_bytes(b"data") + sha = hashlib.sha256(b"data").hexdigest() + snap = repo / "snapshot.json" + snap.write_text(json.dumps({"slugs": {"clean-b": { + "parquet_bytes": 4, "parquet_sha256": sha, + "vortex_bytes": None, "vortex_sha256": None}}})) + monkeypatch.setenv("RAINCLOUD_SNAPSHOT", str(snap)) + monkeypatch.setattr(publish, "load_manifest", + lambda: _scrape_manifest("scraped-a", extra=["clean-b"])) + monkeypatch.setattr(publish, "_outputs_root", lambda m=None: repo / "outputs" / "v1") + rc = publish.main(["--all", "--mirror", f"file://{tmp_path}/mirror", "--dry-run"]) + assert rc == 0 + cap = capsys.readouterr() + assert "refusing scraped-a" in cap.err + assert "v1/clean-b/parquet/clean-b.parquet" in cap.out + + +def test_no_redistribution_slugs_selects_flagged(): + from scripts.pipeline import publish + m = {"schema_version": 1, "datasets": [ + {"slug": "blocked", "license": {"redistribution_permitted": False}}, + {"slug": "ok", "license": {"redistribution_permitted": True}}, + {"slug": "bare"}, # no license block must not trip the filter + ]} + assert publish.no_redistribution_slugs(m, ["blocked", "ok", "bare"]) == ["blocked"] + + +def test_main_refuses_no_redistribution_slug(tmp_path, monkeypatch, capsys): + from scripts.pipeline import publish + m = {"schema_version": 1, "datasets": [ + {"slug": "restricted", "license": {"redistribution_permitted": False, + "scrape_advisory": None}}]} + monkeypatch.setattr(publish, "load_manifest", lambda: m) + rc = publish.main(["restricted", "--mirror", f"file://{tmp_path}/m"]) + assert rc == 1 + err = capsys.readouterr().err + assert "redistribution_permitted=false" in err + assert "--allow-no-redistribution" in err + + +def test_main_both_gates_need_both_flags(tmp_path, monkeypatch, capsys): + """A slug that trips BOTH gates (scrape_advisory + no-redistribution, like the + Amazon corpus) stays refused until BOTH overrides are passed.""" + from scripts.pipeline import publish + m = {"schema_version": 1, "datasets": [ + {"slug": "amazon", "license": {"redistribution_permitted": False, + "scrape_advisory": "scraped; do not mirror"}}]} + monkeypatch.setattr(publish, "load_manifest", lambda: m) + snap = tmp_path / "snapshot.json"; snap.write_text(json.dumps({"slugs": {}})) + monkeypatch.setenv("RAINCLOUD_SNAPSHOT", str(snap)) + monkeypatch.setattr(publish, "_outputs_root", lambda mm=None: tmp_path / "outputs" / "v1") + base = ["amazon", "--mirror", f"file://{tmp_path}/m"] + # Only the scrape override -> still blocked by the redistribution gate. + assert publish.main(base + ["--allow-scrape-advisory"]) == 1 + # Only the redistribution override -> still blocked by the advisory gate. + assert publish.main(base + ["--allow-no-redistribution"]) == 1 + # Both overrides -> reaches plan_uploads (no artifact on disk -> publishes 0). + assert publish.main(base + ["--allow-scrape-advisory", + "--allow-no-redistribution"]) == 0 + + def test_main_dry_run_finds_artifact(tmp_path, monkeypatch, capsys): from scripts.pipeline import publish repo = tmp_path