spiraldb · mprammer · Jun 11, 2026 · Jun 11, 2026 · Jun 11, 2026
diff --git a/docs/v1/datasets.md b/docs/v1/datasets.md
@@ -1,4 +1,4 @@
-<!-- AUTO-GENERATED datasets by scripts/pipeline/docs.py at 2026-05-17T01:07:13Z. Regenerate with: python -m scripts.pipeline.docs datasets. DO NOT EDIT. -->
+<!-- AUTO-GENERATED datasets by scripts/pipeline/docs.py at 2026-06-10T15:47:46Z. Regenerate with: python -m scripts.pipeline.docs datasets. DO NOT EDIT. -->
 
 ## Curated picks
 
@@ -20,6 +20,7 @@
 | Dataset Short Name | Dataset Full Name | Dataset Description | Dataset Source (URL) | Data Kind | License | Row Count | Row Groups - Parquet | File Size - Parquet | File Size - Vortex |
 |--------------------|-------------------|---------------------|----------------------|-----------|---------|-----------|----------------------|---------------------|--------------------|
 | 120 years of Olympic history: athletes and results | 120 years of Olympic history: athletes and results | basic bio data on athletes and medal results from Athens 1896 to Rio 2016. This is a historical dataset on the modern Olympic Games, including all the Games from Athens 1896 to Rio 2016. I scraped this data from www.sports-reference.com in May 2018. The R code I used to scrape and wrangle the data is on GitHub. — adapted from the dataset's Kaggle description (heesoo37/120-years-of-olympic-history-athletes-and-results). | https://github.com/rgriff23/Olympic_history | Tabular (CSV) | CC0-1.0 | 271,116 | 1 | 4.4 MB | 5.6 MB |
+| [⚠ Amazon Reviews 2023 (Subscription Boxes)](#scrape-advisory-amazon-reviews-2023-subscription-boxes) | Amazon Reviews 2023 — Subscription_Boxes user reviews (McAuley Lab, UCSD; Hou et al. 2024) | User reviews for the Subscription_Boxes category from the Amazon Reviews 2023 corpus — rating, title, text, asin, parent_asin, user_id, millisecond timestamp, helpful_vote, verified_purchase, and a nested images list. Smallest of 33 categories; the catalog's smoke-test slice of the 571M-review corpus. | https://huggingface.co/datasets/McAuley-Lab/Amazon-Reviews-2023 | Structured (JSON) | NoAssertion | 16,216 | 1 | 2.4 MB | 3.1 MB |
 | [⚠ C4 (en, validation)](#scrape-advisory-c4-en-validation) | AllenAI/C4 — Colossal Clean Crawled Corpus, English validation split | C4 (Raffel et al., JMLR 2020) — a heavily-filtered scrape of Common Crawl used to pretrain T5. This entry pulls only the 8-shard English validation split (~365k documents), enough for type coverage and as a smoke-test for the Common-Crawl scrape playbook. Flip allow_patterns to `en/c4-train.*.json.gz` to mirror the 327 GB English training set. | https://huggingface.co/datasets/allenai/c4 | Structured (JSON) | ODC-By-1.0 | 364,608 | — | 339.7 MB | 435.3 MB |
 | [⚠ CodeParrot Clean (validation)](#scrape-advisory-codeparrot-clean-valid) | codeparrot/codeparrot-clean — validation split | 61k Python source files scraped from MIT/BSD/Apache-licensed GitHub repos by the CodeParrot project. Validation split (142 MB raw .json.gz). Showcases code-corpus shape: string `content`, numeric quality metrics (line_mean, alpha_frac), boolean autogenerated flag, and per-row license attribution. | https://huggingface.co/datasets/codeparrot/codeparrot-clean-valid | Structured + Blobs | Apache-2.0 | 61,373 | — | 149.7 MB | 343.5 MB |
 | [⚠ FineMath (4+ quality subset)](#scrape-advisory-finemath-4plus) | HuggingFaceTB/finemath — quality≥4 math web pages | Math-focused subset of the fineweb pipeline, filtered to documents with an automated quality score ≥ 4 (the higher of the two quality tiers). 64 parquet shards. Schema mirrors fineweb (`text` + harvest metadata). | https://huggingface.co/datasets/HuggingFaceTB/finemath | Tabular (Parquet) | ODC-By-1.0 | 6,699,493 | 7 | 12,647.7 MB | 21,724.8 MB |
@@ -273,6 +274,9 @@
 
 These datasets aggregate or reference content whose underlying licenses have not been individually cleared. The aggregator's declared license (the License column above) governs only the metadata it ships, not the content it points at. Read each advisory before redistributing or building on top of one of these slugs.
 
+<a id="scrape-advisory-amazon-reviews-2023-subscription-boxes"></a>
+**Amazon Reviews 2023 (Subscription Boxes)** (`amazon-reviews-2023-subscription-boxes`) — Amazon-derived: built by crawling raw HTML from Amazon.com through September 2023 and parsing it to JSON. Amazon's Conditions of Use prohibit the data-mining that produced it and forbid derivative use / redistribution of product listings, descriptions, and prices; review text copyright sits with individual reviewers, not Amazon or the aggregator. No layer in the chain grants third-party redistribution rights. Build and use locally as a research convenience only — these artifacts must NEVER be uploaded to a shared mirror (publish refuses them by default on this advisory).
+
 <a id="scrape-advisory-c4-en-validation"></a>
 **C4 (en, validation)** (`c4-en-validation`) — C4 (Colossal Clean Crawled Corpus) is a heavily-filtered scrape of Common Crawl. Allen AI's CC-BY-4.0 license covers the harvest layer; the underlying web text remains subject to per-publisher copyright. Treat as research convenience, not a license-cleared corpus.
 

diff --git a/docs/v1/handlers.md b/docs/v1/handlers.md
@@ -1,4 +1,4 @@
-<!-- AUTO-GENERATED handlers by scripts/pipeline/docs.py at 2026-05-17T01:07:13Z. Regenerate with: python -m scripts.pipeline.docs handlers. DO NOT EDIT. -->
+<!-- AUTO-GENERATED handlers by scripts/pipeline/docs.py at 2026-06-10T15:47:46Z. Regenerate with: python -m scripts.pipeline.docs handlers. DO NOT EDIT. -->
 | Handler | Purpose | Streaming | Extra Deps | # Manifest Specs | Example Slugs |
 |---------|---------|-----------|------------|------------------|---------------|
 | `beijing_pm25_parse` | Parse UCI Beijing Multi-Site Air Quality (dataset 501). | no | — | 1 | `uci-beijing-multi-site-air-quality` |
@@ -7,7 +7,7 @@
 | `ghcn_daily_parse` | Parse NOAA GHCN-Daily .dly fixed-width records into a long-format parquet. | yes | — | 1 | `ghcn-daily` |
 | `glove_split` | Read a single-dimension GloVe text file and emit a | no | — | 3 | `glove-6b-100d`, `glove-6b-200d` (+1 more) |
 | `har_parse` | Parse UCI Human Activity Recognition Using Smartphones (dataset 240). | no | — | 1 | `uci-human-activity-recognition-using-smartphones` |
-| `hf_concat_splits` | Concat HF parquet shards into one parquet, optionally injecting a | no | — | 52 | `ai2-arc`, `anthropic-hh-rlhf-helpful-base` (+50 more) |
+| `hf_concat_splits` | Concat HF parquet shards into one parquet, optionally injecting a | no | — | 53 | `ai2-arc`, `amazon-reviews-2023-subscription-boxes` (+51 more) |
 | `identity` | Passthrough handler — single parsed table becomes the output. | no | — | 3 | `clickbench-hits`, `emotions-dataset-for-nlp` (+1 more) |
 | `jsonbench_variant_parse` | Parse ClickHouse JSONBench's Bluesky JSONL.gz dumps into a single parquet | yes | — | 1 | `jsonbench-bluesky-100m` |
 | `jsonl_as_string_parse` | Stream a JSONL[.gz] file into a parquet with a single `raw_json: string` | yes | — | 1 | `open-food-facts` |

diff --git a/docs/v1/snapshot.json b/docs/v1/snapshot.json
@@ -1,6 +1,6 @@
 {
   "schema_version": 1,
-  "generated_at": "2026-05-27T18:14:16Z",
+  "generated_at": "2026-06-10T15:47:46Z",
   "note": "Auto-generated by scripts/pipeline/docs.py. Read by the TUI as a fallback when a local parquet isn't built. Regenerate after any build / schema change with `python -m scripts.pipeline.docs snapshot`.",
   "slugs": {
     "clickbench-hits": {
@@ -117070,6 +117070,106 @@
         "wide_row": false,
         "high_cardinality_present": true
       }
+    },
+    "amazon-reviews-2023-subscription-boxes": {
+      "expected_rows": 16216,
+      "last_built_rows": 16216,
+      "last_built_row_groups": 1,
+      "parquet_bytes": 2567134,
+      "vortex_bytes": 3278148,
+      "parquet_sha256": "ce709109a3e505b3e26df331a54e15c09c92c20014f54a27afa2528730fb78be",
+      "vortex_sha256": "f68c40967bc938d8a8b03e5d9582738e84f03d7b7dd130ec3d4bcda917e10823",
+      "columns": [
+        {
+          "name": "rating",
+          "type": "double",
+          "length": 5040,
+          "null_count": 0,
+          "min": 1.0,
+          "max": 5.0
+        },
+        {
+          "name": "title",
+          "type": "string",
+          "length": 215759,
+          "null_count": 0,
+          "min": "!",
+          "max": "\ud83e\udd73\ud83e\udd73\ud83e\udd73"
+        },
+        {
+          "name": "text",
+          "type": "string",
+          "length": 1770531,
+          "null_count": 0,
+          "min": "",
+          "max": "\ud83e\udd1f\ud83d\udc4c\ud83d\ude0e"
+        },
+        {
+          "name": "images",
+          "type": "list<element: struct<small_image_url: string, medium_image_url: string, large_image_url: string, attachment_type: string>>",
+          "length": 98164,
+          "null_count": null,
+          "min": null,
+          "max": null
+        },
+        {
+          "name": "asin",
+          "type": "string",
+          "length": 25015,
+          "null_count": 0,
+          "min": "B01M71IUZ7",
+          "max": "B0CBCVHKG7"
+        },
+        {
+          "name": "parent_asin",
+          "type": "string",
+          "length": 20215,
+          "null_count": 0,
+          "min": "B01M71IUZ7",
+          "max": "B0BXVNMNV6"
+        },
+        {
+          "name": "user_id",
+          "type": "string",
+          "length": 291260,
+          "null_count": 0,
+          "min": "AE224HI2GTJDDOBC3RN3O5SDBNOA",
+          "max": "AHZYSB4Z3JKMWDBSP7KXNXBLIJUA"
+        },
+        {
+          "name": "timestamp",
+          "type": "uint64",
+          "length": 127663,
+          "null_count": 0,
+          "min": 1485302279000,
+          "max": 1693437206732
+        },
+        {
+          "name": "helpful_vote",
+          "type": "uint16",
+          "length": 8769,
+          "null_count": 0,
+          "min": 0,
+          "max": 985
+        },
+        {
+          "name": "verified_purchase",
+          "type": "bool",
+          "length": 1279,
+          "null_count": 0,
+          "min": false,
+          "max": true
+        }
+      ],
+      "size_bucket": "xs",
+      "shape_traits": {
+        "has_nested": true,
+        "has_timestamp": false,
+        "has_variant": false,
+        "string_heavy": false,
+        "wide_row": false,
+        "high_cardinality_present": null
+      }
     }
   }
 }
diff --git a/scripts/pipeline/publish.py b/scripts/pipeline/publish.py
@@ -26,6 +26,37 @@ class PublishMismatch(Exception):
     """On-disk artifact sha256 disagrees with the snapshot."""
 
 
+def scrape_advisory_slugs(manifest, slugs):
+    """Subset of `slugs` whose license carries a non-null `scrape_advisory`.
+
+    These aggregate or reference content whose underlying licenses have not been
+    cleared for redistribution (public-web scrapes, Common Crawl derivatives,
+    Amazon-Conditions-of-Use-governed review corpora). `publish` refuses them by
+    default: building and using such artifacts locally is the customary research
+    posture, but uploading them to a shared mirror is the one act those terms
+    actually forbid. Tolerant of specs with no `license` block (test fixtures).
+    """
+    by_slug = {d["slug"]: d for d in manifest.get("datasets", [])}
+    return [s for s in slugs
+            if (by_slug.get(s, {}).get("license") or {}).get("scrape_advisory")]
+
+
+def no_redistribution_slugs(manifest, slugs):
+    """Subset of `slugs` whose license sets `redistribution_permitted` to False.
+
+    An independent gate from `scrape_advisory_slugs`: the advisory flags a *gap*
+    between an aggregator's declared license and uncleared underlying content,
+    while this is the spec stating outright that the license does not grant
+    redistribution. A slug can trip either or both (the Amazon Reviews corpus
+    trips both); each gate has its own override, so clearing one never silently
+    clears the other. Tolerant of specs with no `license` block (test fixtures).
+    """
+    by_slug = {d["slug"]: d for d in manifest.get("datasets", [])}
+    return [s for s in slugs
+            if (by_slug.get(s, {}).get("license") or {}).get("redistribution_permitted")
+            is False]
+
+
 def plan_uploads(slugs, snapshot, *, outputs_root: Path):
     """Return [(local_path, remote_key)] for present artifacts.
 
@@ -95,6 +126,14 @@ def main(argv=None) -> int:
     ap.add_argument("--all", action="store_true")
     ap.add_argument("--mirror", required=True, help="fsspec base, e.g. s3://b/p")
     ap.add_argument("--dry-run", action="store_true")
+    ap.add_argument(
+        "--allow-scrape-advisory", action="store_true",
+        help="publish slugs whose license carries a scrape_advisory (refused by "
+             "default — their underlying content is not cleared for redistribution)")
+    ap.add_argument(
+        "--allow-no-redistribution", action="store_true",
+        help="publish slugs whose license sets redistribution_permitted=false "
+             "(refused by default — the license does not grant redistribution)")
     args = ap.parse_args(argv)
 
     if args.all and args.slugs:
@@ -105,6 +144,33 @@ def main(argv=None) -> int:
     if not slugs:
         ap.error("pass slugs or --all")
 
+    # Default-block on two independent license gates: a mirror upload IS
+    # redistribution, and neither a scrape_advisory nor redistribution_permitted=
+    # false clears it. Each gate has its own --allow-* override, so clearing one
+    # never silently clears the other (a slug tripping both needs both flags).
+    # --all skips blocked slugs and keeps going; an explicit publish that leaves
+    # nothing to upload fails loudly so it doesn't read as a no-op success.
+    gates = (
+        (scrape_advisory_slugs(manifest, slugs), args.allow_scrape_advisory,
+         "license carries a scrape_advisory — underlying content not cleared for "
+         "redistribution", "--allow-scrape-advisory"),
+        (no_redistribution_slugs(manifest, slugs), args.allow_no_redistribution,
+         "license sets redistribution_permitted=false", "--allow-no-redistribution"),
+    )
+    blocked: set[str] = set()
+    for hit, allowed, reason, flag in gates:
+        if allowed:
+            continue
+        for s in sorted(hit):
+            print(f"refusing {s}: {reason} (pass {flag} to override)", file=sys.stderr)
+        blocked.update(hit)
+    if blocked:
+        slugs = [s for s in slugs if s not in blocked]
+        if not slugs:
+            print(f"refused all {len(blocked)} requested slug(s); nothing to publish",
+                  file=sys.stderr)
+            return 1
+
     # Resolve the snapshot the same way the loader does (RAINCLOUD_SNAPSHOT ->
     # checkout -> wheel-packaged), so publish gates against the file the loader
     # will trust rather than a hardcoded REPO_ROOT path.

diff --git a/sources.json b/sources.json
@@ -16531,6 +16531,68 @@
         "nested-json",
         "prose"
       ]
+    },
+    {
+      "slug": "amazon-reviews-2023-subscription-boxes",
+      "short_name": "Amazon Reviews 2023 (Subscription Boxes)",
+      "full_name": "Amazon Reviews 2023 \u2014 Subscription_Boxes user reviews (McAuley Lab, UCSD; Hou et al. 2024)",
+      "description": "User reviews for the Subscription_Boxes category from the Amazon Reviews 2023 corpus \u2014 rating, title, text, asin, parent_asin, user_id, millisecond timestamp, helpful_vote, verified_purchase, and a nested images list. Smallest of 33 categories; the catalog's smoke-test slice of the 571M-review corpus.",
+      "license": {
+        "spdx": "NoAssertion",
+        "source_url": "https://huggingface.co/datasets/McAuley-Lab/Amazon-Reviews-2023",
+        "redistribution_permitted": false,
+        "attribution_required": true,
+        "notes": "No upstream license of any kind. McAuley Lab requests citation of Hou et al. 2024 (arXiv:2403.03952). Research use only.",
+        "scrape_advisory": "Amazon-derived: built by crawling raw HTML from Amazon.com through September 2023 and parsing it to JSON. Amazon's Conditions of Use prohibit the data-mining that produced it and forbid derivative use / redistribution of product listings, descriptions, and prices; review text copyright sits with individual reviewers, not Amazon or the aggregator. No layer in the chain grants third-party redistribution rights. Build and use locally as a research convenience only \u2014 these artifacts must NEVER be uploaded to a shared mirror (publish refuses them by default on this advisory)."
+      },
+      "fetch": {
+        "type": "huggingface",
+        "urls": [
+          "hf://McAuley-Lab/Amazon-Reviews-2023"
+        ],
+        "auth": "huggingface",
+        "hf_allow_patterns": [
+          "raw/review_categories/Subscription_Boxes.jsonl"
+        ],
+        "expected_bytes": null,
+        "expected_sha256": null,
+        "notes": "Fetches one category file (~9 MB raw JSONL). Sibling categories are added by swapping the allow-pattern + slug."
+      },
+      "extract": {
+        "type": "passthrough",
+        "include": [
+          "*.jsonl"
+        ],
+        "exclude": [],
+        "post": null
+      },
+      "parse": {
+        "reader": "jsonl",
+        "options": {}
+      },
+      "transform": {
+        "handler": "hf_concat_splits",
+        "params": {
+          "add_split_column": false
+        }
+      },
+      "write": {
+        "output": "amazon-reviews-2023-subscription-boxes.parquet",
+        "compression": "zstd",
+        "row_group_size_rows": 1048576,
+        "statistics": true,
+        "page_index": false
+      },
+      "expect": {
+        "rows": 16216,
+        "schema_hash": null,
+        "notes": "16,216 reviews; confirmed on first build.",
+        "row_stability": "static"
+      },
+      "convert": {
+        "vortex": true,
+        "vortex_skip_reason": null
+      }
     }
   ]
 }