Add CSI coarse-sense layer as an add-on over frozen lexen-v1

CHANGELOG.md

@@ -2,6 +2,19 @@
 
 All notable dataset and tooling changes are recorded here.
 
+## Unreleased - CSI coarse-sense add-on - 2026-06-16
+
+* Added a second, public coarse-sense layer (**CSI**, Coarse Sense Inventory; Lacerra et al., AAAI
+  2020, CC BY-NC-SA 4.0) as an **add-on resource** under `coarsenings/csi/`, alongside the native
+  in-record Glite layer, so coarse-sense results are reproducible under a third-party inventory.
+* Shipped the per-item CSI block as the sidecar `coarsenings/csi/csi_layer.jsonl` (keyed by
+  `item_id`, same shape as `item["glite"]`), plus the forward map, aliases, and coverage summary.
+* **Additive only:** the frozen `lexen-v1` artifacts (`data/lexen-v1/items.jsonl`,
+  `dataset.json`) and `sources/manifest.json` are unchanged; the release content hash is unchanged
+  and `lexen-verify-release --release lexen-v1` still passes. No new dataset version.
+* Added `scripts/apply_csi.py` to (re)build the sidecar; it asserts byte-for-value reproduction of
+  the existing Glite block before emitting CSI, guaranteeing structural parity.
+
 ## lexen-v1 - 2026-06-13
 
 Initial lexEN release.

LICENSE

@@ -27,3 +27,15 @@ artifacts are released for research use and non-commercial evaluation.
 The source manifest in `sources/manifest.json` records the source package, local files, license
 label, access status, and SHA-256 hashes for each input package. The release metadata in
 `data/lexen-v1/dataset.json` repeats the release-level policy under `license`.
+
+## Add-on coarsening layers (`coarsenings/`)
+
+`coarsenings/` holds optional coarse-sense layers that map onto the frozen `lexen-v1` release
+without modifying it (the verified release surface — `data/lexen-v1/` and `sources/manifest.json` —
+is unchanged). The **CSI** layer (`coarsenings/csi/`) is derived from the Coarse Sense Inventory of
+Lacerra, Bevilacqua, Pasini & Navigli (AAAI 2020), <https://sapienzanlp.github.io/csi/>, which is
+licensed **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)**.
+This derived subset is distributed under the same CC BY-NC-SA 4.0 terms, with attribution to Lacerra
+et al. (2020); the only modification is the join to lexEN's referenced sense keys and the reduction
+of each synset's multi-domain CSI label set to a single composite class (declared as a change per the
+ShareAlike terms). See `coarsenings/csi/details.json`.

README.md

@@ -61,8 +61,8 @@ The construction process is:
 
 1. Take Maru2022 ALLamended as the base benchmark.
 2. Run a [WSD prediction panel](sources/model-panel/) over all 4,917 Maru2022 items.
-3. Flag 363 suspicious items where the model panel strongly disagrees with the Maru2022 label
-   using the [selection procedure](docs/selection.md).
+3. Flag 363 suspicious items where the model panel strongly disagrees with the Maru2022 label using
+   the [selection procedure](docs/selection.md).
 4. Review those 363 items at [marureview.com](https://marureview.com/).
 5. Give the same [reviewer brief](https://marureview.com/brief)
    ([repository copy](sources/reviews/protocols/marureview-brief-2026-05-26.md)) to three
@@ -81,10 +81,9 @@ The construction process is:
 
 ## Suspicious-Item Selection
 
-Selection: a model panel of three families - SANDWiCH; GPT-5.5; and a CatBoost
-ensemble of WSD models (ConSeC, ESCHER, BEM, MFS, gpt-5-mini) - flagged 363
-items for review via an S1-S6 waterfall (S1 55, S2 138, S3 41, S4 75, S5 48,
-S6 6; not flagged 4,554).
+Selection: a model panel of three families - SANDWiCH; GPT-5.5; and a CatBoost ensemble of WSD
+models (ConSeC, ESCHER, BEM, MFS, gpt-5-mini) - flagged 363 items for review via an S1-S6 waterfall
+(S1 55, S2 138, S3 41, S4 75, S5 48, S6 6; not flagged 4,554).
 
 The S1-S6 waterfall classes are recomputed from committed source predictions in
 [`sources/model-panel/`](sources/model-panel/). The GPT prediction files keep prompts and normalized

coarsenings/README.md

@@ -0,0 +1,32 @@
+# Coarse-sense add-on layers for lexEN
+
+This directory holds **add-on coarse-sense layers** that map onto the frozen `lexen-v1`
+release **without modifying it**. The released dataset (`data/lexen-v1/items.jsonl`,
+`data/lexen-v1/dataset.json`) and its source ledger (`sources/manifest.json`) are
+unchanged and still pass `lexen-verify-release`; these layers live entirely outside that
+verified surface.
+
+Two coarsenings are available for lexEN:
+
+- **Glite** — shipped *in-record* as `item["glite"]` inside `data/lexen-v1/items.jsonl`
+  (the dataset's native coarse layer; a learner-dictionary inventory).
+- **CSI** — shipped *here* as an add-on (`coarsenings/csi/`), a public third-party
+  inventory (Coarse Sense Inventory, Lacerra et al., AAAI 2020) provided so coarse-sense
+  results can be reproduced under an inventory no lexEN author controls.
+
+## `coarsenings/csi/`
+
+| Path | Purpose |
+| --- | --- |
+| `files/wordnet_sense_key_to_csi_concept.jsonl` | Forward map: WordNet sense key → CSI composite concept id. |
+| `files/csi_report_aliases.json` | Aliases for legacy/British keys, remapped to their CSI concept. |
+| `files/mapping_coverage.json` | Coverage over the lexEN candidate/gold/review inventory (79.2%). |
+| `csi_layer.jsonl` | **Per-item sidecar**, one line `{ "item_id", "csi": {…} }`, the `csi` block in the same shape as `item["glite"]`. Join to `items.jsonl` on `item_id`. |
+| `details.json`, `description.md` | Package metadata, provenance, and license. |
+
+Regenerate the sidecar with `python scripts/apply_csi.py` (stdlib only). That script also
+asserts it reproduces the existing `item["glite"]` block exactly before emitting CSI, so
+the CSI block is structurally identical to the native Glite layer.
+
+**License:** CSI is **CC-BY-NC-SA 4.0** (Lacerra et al. 2020); this derived subset is
+distributed under the same terms with attribution. See `coarsenings/csi/details.json`.

coarsenings/csi/description.md

@@ -0,0 +1,44 @@
+---
+spec_version: "1"
+dataset_id: "csi-coarsening-maru2022-subset"
+source_dataset_id: "csi"
+source_package_generated_at: "2026-06-16"
+date_added_to_lexen: "2026-06-16"
+---
+# CSI Coarsening Subset (third-party public coarse inventory)
+
+This directory contains a **CSI** coarse-sense layer for lexEN — an independent, third-party
+alternative to the Glite coarsening. CSI (Coarse Sense Inventory) is the 45-domain inventory of
+Lacerra, Bevilacqua, Pasini & Navigli (AAAI 2020). It is provided so coarse-sense results computed
+against lexEN can be reproduced under a public inventory that no lexEN author controls.
+
+This layer is an **add-on resource over the frozen `lexen-v1` release**: it does not modify
+`data/lexen-v1/items.jsonl` or the release content hash. Per-item CSI labels are shipped as the
+sidecar `data/lexen-v1/csi_layer.jsonl`, keyed by `item_id`, mirroring the in-record `glite` block.
+
+## Included Files
+
+| File | Purpose |
+| --- | --- |
+| `files/wordnet_sense_key_to_csi_concept.jsonl` | Forward map: WordNet sense key → CSI composite concept id. |
+| `files/mapping_coverage.json` | Coverage summary over the lexEN candidate/gold/review inventory. |
+| `files/csi_report_aliases.json` | Report-local aliases for legacy/British WordNet sense keys, remapped to their CSI concept. |
+
+## Coarsening Policy
+
+Each WordNet synset that lexEN references is mapped to a single **composite CSI concept**
+`csi:<sorted+joined domain labels>`. CSI assigns some synsets more than one domain; we collapse a
+synset's full domain set into one composite class. This keeps the layer a clean **partition** that
+plugs into the same single-concept coarse-scoring contract as Glite (a coarse hit inherits a fine hit,
+else predicted concept ∈ gold concepts). If a sense key is not present in the public CSI map, the
+coarsener first checks the alias file; if no alias exists it keeps an explicit `unmapped:<sense_key>`
+marker rather than guessing a concept. CSI covers 79.2% of the referenced keys; the remainder are
+`unmapped:` and therefore never silently merged.
+
+## Provenance & License
+
+Derived from the released CSI resource (`wn_synset2csi.txt`, WordNet 3.0 synset offsets) at
+<https://sapienzanlp.github.io/csi/>. CSI is licensed **CC-BY-NC-SA 4.0**; this derived subset is
+distributed under the same terms, with attribution to Lacerra et al. (2020). The only modification is
+the join to lexEN's referenced sense keys plus the multi-domain→composite reduction described above
+(declared as a change per the ShareAlike terms).

coarsenings/csi/details.json

@@ -0,0 +1,42 @@
+{
+  "access_kind": "public",
+  "authors": [
+    {"country": "Italy", "institution": "Sapienza University of Rome", "name": "Caterina Lacerra", "orcid": null},
+    {"country": "Italy", "institution": "Sapienza University of Rome", "name": "Michele Bevilacqua", "orcid": null},
+    {"country": "Italy", "institution": "Sapienza University of Rome", "name": "Tommaso Pasini", "orcid": null},
+    {"country": "Italy", "institution": "Sapienza University of Rome", "name": "Roberto Navigli", "orcid": null}
+  ],
+  "categories": [
+    "sense-mapping",
+    "knowledge-base",
+    "wsd"
+  ],
+  "checksums": {
+    "csi_report_aliases.json": "sha256:fb4996937b97e866a6741f71a1814874f79838819f961ae70e2667ccce02fd3a",
+    "mapping_coverage.json": "sha256:35318031756ad4151a53eb8716cc5d7065bb8417616430659f89244caca85848",
+    "wordnet_sense_key_to_csi_concept.jsonl": "sha256:b84ee722e7cb6df6ad86e2fe55985e736b7d28e5871d6baef23e9f754f2b5e2a"
+  },
+  "dataset_id": "csi-coarsening-maru2022-subset",
+  "date_added": "2026-06-16",
+  "date_added_to_lexen": "2026-06-16",
+  "date_published": "2020-02-07",
+  "download_url": "https://raw.githubusercontent.com/SapienzaNLP/csi/master/docs/_download/csi_data.zip",
+  "files": [
+    {"description": "Forward map from Maru2022/lexEN WordNet sense keys to CSI composite concept ids (sorted+joined domain labels).", "format": "jsonl", "path": "files/wordnet_sense_key_to_csi_concept.jsonl"},
+    {"description": "Machine-readable coverage summary for the CSI coarsening subset.", "format": "json", "path": "files/mapping_coverage.json"},
+    {"description": "Report-local alias table for legacy/British WordNet sense keys, remapped to their CSI concept.", "format": "json", "path": "files/csi_report_aliases.json"}
+  ],
+  "institutions": ["Sapienza NLP Group, Sapienza University of Rome"],
+  "license": "CC-BY-NC-SA-4.0",
+  "modifications": "Derived from the released CSI WordNet-3.0 synset->domain mapping (wn_synset2csi.txt). We (1) joined each synset's CSI sense keys to the WordNet sense keys lexEN references, and (2) reduced each synset's multi-domain label set to a single composite class (sorted+joined), a conservative partition for single-concept coarse scoring. Distributed under the same CC-BY-NC-SA 4.0 license as the original.",
+  "name": "CSI Coarse Sense Inventory subset for Maru2022/lexEN",
+  "short_description": "Public WordNet sense-key to CSI composite-concept subset, a third-party coarse inventory used as an independent coarse-sense label layer alongside Glite.",
+  "size_description": "3 files; the forward map contains 7,783 WordNet sense-key mappings to 413 composite CSI concepts over the lexEN candidate/gold/review inventory (79.2% of 9,833 referenced keys; uncovered keys carry an explicit unmapped: marker).",
+  "source_dataset_id": "csi",
+  "source_package_generated_at": "2026-06-16",
+  "source_paper_id": "lacerra2020",
+  "spec_version": "1",
+  "url": "https://sapienzanlp.github.io/csi/",
+  "version": null,
+  "year": 2020
+}

coarsenings/csi/files/csi_report_aliases.json

@@ -0,0 +1,35 @@
+{
+  "spec_version": "1",
+  "aliases": [
+    {
+      "source_sense_key": "court%1:06:05::",
+      "target_sense_key": "court%1:14:00::",
+      "concept_id": "csi:LAW_AND_CRIME_",
+      "reason": "Legacy WordNet sense key for the legal-tribunal noun sense; Glite maps the equivalent legal-court concept through court%1:14:00:: and court%1:06:02::."
+    },
+    {
+      "source_sense_key": "organisation%1:04:01::",
+      "target_sense_key": "organization%1:04:01::",
+      "concept_id": "csi:POLITICS_GOVERNMENT_AND_NOBILITY_",
+      "reason": "British-spelling WordNet sense key; canonical Glite map contains the equivalent American-spelling organization sense key."
+    },
+    {
+      "source_sense_key": "organisation%1:14:00::",
+      "target_sense_key": "organization%1:14:00::",
+      "concept_id": "csi:POLITICS_GOVERNMENT_AND_NOBILITY_",
+      "reason": "British-spelling WordNet sense key; canonical Glite map contains the equivalent American-spelling organization sense key."
+    },
+    {
+      "source_sense_key": "organisation%1:14:01::",
+      "target_sense_key": "organization%1:14:01::",
+      "concept_id": "csi:BUSINESS_ECONOMICS_AND_FINANCE_+POLITICS_GOVERNMENT_AND_NOBILITY_",
+      "reason": "British-spelling WordNet sense key; canonical Glite map contains the equivalent American-spelling organization sense key."
+    },
+    {
+      "source_sense_key": "show%1:10:00::",
+      "target_sense_key": "show%1:04:00::",
+      "concept_id": "csi:ART_ARCHITECTURE_AND_ARCHAEOLOGY_+MEDIA_",
+      "reason": "Legacy WordNet noun sense for a public entertainment/performance; Glite maps the equivalent event concept through show%1:04:00::."
+    }
+  ]
+}