Add per-asset dataStandard, HED standard, and extensions to StandardsType by yarikoptic · Pull Request #371 · dandi/dandi-schema

yarikoptic · 2026-02-24T22:11:01Z

Summary

Add dataStandard: Optional[List[StandardsType]] field to BareAsset so dandi-cli can declare per-asset data standards (NWB, BIDS, HED, OME/NGFF) that flow through to AssetsSummary during aggregation
Add version: Optional[str] field to StandardsType to track standard versions (e.g. NWB 2.7.0, BIDS 1.9.0, HED 8.2.0)
Add extensions: Optional[List[StandardsType]] self-referencing field to StandardsType for standard extensions (NWB ndx-* extensions, HED library schemas)
Add hed_standard constant alongside existing nwb_standard, bids_standard, ome_ngff_standard
Collect per-asset dataStandard entries in aggregate_assets_summary(), with deprecated heuristic fallbacks for older clients (remove after 2026-12-01)
Register dataStandard in migrate() _FIELDS_INTRODUCED for downgrade pruning
Comprehensive CLAUDE.md with architecture docs and schema change checklist

Companion dandi-cli PR: dandi/dandi-cli#XXXX

Test plan

test_hed_standard_structure — verifies hed_standard has correct name, identifier, schemaKey
test_aggregate_per_asset_datastandard — HED declared per-asset flows to summary
test_aggregate_per_asset_datastandard_no_duplication — BIDS not duplicated when declared both per-asset and via heuristic
All 243 existing tests pass

PR in dandi-cli:

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions dandi-cli#1811

🤖 Generated with Claude Code

codecov · 2026-02-24T22:58:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.93%. Comparing base (e3ea45b) to head (571978e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #371      +/-   ##
==========================================
+ Coverage   97.91%   97.93%   +0.02%     
==========================================
  Files          18       18              
  Lines        2401     2427      +26     
==========================================
+ Hits         2351     2377      +26     
  Misses         50       50

Flag	Coverage Δ
unittests	`97.93% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dandischema/metadata.py

…s on StandardsType - Bump DANDI_SCHEMA_VERSION to 0.8.0, add 0.7.0 to ALLOWED_INPUT_SCHEMAS - Add `dataStandard: Optional[List[StandardsType]]` to BareAsset for per-asset standard declarations (NWB, BIDS, HED, OME/NGFF) - Add `version: Optional[str]` to StandardsType - Add `extensions: Optional[List["StandardsType"]]` (self-referencing) to StandardsType for NWB ndx-* extensions, HED library schemas, etc. - Add `hed_standard` constant (RRID:SCR_014074) - Collect per-asset dataStandard in aggregate_assets_summary(), with deprecated path/encoding heuristic fallbacks (remove after 2026-12-01) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…cklist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

candleindark · 2026-03-03T19:07:02Z

dandischema/models.py

+    dataStandard: Optional[List[StandardsType]] = Field(
+        None,
+        description="Data standard(s) applicable to this asset.",
+        json_schema_extra={"readOnly": True},


readOnly should set to true only in the case that the dataStandard field is assigned and managed solely by the server per https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-validation-00#rfc.section.9.4. Is it the case that the dataStandard field is intended to be assigned and managed solely by the server?

I would guess the answer is no -- that it is intended to be extracted from the source files by the DANDI CLI running client-side.

INTERESTING! We also have readOnly on above "TODO" attributes which I guess nobody ever assigned to any asset. Although we are indeed to populate it from client (or on server) but since we have other similar wasGeneratedBy also editable, I see no point in making this readOnly, and I guess it is really our own thing, hence

Suggested change

json_schema_extra={"readOnly": True},

json_schema_extra={"nskey": DANDI_NSKEY},

right?

bendichter · 2026-03-03T19:46:42Z

@yarikoptic this looks great! It will be exciting to be able to query the archive for NWB version and extension usage for data going forward. I agree with @candleindark that this probably should not be readOnly. Other than that, looks good. It would be great to also see a draft of dandi-cli that extracts this information. I'd prefer to have a draft of that before merging this, in case we are able to catch any edge cases.

yarikoptic · 2026-03-03T19:50:46Z

It would be great to also see a draft of dandi-cli that extracts this information

It is there waiting for you

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions dandi-cli#1811

candleindark

Left some minor suggestions for change

candleindark · 2026-03-03T19:13:13Z

dandischema/models.py

+    dataStandard: Optional[List[StandardsType]] = Field(
+        None,
+        description="Data standard(s) applicable to this asset.",
+        json_schema_extra={"readOnly": True},


Shouldn't we have an "nskey" key in the json_schema_extra dict? If dataStandard is not defined in a known ontology, we I think we should use ours, i.e.,

json_schema_extra={"nskey": DANDI_NSKEY}

candleindark · 2026-03-03T19:39:42Z

dandischema/models.py

+    )
+    extensions: Optional[List["StandardsType"]] = Field(
+        None,
+        description="Extensions to the standard used in this dataset "


Suggested change

description="Extensions to the standard used in this dataset "

description="Extensions to the standard used"

The standard extended can be just one applicable to an asset, not an entire dataset per definition of dataStandard in BareAsset.

candleindark · 2026-03-03T19:58:17Z

dandischema/models.py

+        description="Version of the standard used.",
+        json_schema_extra={"nskey": "schema"},
+    )
+    extensions: Optional[List["StandardsType"]] = Field(


Suggested change

extensions: Optional[List["StandardsType"]] = Field(

extensions: Optional[List[StandardsType]] = Field(

With from __future__ import annotations at the beginning of the file, you don't need the type to be wrapped in a string.

candleindark · 2026-03-03T20:07:28Z

dandischema/models.py

+    extensions: Optional[List["StandardsType"]] = Field(
+        None,
+        description="Extensions to the standard used in this dataset "
+        "(e.g. NWB extensions like ndx-*, HED library schemas).",


Shouldn't we set "nskey" key in the json_schema_extra dict? If extensions is not defined in a known ontology, we I think we should use ours, i.e.,

json_schema_extra={"nskey": DANDI_NSKEY}

candleindark · 2026-03-03T20:22:25Z

dandischema/models.py

    identifier="DOI:10.25504/FAIRsharing.9af712",
 ).model_dump(mode="json", exclude_none=True)

+hed_standard = StandardsType(


Defining hed_standard in lower case is consistent with the established pattern, as in ome_ngff_standard. However, all the standard variables, nwb_standard, bids_standard, ome_ngff_standard, and hed_standard should be specified in upper case laters since they function as constants.

I think that should be done in a separate PR though.

bendichter · 2026-03-03T20:27:59Z

I approve the dandi dli PR. Once we respond to @candleindark 's comments, this is good to go from my end

yarikoptic mentioned this pull request Feb 24, 2026

Populate per-asset dataStandard for BIDS, HED, NWB, and extensions dandi/dandi-cli#1811

Open

5 tasks

yarikoptic commented Mar 3, 2026

View reviewed changes

dandischema/metadata.py Outdated Show resolved Hide resolved

yarikoptic requested review from bendichter, candleindark and satra March 3, 2026 16:31

yarikoptic and others added 2 commits March 3, 2026 11:33

docs: comprehensive CLAUDE.md with architecture, conventions, and che…

571978e

…cklist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

yarikoptic force-pushed the enh-hed branch from be40b26 to 571978e Compare March 3, 2026 16:33

candleindark reviewed Mar 3, 2026

View reviewed changes

	json_schema_extra={"readOnly": True},
	json_schema_extra={"nskey": DANDI_NSKEY},

	description="Extensions to the standard used in this dataset "
	description="Extensions to the standard used"

	extensions: Optional[List["StandardsType"]] = Field(
	extensions: Optional[List[StandardsType]] = Field(

Conversation

yarikoptic commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

codecov bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

yarikoptic commented Mar 3, 2026

Uh oh!

candleindark left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bendichter commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yarikoptic commented Feb 24, 2026 •

edited

Loading

codecov bot commented Feb 24, 2026 •

edited

Loading