Design document for the Zenodo like DOI per dandiset

[yarikoptic]

A design doc composed with @djarecka to avoid dummy DOIs for dandisets

refs:

• Stop injecting "fake" DOIs into draft dandisets #1709
• Unpublished Dandisets display a DOI under Cite As #1932

TODOs

• complete initial pass
• seek review
• make explicit that DOI generation is optional overall, since LINC does not even need it (ref)
• potentially add a sequence diagram of interactions between user, archive, datacite across different stages of embargoed -> public dandisets

but could already be checked out by @dandi/archive-maintainers folks since overall idea is formulated already and some early concerns/questions could already be asked/answered

[djarecka]

I created some test to simulate the workflow in dandi/dandi-schema#275

[yarikoptic]

@djarecka we are to answer those I think before we could call this "done" ;-)

[yarikoptic]

@djarecka I feel like you clarified on this but we did not put it "in writing" here. What do you remember on this aspect?

[yarikoptic]

ping @djarecka

[djarecka]

yes, sorry, i just gave "thumbs up", since you were right, Draft DOI doesn't have to have metadata, it can only have url, and if has more, it will be added

[asmacdo]

In the POC I've added a doi field to the Dandiset model which does add a migration.

[yarikoptic]

isn't there also a "draft" Version with DOI (and that's where I guess we inject a fake one), i.e. could we avoid changing DB model?

[asmacdo]

I have to admit that I "just dont like that" since the Dandiset DOI semantically belongs to the Dandiset. I cant think of a reason this wouldnt work, but just feels messy.

To retrieve the Dandiset DOI via Django, someversion.dandiset.draft_version.doi IMO violates the "principle of least surprise".

Prior to publication the Dandiset DOI will point to the draft version (via the DLP) but after publication the dandiset DOI will point to the latest publication, so that would also be surprising

[asmacdo]

Do we have any more information about this?

[yarikoptic]

It might have been a bad memory, @djarecka do you have any information on this or should we just remove this?

Suggested change

- Test site of datacite had different result of validation that the primary one

[djarecka]

let's remove it. It was based on my memory of some old issue, but wasn't able to reproduce, or find old records.

[asmacdo]

For dandi-schema, we may also want to pull the validation out of to_datacite. This validation does not use the datacite API, rather validation occurs against the schema which has been pre-fetched and committed into the dandi-schema repo.

Currently to_datacite accepts an optional arg validate, which defaults to False-- the only use of to_datacite in dandi-archive DOES NOT enable validation!. I'm curious what problems this could cause, and if those problems have actually occurred-- maybe there are more Dandisets with DOIs stored in the data model without a DOI minted? Or maybe they were created but set to Draft due to validation errors? We would have logged exceptions if the datacite API did not accept a new DOI.

From this design doc only, I gather that we believe Draft DOIs have less stringent validation than Findable, but I have not found any upstream documentation that confirms this. If it is the case, I suggest we pull the validation out of the to_datacite function, which would only be responsible for constructing the API payload. Then, in dandi-archive, we perform a validation. If valid we publish a Findable DOI, and if invalid we fallback to Draft DOI if it fails validation.

Either way, we need to consider using some kind of validation. @djarecka @yarikoptic I suggest this as topic for our discussion tomorrow.

[djarecka]

For dandi-schema, we may also want to pull the validation out of to_datacite. This validation does not use the datacite API, rather validation occurs against the schema which has been pre-fetched and committed into the dandi-schema repo.

correct!

Currently to_datacite accepts an optional arg validate, which defaults to False-- the only use of to_datacite in dandi-archive DOES NOT enable validation!. I'm curious what problems this could cause, and if those problems have actually occurred-- maybe there are more Dandisets with DOIs stored in the data model without a DOI minted? Or maybe they were created but set to Draft due to validation errors? We would have logged exceptions if the datacite API did not accept a new DOI.

That's probably not good, if the validation is not used before publishing. Was not aware.

From this design doc only, I gather that we believe Draft DOIs have less stringent validation than Findable, but I have not found any upstream documentation that confirms this. If it is the case, I suggest we pull the validation out of the to_datacite function, which would only be responsible for constructing the API payload. Then, in dandi-archive, we perform a validation. If valid we publish a Findable DOI, and if invalid we fallback to Draft DOI if it fails validation.

Yes, no documentation, but that's the case...

[yarikoptic]

@djarecka did you check in your experiments?

[yarikoptic]

@asmacdo -- do you think this one is good enough and reflects everything you learned from implementing #2350 ? then we likely should just merge it. Any objections? @djarecka what are you thoughts?

[asmacdo]

@yarikoptic I think its good enough to merge. There are remaining "unknowns" relating to how datacite will respond with requests for Findable DOIs that might not have fully valid metadata, which will hopefully be illuminated by incorporating NSKEY so I can create permanent Findable DOIs that will not cause collisions. Whatever comes of that, I think its fine to open a new PR to the design doc if necessary.

[yarikoptic]

@asmacdo then please approve this PR "officially" -- I can't since I am the author. I have enabled auto-merge though ;)

[asmacdo]

Oh right, forgot this wasnt mine :)

[dandibot]

🚀 PR was released in v0.11.1 🚀