Full DCAT-US1.1 translator testing and iteration

[jbrown-xentity]

User Story

In order to evaluate the translator from 1.1 to 3.0, data.gov team wants to run the translator against all DCAT-US1.1 sources today and evaluate the output.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN the translation script exists
  WHEN I run it against all DCAT-US1.1 harvest endpoints
  THEN all documents are valid
  OR any invalid documents are marked as not fixable and documented clearly.

Background

Previous ticket for translator creation

Security Considerations (required)

None

Sketch

This should result in some iterative improvement across various data sources, finding edge cases and other items that should improve the translator.

This is not meant to be perfect and work for every situation; if something can't be fixed/translated then that should be documented. If we have agency contact points then we should do pre-emptive outreach to make them aware of the complications that may arise.

[akuny]

After combing through the harvest sources available by way of https://harvest.data.gov/harvest_sources/?per_page=1000

Edit: superseded by subsequent results below.

Status	Count of URLs
invalid v1.1, no valid subset	55
converts	46
fetch error	13
invalid v1.1, has valid subset	12
valid v1.1 but uses reserved field name	1
malformed	1

[jbrown-xentity]

@akuny this looks a little surprising to me (55 are completely invalid, yikes), but not shocking.
Can you get a spreadsheet or a list going with the individual sources and it's success or failure status? And maybe a summary of what is not valid for 1.1? If it's program or bureau code, then it can probably be ignored...

[akuny]

@jbrown-xentity This PR contains a CSV file breaking down the validation and conversion status of each harvest source URL, along with log output demonstrating which fields were invalid in the case of the 67 URLs that don't point to valid v1.1 data.

[jbrown-xentity]

To summarize from a business perspective:

Every valid 1.1 dataset converts to valid 3.0 datasets except for NIST, NIST has extra metadata for a field called replaces in 17 different datasets (of their 1222: https://harvest.data.gov/harvest_source/11fee1a2-8298-4fae-919e-81aeb342afa7). While this is fine in 1.1; it is a reserved field in DCAT-US3.0 (see https://resources.data.gov/resources/dcat-us-3-dataset/, it should be an array). So there is at least 1 use case where problems may be encountered due to the "openness" of 1.1 and the number of new fields in 3.0.

Tagging @SueValente as we should consider following up and preparing NIST.

[SueValente]

Thank you - yes can you help me prepare the wording and we can add it to the webpage(s), and then communicate it more broadly from there.

[akuny]

After swapping the v1.1 dataset file used for validation per @jbrown-xentity 's comment here, here's the update overview:

Status	Count of URLs
invalid v1.1, has valid subset	54
converts	49
fetch error	14
invalid v1.1, no valid subset	9
valid v1.1, but uses reserved field name	1