SPIKE Research: Ingest & Display DCAT US 3 Metadata

[SueValente]

Purpose

We want to be able to ingest and display DCAT US 3 metadata when provided by agencies, but we're not sure how many or which parts of the system need to change to make this happen

Given above need, conducting researching and investigation, along with test acceptance of sample DCAT US 3 files is needed to provide factual knowledge on future steps.

16 hours of effort has been allocated and once compete, findings will be demonstrated and specific future actions will be decided.

Steps

• Daily stand up report out on this specific item, from each team member.

Tracking

• Track ideas and thoughts here: https://docs.google.com/spreadsheets/d/1RdAN89XXQOO7fTRpXOe2jdEyplqVmzp_lVspRhVRres/edit?usp=sharing

Baseline Considerations

• New DCAT schema for records model
• Update validation schema
• New writer for DCAT-3 standard
• Create a DCAT 1.1 Reader
• Manual input of universe of errors for new/changed properties

Adjacent but NOT part of this ticket:

Create Validator Tool:

• Creation of a new translator & validator that changes 1.1 to 3 to aid agency adoption
• DCAT-3 Validator

Modify Inventory.data.gov

• Inventory: Update data entry form and validation criteria
• Modification of inventory.data.gov to output DCAT-US 3 as a “base point” for agencies to get started

[jbrown-xentity]

Alright, I would cut up this ticket quite a bit. I don't see any reason to change the MDTranslator, or Inventory, as part of this work.

The goal is to be able to harvest DCAT-US3.0 metadata and have it available for searching/filtering/viewing in catalog.

I've backlogged the tickets I believe necessary to complete this work inside the parent epic. There are some gotchas, specifically around catalog level metadata other than dataset. Historically we've never cared or captured that information, in fact we didn't even validate it! That is changing with DCAT-US3.0. Regardless of how valuable most of the catalog level metadata is, there are some complexities. The service field (class DataService) (see here) needs to be ingested. The differences between dataService and dataset may be so trivial that we can harvest them together and save the class difference and use the same model, or we may not. We also may decide we want to capture the catalog level metadata in order to display, search, filter, etc in the harvest application (but probably diminishing returns for catalog application). These things need to be decided early for the rest of the work to flow. Right now that's embedded in #6000 , but I'd be ok making that definition and design it's own ticket as well.

@rshewitt @jpyuda @FuhuXia please feel free to comment or change the tickets in this epic where appropriate. I'm happy to do some of the analysis and review when I get back.

[jbrown-xentity]

@SueValente other than the backlogged plan (overview above, and list of work in tickets on the epic: #5965), what do you want to see to call this ticket done?