External sources data cleaning#4
External sources data cleaning#4jappareti wants to merge 4 commits intoResiliencyWorkingGroup:devfrom
Conversation
| async function fetchById(mapDataset) { | ||
| try { | ||
| const response = await fetch(external[mapDataset]); | ||
| const response = await fetch(`${external[mapDataset].url}?${external[mapDataset].query}`); |
There was a problem hiding this comment.
Requires query to be defined on all the external datasets. We can run can JavaScript inside curly braces.
const response = await fetch(`${external[mapDataset].url}?${external[mapDataset].query || ''}`);| privatelyOwnedPublicOpenSpaces: { | ||
| url: "https://data.sfgov.org/resource/3ub7-d4yy.geojson", | ||
| query: | ||
| "$query=SELECT name AS title, descriptio AS description, the_geom WHERE the_geom IS NOT NULL", |
There was a problem hiding this comment.
This works and eliminates the need for author: 'DataSF'.
query: '$query=SELECT name AS title, "DataSF" as author, descriptio AS description, the_geom WHERE the_geom IS NOT NULL'
| const dataset = await response.json(); | ||
|
|
||
| return dataset; | ||
| const modifiedDataset = addAuthor(dataset, external[mapDataset].author); |
There was a problem hiding this comment.
If each dataset needs to be modified differently this might be a better choice.
return external[mapDataset].modifyDataset
? external[mapDataset].modifyDataset(dataset)
: dataset;| @@ -0,0 +1,8 @@ | |||
| function addAuthor(dataset, author) { | |||
There was a problem hiding this comment.
With the modified query string I don't think we'll need this function with the current external datasets.
|
We want to mirror the internal data as close as possible so the properties should include _id, title, description, author and asset. Also, it will make the code easier to read if you keep the same order on all of the query strings. Let's use single quotes for strings as well. |
2 participants
Updated SOQL query strings for external data from DataSF. Formatted to be consistent with the same data model as internal data sources.