Generate JsonLD datasets from websites

[valentinedwv]

There is a google sheet with earthsurface processes.

Can we use AI to do:

1. generate a JSON-LD webpage description

Spend a little human time, or ask AI if it sees datasets. eg the merit dem site has several.
and generate JSON-LD.

So, need to generate a series/set of prompts to do this.

addendum:
One time I tried this. And Doug had a project that used BAML to manage whatever BAML does.

siteFromPrompt

Experiments

[ywkim312]

@valentinedwv What is the " merit dem site"?

NVM I found it from the google sheet

[ywkim312]

@valentinedwv @ylyangtw I've been working on the JSON-LD generation for the datasets in the Google Sheet. The script is set up, and I've tested fetching webpages (like the MERIT DEM site), so that part works.

To finish, I need an API key for an AI service to generate the JSON-LD. The script supports OpenAI (like GPT-4) or Anthropic (Claude), and we can add others if needed.

A few questions:

• Do you have an API key we can use, or should we set up a new account?
• Any preference on which service (OpenAI, Anthropic, or something else)?
• Any budget or usage limits to keep in mind?

Once we have the key, I can test it on the MERIT DEM site and then run it on all the datasets that need JSON-LD. Let me know what works best for you.

[valentinedwv]

Lets use the National Research Platform LLM
https://nrp.ai/documentation/

[ywkim312]

Lets use the National Research Platform LLM
https://nrp.ai/documentation/

Do you have a api key for this? Or should I create one myself?

[valentinedwv]

Sent you a new one.
You can also create these, yourself

[ywkim312]

Sent you a new one. You can also create these, yourself

I got it. Thank you!

[valentinedwv]

Once wrote a prompt, and actually stored it.

experiments/siteFromPrompt

[valentinedwv]

Think we might need to rethink this, and put a human in the loop with our partners.

You want to collect multiple datasets from merit DEM


On the sidebar under links there are datasets, like MERIT DEM and MERIT Hydro

file naming convention:
The naming format is:
Filename represents the center of the lower left pixel of the data domain; e.g. the file "n30w120_dir.tif" is for the Flow Direction of the domain N30-N35, W120-W115. (more accurately, N29.99958333-N34.99958333,W120.0004167-W115.0004167)

The 5 deg tiles are compressed into 30 degree (lat) by 30 degree (lon) package.
Package name represents the lower left corner of the package domain; e.g. the package "dir_n30w120.tar" contains the files in the domain of N30-N60, W120-W090.


If you go to:
http://hydro.iis.u-tokyo.ac.jp/~yamadai/MERIT_DEM/index.html
There is a product description, and download links for multiple data formats, ESRI, GeoTIFF, and MMR
this should be one datacatalog, with files for different spatial regions. 



if you go to:
http://hydro.iis.u-tokyo.ac.jp/~yamadai/MERIT_Hydro/index.html
There is a product description, and download links for multiple datasets

Flow Direction (Local Drainage Direction)
Hydrologically Adjusted Elevations
Upstream Drainage Area (Flow Accumulation Area)
River Channel Width
HAND: Height Above Nearest Drainage

[valentinedwv]

Notes from a slack exchange:
Prompt:
step 1, for each webpage listed at https://docs.google.com/spreadsheets/d/1pqZpMWqQFwUrleHXPbvXqXX59Xcj1Yrtqt2nJTh1reM/edit?gid=1162616600#gid=1162616600
Create a schema.org webpage description. Title, description, org is possible,
Ouptut is a a set of jsonld files for each website.
Google Sheets

Google Sheets Logo
Geochemistry: Sources Config
Spreadsheet in Google Sheets
Open in Google Sheets
Jan 12
Geochemistry: Sources Config

12:41
step1... maybe output descriptions as title, url, description in an excel file...
step 1.1 do any of these have an xml sitemap.
step 2. manually at first, ask chatgpt to see if if can find datasets, and describe them. (I asked for a csv first). Are these groups. f the list seems good, then ask it to create jsonld datasets, as individual files,
12:46
step 1.2 if it has a sitemap, look at some of the urls. are they dataset landing pages. if so check the schema.org validator to see if they have jsonld...
if they do, then workflow to to add them as a repository to the sources list.
12:49
If there is no jsonld, then ask chatgpt to see if you can't do a jsonld dataset for a couple of urls from the site map... cleanup (are the title, description, distributions correct, keyworkd, temportal, spatial),
When generation looks good, ask chatgp to give you a clean propmpt, save...
then ask chatgpt to run over the urls in the sitemap, and generate a single jsonld for each dataset in the sitemap. (not all pages will have a dataset).
12:51
step 2. continued. look a the csv file, see if it compares to the datasets on the website, work with chatgpt to clean up. ask for a clean prompt at end. run clean prompt to generate jsonld for site... save prompt to directory in https://github.com/earthcube/pageinspection/tree/master/experiments/siteFromPrompt
12:52
step 3. When we get three of four sites done, see if we can't get chatgpt to zero shot the rest of the website urls.

[ywkim312]

Manual Workflow for Dataset JSON-LD Generation

Overview

This document outlines a manual, iterative workflow for processing webpages from a CSV file and generating JSON-LD descriptions for datasets. The process involves working with ChatGPT to analyze websites, identify datasets, and create structured metadata.

Source Data

• Input: Google Sheets spreadsheet with list of webpages
• URL: https://docs.google.com/spreadsheets/d/1pqZpMWqQFwUrleHXPbvXqXX59Xcj1Yrtqt2nJTh1reM/edit?gid=1162616600#gid=1162616600
• Sheet Name: Geochemistry: Sources Config

Step 1: Initial Webpage Descriptions

Step 1.0: Create Schema.org Webpage Descriptions

For each webpage listed in the spreadsheet:

• Create a schema.org webpage description
• Include: Title, Description, Organization (if possible)
• Output: Set of JSON-LD files (one per website)
• Alternative Output: Excel file with columns: title, url, description

Step 1.1: Check for XML Sitemaps

• For each website, check if it has an XML sitemap
• Document which sites have sitemaps available

Step 1.2: Analyze Sitemap URLs

If a sitemap exists:

1. Examine URLs: Look at some URLs from the sitemap
   • Determine if they are dataset landing pages
2. Check for Existing JSON-LD:
   • Use schema.org validator to check if pages already have JSON-LD
   • If JSON-LD exists and looks good:
     • Add the repository to the sources list
     • Workflow: Add as repository to sources list
3. If No JSON-LD Exists:
   • Use ChatGPT to generate JSON-LD for a couple of sample URLs from the sitemap
   • Validation Checklist:
     • Are title, description, distributions correct?
     • Are keywords, temporal, spatial information included?
   • If generation looks good:
     • Ask ChatGPT to provide a clean prompt
     • Save the prompt
   • Then: Ask ChatGPT to process all URLs in the sitemap
     • Generate a single JSON-LD file for each dataset found
     • Note: Not all pages in sitemap will contain datasets

Step 2: Manual Dataset Discovery and Description

Step 2.1: Initial Dataset Discovery

• Manually use ChatGPT to analyze the website
• Ask ChatGPT to find datasets and describe them
• First Output: Request CSV format with dataset information
• Review the list to see if these are groups or individual datasets
• If the list seems good:
  • Ask ChatGPT to create JSON-LD for datasets
  • Output as individual JSON-LD files

Step 2.2: Cleanup and Prompt Refinement

1. Review CSV File:
   • Compare CSV output with actual datasets on the website
   • Work with ChatGPT to clean up discrepancies
2. Generate Clean Prompt:
   • Ask ChatGPT to provide a clean, refined prompt
   • Save the prompt to directory: https://github.com/earthcube/pageinspection/tree/master/experiments/siteFromPrompt
3. Run Clean Prompt:
   • Use the clean prompt to generate JSON-LD for the entire site
   • Save all generated JSON-LD files

Step 3: Scaling to Remaining Sites

After completing 3-4 sites using the manual workflow:

• Attempt zero-shot approach for remaining website URLs
• Use ChatGPT to process remaining sites without manual intervention
• Validate outputs against the patterns learned from manual processing

Key Outputs

1. JSON-LD Files: Individual JSON-LD files for each dataset/website
2. Prompts: Clean, reusable prompts saved in the pageinspection repository
3. CSV Files: Intermediate dataset listings for review and cleanup
4. Documentation: Notes on which sites have sitemaps, existing JSON-LD, etc.

Notes

• This is a manual, iterative process designed to refine prompts and workflows
• Each site may require different approaches depending on structure
• The goal is to eventually automate the process after learning patterns from manual work
• All prompts should be saved for future reference and reuse