Skip to content

Add structured JSON outputs for ChatBob integration#83

Open
ChrisCollison wants to merge 22 commits into
masterfrom
chatbob
Open

Add structured JSON outputs for ChatBob integration#83
ChrisCollison wants to merge 22 commits into
masterfrom
chatbob

Conversation

@ChrisCollison

Copy link
Copy Markdown
Collaborator

This PR adds a structured JSON evidence layer to ROBERT without changing its standard behavior.

The new JSON files capture information that ROBERT already generates during CURATE, GENERATE, VERIFY, PREDICT, and REPORT. All ChatBob-related JSON files are written to a separate top-level JSON/ directory, while the existing DAT, CSV, image, and PDF outputs remain in their normal locations.

I'm including:

Runtime JSON audit files are implemented for:

  • CURATE
  • GENERATE
  • VERIFY
  • PREDICT

REPORT currently includes figure provenance only, rather than a complete report audit.

But AQME and EVALUATE JSON audits are not included in this PR.

JSON writing is fail-soft, so a problem writing a JSON file should not interrupt or alter a normal ROBERT run.

I validated, as follows:

The modified ROBERT 2.1.2 workflow was compared with an unmodified ROBERT 2.1.2 workflow using the same input dataset.

The comparison found:

  • the four primary DAT files matched after normalizing timestamps, absolute paths, execution times, and trailing whitespace;
  • scientific values in 28 matching CSV files were identical within strict tolerance;
  • the only expected CSV text difference was the recorded input path in the CURATE options file.

The generated JSON files were also reviewed against the corresponding ROBERT DAT information.

For the tested workflow, the JSON additions did not change ROBERT’s scientific outputs.

Scope

This PR is intended to add structured access to information ROBERT already produces. It does not intentionally change:

  • scoring
  • model selection
  • descriptor selection
  • thresholds
  • predictions
  • plotting
  • command-line behavior

I next want to work on the JSON schema so I am calling the current set up 'experimental' while it is aligned with the questions ChatBob users are likely to ask.

Follow-up work

The next steps are to:

  • add automated tests for JSON content and DAT-to-JSON fidelity since most of this was done 'manually';
  • continue refining the schema;
  • implement AQME and EVALUATE audits for chatbob use;
  • decide whether REPORT should later include a full audit, although I don't think this is a high priority since it all comes from the data extracted to DAT and JSON files;
  • build deterministic question-to-evidence mapping before adding the LLM response layer.

…rmation about the incoming dataset such that a ChatBOB can use that information downstream
…the variable desc_keep, which is a variable built at source in the original ROBERT. Here we add a label to the JSON auditing files so that the variable can be better understood by ChatBob
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant