Add financial fact foundation and roadmap#12
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR lays the groundwork for a canonical “financial facts” layer in Jetbot, expanding evidence metadata, adding fact conversion/corrections utilities, persisting facts through the pipeline, exposing a new facts API endpoint, and upgrading evaluation tooling to produce machine-readable reports.
Changes:
- Introduces new canonical schemas (
FinancialFact,ExtractionTrace,Correction) and enriches evidence fields onSourceRefandTableCell. - Adds a fact conversion layer (
facts_from_statements) + correction application, wires facts into agent state/finalization, and exposesGET /v1/documents/{doc_id}/facts. - Upgrades
scripts/eval.pyinto a golden-case runner that outputs JSON + Markdown reports, plus adds/extends tests for facts, metrics, eval reporting, and the new endpoint.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| web/src/api/types.ts | Adds FinancialFact type and extends evidence-related types for richer provenance metadata. |
| web/src/api/docs.ts | Adds docsApi.facts() and normalizes richer SourceRef fields in existing normalizers. |
| src/schemas/models.py | Adds canonical fact/correction/trace schemas; extends SourceRef and TableCell evidence fields. |
| src/finance/facts.py | Implements statement→facts conversion, correction application, fact ID generation, and scale inference. |
| src/agent/state.py | Extends AgentState to carry facts, corrections, and extraction traces. |
| src/agent/nodes.py | Persists facts/corrections/traces on finalize; auto-generates facts from statements if missing. |
| src/api/routes.py | Adds GET /documents/{doc_id}/facts and persists partial facts on failure. |
| src/utils/metrics.py | Adds fact-level metrics and aggregates them into golden metrics. |
| scripts/eval.py | Reworks eval script into a runner that executes golden cases and writes JSON/MD reports. |
| tests/test_finance_facts.py | Adds unit tests for fact conversion and correction application. |
| tests/test_metrics.py | Adds unit tests for fact-level metrics. |
| tests/test_eval_script.py | Adds tests ensuring eval report writer produces JSON/MD and marks failures correctly. |
| tests/test_routes_web.py | Adds API tests for facts endpoint success and missing-file 404. |
| docs/financial_fact_platform_roadmap.md | Adds detailed roadmap document for the fact platform implementation phases. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+45
to
+54
| def apply_corrections(facts: Iterable[FinancialFact], corrections: Iterable[Correction]) -> list[FinancialFact]: | ||
| by_id = {fact.fact_id: fact for fact in facts} | ||
| valid_fields = set(FinancialFact.model_fields) | ||
|
|
||
| for correction in corrections: | ||
| fact = by_id.get(correction.fact_id) | ||
| if fact is None or correction.field_name not in valid_fields: | ||
| continue | ||
| by_id[correction.fact_id] = fact.model_copy(update={correction.field_name: correction.new_value}) | ||
| return list(by_id.values()) |
Comment on lines
+110
to
+112
| def _fact_id(doc_id: str, statement_type: str, concept: str, period_end: object, label: str) -> str: | ||
| raw = "|".join([doc_id, statement_type, concept, str(period_end or ""), label]) | ||
| return "fact_" + hashlib.sha1(raw.encode("utf-8")).hexdigest()[:16] |
Comment on lines
+142
to
+153
| appear in different statements. | ||
| """ | ||
| indexed: dict[str, FinancialFact] = {} | ||
| for fact in actual_facts: | ||
| indexed.setdefault(fact.concept, fact) | ||
| indexed[f"{fact.statement_type}:{fact.concept}"] = fact | ||
|
|
||
| matched: list[str] = [] | ||
| mismatched: list[dict[str, Any]] = [] | ||
| missing: list[str] = [] | ||
|
|
||
| for key, expected_val in expected_values.items(): |
| @@ -557,6 +558,21 @@ def finalize(state: AgentState) -> AgentState: | |||
| store.save_json(state.doc_meta.doc_id, "extracted/pages.json", [p.model_dump() for p in state.pages]) | |||
| store.save_json(state.doc_meta.doc_id, "extracted/tables.json", [t.model_dump() for t in state.tables]) | |||
| store.save_json(state.doc_meta.doc_id, "extracted/statements.json", {k: v.model_dump() for k, v in state.statements.items()}) | |||
| @@ -522,6 +530,8 @@ def _save_partial_results(doc_id: str) -> None: | |||
| s.save_json(doc_id, "extracted/tables.json", [t.model_dump() for t in partial.tables]) | |||
| if partial.statements: | |||
| s.save_json(doc_id, "extracted/statements.json", {k: v.model_dump() for k, v in partial.statements.items()}) | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR starts the Filing-to-Model roadmap by adding the first financial fact foundation slice.
It introduces a canonical fact layer, richer evidence metadata, a report-producing evaluation runner, a facts API endpoint, and a detailed local roadmap document for the next implementation phases.
What Changed
FinancialFact,ExtractionTrace, andCorrectionschemas.SourceRefandTableCellwith richer evidence fields such asrow,col,bbox,engine, andartifact_path.src/finance/facts.pyto convert currentFinancialStatementoutputs into canonical facts and apply corrections.AgentState, pipeline finalization, and partial-failure persistence.GET /v1/documents/{doc_id}/facts.scripts/eval.pyinto a golden evaluation runner that produces JSON and Markdown reports.docs/financial_fact_platform_roadmap.md.Why
Jetbot already has a functional PDF analysis pipeline, but the next bottleneck is trust and workflow fit: exact facts, evidence traceability, reviewability, and exportability.
This PR establishes the data model and evaluation base needed for the next phases:
Validation
python -m ruff check src tests scriptspython -m mypy src --ignore-missing-importspython -m pytest -q --timeout=60python scripts/eval.py --output-dir data/eval-devcd web && npm run lintcd web && npm run typecheckcd web && npm run buildFollow-up
Follow-up PRs should focus on: