Move ord-data's pipeline scripts in from ord-schema

[skearnes]

Companion to ord-schema#908, which removes these same files. process_dataset.py, pb_to_parquet_dataset.py, and parse_uspto.py are this repository's pipeline and one-off ingestion code rather than schema use cases, so they land here with their tests. That is 25 tests arriving on top of the 10 from #262 — the point of standing up the harness first was that process_dataset_test.py is the best-covered thing in ord_schema/scripts/ and moving it into a repository with nowhere to run it would have quietly dropped that coverage.

Background: ord-logbook: Where ord-schema ends.

What changed beyond the copy

Imports. Test imports become bare import process_dataset, matching the co-located convention upload_to_huggingface_test.py already uses, and pb_to_parquet_dataset_test.py's caplog logger name follows __name__ to its new value.

The process_dataset.py docstring. It called itself "a one-stop shop for preparing submissions," which reads as contributor-facing and is exactly what made the earlier read of this boundary wrong. It is the submission workflow's machinery, and now says so — including a pointer to validate_dataset.py for contributors who do want a pre-submission check.

A pipeline dependency group holds what the dataset-touching scripts need (ord-schema, pygithub); dev includes it, so the mirror job's uv sync --no-dev still skips the rdkit/pandas chain.

submission.yml drops its pinned ord-schema checkout and pip install for uv sync --only-group pipeline. ORD_SCHEMA_TAG leaves that workflow entirely.

Ruff's ignore list picks up the rules the incoming code trips — G004, PLR2004, RUF001, S101, PTH207, TD003, FIX002 — each already ignored in ord-schema for the same reason, and the test per-file ignore widens from D103 to D.

Two things worth a close look

The submission path now validates at ord-schema 0.8, not v0.6.3. This is forced, not chosen: the moved process_dataset.py imports ord_schema.datasets, which does not exist in v0.6.3. Full-corpus validation in validation.yml still runs at the pinned tag, so the two paths now disagree about which schema version they validate against. Reconciling them means bumping validation.yml too, which deserves its own trial run against the whole corpus rather than riding along here.

process_submission gains needs: check_file_types. The job now runs scripts/process_dataset.py out of the pull request's own checkout, where previously the script came from a pinned ord-schema tag a fork could not touch. check_file_types already rejects fork submissions containing anything but dataset files, but it is a parallel job, so without the dependency a fork could edit the script and have it executed while that check was still running. Fork pull_request runs get no secrets and a read-only token, so the exposure was bounded either way — but the ordering should be explicit.

Verification

ruff check, ruff format --check, ty check, and all four pre-commit hooks pass; 35 tests pass under pytest -n auto.

Stacked on #262 and will retarget to main when that merges.

🤖 Generated with Claude Code

Greptile Summary

This PR relocates ORD dataset-processing and ingestion tooling into ord-data.

• Adds the processing, protobuf-to-Parquet, and USPTO parsing scripts with their tests.
• Introduces a dedicated pipeline dependency group and updates the submission workflow to run the local processing script through uv.
• Updates validation documentation, conversion APIs, lint configuration, and contributor guidance for the new repository boundary.

Confidence Score: 3/5

The PR is not yet safe to merge because cross-file reaction moves still produce incorrect submission statistics and the token-backed workflow still invokes a mutable third-party action reference.

The processing code computes and unions statistics independently per file, so moving one reaction between files remains visible as an addition and removal; the submission workflow also continues to execute astral-sh/setup-uv@v7 before processing and pushing with repository credentials.

Files Needing Attention: scripts/process_dataset.py; .github/workflows/submission.yml

Important Files Changed

Filename	Overview
.github/workflows/submission.yml	Switches submission processing from a pinned ord-schema checkout to the local script and uv environment, but retains the previously reported mutable setup-action reference.
scripts/process_dataset.py	Moves the submission processing implementation and tests into this repository; the previously reported cross-file statistics defect remains.
scripts/pb_to_parquet_dataset.py	Adds the protobuf-to-Parquet conversion utility and its co-located tests.
scripts/parse_uspto.py	Adds the USPTO CML ingestion utility.
pyproject.toml	Adds pipeline dependencies and adjusts test, type-checking, and lint configuration for the moved scripts.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  PR[Dataset pull request] --> Types[Check changed file types]
  Types --> Checkout[Checkout PR revision]
  Checkout --> LFS[Fetch changed LFS objects]
  LFS --> Deps[Install pipeline dependency group]
  Deps --> Process[Run scripts/process_dataset.py]
  Process --> Comment[Post submission summary]
  Process --> Update[Update and push datasets]

Loading

_{Reviews (2): Last reviewed commit: "Drop deprecated ord-schema API from scri..." | Re-trigger Greptile}

[greptile-apps]

Cross-file moves corrupt statistics

When a reaction moves between two dataset files modified in the same pull request, get_change_stats() compares each file independently and the later unions classify the reaction as both added and removed instead of reconciling it across the submission, producing incorrect totals in the submission comment.

[greptile-apps]

Setup action uses mutable tag

The new submission job executes astral-sh/setup-uv@v7 before processing datasets and performing token-backed repository operations, so repointing this mutable tag changes the code executed with access to the job checkout and available workflow token; pin the action to a reviewed full commit SHA. How this was verified: The workflow runs the mutable action before its GITHUB_TOKEN-backed processing and push steps.