Skip to content

[rollout] feat: add best-effort file artifacts for workflows/graders#250

Open
JoyboyBrian wants to merge 2 commits into
mainfrom
brian/artifacts
Open

[rollout] feat: add best-effort file artifacts for workflows/graders#250
JoyboyBrian wants to merge 2 commits into
mainfrom
brian/artifacts

Conversation

@JoyboyBrian

@JoyboyBrian JoyboyBrian commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

What

  • Add ctx.artifacts_dir to AgentWorkflowContext and GraderContext so user code can write rollout artifacts (logs, traces, generated files) to a known directory.
  • Add osmosis_ai/rollout/utils/file_artifacts.py with HARBOR_ARTIFACTS_DIR (sandbox /logs/artifacts/) and default_artifact_root() (~/.osmosis/artifacts/).
  • Wire artifact collection through HarborBackend (agent runner + grader runner) and LocalBackend, landing files on the host under ~/.osmosis/artifacts/<rollout_id>/artifacts/.
  • Update the rollout scaffold template and docs/rollout-sdk.md with an Artifacts section and usage example.
  • Add unit tests covering context wiring, Harbor backend collection, and local backend collection.

Why

Rollouts often produce files (traces, screenshots, large/binary outputs) that don't belong in the sample payload. This gives workflows and graders a single convention — write under ctx.artifacts_dir — and the backends collect the files to the host after each rollout. Artifact handling is best-effort and never affects rewards or rollout status.

How to Test

  • uv run pytest tests/unit/rollout/test_context.py tests/unit/rollout/test_harbor_backend.py tests/unit/rollout/test_local_backend.py
  • uv run ruff check . and uv run ruff format --check .
  • uv run pyright osmosis_ai/

Checklist

  • PR title follows [module] type: description format
  • Appropriate labels added (e.g. enhancement, bug, breaking)
  • ruff check . and ruff format --check . pass
  • pyright osmosis_ai/ passes
  • pytest passes (new tests added if applicable)
  • Public API changes are documented
  • No secrets or credentials included

Summary by cubic

Add best-effort file artifact support for rollouts. Workflows and graders can write to ctx.artifacts_dir, and Harbor/Local backends collect files to ~/.osmosis/artifacts/<rollout_id>/artifacts/ without affecting rollout status or rewards.

  • New Features
    • Added artifacts_dir to AgentWorkflowContext and GraderContext.
    • Harbor: runners set artifacts_dir to /logs/artifacts; backend moves artifacts to host before cleanup, copies when trials are kept, and keeps the trial dir if relocation fails (logs, doesn’t fail rollout).
    • Local: creates per-rollout dir at ~/.osmosis/artifacts/<id>/artifacts/logs/artifacts; passes it to contexts and degrades to None if creation fails.
    • New osmosis_ai/rollout/utils/file_artifacts.py with HARBOR_ARTIFACTS_DIR and default_artifact_root().
    • Updated scaffold and docs/rollout-sdk.md with an Artifacts section and example.
    • Added unit tests for context wiring, Harbor relocation/failure behavior, and Local creation/degradation.

Written for commit f4d564f. Summary will update on new commits.

Review in cubic

@JoyboyBrian JoyboyBrian added the enhancement New feature or request label Jul 4, 2026
@JoyboyBrian JoyboyBrian requested a review from BaiqingL as a code owner July 4, 2026 23:48
@JoyboyBrian JoyboyBrian added rollout Remote Rollout module enhancement New feature or request labels Jul 4, 2026
@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 11 files

Confidence score: 2/5

  • osmosis_ai/rollout/backend/local/backend.py and osmosis_ai/rollout/backend/harbor/backend.py build artifact paths from rollout_id without sanitization, so traversal/absolute segments could write, copy, or delete files outside the intended artifacts root; merging as-is carries a concrete host-path safety risk — validate/normalize rollout_id and enforce containment within the artifact root before any filesystem operation.
  • osmosis_ai/templates/_scaffolds/rollout/main.py.tpl assumes artifacts_dir is always present in the grader example, but GraderContext.artifacts_dir can be None; copied scaffold code may raise runtime errors in environments where artifact directory creation fails — add a None guard in the template example before using the path.
  • docs/rollout-sdk.md shows grade(...) -> None while the ABC uses -> Any, which can mislead implementers and static typing expectations; this is low risk but increases integration friction — align the doc signature to -> Any before or shortly after merge.

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread osmosis_ai/rollout/backend/local/backend.py
Comment thread osmosis_ai/rollout/backend/harbor/backend.py
Comment thread osmosis_ai/templates/_scaffolds/rollout/main.py.tpl Outdated
Comment thread docs/rollout-sdk.md Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 11 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request rollout Remote Rollout module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant