[rollout] feat: add best-effort file artifacts for workflows/graders#250
Open
JoyboyBrian wants to merge 2 commits into
Open
[rollout] feat: add best-effort file artifacts for workflows/graders#250JoyboyBrian wants to merge 2 commits into
JoyboyBrian wants to merge 2 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
There was a problem hiding this comment.
4 issues found across 11 files
Confidence score: 2/5
osmosis_ai/rollout/backend/local/backend.pyandosmosis_ai/rollout/backend/harbor/backend.pybuild artifact paths fromrollout_idwithout sanitization, so traversal/absolute segments could write, copy, or delete files outside the intended artifacts root; merging as-is carries a concrete host-path safety risk — validate/normalizerollout_idand enforce containment within the artifact root before any filesystem operation.osmosis_ai/templates/_scaffolds/rollout/main.py.tplassumesartifacts_diris always present in the grader example, butGraderContext.artifacts_dircan beNone; copied scaffold code may raise runtime errors in environments where artifact directory creation fails — add aNoneguard in the template example before using the path.docs/rollout-sdk.mdshowsgrade(...) -> Nonewhile the ABC uses-> Any, which can mislead implementers and static typing expectations; this is low risk but increases integration friction — align the doc signature to-> Anybefore or shortly after merge.
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ctx.artifacts_dirtoAgentWorkflowContextandGraderContextso user code can write rollout artifacts (logs, traces, generated files) to a known directory.osmosis_ai/rollout/utils/file_artifacts.pywithHARBOR_ARTIFACTS_DIR(sandbox/logs/artifacts/) anddefault_artifact_root()(~/.osmosis/artifacts/).HarborBackend(agent runner + grader runner) andLocalBackend, landing files on the host under~/.osmosis/artifacts/<rollout_id>/artifacts/.docs/rollout-sdk.mdwith an Artifacts section and usage example.Why
Rollouts often produce files (traces, screenshots, large/binary outputs) that don't belong in the sample payload. This gives workflows and graders a single convention — write under
ctx.artifacts_dir— and the backends collect the files to the host after each rollout. Artifact handling is best-effort and never affects rewards or rollout status.How to Test
uv run pytest tests/unit/rollout/test_context.py tests/unit/rollout/test_harbor_backend.py tests/unit/rollout/test_local_backend.pyuv run ruff check .anduv run ruff format --check .uv run pyright osmosis_ai/Checklist
[module] type: descriptionformatenhancement,bug,breaking)ruff check .andruff format --check .passpyright osmosis_ai/passespytestpasses (new tests added if applicable)Summary by cubic
Add best-effort file artifact support for rollouts. Workflows and graders can write to
ctx.artifacts_dir, and Harbor/Local backends collect files to~/.osmosis/artifacts/<rollout_id>/artifacts/without affecting rollout status or rewards.artifacts_dirtoAgentWorkflowContextandGraderContext.artifacts_dirto/logs/artifacts; backend moves artifacts to host before cleanup, copies when trials are kept, and keeps the trial dir if relocation fails (logs, doesn’t fail rollout).~/.osmosis/artifacts/<id>/artifacts/logs/artifacts; passes it to contexts and degrades toNoneif creation fails.osmosis_ai/rollout/utils/file_artifacts.pywithHARBOR_ARTIFACTS_DIRanddefault_artifact_root().docs/rollout-sdk.mdwith an Artifacts section and example.Written for commit f4d564f. Summary will update on new commits.