feat(api): expose optics dry_run over REST (inline + upload)#315
Open
chinmayajha wants to merge 8 commits into
Open
feat(api): expose optics dry_run over REST (inline + upload)#315chinmayajha wants to merge 8 commits into
chinmayajha wants to merge 8 commits into
Conversation
Extract the suite discovery/loading logic that was inlined in BaseRunner into reusable module-level functions (load_test_cases_data, load_modules_data, load_elements_data, load_api_data_files, load_suite_from_folder) plus an inline builder (build_suite_from_inline), and add a LoadedSuite model. BaseRunner now delegates to these so the CLI and future REST callers share one code path. find_files gains a validate flag: CLI keeps its sys.exit-on-missing-files behaviour (default True); library/server callers pass validate=False to get the (possibly empty) collections back and surface the problem themselves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a require_driver flag (default True) to Session and SessionManager.create_session, threaded into OpticsBuilder. When False and no driver/element source is configured, get_driver()/get_element_source() return an empty InstanceFallback([]) instead of raising E0501, and Session skips the get_driver() fail-fast (driver=None). This lets callers build the full keyword registry and validate a suite without any device. Defaults preserve the existing fail-fast behaviour for execute/live/action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dry-run is a validator, so a missing ${variable} or an unknown keyword should be
reported as a per-keyword FAIL, not crash the run:
- _init_keywords resolves the display name defensively (_safe_resolve) so an
undefined variable can no longer raise OpticsError during runner construction.
- _dry_run_module catches OpticsError (not just ValueError) and records the
exception message as the keyword's reason.
Previously an unresolved variable aborted the whole dry run; now it surfaces as a
clear, actionable failure. Improves both the CLI dry-run and the REST endpoint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DryRunExecutor.execute now returns runner.result_printer.test_state (Dict[str, TestCaseResult]); ExecutionEngine.execute already propagates the executor's return value. The CLI ignores it; the REST dry-run endpoints surface it as the response payload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dry-run upload endpoint uses multipart/form-data (UploadFile/Form), which FastAPI requires python-multipart for. It was only present transitively; declare it explicitly and refresh the lock (the regeneration also picks up the previously unlocked optional google-genai/llm dependency tree). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose 'optics dry_run' over HTTP with two endpoints sharing one device-less core: - POST /v1/dry_run - inline JSON suite (test_cases/modules/elements/api) - POST /v1/dry_run/upload - multipart CSV/YAML files or a single .zip Both build a LoadedSuite and run it on an ephemeral, device-less session (require_driver=False), strip all source configs so no driver is ever instantiated (even if an uploaded config.yaml enables one), run mode=dry_run with use_printer=False, return a DryRunResponse, and always tear down the session/temp dir in finally. Blocking suite-building/extraction is offloaded via asyncio.to_thread. Hardening (common/dry_run.py): zip-slip via commonpath confinement, zip-bomb via a chunked written-byte counter that does not trust header sizes plus entry/ratio caps, streaming size caps on the inline body and on each uploaded file, and filename sanitisation. Endpoints are unauthenticated like the rest of the server. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TestClient coverage for both endpoints: inline happy path, unknown keyword and unresolved variable reported as FAIL (200, with reason), empty/invalid/oversized payloads, include/exclude filtering, zip + individual-file uploads, and the security guards (zip-slip, zip-bomb, oversized upload, no-test-case upload -> 400). Plus helper-level unit tests for the loaders, build_suite_from_inline, find_files(validate=False), and a device-less session building the full registry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Dry Run section to docs/usage/REST_API_usage.md (both endpoints, request/response, curl examples, status-code semantics, limits, and the no-auth caveat). Update CLAUDE.md to describe the endpoints, device-less sessions, reusable loaders, and the dry-run validator behaviour with fresh anchors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
|
Follow-up work intentionally left out of this PR (filed separately):
This PR also closes out the work from #309. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You can now validate a whole test suite over HTTP — no device, no Appium session. Two ways in:
POST /v1/dry_run— inline JSON suite (test_cases/modules/elements/api)POST /v1/dry_run/upload— upload CSV/YAML files or a single.zipBoth return a structured per-test-case → module → keyword PASS/FAIL report. Great for catching unknown keywords, unresolved
${vars}, and broken modules in CI or an editor before anything gets provisioned.How it works
Each request spins up an ephemeral, device-less session, runs it through the existing
ExecutionEnginein dry-run mode, and tears everything down infinally. Device-less is a newrequire_driverflag (defaultTrue, so execute/live are untouched) threaded into the session builder — with no driver configured it hands back empty fallbacks instead of failing. Uploadedconfig.yamldriver configs are stripped, so a dry run can never accidentally connect to a device.The suite-loading logic that was inlined in
BaseRunneris now shared module-level functions (load_suite_from_folder,build_suite_from_inline), so CLI and REST can't drift.Worth knowing
${var}comes back as a keywordFAILwith a reason — not a crash. Also fixed asys.exitinfind_filesthat was turning bad uploads into 500s.200= ran (PASS/FAIL in body),400= no/invalid suite,413= too large,422= malformed JSON.python-multipart(FastAPI needs it for uploads); lock regenerated.Testing
25 unit/API tests plus an exhaustive live-server harness (33 cases: all paths, 40-way concurrency, every bundled sample, enabled-Appium upload confirmed to not connect). Full suite 317 passed; pre-commit + commitizen green.
Commits
Split by concern, each leaves the tree green: loaders refactor → device-less sessions → dry-run validator fix → executor return → deps → endpoints+hardening → tests → docs.
Follow-ups
Async job/polling for huge suites (#312), in-memory zip parsing (#313), optional API auth (#314), Pydantic v2 migration (#311).
🤖 Generated with Claude Code