STG-4451/4475/4478: dual-mode CLI (monolith + microservices/Clark)#137
Merged
Conversation
…lation support
- mdast_cli_core/microservices.py: Clark facade client (Bearer auth, md5-first
multipart upload with X-File-Size, application_md5 precheck, fsm_locked two-phase
scan creation, scanyon-native (stage, status) polling, /rest/scans/{id}/report)
- mdast_cli_core/factory.py: installation mode resolution (MDAST_CLI_MODE env or
probe autodetect), monolith path unchanged
- mdast_cli/ms_flow.py: microservices scan flow with pre-check gate (exit code 8),
engines preflight by type/status, reports to user-provided paths
- tests/: 81 tests (unit, contract, smoke incl. monolith regression, security)
- CI: tests workflow + test gates before PyPI/Docker publishing
…start 409
Found by live e2e on dev:
- scanyon rejects type=AUTO without testcase_id (services/scan.py: 'Testcase ID
should be set for auto scan'). The DEC-671-02 premise 'AUTO is the only CLI
mode' is wrong: a scan without a test case must be MANUAL (server sets
wait=True, install/launch/wait/stop semantics), only test-case replays are
AUTO. create_manual_scan now sends type=MANUAL, precheck mirrors the type.
- POST /scans/{id}/start/ is not idempotent; the facade retries it, so a retry
after the first (successful) unlock returns 409 'not in initial state' though
the scan did start. CLI now confirms scan state on 409 and continues if the
scan is past CREATED, fails only on a genuine conflict.
- Tests updated (MANUAL body, precheck type) + 2 new smoke tests for the 409
retry-tolerance and genuine-conflict paths. 83 passed.
Live e2e hit a transient 502 (scanyon momentarily unavailable) on GET
/scans/{id}/ mid-poll and the CLI died, though the scan kept running. A long
CI poll must survive flaky downstream blips: _get_scan now retries transient
5xx (502/503/504) and network errors a few times before failing; 4xx stays
terminal. +4 unit tests. 87 passed.
…polling) Report download (Pepper via Clark) can hit transient 502 right after a scan finishes; wrap download in the same retry as _get_scan so a blip doesn't lose a finished scan's report. Confirmed live: JSON report downloads, PDF currently 502s server-side (Pepper WeasyPrint CSS issue - separate server bug).
… and e2e tests
Report handling:
- --report_format {pdf,json,all,none} (default pdf), shared by both flows.
resolve_report_targets() centralises output paths (scan_report_<id>.pdf/.json
defaults; --pdf_report_file_name / --summary_report_json_file_name still force a
format and keep backward compatibility). Microservices reports are soft-fail: a
Pepper render failure keeps a finished scan green (exit 0 with a warning).
Exit-code parity and robustness:
- Required url/company_id/token validation, app-file existence guard, cr_report
requires stingray creds. Both flows wrap RequestException -> NETWORK_ERROR(6).
- 401/403 -> AUTH_ERROR(7) consistently in both installations.
- Microservices upload accepts any 2xx (not only 201); server-controlled fields
sanitised before logging (CWE-117); monolith architecture resolution guards
non-dict payload items instead of crashing.
- factory: TLS verify secure-by-default (unset/empty stay on), architectures
content-fingerprint probe (not a bare HTTP 200), clearer ambiguous/empty errors.
- nexus2: fix crash when --nexus2_file_name omitted (None into os.path.join).
Docs: README default timeout corrected to ~1h, CR report relabelled Compliance
(not Change Request), Docker report-path note added.
Tests: positive store-download -> scan e2e (8 stores), negative Sting-side e2e,
report target resolution, report-download resilience, mode-probe path fix. 151 passed.
.gitignore: ignore .venv/ and scan_report_* CLI artifacts.
npinaev
force-pushed
the
feature/ms-dual-mode
branch
from
1b44d3e to
a5c96c6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Makes
mdast-clidual-mode: a single package that works against both the classic monolith installation of the DAST platform and the new microservices one (Kubernetes, behind the Clark facade). The command line is identical and the flags are the same; the mode is auto-detected or forced via an env var.The monolith path is behaviourally unchanged: all microservices logic lives on a separate execution branch, and the monolith regression tests are in the suite.
How the mode is selected
MDAST_CLI_MODE=auto|monolith|microservices(defaultauto).auto, the CLI probesGET /rest/architectures/and classifies the installation by the shape of the payload (stringtype+os_version-> microservices; integertype-> monolith), not by a bare HTTP 200. A monolith that happens to answer 200 to a Bearer probe is not misclassified.Microservices flow (Clark facade)
architectures -> resolve platform by ext -> testcase -> engines (active STARTED) -> dedup by md5 -> upload_info (md5-first multipart + X-File-Size, presigned MinIO POST) -> precheck gate -> create (MANUAL without testcase / AUTO with testcase) -> start (409 tolerated) -> poll to terminal (stage, status) -> reports (soft-fail).type=AUTOwithout atestcase_id.POST /scans/{id}/start/is not idempotent: a retry after a successful start returns 409, so the CLI confirms the scan state and continues, failing only on a genuine conflict.Unified exit-code contract (both installations)
0 SUCCESS, 1 INTERNAL_ERROR, 2 INVALID_ARGS, 4 DOWNLOAD_FAILED, 5 SCAN_FAILED, 6 NETWORK_ERROR, 7 AUTH_ERROR, 8 PRECHECK_BLOCKED(there is no code 3).401/403 -> AUTH_ERROR(7)in both installations; network errors / transient 5xx on read/poll/upload ->NETWORK_ERROR(6)(the CI "retry" signal).--architecture_id,--appium_script_path,--cr_reporton microservices are either ignored or exitINVALID_ARGS(2)(not supported by the platform).Reports
--report_format {pdf,json,all,none}(defaultpdf), shared by both flows;--pdf_report_file_name/--summary_report_json_file_namestill force a format (backward compatible).Security and robustness
MDAST_TLS_VERIFY; unset/empty stay secure, disabled only by an explicit0/false/no/off).message, ids, package_name) are sanitised before logging (CWE-117: guards against log-line forgery via CR/LF/ANSI).2xx; it is idempotent by md5 (a retry does not re-upload the file).--nexus2_file_nameis omitted (None intoos.path.join).Tests
151 tests (unit / contract / smoke / e2e), including:
Documentation
README: installation-modes + env section, corrected default timeout (~1 hour), CR report relabelled Compliance (a Bank-of-Russia call-trace-immutability report, not a "change request"), Docker report-path note.
Compatibility
Monolith installation: behaviour and exit codes unchanged. Microservices installation: new support. Verified live on the dev stand (upload, MANUAL/AUTO scan, JSON/PDF reports).