feat: on-screen error detection for runs and library consumers#323
Merged
thakur-patel merged 12 commits intoJun 30, 2026
Merged
Conversation
Introduce an ErrorDefinitions container that maps an error code to its match_string, description and severity. This is the in-memory shape that the runner and the library API will populate from a user-supplied CSV and hand to the screen-scanning logic later on. match_string is documented as a case-insensitive substring match (not a regex) so the matching contract is unambiguous from the model down.
Add CSVDataReader.read_error_definitions to read the error_code, match_string, description and severity columns into the dict shape the ErrorDefinitions model expects. Rows missing either error_code or match_string are skipped so a half-filled row never produces a useless definition. While here, collapse the chained startswith() check in _is_param into a single tuple call to keep ruff quiet.
Add error_detection.py with two pure functions shared by every caller: - extract_visible_text(): pulls only user-visible text out of a page source. HTML (Selenium/Playwright) is stripped via BeautifulSoup so CSS class names like class="error" can't cause false positives; mobile XML (Appium/XCUITest) is parsed with ElementTree, collecting the text-bearing attributes and falling back to a careful regex when the XML is malformed. - detect_errors_in_text(): case-insensitive substring match of each definition's match_string (or the code itself) against the visible text. Pull in beautifulsoup4 for the HTML path and refresh the lockfile. The bs4 import is guarded by a try/except so its absence degrades to a warning rather than a hard import error.
Carry an optional ErrorDefinitions object on Session and accept it through SessionHandler.create_session / SessionManager.create_session so both the runner and the library API can attach user-defined errors to a live session. Defaults to None to keep every existing caller working unchanged. The remaining positional params are also relaxed to Optional to match how they are already used by sessions created without a full test suite.
Teach find_files to recognise an error_definitions CSV by its error_code/match_string header and route it into a new collection, then load it via _load_error_definitions into an ErrorDefinitions store that is passed to create_session. This makes a project's error_definitions.csv picked up automatically by 'optics execute' with no extra wiring, the same way modules and elements are discovered.
Add JUnitEventHandler.add_detected_errors, which appends a synthetic 'on-screen-error-detection' testcase carrying a <failure> per detected error and bumps the suite's tests/failures counters. Emitting a real failing testcase (rather than <properties>) means Jenkins, GitLab CI and GitHub Actions surface the errors and fail the build accordingly. Also drop the now-unused per-testcase keyword_elements buffer; keyword elements are already attached as they arrive, so the second pass and its bookkeeping were dead.
After the live run finishes, capture a final screenshot and page source and, when the session carries error definitions, scan the page source for on-screen errors via the shared error_detection helpers. Matches are logged at WARNING, written to detected_errors_<session>.json under the execution output dir, and forwarded to the JUnit handler so they land in the report. The whole pass is wrapped in try/except and split into small helpers (_save_end_of_run_screenshot / _pagesource / _detection) so a flaky driver can never abort a run that otherwise succeeded.
A test case that ran to COMPLETED_PASSED was emitting its final 'test_case' event with EventStatus.FAIL, so a fully passing test case could still show up as failed in the live tree and the JUnit report. Emit EventStatus.PASS to match the actual outcome (_update_status already recorded PASS).
Add two keywords to the Optics library class so SDK/Robot Framework users get the same capability the runner has: - 'Add Error Definitions' attaches a dict or ErrorDefinitions to the active session, merging on repeated calls. - 'Capture And Detect Errors' captures the page source and returns the matched errors as a list of dicts, writing nothing and swallowing all driver I/O errors so it stays a safe diagnostic primitive. Also tidy the chained startswith() in _parse_config_string into a single tuple call.
Ship an example error_definitions.csv in the contact sample so users have a ready-made template covering common crash/network/session failure strings and can see the auto-discovery wiring end to end.
Scope the nosec on the dumpsys subprocess call to the specific B603 check and note that the package name comes from session capabilities, so the suppression documents exactly what is being waived instead of blanketing the line.
|
thakur-patel
approved these changes
Jun 30, 2026
Add a usage guide covering the error_definitions.csv format, auto-discovery by the runner, the library keywords (Add Error Definitions / Capture And Detect Errors), and how detected errors surface in logs, the JSON sidecar and the JUnit report. Link it from the docs nav.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds on-screen error detection: a project can declare a set of error strings, and the framework scans the screen at the end of a run (and on demand from the library API) and reports any that appear.
How it works
error_definitions.csv(error_code, match_string, description, severity).match_string(or the code itself) is matched as a case-insensitive substring — not a regex. HTML is stripped via BeautifulSoup so CSS class names can't cause false positives; mobile XML is parsed for text-bearing attributes only.detected_errors_<session>.json, and surfaced as a failing JUnit testcase so CI (Jenkins/GitLab/GitHub Actions) fails the build.Changes
common/error_detection.pyextract_visible_text/detect_errors_in_textprimitivescommon/models.pyErrorDefinitionsmodelcommon/runner/data_reader.pyerror_definitions.csvhelper/execute.pycommon/session_manager.pySessioncommon/runner/test_runnner.pycommon/Junit_eventhandler.pyoptics.pysamples/contact/...error_definitions.csvdocs/usage/error_detection.mdAlso folds in two small fixes found along the way: a completed test case now emits
PASS(wasFAIL) on its completion event, and the banditnosecon the adb dumpsys call is scoped toB603with a real justification.Notes
This supersedes #285 with a cleaner, incremental commit history; the net diff is identical.