Explore-then-search trace generation

.github/workflows/run-tests.yml

@@ -1,7 +1,7 @@
 # This workflow installs required Python dependencies and then runs the available tests.
 # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
 
-name: Run Acceptance and Unit tests
+name: Run Acceptance and Unit tests (with and without visualisation dependencies)
 
 on:
   pull_request:
@@ -21,10 +21,18 @@ jobs:
       uses: actions/setup-python@v3
       with:
         python-version: "3.10"  # Only the oldest supported Python version is included here
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip   # upgrade pip to latest version
-        pip install .                         # install pyproject.toml dependencies - excludes optional dependencies (such as visualisation)
-    - name: Run tests
+
+    - name: Install dependencies (without extra visualisation dependencies)
       run: |
-        python run_tests.py
+        python -m pip install --upgrade pip           # upgrade pip to latest version
+        pip install .                                 # no additional [visualisation] dependencies
+
+    - name: Run tests (without visualisation dependencies)
+      run:  python run_tests.py
+
+    - name: Install extra visualisation dependencies
+      run:  pip install ".[visualisation]"            # extra [visualisation] dependencies in pyproject.toml
+
+    - name: Run Tests (with visualisation dependencies)
+      run:  python run_tests.py
+

README.md

@@ -36,7 +36,7 @@ The recommended installation method is using [pip](http://pip-installer.org)
 
 After installation include `robotmbt` as library in your robot file to get access to the new functionality. To run your test suite model-based, use the __Treat this test suite model-based__ keyword as suite setup. Check the _How to model_ section to learn how to make your scenarios suitable for running model-based.
 
-```
+```robotframework
 *** Settings ***
 Library        robotmbt
 Suite Setup    Treat this test suite model-based
@@ -50,7 +50,8 @@ Modelling can be done directly from [Robot framework](https://robotframework.org
 
 Consider these two scenarios:
 
-```
+```robotframework
+*** Test Cases ***
 Buying a postcard
     When you buy a new postcard
     then you have a blank postcard
@@ -63,7 +64,8 @@ Preparing for a birthday party
 
 Mapping the dependencies between scenarios is done by annotating the steps with modelling info. Modelling info is added to the documentation of the step as shown below. Regular documentation can still be added, as long as `*model info*` starts on a new line and a white line is included after the last `:OUT:` expression.
 
-```
+```robotframework
+*** Keywords ***
 you buy a new postcard
     [Documentation]    *model info*
     ...    :IN: None
@@ -117,7 +119,8 @@ All example scenarios naturally contain data. This information is embedded in th
 
 #### Step argument modifiers
 
-```
+```robotframework
+*** Test Cases ***
 Personalising a birthday card
     Given there is a birthday card
     when Johan writes their name on the birthday card
@@ -126,7 +129,8 @@ Personalising a birthday card
 
 The above scenario uses the name `Johan` to create a concrete example. But now suppose that from a testing perspective `Johan` and `Frederique` are part of the same equivalence class. Then the step `Frederique writes their name on the birthday card` would yield an equally valid scenario. This can be achieved by adding a modifier (`:MOD:`) to the model info of the step. The format of a modifier is a Robot argument to which you assign a list of options. The modifier updates the argument value to a randomly chosen value from the specified options.
 
-```
+```robotframework
+*** Keywords ***
 ${person} writes their name on the birthday card
     [Documentation]    *model info*
     ...    :MOD: ${person}= [Johan, Frederique]
@@ -138,7 +142,8 @@ ${person} writes their name on the birthday card
 
 When constructing examples, they often express relations between multiple actors, where each actor can appear in multiple steps. This makes it important to know how modifiers behave when there are multiple modifiers in a scenario.
 
-```
+```robotframework
+*** Test Cases ***
 Addressing a birthday card
    Given Tannaz is having their birthday
    and Johan has a birthday card
@@ -148,7 +153,8 @@ Addressing a birthday card
 
 Have a look at the when-step above. We will assume the model already contains a domain term with two properties: `birthday.celebrant = Tannaz` and `birthday.guests = [Johan, Frederique]`.
 
-```
+```robotframework
+*** Keywords ***
 ${sender} writes the address of ${receiver} on the birthday card
     [Documentation]    *model info*
     ...    :MOD: ${sender}= birthday.guests
@@ -175,10 +181,12 @@ It is not possible to add new options to an existing example value. Any constrai
 
 It is possible for a step to keep the same options. The special `.*` notation lets you keep the available options as-is. Preceding steps must then supply the possible options. Some steps can, or must, deal with multiple independent sets of options that must not be mixed, because the expected results should differ. Suppose you have a set of valid and invalid passwords. You might be reluctant to include the superset of these as options to an authentication step. Instead, you can use `:MOD: ${password}= .*` as the modifier for that step. Like in the when-step for this scenario:
 
-```
-Given 'secret' is too weak a password
-When user tries to update their password to 'secret'
-then the password is rejected
+```robotframework
+*** Test Cases ***
+Reject password
+    Given 'secret' is too weak a password
+    When user tries to update their password to 'secret'
+    then the password is rejected
 ```
 
 In a then-step, modifiers behave slightly different. In then-steps no new option constraints are accepted for an argument. Its value must already have been determined during the given- and when-steps. In other words, regardless of the actual modifier, the expression behaves as if it were `.*`. The exception to this is when a then-step signals the first use of a new example value. In that case the argument value from the original scenario text is used.
@@ -193,12 +201,47 @@ For now, variable data considers strict equivalence classes only. This means tha
 
 By default, trace generation is random. The random seed used for the trace is logged by _Treat this test suite model-based_. This seed can be used to rerun the same trace, if no external random factors influence the test run. To activate the seed, pass it as argument:
 
-```
+```robotframework
 Treat this test suite model-based    seed=eag-etou-cxi-leamv-jsi
 ```
 
 Using `seed=new` will force generation of a new reusable seed and is identical to omitting the seed argument. To completely bypass seed generation and use the system's random source, use `seed=None`. This has even more variation but does not produce a reusable seed.
 
+### Graphs
+
+A graph can be included in the log file to visualise how scenarios are linked. This helps in understanding a test suite's structure and reveals alternative paths that did not make it into the final trace.
+
+To enable graph generation, some extra dependencies must be installed: `pip install robotframework-mbt[visualisation]`
+
+Generate the graph by setting the graph style for the model-based suite. The graph will be included in the Robot log file as part of the keyword's logging.
+
+```robotframework
+Treat this test suite Model-based  graph=scenario
+```
+
+Available graph styles:
+
+* scenario
+  * Compact view: Each scenario is shown as one node.
+* scenario-delta-value
+  * Expanded view: Scenarios can become multiple nodes if they affect system state in different ways.
+
+#### Exporting and importing graph data
+
+Graph data can be stored by setting an output directory using argument `export_graph_data`. This createss a json file, named after the test suite, in the selected folder. Any accessable path can be used. For your convenience, Robot Framework offers an [automatic variable](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#automatic-variables) `${OUTPUT_DIR}` that points to this run's output directory. The `export_graph_data` argument can be used independent of the `graph` argument.
+
+```robotframework
+Treat this test suite Model-based  export_graph_data=${OUTPUT_DIR}
+```
+
+To recreate a graph from previously exported graph data, use:
+
+```robotframework
+Show model graph from exported file    json_file_path=<file_path>    graph_style=scenario
+```
+
+This will draw a graph from the exported file, without the need to rerun the test suite. It is possible to select a different graph style than was used during the test run. If no graph style is selected, then the scenario graph style is used.
+
 ### Option management
 
 If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite.

atest/robotMBT tests/07__processor_options/random_seeds/02__reusing_seed_reproduces_trace.robot

@@ -5,7 +5,7 @@ Documentation     This suite uses the `seed` argument to reproduce a previously
 ...               indicated by the number string.
 Suite Setup       Run keywords    Set suite variable    ${trace}    ${empty}
 ...                        AND    Treat this test suite Model-based    seed=aqmou-eelcuu-sniu-ugsyek-jyhoor
-Suite Teardown    Should be equal    ${trace}    6930142758
+Suite Teardown    Should be equal    ${trace}    6879523014
 Library           robotmbt
 
 *** Test Cases ***

atest/robotMBT tests/07__processor_options/random_seeds/03__retrace_with_refinement.robot

@@ -9,7 +9,7 @@ Documentation     This suite uses the `seed` argument to reproduce a previously
 ...               traces possible of varying length.
 Suite Setup       Run keywords    Set suite variable    ${trace}    ${empty}
 ...                        AND    Treat this test suite Model-based    seed=xou-pumj-ihj-oibiyc-surer
-Suite Teardown    Should be equal    ${trace}    B1A5BY3B4BX6B2
+Suite Teardown    Should be equal    ${trace}    B3AY2A5A6B4BX1
 Library           robotmbt
 
 *** Test Cases ***

atest/robotMBT tests/07__processor_options/random_seeds/04__retrace_with_step_modifiers.robot

@@ -6,7 +6,7 @@ Documentation     This suite uses the `seed` argument to reproduce a previously
 ...               included, the modifers will number the scenarios at random. The reproduced trace
 ...               must match for both the scenario order and the data order.
 Suite Setup       Treat this test suite Model-based    seed=iulr-vih-esycu-eyl-yfa
-Suite Teardown    Should be equal    ${trace}    H6G3E5I4D9F8J2B1A7C0
+Suite Teardown    Should be equal    ${trace}    H2G3C1E7I5B9F0A4D6J8
 Library           robotmbt
 
 *** Test Cases ***

atest/robotMBT tests/07__processor_options/random_seeds/05__retrace_combined.robot

@@ -15,8 +15,8 @@ Documentation     This suite uses the `seed` argument to reproduce a previously
 ...               a data choice by using step modifiers. The lower level includes a path choice.
 ...               Both low-level scenarios are equally valid, the only difference is that the data
 ...               choice is included either once or twice.
-Suite Setup       Treat this test suite Model-based    seed=gujuqt-iakm-oexo-xnu-huba
-Suite Teardown    Should be equal    ${trace}    ATQQPRSPRPXY
+Suite Setup       Treat this test suite Model-based    seed=kece-zwu-eihho-yli-rbixx
+Suite Teardown    Should be equal    ${trace}    APPSRTTPPXY
 Library           robotmbt
 
 *** Test Cases ***

atest/robotMBT tests/10__visualisation/01__scenario_graph/__init__.robot

@@ -0,0 +1,3 @@
+*** Settings ***
+Suite Setup       Clear prior exports    ${OUTPUT_DIR}${/}run_model_with_graph.json
+Library           ../graph_checker.py

atest/robotMBT tests/10__visualisation/01__scenario_graph/run_model_with_graph.robot

@@ -0,0 +1,29 @@
+*** Settings ***
+Documentation     This suite runs model-based with the graphing option enabled. The checks for the contents
+...               of the graph are delegated to the second robot file in the same folder. Export/import
+...               functionality is used to gain access to the graph contents.
+Suite Setup       Treat this test suite Model-based    graph=scenario    export_graph_data=${OUTPUT_DIR}
+Resource          ../../../resources/birthday_cards_data_variation.resource
+Library           robotmbt
+
+*** Test Cases ***
+Buying a card
+    Given Jonathan is having their birthday
+    and Douwe is a friend of Jonathan
+    and Diogo is a friend of Jonathan
+    and Tycho is a friend of Jonathan
+    and Thomas is a friend of Jonathan
+    When Douwe buys a birthday card
+    then there is a blank birthday card available
+
+Someone writes their name on the card
+    Given there is a birthday card
+    and Diogo's name is not yet on the birthday card
+    when Diogo writes their name on the birthday card
+    then the birthday card has 'Diogo' written on it
+
+At least 3 people can write their name on the card
+    Given the birthday card has 2 different names written on it
+    and Tycho's name is not yet on the birthday card
+    when Tycho writes their name on the birthday card
+    then the birthday card has 3 different names written on it

atest/robotMBT tests/10__visualisation/01__scenario_graph/verify_model_graph.robot

@@ -0,0 +1,33 @@
+*** Settings ***
+Documentation     This suite takes the graph generated in the first suite in this folder
+...               and checks the content's properties.
+Library           ../graph_checker.py
+Library           Collections
+Suite Setup       Import graph data from    ${OUTPUT_DIR}${/}run_model_with_graph.json
+
+*** Test Cases ***
+Scenarios are nodes in the graph
+    VAR    @{expected_nodes}    start
+    ...                         Buying a card
+    ...                         Someone writes their name on the card
+    ...                         At least 3 people can write their name on the card
+    ${node_count}=    Number of graph nodes
+    Should be equal    ${node_count}    ${4}
+    @{all_nodes}=    List of node titles
+    Lists should be equal    ${all_nodes}    ${expected_nodes}    ignore_order=True
+
+Dependent nodes are connected by directed edges
+    @{successors}=    All successors to node    Buying a card
+    VAR    @{next_node}     Someone writes their name on the card
+    Should be equal    ${successors}    ${next_node}
+    @{successors}=    All successors to node    Someone writes their name on the card
+    Should not contain    ${successors}    Buying a card
+
+The dependency order for nodes is top-down
+    ${first_pos}=    Vertical position of node    Buying a card
+    ${second_pos}=    Vertical position of node    Someone writes their name on the card
+    Should be true    ${second_pos} < ${first_pos}
+
+Repeating scenarios have a self-looping edge
+    @{successors}=    All successors to node    Someone writes their name on the card
+    Should contain    ${successors}    Someone writes their name on the card

atest/robotMBT tests/10__visualisation/02__scenario-delta-value_graph/__init__.robot

@@ -0,0 +1,3 @@
+*** Settings ***
+Suite Setup       Clear prior exports    ${OUTPUT_DIR}${/}run_model_with_graph.json
+Library           ../graph_checker.py

atest/robotMBT tests/10__visualisation/02__scenario-delta-value_graph/run_model_with_graph.robot

@@ -0,0 +1,29 @@
+*** Settings ***
+Documentation     This suite runs model-based with the graphing option enabled. The checks for the contents
+...               of the graph are delegated to the second robot file in the same folder. Export/import
+...               functionality is used to gain access to the graph contents.
+Suite Setup       Treat this test suite Model-based    graph=scenario-delta-value    export_graph_data=${OUTPUT_DIR}
+Resource          ../../../resources/birthday_cards_data_variation.resource
+Library           robotmbt
+
+*** Test Cases ***
+Buying a card
+    Given Jonathan is having their birthday
+    and Douwe is a friend of Jonathan
+    and Diogo is a friend of Jonathan
+    and Tycho is a friend of Jonathan
+    and Thomas is a friend of Jonathan
+    When Douwe buys a birthday card
+    then there is a blank birthday card available
+
+Someone writes their name on the card
+    Given there is a birthday card
+    and Diogo's name is not yet on the birthday card
+    when Diogo writes their name on the birthday card
+    then the birthday card has 'Diogo' written on it
+
+At least 3 people can write their name on the card
+    Given the birthday card has 2 different names written on it
+    and Tycho's name is not yet on the birthday card
+    when Tycho writes their name on the birthday card
+    then the birthday card has 3 different names written on it

atest/robotMBT tests/10__visualisation/02__scenario-delta-value_graph/verify_model_graph.robot

@@ -0,0 +1,36 @@
+*** Settings ***
+Documentation     This suite takes the graph generated in the first suite in this folder
+...               and checks how the content's properties for the scenario-delta-value
+...               graph are different compared to the scenario graph.
+Library           ../graph_checker.py
+Library           Collections
+Suite Setup       Import graph data from    ${OUTPUT_DIR}${/}run_model_with_graph.json    graph_type=scenario-delta-value
+
+*** Test Cases ***
+Repeated scenario with different states become separate nodes
+    [Documentation]    In the standard scenario graph, the repeated scenario would show up
+    ...                as a self-looping edge. In the scenario-delta-value variant, also
+    ...                state changes are taken into account, meaning that the two variants
+    ...                will each get their own node.
+    ${node_count}=    Number of graph nodes
+    Should be true    ${node_count} > ${4}
+    @{all_nodes}=    List of node titles
+    Should contain    ${all_nodes}     Someone writes their name on the card
+    Should contain    ${all_nodes}     Someone writes their name on the card (rep 2)
+
+Full node text contains state info
+    [Documentation]    Next to the scenario name, the node text for scenario-delta-value graphs
+    ...                also contians information about the model state. 'Buying a card'
+    ...                initialises the modeland contains all properties in their initial state.
+    ...                'host' and 'names' are properties being tracked by the model.
+    ${full_text}=    Full node text of node     Buying a card
+    Should contain    ${full_text}   host
+    Should contain    ${full_text}   names
+
+Full node text only contains changed information
+    [Documentation]    When someone writes their name on the card, then the host or celebrant
+    ...                does not change. The node for this scenario should only show the state
+    ...                that changed during this scenario.
+    ${full_text}=    Full node text of node     Someone writes their name on the card (rep 2)
+    Should contain    ${full_text}   names
+    Should not contain    ${full_text}   host