Longer traces

README.md

@@ -1,6 +1,6 @@
 # RobotMBT - the oneliner
 
- Model-based testing in Robot framework with test case generation
+ Model-based testing in Robot framework with dynamic trace and test case generation
 
 ## Introduction
 
@@ -24,7 +24,7 @@ RobotMBT offers features to cover both _when_ and _what_ variations.
 
 RobotMBT is suitable for sequencing complete scenarios, including action refinement for when-steps. Concrete example scenarios can be generalised for added data-driven variation. When all steps are properly annotated with modelling info, the library can resolve their dependencies and figure out the correct execution order. Each run a new test sequence is generated from the available options.
 
-To be successful, the set of scenarios in the model must (for now) be composable into a single complete sequence, without leftovers. The same scenario can be inserted into the trace multiple times, creating loops, if repetition helps to reach the entry condition for later scenarios. Dead ends should be prevented, i.e., sequences from which there is no way forward and no way to loop back.
+To be successful, the set of scenarios in the model must be composable into a single complete sequence. There are no automatic resets or retries. The same scenario can be inserted into the trace multiple times, creating loops. Either to reach otherwise unreachable scenarios, or simply to create longer test runs. Dead ends should be prevented, i.e., sequences from which there is no way forward and no way to loop back.
 
 ## Getting started
 
@@ -207,6 +207,36 @@ Modified example values do not cascade. If a modifier expression references anot
 
 ## Configuration options
 
+Configure your test run by using any of the options from the list.
+
+| option                                  | purpose                          | values (default marked with *) |
+|-----------------------------------------|----------------------------------|--------------------------------|
+| [coverage_target](#setting-run-targets) | Each scenario must be executed at least this many times | 0 or 1* |
+| [scenario_target](#setting-run-targets) | The trace must have at least this many scenarios   | 0* or higher |
+| [time_target](#setting-run-targets)     | Setting a minimum test run duration           | Robot time string |
+| [seed](#random-seed)                    | Re-running a prior trace          | a specific seed, new* or None |
+| [batch_size](#batch-size)               | Phased trace generation              | 1 or higher (default 100*) |
+| [graph](#graphs)                        | Visualising the model   | None*, scenario or scenario-delta-value |
+| [export_graph_data](#exporting-and-importing-graph-data) | Storing graphs as json data | None* or file path |
+
+Options are available as named arguments:
+
+```robotframework
+Treat this test suite model-based    coverage_target=1    scenario_target=250
+```
+
+If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite.
+
+Tip: [Robot dictionaries](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#dictionary-variable) (`&{ }`) can be used to group related options and pass them as one set.
+
+### Setting run targets
+
+By default, a trace will be generated that runs to _single coverage_, meaning that each scenario must be included in the trace at least once. This is equivalent to setting `coverage_target=1`. To continue generating longer traces after single coverage is achieved, a second target can be enabled. For example, `scenario_target=500` will continue to run until there are 500 scenarios in the trace. Note that when scenarios are split up due to when-step refinement, that each part will count as one scenario. Alternatively, setting `time_target=1 hour 30 min` will cause the test run to run at least this long. Refer to the Robot Framework documentation to find [supported time formats](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#time-as-time-string).
+
+The test run will finish once all enabled targets are achieved. By default, only the coverage target is set. If you do not need guaranteed coverage, then `coverage_target=0` will disable the coverage check. Test runs can now finish before all scenarios are executed. This does not affect the trace generation process, which will still prefer new coverage over repetition.
+
+Tip: _When generating large test suites, use Robot Framework's [Split log](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#splitting-logs)  feature (`--splitlog`), to keep log file sizes manageable._
+
 ### Random seed
 
 By default, trace generation is random. The random seed used for the trace is logged by _Treat this test suite model-based_. This seed can be used to rerun the same trace, if no external random factors influence the test run. To activate the seed, pass it as argument:
@@ -217,6 +247,14 @@ Treat this test suite model-based    seed=eag-etou-cxi-leamv-jsi
 
 Using `seed=new` will force generation of a new reusable seed and is identical to omitting the seed argument. To completely bypass seed generation and use the system's random source, use `seed=None`. This has even more variation but does not produce a reusable seed.
 
+### Batch size
+
+Trace generation is done in _Batches_, so that the test run can already start before the full trace is generated. Small batch sizes cause the test run to start quickly, whereas larger batch sizes give more room to find suitable traces. Batch size is configurable by setting `batch_size=`.
+
+If the batch size is large enough to reach all run targets in a single batch, then the run will finish without further extensions. If not all targets are achieved in the first batch, then trace generation continues whenever new scenarios are needed. This is always at the end of a scenario. This scenario is tagged `mbt trace extension` and also contains the logging for the extended trace generation.
+
+Tip: _Small batch sizes are good at exposing dead ends in your model._
+
 ### Graphs
 
 A graph can be included in the log file to visualise how scenarios are linked. This helps in understanding a test suite's structure and reveals alternative paths that did not make it into the final trace.
@@ -252,12 +290,6 @@ Show model graph from exported file    json_file_path=<file_path>    graph_style
 
 This will draw a graph from the exported file, without the need to rerun the test suite. It is possible to select a different graph style than was used during the test run. If no graph style is selected, then the scenario graph style is used.
 
-### Option management
-
-If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite.
-
-Tip: [Robot dictionaries](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#dictionary-variable) (`&{ }`) can be used to group related options and pass them as one set.
-
 ## Contributing
 
 If you have feedback, ideas, or want to get involved in coding, then check out the [Contribution guidelines](https://github.com/JFoederer/robotframeworkMBT/blob/main/CONTRIBUTING.md).

atest/robotMBT tests/03__parse_model_info/MyProcessor.py

@@ -1,6 +1,9 @@
-class MyProcessor:
+from robotmbt import SuiteProcessor
 
-    def process_test_suite(self, in_suite):
+
+class MyProcessor(SuiteProcessor):
+    def process_test_suite(self, in_suite, **kwargs):
+        super().process_test_suite(in_suite, **kwargs)
         self.in_suite = in_suite
         self._fail_on_step_errors()
         msg = "Model info not properly parsed"

atest/robotMBT tests/03__parse_model_info/correct_model_info.robot

@@ -1,7 +1,7 @@
 *** Settings ***
 Suite Setup       Treat this test suite Model-based
 Library           MyProcessor.py
-Library           robotmbt    processor_lib=MyProcessor
+Library           robotmbt    processor=MyProcessor
 
 *** Test cases ***
 concise model info

atest/robotMBT tests/03__parse_model_info/incorrect_model_info.robot

@@ -1,7 +1,7 @@
 *** Settings ***
 Suite Setup       Expect failing suite processing
 Library           MyProcessor.py
-Library           robotmbt    processor_lib=MyProcessor
+Library           robotmbt    processor=MyProcessor
 
 *** Test Cases ***
 fail on empty model info

atest/robotMBT tests/07__processor_options/option_handling/01__pass_option_directly.robot → atest/robotMBT tests/07__processor_options/01__option_handling/01__pass_option_directly.robot

@@ -3,7 +3,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    repeat=2
 Suite Teardown    Should be equal    ${test_count}    ${2}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/02__set_option_by_keyword.robot → atest/robotMBT tests/07__processor_options/01__option_handling/02__set_option_by_keyword.robot

@@ -4,7 +4,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${2}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/03__update_option_by_keyword.robot → atest/robotMBT tests/07__processor_options/01__option_handling/03__update_option_by_keyword.robot

@@ -5,7 +5,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${3}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/04__update_option_at_trigger.robot → atest/robotMBT tests/07__processor_options/01__option_handling/04__update_option_at_trigger.robot

@@ -4,7 +4,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    repeat=3
 Suite Teardown    Should be equal    ${test_count}    ${3}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/05__use_update_without_setter.robot → atest/robotMBT tests/07__processor_options/01__option_handling/05__use_update_without_setter.robot

@@ -4,7 +4,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${2}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/06__pass_multiple_options.robot → atest/robotMBT tests/07__processor_options/01__option_handling/06__pass_multiple_options.robot

@@ -3,7 +3,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    repeat=2    bonus_scenario=${True}
 Suite Teardown    Should be equal    ${test_count}    ${3}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/07__set_clears_other_options.robot → atest/robotMBT tests/07__processor_options/01__option_handling/07__set_clears_other_options.robot

@@ -5,7 +5,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${2}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/08__empty_setter_clears_all_options.robot → atest/robotMBT tests/07__processor_options/01__option_handling/08__empty_setter_clears_all_options.robot

@@ -5,7 +5,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${1}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/09__multiple_options_from_dict.robot → atest/robotMBT tests/07__processor_options/01__option_handling/09__multiple_options_from_dict.robot

@@ -3,7 +3,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    &{mbt_options}
 Suite Teardown    Should be equal    ${test_count}    ${3}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Variables ***
 &{mbt_options}    repeat=2    bonus_scenario=${True}

atest/robotMBT tests/07__processor_options/option_handling/10__partial_option_update.robot → atest/robotMBT tests/07__processor_options/01__option_handling/10__partial_option_update.robot

@@ -4,7 +4,7 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    bonus_scenario=${False}
 Suite Teardown    Should be equal    ${test_count}    ${2}
 Library           suiterepeater.py
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater
 
 *** Variables ***
 &{mbt_options}    repeat=2    bonus_scenario=${True}

atest/robotMBT tests/07__processor_options/01__option_handling/11__argument_restrictions.robot

@@ -0,0 +1,12 @@
+*** Settings ***
+Library           strictsuiterepeater.py
+Library           robotmbt    processor=strictsuiterepeater
+
+*** Test Cases ***
+arguments can be mandatory
+    Run keyword and expect error    *StrictSuiteRepeater.process_test_suite() missing 1 required keyword-only argument: 'repeat'
+    ...    Treat this test suite Model-based    bonus_scenario=${True}
+
+can fail on unknown arguments
+    Run keyword and expect error    *StrictSuiteRepeater.process_test_suite() got an unexpected keyword argument 'intentional_fail'
+    ...    Treat this test suite Model-based    repeat=1    bonus_scenario=${True}    intentional_fail=${True}

atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/__init__.robot → atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/__init__.robot

@@ -3,4 +3,4 @@ Documentation     In this suite one of the processor options is set on the highe
 ...               which is then reused in both sub suites. Each sub suite adds their own value
 ...               for a second configuration option.
 Suite Setup       Set model-based options    repeat=2
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater

atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot → atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot

@@ -2,7 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    bonus_scenario=${True}
 Suite Teardown    Should be equal    ${test_count}    ${3}
-Library           robotmbt    processor_lib=suiterepeater
+Library           ../suiterepeater.py
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot → atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot

@@ -2,7 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    bonus_scenario=${False}
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           robotmbt    processor_lib=suiterepeater
+Library           ../suiterepeater.py
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/01__with_bonus_scenario_option.robot → atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/01__with_bonus_scenario_option.robot

@@ -2,7 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    bonus_scenario=${True}
 Suite Teardown    Should be equal    ${test_count}    ${3}
-Library           robotmbt    processor_lib=suiterepeater
+Library           ../suiterepeater.py
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/02__without_using_bonus_scenario_option.robot → atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/02__without_using_bonus_scenario_option.robot

@@ -2,7 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           robotmbt    processor_lib=suiterepeater
+Library           ../suiterepeater.py
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/__init__.robot → atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/__init__.robot

@@ -5,4 +5,4 @@ Documentation     In this suite one of the processor options is set on the highe
 ...               all. The second suite should be unaffected by the option set in the preceeding
 ...               suite.
 Suite Setup       Set model-based options    repeat=2
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater

atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/03__direct_setting_overrules_library_setting.robot → atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/03__direct_setting_overrules_library_setting.robot

@@ -2,7 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    repeat=3
 Suite Teardown    Should be equal    ${test_count}    ${3}
-Library           robotmbt    processor_lib=suiterepeater
+Library           ../suiterepeater.py
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/04__prior_overrule_does_not_persist.robot → atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/04__prior_overrule_does_not_persist.robot

@@ -2,7 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           robotmbt    processor_lib=suiterepeater
+Library           ../suiterepeater.py
+Library           robotmbt    processor=suiterepeater
 
 *** Test Cases ***
 only test case

atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/__init__.robot → atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/__init__.robot

@@ -4,4 +4,4 @@ Documentation     In this suite one of the processor options is set on the highe
 ...               setting with their own value, the second library doesn't. The second suite should
 ...               be unaffected by the overruled option from the preceeding suite.
 Suite Setup       Set model-based options    repeat=2
-Library           robotmbt    processor_lib=suiterepeater
+Library           robotmbt    processor=suiterepeater

atest/robotMBT tests/07__processor_options/01__option_handling/strictsuiterepeater.py

@@ -0,0 +1,15 @@
+from robot.api.deco import library
+
+from suiterepeater import SuiteRepeater
+
+
+@library(auto_keywords=None, listener=True)
+class StrictSuiteRepeater(SuiteRepeater):
+    """
+    Nearly identical to SuiteRepeater as used in other test cases. The difference is that
+    this variant is strict in its argument handling and will fail if mandatory arguments
+    are missing or unknown arguments are provided.
+    """
+
+    def process_test_suite(self, in_suite, *, repeat, bonus_scenario=False):
+        return super().process_test_suite(in_suite, repeat=repeat, bonus_scenario=bonus_scenario)

atest/robotMBT tests/07__processor_options/option_handling/suiterepeater.py → atest/robotMBT tests/07__processor_options/01__option_handling/suiterepeater.py

@@ -1,27 +1,27 @@
 import copy
 
 from robot.api.deco import library
+from robotmbt import SuiteProcessor
 
 
 @library(auto_keywords=None, listener=True)
-class SuiteRepeater:
+class SuiteRepeater(SuiteProcessor):
     """
     Given a test suite, repeats all scenarios 'repeat' times (default=1)
     Setting bonus_scenario=${True} repeats 1 additional time
     sub-suites are ignored
     """
 
     def process_test_suite(self, in_suite, repeat=1, **kwargs):
+        super().process_test_suite(in_suite, **kwargs)
         n_repeats = int(repeat)
         if kwargs.get('bonus_scenario', False):
             n_repeats += 1
+        self.scenario_count *= n_repeats
         out_suite = copy.deepcopy(in_suite)
         out_suite.scenarios = n_repeats*out_suite.scenarios
         for i in range(len(out_suite.scenarios)):
             out_suite.scenarios[i] = out_suite.scenarios[i].copy()
             if i:
                 out_suite.scenarios[i].name += f" (rep {i+1})"
         return out_suite
-
-    def mandatory_repeat_argument(self, in_suite, *, repeat, bonus_scenario=False):
-        return self.process_test_suite(in_suite, repeat=repeat, bonus_scenario=bonus_scenario)