diff --git a/README.md b/README.md index 71f25a6f..d84fda1f 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # RobotMBT - the oneliner - Model-based testing in Robot framework with test case generation + Model-based testing in Robot framework with dynamic trace and test case generation ## Introduction @@ -24,7 +24,7 @@ RobotMBT offers features to cover both _when_ and _what_ variations. RobotMBT is suitable for sequencing complete scenarios, including action refinement for when-steps. Concrete example scenarios can be generalised for added data-driven variation. When all steps are properly annotated with modelling info, the library can resolve their dependencies and figure out the correct execution order. Each run a new test sequence is generated from the available options. -To be successful, the set of scenarios in the model must (for now) be composable into a single complete sequence, without leftovers. The same scenario can be inserted into the trace multiple times, creating loops, if repetition helps to reach the entry condition for later scenarios. Dead ends should be prevented, i.e., sequences from which there is no way forward and no way to loop back. +To be successful, the set of scenarios in the model must be composable into a single complete sequence. There are no automatic resets or retries. The same scenario can be inserted into the trace multiple times, creating loops. Either to reach otherwise unreachable scenarios, or simply to create longer test runs. Dead ends should be prevented, i.e., sequences from which there is no way forward and no way to loop back. ## Getting started @@ -207,6 +207,36 @@ Modified example values do not cascade. If a modifier expression references anot ## Configuration options +Configure your test run by using any of the options from the list. + +| option | purpose | values (default marked with *) | +|-----------------------------------------|----------------------------------|--------------------------------| +| [coverage_target](#setting-run-targets) | Each scenario must be executed at least this many times | 0 or 1* | +| [scenario_target](#setting-run-targets) | The trace must have at least this many scenarios | 0* or higher | +| [time_target](#setting-run-targets) | Setting a minimum test run duration | Robot time string | +| [seed](#random-seed) | Re-running a prior trace | a specific seed, new* or None | +| [batch_size](#batch-size) | Phased trace generation | 1 or higher (default 100*) | +| [graph](#graphs) | Visualising the model | None*, scenario or scenario-delta-value | +| [export_graph_data](#exporting-and-importing-graph-data) | Storing graphs as json data | None* or file path | + +Options are available as named arguments: + +```robotframework +Treat this test suite model-based coverage_target=1 scenario_target=250 +``` + +If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite. + +Tip: [Robot dictionaries](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#dictionary-variable) (`&{ }`) can be used to group related options and pass them as one set. + +### Setting run targets + +By default, a trace will be generated that runs to _single coverage_, meaning that each scenario must be included in the trace at least once. This is equivalent to setting `coverage_target=1`. To continue generating longer traces after single coverage is achieved, a second target can be enabled. For example, `scenario_target=500` will continue to run until there are 500 scenarios in the trace. Note that when scenarios are split up due to when-step refinement, that each part will count as one scenario. Alternatively, setting `time_target=1 hour 30 min` will cause the test run to run at least this long. Refer to the Robot Framework documentation to find [supported time formats](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#time-as-time-string). + +The test run will finish once all enabled targets are achieved. By default, only the coverage target is set. If you do not need guaranteed coverage, then `coverage_target=0` will disable the coverage check. Test runs can now finish before all scenarios are executed. This does not affect the trace generation process, which will still prefer new coverage over repetition. + +Tip: _When generating large test suites, use Robot Framework's [Split log](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#splitting-logs) feature (`--splitlog`), to keep log file sizes manageable._ + ### Random seed By default, trace generation is random. The random seed used for the trace is logged by _Treat this test suite model-based_. This seed can be used to rerun the same trace, if no external random factors influence the test run. To activate the seed, pass it as argument: @@ -217,6 +247,14 @@ Treat this test suite model-based seed=eag-etou-cxi-leamv-jsi Using `seed=new` will force generation of a new reusable seed and is identical to omitting the seed argument. To completely bypass seed generation and use the system's random source, use `seed=None`. This has even more variation but does not produce a reusable seed. +### Batch size + +Trace generation is done in _Batches_, so that the test run can already start before the full trace is generated. Small batch sizes cause the test run to start quickly, whereas larger batch sizes give more room to find suitable traces. Batch size is configurable by setting `batch_size=`. + +If the batch size is large enough to reach all run targets in a single batch, then the run will finish without further extensions. If not all targets are achieved in the first batch, then trace generation continues whenever new scenarios are needed. This is always at the end of a scenario. This scenario is tagged `mbt trace extension` and also contains the logging for the extended trace generation. + +Tip: _Small batch sizes are good at exposing dead ends in your model._ + ### Graphs A graph can be included in the log file to visualise how scenarios are linked. This helps in understanding a test suite's structure and reveals alternative paths that did not make it into the final trace. @@ -252,12 +290,6 @@ Show model graph from exported file json_file_path= graph_style This will draw a graph from the exported file, without the need to rerun the test suite. It is possible to select a different graph style than was used during the test run. If no graph style is selected, then the scenario graph style is used. -### Option management - -If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite. - -Tip: [Robot dictionaries](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#dictionary-variable) (`&{ }`) can be used to group related options and pass them as one set. - ## Contributing If you have feedback, ideas, or want to get involved in coding, then check out the [Contribution guidelines](https://github.com/JFoederer/robotframeworkMBT/blob/main/CONTRIBUTING.md). diff --git a/atest/robotMBT tests/03__parse_model_info/MyProcessor.py b/atest/robotMBT tests/03__parse_model_info/MyProcessor.py index 09cb13bb..c2bd9d78 100644 --- a/atest/robotMBT tests/03__parse_model_info/MyProcessor.py +++ b/atest/robotMBT tests/03__parse_model_info/MyProcessor.py @@ -1,6 +1,9 @@ -class MyProcessor: +from robotmbt import SuiteProcessor - def process_test_suite(self, in_suite): + +class MyProcessor(SuiteProcessor): + def process_test_suite(self, in_suite, **kwargs): + super().process_test_suite(in_suite, **kwargs) self.in_suite = in_suite self._fail_on_step_errors() msg = "Model info not properly parsed" diff --git a/atest/robotMBT tests/03__parse_model_info/correct_model_info.robot b/atest/robotMBT tests/03__parse_model_info/correct_model_info.robot index bbab3e87..1d6cd4ea 100644 --- a/atest/robotMBT tests/03__parse_model_info/correct_model_info.robot +++ b/atest/robotMBT tests/03__parse_model_info/correct_model_info.robot @@ -1,7 +1,7 @@ *** Settings *** Suite Setup Treat this test suite Model-based Library MyProcessor.py -Library robotmbt processor_lib=MyProcessor +Library robotmbt processor=MyProcessor *** Test cases *** concise model info diff --git a/atest/robotMBT tests/03__parse_model_info/incorrect_model_info.robot b/atest/robotMBT tests/03__parse_model_info/incorrect_model_info.robot index 34c9fb29..a39197fe 100644 --- a/atest/robotMBT tests/03__parse_model_info/incorrect_model_info.robot +++ b/atest/robotMBT tests/03__parse_model_info/incorrect_model_info.robot @@ -1,7 +1,7 @@ *** Settings *** Suite Setup Expect failing suite processing Library MyProcessor.py -Library robotmbt processor_lib=MyProcessor +Library robotmbt processor=MyProcessor *** Test Cases *** fail on empty model info diff --git a/atest/robotMBT tests/07__processor_options/option_handling/01__pass_option_directly.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/01__pass_option_directly.robot similarity index 86% rename from atest/robotMBT tests/07__processor_options/option_handling/01__pass_option_directly.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/01__pass_option_directly.robot index d4820835..21817a1b 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/01__pass_option_directly.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/01__pass_option_directly.robot @@ -3,7 +3,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based repeat=2 Suite Teardown Should be equal ${test_count} ${2} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/02__set_option_by_keyword.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/02__set_option_by_keyword.robot similarity index 87% rename from atest/robotMBT tests/07__processor_options/option_handling/02__set_option_by_keyword.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/02__set_option_by_keyword.robot index bf673c84..8015bdbe 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/02__set_option_by_keyword.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/02__set_option_by_keyword.robot @@ -4,7 +4,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based Suite Teardown Should be equal ${test_count} ${2} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/03__update_option_by_keyword.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/03__update_option_by_keyword.robot similarity index 89% rename from atest/robotMBT tests/07__processor_options/option_handling/03__update_option_by_keyword.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/03__update_option_by_keyword.robot index f95ef80c..388fc662 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/03__update_option_by_keyword.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/03__update_option_by_keyword.robot @@ -5,7 +5,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based Suite Teardown Should be equal ${test_count} ${3} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/04__update_option_at_trigger.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/04__update_option_at_trigger.robot similarity index 88% rename from atest/robotMBT tests/07__processor_options/option_handling/04__update_option_at_trigger.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/04__update_option_at_trigger.robot index 4d34f75d..55289dc5 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/04__update_option_at_trigger.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/04__update_option_at_trigger.robot @@ -4,7 +4,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based repeat=3 Suite Teardown Should be equal ${test_count} ${3} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/05__use_update_without_setter.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/05__use_update_without_setter.robot similarity index 87% rename from atest/robotMBT tests/07__processor_options/option_handling/05__use_update_without_setter.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/05__use_update_without_setter.robot index b32f658f..cdcd3bc9 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/05__use_update_without_setter.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/05__use_update_without_setter.robot @@ -4,7 +4,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based Suite Teardown Should be equal ${test_count} ${2} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/06__pass_multiple_options.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/06__pass_multiple_options.robot similarity index 87% rename from atest/robotMBT tests/07__processor_options/option_handling/06__pass_multiple_options.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/06__pass_multiple_options.robot index 530739c5..c589ec2c 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/06__pass_multiple_options.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/06__pass_multiple_options.robot @@ -3,7 +3,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based repeat=2 bonus_scenario=${True} Suite Teardown Should be equal ${test_count} ${3} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/07__set_clears_other_options.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/07__set_clears_other_options.robot similarity index 89% rename from atest/robotMBT tests/07__processor_options/option_handling/07__set_clears_other_options.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/07__set_clears_other_options.robot index 0f9ff6c6..6d472f00 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/07__set_clears_other_options.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/07__set_clears_other_options.robot @@ -5,7 +5,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based Suite Teardown Should be equal ${test_count} ${2} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/08__empty_setter_clears_all_options.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/08__empty_setter_clears_all_options.robot similarity index 89% rename from atest/robotMBT tests/07__processor_options/option_handling/08__empty_setter_clears_all_options.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/08__empty_setter_clears_all_options.robot index 292ee346..6b7fe5b7 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/08__empty_setter_clears_all_options.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/08__empty_setter_clears_all_options.robot @@ -5,7 +5,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based Suite Teardown Should be equal ${test_count} ${1} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/09__multiple_options_from_dict.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/09__multiple_options_from_dict.robot similarity index 88% rename from atest/robotMBT tests/07__processor_options/option_handling/09__multiple_options_from_dict.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/09__multiple_options_from_dict.robot index 1d0443c7..f7ee4f3c 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/09__multiple_options_from_dict.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/09__multiple_options_from_dict.robot @@ -3,7 +3,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based &{mbt_options} Suite Teardown Should be equal ${test_count} ${3} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Variables *** &{mbt_options} repeat=2 bonus_scenario=${True} diff --git a/atest/robotMBT tests/07__processor_options/option_handling/10__partial_option_update.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/10__partial_option_update.robot similarity index 90% rename from atest/robotMBT tests/07__processor_options/option_handling/10__partial_option_update.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/10__partial_option_update.robot index e840d71e..80a88dd8 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/10__partial_option_update.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/10__partial_option_update.robot @@ -4,7 +4,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based bonus_scenario=${False} Suite Teardown Should be equal ${test_count} ${2} Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater *** Variables *** &{mbt_options} repeat=2 bonus_scenario=${True} diff --git a/atest/robotMBT tests/07__processor_options/01__option_handling/11__argument_restrictions.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/11__argument_restrictions.robot new file mode 100644 index 00000000..e6fc9f6f --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/11__argument_restrictions.robot @@ -0,0 +1,12 @@ +*** Settings *** +Library strictsuiterepeater.py +Library robotmbt processor=strictsuiterepeater + +*** Test Cases *** +arguments can be mandatory + Run keyword and expect error *StrictSuiteRepeater.process_test_suite() missing 1 required keyword-only argument: 'repeat' + ... Treat this test suite Model-based bonus_scenario=${True} + +can fail on unknown arguments + Run keyword and expect error *StrictSuiteRepeater.process_test_suite() got an unexpected keyword argument 'intentional_fail' + ... Treat this test suite Model-based repeat=1 bonus_scenario=${True} intentional_fail=${True} diff --git a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/__init__.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/__init__.robot similarity index 84% rename from atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/__init__.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/__init__.robot index 37052f90..43ef3701 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/__init__.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/__init__.robot @@ -3,4 +3,4 @@ Documentation In this suite one of the processor options is set on the highe ... which is then reused in both sub suites. Each sub suite adds their own value ... for a second configuration option. Suite Setup Set model-based options repeat=2 -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater diff --git a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot similarity index 78% rename from atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot index b7289d59..cb71d77f 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot @@ -2,7 +2,8 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based bonus_scenario=${True} Suite Teardown Should be equal ${test_count} ${3} -Library robotmbt processor_lib=suiterepeater +Library ../suiterepeater.py +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot similarity index 78% rename from atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot index f3d6b994..73af861f 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot @@ -2,7 +2,8 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based bonus_scenario=${False} Suite Teardown Should be equal ${test_count} ${2} -Library robotmbt processor_lib=suiterepeater +Library ../suiterepeater.py +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/01__with_bonus_scenario_option.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/01__with_bonus_scenario_option.robot similarity index 78% rename from atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/01__with_bonus_scenario_option.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/01__with_bonus_scenario_option.robot index b7289d59..cb71d77f 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/01__with_bonus_scenario_option.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/01__with_bonus_scenario_option.robot @@ -2,7 +2,8 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based bonus_scenario=${True} Suite Teardown Should be equal ${test_count} ${3} -Library robotmbt processor_lib=suiterepeater +Library ../suiterepeater.py +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/02__without_using_bonus_scenario_option.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/02__without_using_bonus_scenario_option.robot similarity index 77% rename from atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/02__without_using_bonus_scenario_option.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/02__without_using_bonus_scenario_option.robot index 70f501af..f06478b5 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/02__without_using_bonus_scenario_option.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/02__without_using_bonus_scenario_option.robot @@ -2,7 +2,8 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based Suite Teardown Should be equal ${test_count} ${2} -Library robotmbt processor_lib=suiterepeater +Library ../suiterepeater.py +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/__init__.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/__init__.robot similarity index 89% rename from atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/__init__.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/__init__.robot index 882097c5..eb7426d8 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/13__direct_settings_affect_current_suite_only/__init__.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/13__direct_settings_affect_current_suite_only/__init__.robot @@ -5,4 +5,4 @@ Documentation In this suite one of the processor options is set on the highe ... all. The second suite should be unaffected by the option set in the preceeding ... suite. Suite Setup Set model-based options repeat=2 -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater diff --git a/atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/03__direct_setting_overrules_library_setting.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/03__direct_setting_overrules_library_setting.robot similarity index 78% rename from atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/03__direct_setting_overrules_library_setting.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/03__direct_setting_overrules_library_setting.robot index f92713d2..db5ce717 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/03__direct_setting_overrules_library_setting.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/03__direct_setting_overrules_library_setting.robot @@ -2,7 +2,8 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based repeat=3 Suite Teardown Should be equal ${test_count} ${3} -Library robotmbt processor_lib=suiterepeater +Library ../suiterepeater.py +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/04__prior_overrule_does_not_persist.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/04__prior_overrule_does_not_persist.robot similarity index 77% rename from atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/04__prior_overrule_does_not_persist.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/04__prior_overrule_does_not_persist.robot index 70f501af..f06478b5 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/04__prior_overrule_does_not_persist.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/04__prior_overrule_does_not_persist.robot @@ -2,7 +2,8 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0} ... AND Treat this test suite Model-based Suite Teardown Should be equal ${test_count} ${2} -Library robotmbt processor_lib=suiterepeater +Library ../suiterepeater.py +Library robotmbt processor=suiterepeater *** Test Cases *** only test case diff --git a/atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/__init__.robot b/atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/__init__.robot similarity index 88% rename from atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/__init__.robot rename to atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/__init__.robot index 8ae64cf4..a4b01e4a 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/14__overruling_library_setting_affects_current_suite_only/__init__.robot +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/14__overruling_library_setting_affects_current_suite_only/__init__.robot @@ -4,4 +4,4 @@ Documentation In this suite one of the processor options is set on the highe ... setting with their own value, the second library doesn't. The second suite should ... be unaffected by the overruled option from the preceeding suite. Suite Setup Set model-based options repeat=2 -Library robotmbt processor_lib=suiterepeater +Library robotmbt processor=suiterepeater diff --git a/atest/robotMBT tests/07__processor_options/01__option_handling/strictsuiterepeater.py b/atest/robotMBT tests/07__processor_options/01__option_handling/strictsuiterepeater.py new file mode 100644 index 00000000..6344afb7 --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/strictsuiterepeater.py @@ -0,0 +1,15 @@ +from robot.api.deco import library + +from suiterepeater import SuiteRepeater + + +@library(auto_keywords=None, listener=True) +class StrictSuiteRepeater(SuiteRepeater): + """ + Nearly identical to SuiteRepeater as used in other test cases. The difference is that + this variant is strict in its argument handling and will fail if mandatory arguments + are missing or unknown arguments are provided. + """ + + def process_test_suite(self, in_suite, *, repeat, bonus_scenario=False): + return super().process_test_suite(in_suite, repeat=repeat, bonus_scenario=bonus_scenario) diff --git a/atest/robotMBT tests/07__processor_options/option_handling/suiterepeater.py b/atest/robotMBT tests/07__processor_options/01__option_handling/suiterepeater.py similarity index 79% rename from atest/robotMBT tests/07__processor_options/option_handling/suiterepeater.py rename to atest/robotMBT tests/07__processor_options/01__option_handling/suiterepeater.py index bdbfa9fb..ede1eded 100644 --- a/atest/robotMBT tests/07__processor_options/option_handling/suiterepeater.py +++ b/atest/robotMBT tests/07__processor_options/01__option_handling/suiterepeater.py @@ -1,10 +1,11 @@ import copy from robot.api.deco import library +from robotmbt import SuiteProcessor @library(auto_keywords=None, listener=True) -class SuiteRepeater: +class SuiteRepeater(SuiteProcessor): """ Given a test suite, repeats all scenarios 'repeat' times (default=1) Setting bonus_scenario=${True} repeats 1 additional time @@ -12,9 +13,11 @@ class SuiteRepeater: """ def process_test_suite(self, in_suite, repeat=1, **kwargs): + super().process_test_suite(in_suite, **kwargs) n_repeats = int(repeat) if kwargs.get('bonus_scenario', False): n_repeats += 1 + self.scenario_count *= n_repeats out_suite = copy.deepcopy(in_suite) out_suite.scenarios = n_repeats*out_suite.scenarios for i in range(len(out_suite.scenarios)): @@ -22,6 +25,3 @@ def process_test_suite(self, in_suite, repeat=1, **kwargs): if i: out_suite.scenarios[i].name += f" (rep {i+1})" return out_suite - - def mandatory_repeat_argument(self, in_suite, *, repeat, bonus_scenario=False): - return self.process_test_suite(in_suite, repeat=repeat, bonus_scenario=bonus_scenario) diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/__init__.robot b/atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/__init__.robot similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/__init__.robot rename to atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/__init__.robot diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/no_seed.robot b/atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/no_seed.robot similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/no_seed.robot rename to atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/no_seed.robot diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/seed_new.robot b/atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/seed_new.robot similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/seed_new.robot rename to atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/seed_new.robot diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/seed_none.robot b/atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/seed_none.robot similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/seed_none.robot rename to atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/seed_none.robot diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/traces.py b/atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/traces.py similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/traces.py rename to atest/robotMBT tests/07__processor_options/02__random_seeds/01__generating_random_traces/traces.py diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/02__reusing_seed_reproduces_trace.robot b/atest/robotMBT tests/07__processor_options/02__random_seeds/02__reusing_seed_reproduces_trace.robot similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/02__reusing_seed_reproduces_trace.robot rename to atest/robotMBT tests/07__processor_options/02__random_seeds/02__reusing_seed_reproduces_trace.robot diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/03__retrace_with_refinement.robot b/atest/robotMBT tests/07__processor_options/02__random_seeds/03__retrace_with_refinement.robot similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/03__retrace_with_refinement.robot rename to atest/robotMBT tests/07__processor_options/02__random_seeds/03__retrace_with_refinement.robot diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/04__retrace_with_step_modifiers.robot b/atest/robotMBT tests/07__processor_options/02__random_seeds/04__retrace_with_step_modifiers.robot similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/04__retrace_with_step_modifiers.robot rename to atest/robotMBT tests/07__processor_options/02__random_seeds/04__retrace_with_step_modifiers.robot diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/05__retrace_combined.robot b/atest/robotMBT tests/07__processor_options/02__random_seeds/05__retrace_combined.robot similarity index 100% rename from atest/robotMBT tests/07__processor_options/random_seeds/05__retrace_combined.robot rename to atest/robotMBT tests/07__processor_options/02__random_seeds/05__retrace_combined.robot diff --git a/atest/robotMBT tests/07__processor_options/03__stop_conditions/01__stop_at_single_coverage_by_default.robot b/atest/robotMBT tests/07__processor_options/03__stop_conditions/01__stop_at_single_coverage_by_default.robot new file mode 100644 index 00000000..cbdd4b08 --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/03__stop_conditions/01__stop_at_single_coverage_by_default.robot @@ -0,0 +1,26 @@ +*** Settings *** +Documentation At single coverage (the default), the final suite should repeat the middle +... exactly once, then insert the last scenario and stop. +Suite Setup Treat this test suite Model-based +Suite Teardown Should be equal ${scenario_count} ${4} +Test Teardown Set suite variable ${scenario_count} ${scenario_count+1} +Resource ../../../resources/birthday_cards_flat.resource +Library robotmbt + +*** variables *** +${scenario_count} ${0} + +*** Test Cases *** +Buying a card + When someone buys a birthday card + then there is a blank birthday card available + +Someone writes their name on the card + Given there is a birthday card + when Someone writes their name on the birthday card + then the birthday card has 'Someone' written on it + +At least 3 people can write their name on the card + Given the birthday card has 2 names written on it + when someone writes their name on the birthday card + then the birthday card has 3 names written on it diff --git a/atest/robotMBT tests/07__processor_options/03__stop_conditions/02__stop_beyond_single_coverage.robot b/atest/robotMBT tests/07__processor_options/03__stop_conditions/02__stop_beyond_single_coverage.robot new file mode 100644 index 00000000..92590080 --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/03__stop_conditions/02__stop_beyond_single_coverage.robot @@ -0,0 +1,26 @@ +*** Settings *** +Documentation Create a test trace that extends beyond single coverage by using a scenario +... target that is one higher than needed to reach single coverage. +Suite Setup Treat this test suite Model-based scenario_target=5 +Suite Teardown Should be equal ${scenario_count} ${5} +Test Teardown Set suite variable ${scenario_count} ${scenario_count+1} +Resource ../../../resources/birthday_cards_flat.resource +Library robotmbt + +*** variables *** +${scenario_count} ${0} + +*** Test Cases *** +Buying a card + When someone buys a birthday card + then there is a blank birthday card available + +Someone writes their name on the card + Given there is a birthday card + when Someone writes their name on the birthday card + then the birthday card has 'Someone' written on it + +At least 3 people can write their name on the card + Given the birthday card has 2 names written on it + when someone writes their name on the birthday card + then the birthday card has 3 names written on it diff --git a/atest/robotMBT tests/07__processor_options/03__stop_conditions/03__stop_before_single_coverage.robot b/atest/robotMBT tests/07__processor_options/03__stop_conditions/03__stop_before_single_coverage.robot new file mode 100644 index 00000000..4c300326 --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/03__stop_conditions/03__stop_before_single_coverage.robot @@ -0,0 +1,26 @@ +*** Settings *** +Documentation Create a test trace that stops before single coverage is reached by using a +... scenario target that is lower than what is needed to reach single coverage. +Suite Setup Treat this test suite Model-based coverage_target=0 scenario_target=2 +Suite Teardown Should be equal ${scenario_count} ${2} +Test Teardown Set suite variable ${scenario_count} ${scenario_count+1} +Resource ../../../resources/birthday_cards_flat.resource +Library robotmbt + +*** variables *** +${scenario_count} ${0} + +*** Test Cases *** +Buying a card + When someone buys a birthday card + then there is a blank birthday card available + +Someone writes their name on the card + Given there is a birthday card + when Someone writes their name on the birthday card + then the birthday card has 'Someone' written on it + +At least 3 people can write their name on the card + Given the birthday card has 2 names written on it + when someone writes their name on the birthday card + then the birthday card has 3 names written on it diff --git a/atest/robotMBT tests/07__processor_options/03__stop_conditions/04__refinement_must_complete_to_reach_coverage_target.robot b/atest/robotMBT tests/07__processor_options/03__stop_conditions/04__refinement_must_complete_to_reach_coverage_target.robot new file mode 100644 index 00000000..5384b607 --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/03__stop_conditions/04__refinement_must_complete_to_reach_coverage_target.robot @@ -0,0 +1,41 @@ +*** Settings *** +Documentation Due to refinement, the high-level scenario is split up to insert the low-level +... scenario. Coverage is not completed until the final part of the split-up +... scenario is executed. +Suite Setup Treat this test suite Model-based coverage_target=1 +Suite Teardown Final checks +Test Teardown Set suite variable ${scenario_count} ${scenario_count+1} +Resource ../../../resources/birthday_cards_composed.resource +Library robotmbt + + +*** variables *** +${scenario_count} ${0} +${high_level_started} ${0} +${high_level_completed} ${0} + + +*** Test Cases *** +Buying a card + When someone buys a birthday card + then there is a blank birthday card available + +high-level scenario + Set suite variable ${high_level_started} ${high_level_started+1} + Given there is a birthday card + when Someone writes their name on the birthday card + then the birthday card has 'Someone' written on it + Set suite variable ${high_level_completed} ${high_level_completed+1} + +low-level scenario + Given there is a birthday card + when Someone writes their name in pen on the birthday card + then the birthday card has 'Someone' written on it + and there is text added in ink on the birthday card + + +*** Keywords *** +Final checks + Should be equal ${scenario_count} ${3} + Should be equal ${high_level_started} ${1} + Should be equal ${high_level_completed} ${1} diff --git a/atest/robotMBT tests/07__processor_options/03__stop_conditions/05__refinement_can_stay_incomplete_once_coverage_reached.robot b/atest/robotMBT tests/07__processor_options/03__stop_conditions/05__refinement_can_stay_incomplete_once_coverage_reached.robot new file mode 100644 index 00000000..3b478b54 --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/03__stop_conditions/05__refinement_can_stay_incomplete_once_coverage_reached.robot @@ -0,0 +1,50 @@ +*** Settings *** +Documentation Due to refinement, the high-level scenario is split up to insert a low-level +... scenario. Because there are two low-level scenarios, the high-level scenario +... must be repeated. Coverage for the high-level scenario is reached as a soon +... as the first one completes. Therefore, full coverage is reached as soon as the +... second lower-level scenario completes, causing the trace to end, even though +... that high-level scenario did not complete yet. +Suite Setup Treat this test suite Model-based coverage_target=1 +Suite Teardown Final checks +Test Teardown Set suite variable ${scenario_count} ${scenario_count+1} +Resource ../../../resources/birthday_cards_composed.resource +Library robotmbt + + +*** variables *** +${scenario_count} ${0} +${high_level_started} ${0} +${high_level_completed} ${0} + + +*** Test Cases *** +Buying a card + When someone buys a birthday card + then there is a blank birthday card available + +high-level scenario + Set suite variable ${high_level_started} ${high_level_started+1} + Given there is a birthday card + when Someone writes their name on the birthday card + then the birthday card has 'Someone' written on it + Set suite variable ${high_level_completed} ${high_level_completed+1} + +low-level scenario A + Given there is a birthday card + when Someone writes their name in pen on the birthday card + then the birthday card has 'Someone' written on it + and there is text added in ink on the birthday card + +low-level scenario B + Given there is a birthday card + when Someone writes their name in pen on the birthday card + then the birthday card has 'Someone' written on it + and there is text added in ink on the birthday card + + +*** Keywords *** +Final checks + Should be equal ${scenario_count} ${4} + Should be equal ${high_level_started} ${2} + Should be equal ${high_level_completed} ${1} diff --git a/atest/robotMBT tests/07__processor_options/03__stop_conditions/06__time_target_can_stop_test_run.robot b/atest/robotMBT tests/07__processor_options/03__stop_conditions/06__time_target_can_stop_test_run.robot new file mode 100644 index 00000000..8b2b1062 --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/03__stop_conditions/06__time_target_can_stop_test_run.robot @@ -0,0 +1,33 @@ +*** Settings *** +Documentation This test suite confirms that a time target can stop a test run if it is the +... last condition to be satisfied. For run duration reasons an unrealistically +... short time span is used, while all other conditions are disabled. This has a +... double effect. One effect is that the test run should always stop after the +... first test case, even though a longer trace was created. We know that a longer +... trace was created due to the second effect. With the coverage target disabled +... the coverage drought limit does not kick in, so it is also the time target that +... is responsible for stopping the trace generation without getting stuck in an +... infinite loop. +Suite Setup Treat this test suite Model-based coverage_target=0 time_target=0.1 sec +Suite Teardown Should be equal ${scenario_count} ${1} +Test Teardown Set suite variable ${scenario_count} ${scenario_count+1} +Resource ../../../resources/birthday_cards_flat.resource +Library robotmbt + +*** variables *** +${scenario_count} ${0} + +*** Test Cases *** +Buying a card + When someone buys a birthday card + then there is a blank birthday card available + +Someone writes their name on the card + Given there is a birthday card + when Someone writes their name on the birthday card + then the birthday card has 'Someone' written on it + +At least 3 people can write their name on the card + Given the birthday card has 2 names written on it + when someone writes their name on the birthday card + then the birthday card has 3 names written on it diff --git a/atest/robotMBT tests/07__processor_options/04__batch_size/tag_listener.py b/atest/robotMBT tests/07__processor_options/04__batch_size/tag_listener.py new file mode 100644 index 00000000..373b3f99 --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/04__batch_size/tag_listener.py @@ -0,0 +1,29 @@ +from robot.api import logger +from robot.api.deco import library +from robot.libraries.BuiltIn import BuiltIn + + +@library(scope='SUITE', listener='SELF') +class TagListener: + ROBOT_LISTENER_PRIORITY = -1 # Set lower priority to make sure tags are already set by robotmbt + TRACE_TAG = 'mbt trace extension' + + def __init__(self): + self.test_count = 0 + + def end_test(self, tc, result): + self.test_count += 1 + if self.test_count % 2: + if self.TRACE_TAG in result.tags: + result.status = 'FAIL' + result.message = f"Unexpected test tag '{self.TRACE_TAG}'" + else: + if self.TRACE_TAG not in result.tags: + result.status = 'FAIL' + result.message = f"Test tag '{self.TRACE_TAG}' missing" + + if 'my tag' not in result.tags: + result.status = 'FAIL' + result.message = "Test tag 'my tag' missing" + BuiltIn().set_suite_variable('${confirmed_passes}', BuiltIn().get_variable_value('${confirmed_passes}') + 1) + logger.info("PASS confirmed by listener") diff --git a/atest/robotMBT tests/07__processor_options/04__batch_size/trace_extension_is_tagged.robot b/atest/robotMBT tests/07__processor_options/04__batch_size/trace_extension_is_tagged.robot new file mode 100644 index 00000000..8b7a1ebf --- /dev/null +++ b/atest/robotMBT tests/07__processor_options/04__batch_size/trace_extension_is_tagged.robot @@ -0,0 +1,30 @@ +*** Settings *** +Documentation This test suite checks that Batch size for trace generation is respected and that +... the (logging for) trace generation can be found, even when it is not part of the +... initial 'Treat this test suite model-based' keyword. To find the delayed parts of +... trace generation, the scenarios that trigger trace extension are tagged. Since +... tagging is not done until `end_test`, a listener is used to check and confirm +... that the expected scenarios are tagged. +... +... Note that tagging is not the preferred solution, but alternatives failed due to +... Robot Framework's scoping limitations. +Suite Setup Treat this test suite Model-based batch_size=2 +Suite Teardown Should Be Equal ${confirmed_passes} ${3} +Test Tags my tag +Library robotmbt +Library tag_listener.py + + +*** Variables *** +${confirmed_passes} ${0} + + +*** Test Cases *** +Scenario 1 + No Operation + +Scenario 2 + No Operation + +Scenario 3 + No Operation diff --git a/atest/robotMBT tests/07__processor_options/option_handling/11__argument_restrictions.robot b/atest/robotMBT tests/07__processor_options/option_handling/11__argument_restrictions.robot deleted file mode 100644 index 3da3a8f5..00000000 --- a/atest/robotMBT tests/07__processor_options/option_handling/11__argument_restrictions.robot +++ /dev/null @@ -1,12 +0,0 @@ -*** Settings *** -Library suiterepeater.py -Library robotmbt processor_lib=suiterepeater processor=mandatory_repeat_argument - -*** Test Cases *** -arguments can be mandatory - Run keyword and expect error *SuiteRepeater.mandatory_repeat_argument() missing 1 required keyword-only argument: 'repeat' - ... Treat this test suite Model-based bonus_scenario=${True} - -can fail on unknown arguments - Run keyword and expect error *SuiteRepeater.mandatory_repeat_argument() got an unexpected keyword argument 'intentional_fail' - ... Treat this test suite Model-based repeat=1 bonus_scenario=${True} intentional_fail=${True} diff --git a/pyproject.toml b/pyproject.toml index d742728c..f8fd9fa7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta" [project] name = "robotframework-mbt" -version = "0.13.0" +version = "0.14.0" description = "Model-Based Testing in Robot framework with test case generation" readme = "README.md" authors = [{ name = "Johan Foederer", email = "github@famfoe.nl" }] diff --git a/robotmbt/__init__.py b/robotmbt/__init__.py index bcdfcc7b..236550ea 100644 --- a/robotmbt/__init__.py +++ b/robotmbt/__init__.py @@ -32,6 +32,7 @@ from .version import VERSION from .suitereplacer import SuiteReplacer +from .suiteprocessors import SuiteProcessor class robotmbt(SuiteReplacer): diff --git a/robotmbt/modeller.py b/robotmbt/modeller.py index 0d742005..901c4ca4 100644 --- a/robotmbt/modeller.py +++ b/robotmbt/modeller.py @@ -146,7 +146,7 @@ def handle_refinement_exit(inserted_refinement: Scenario, tracestate: TraceState tail_inserted, remainder, extra_data = process_scenario(refinement_tail, model) if not tail_inserted: logger.debug(extra_data['fail_msg']) - # Confirm then rewind, to roll back complete scenario, including its refiements + # Confirm then rewind, to roll back complete scenario, including its refinements # Because that exit check passed, this is an error in the refined scenario itself tracestate.confirm_full_scenario(refinement_tail.src_id, refinement_tail, model) tail = rewind(tracestate) @@ -256,10 +256,11 @@ def _parse_modifier_expression(expression: str, args: StepArguments) -> tuple[st def rewind(tracestate: TraceState, drought_recovery: bool = False) -> TraceSnapShot | None: - if tracestate[-1].remainder and tracestate.highest_part(tracestate[-1].remainder.src_id) > 1: - # When rewinding an 'in between' part, rewind both the part and the refinement - tracestate.rewind() tail = tracestate.rewind() while drought_recovery and tracestate.coverage_drought: + if not tracestate.can_rewind(): + logger.debug( + f"Coverage drought recovery stalled. {tracestate.coverage_drought} Scenarios are already committed.") + break tail = tracestate.rewind() return tail diff --git a/robotmbt/suitedata.py b/robotmbt/suitedata.py index 913f0f3b..359b9fb2 100644 --- a/robotmbt/suitedata.py +++ b/robotmbt/suitedata.py @@ -69,6 +69,9 @@ def steps_with_errors(self): + [e for s in map(Scenario.steps_with_errors, self.scenarios) for e in s] + ([self.teardown] if self.teardown and self.teardown.has_error() else [])) + def scenario_count(self): + return len(self.scenarios) + sum([s.scenario_count() for s in self.suites]) + class Scenario: def __init__(self, name: str, parent: Suite, og_tc): diff --git a/robotmbt/suiteprocessors.py b/robotmbt/suiteprocessors.py index 8e06c857..37731dc1 100644 --- a/robotmbt/suiteprocessors.py +++ b/robotmbt/suiteprocessors.py @@ -32,9 +32,11 @@ import copy import random +import time from robot.api import logger from robot.errors import TimeoutExceeded +from robot.utils import timestr_to_secs from . import modeller from .modelspace import ModelSpace @@ -47,17 +49,95 @@ Visualiser = None -class SuiteProcessors: - @staticmethod - def echo(in_suite: Suite) -> Suite: +class SuiteProcessor: + def __init__(self): + self.scenario_count: int = 0 + self.commit_count: int = 0 + self.coverage_target: int = 1 + self.scenario_target: int = 0 + self.time_target: float = 0 + + def process_test_suite(self, in_suite: Suite, + **kwargs) -> Suite: + self._handle_target_options(**kwargs) + self.scenario_count = in_suite.scenario_count() + # Counts the scenarios committed by the runner. I.e. the scenarios that are scheduled for execution + # and cannot be touched anymore + self.commit_count = 0 + return Suite('not implemented') + + def next_scenario_request(self): + """Indicates the wish for (at least) one more scenario and triggers trace genaration when needed.""" + # This basic implementation assumes that the complete target test suite is returned directly + # by process_test_suite() in an overridden method. No further generation is triggered. + if self.scenario_count >= self.commit_count + 1: + self.commit_count += 1 + + @property + def scenarios_committed(self) -> int: + """ + Each time next_scenario_request() is called, you commit to one more scenario, if there is at + least one more scenario available. This can be an already pending scenario, or new trace + generation can be triggered to generate a next scenario (or batch). Committed scenarios are + assumed to have been executed when determining the achieved targets. + """ + return self.commit_count + + @property + def scenarios_pending(self) -> int: + """ + The number of scenarios that are waiting in the buffer for when more scenarios are requested. + + For practical reasons, trace generation can run ahead of the actual test execution, creating + a buffer of pending scenarios ahead of time. + + Note that pending scenarios are still subject to change. Only committed scenarios are frozen + in place. + """ + return self.scenario_count - self.commit_count + + def are_all_targets_reached(self, committed_only: bool = True) -> bool: + if not committed_only: + return True + if self.coverage_target and self.commit_count < self.scenario_count: + return False + if self.scenario_target and self.commit_count < self.scenario_target: + return False + if self.time_target and time.time() < self.time_target: + return False + return True + + def _handle_target_options(self, + coverage_target: str | int | None = 1, + scenario_target: str | int = 0, + time_target: str | None = None, + **kwargs): + self.coverage_target = 0 if coverage_target is None else int(coverage_target) + if self.coverage_target not in [0, 1]: + logger.warn(f"Unsupported coverage target request '{coverage_target}'. Using default coverage target of 1") + self.coverage_target = 1 + self.scenario_target = int(scenario_target) + self.time_target = (time.time() + timestr_to_secs(time_target)) if time_target else 0 + + +class Echo(SuiteProcessor): + def process_test_suite(self, in_suite: Suite, **kwargs) -> Suite: + super().process_test_suite(in_suite, **kwargs) return in_suite - def flatten(self, in_suite: Suite) -> Suite: + +class Flatten(SuiteProcessor): + def process_test_suite(self, in_suite: Suite, **kwargs) -> Suite: """ Takes a Suite as input and returns a Suite as output. The output Suite does not have any sub-suites, only scenarios. The scenarios do not have a setup. Any setup keywords are inserted at the front of the scenario as regular steps. """ + super().process_test_suite(in_suite, **kwargs) + return self.flatten(in_suite) + + @staticmethod + def flatten(in_suite: Suite) -> Suite: out_suite = copy.deepcopy(in_suite) outer_scenarios = out_suite.scenarios for scenario in outer_scenarios: @@ -69,7 +149,7 @@ def flatten(self, in_suite: Suite) -> Suite: scenario.teardown = None out_suite.scenarios = [] for suite in in_suite.suites: - subsuite = self.flatten(suite) + subsuite = Flatten.flatten(suite) for scenario in subsuite.scenarios: if subsuite.setup: scenario.steps.insert(0, subsuite.setup) @@ -80,44 +160,89 @@ def flatten(self, in_suite: Suite) -> Suite: out_suite.suites = [] return out_suite + +class ModelBased(SuiteProcessor): def process_test_suite(self, in_suite: Suite, *, seed: str | int | bytes | bytearray = 'new', - graph: str = '', export_graph_data: str = '') -> Suite: + batch_size: str | int = 100, + graph: str = '', export_graph_data: str = '', **kwargs) -> Suite: + # handle options + super().process_test_suite(in_suite, **kwargs) + self.batch_size = int(batch_size) + self._init_randomiser(seed) + self._visualiser = self._init_visualiser(in_suite.name) if graph or export_graph_data else None + self.out_suite = Suite(in_suite.name) self.out_suite.filename = in_suite.filename self.out_suite.parent = in_suite.parent self._fail_on_step_errors(in_suite) - self.flat_suite = self.flatten(in_suite) + self.flat_suite = Flatten.flatten(in_suite) for id, scenario in enumerate(self.flat_suite.scenarios, start=1): scenario.src_id = id self.scenarios: list[Scenario] = self.flat_suite.scenarios[:] logger.debug("Use these numbers to reference scenarios from traces\n\t" + "\n\t".join([f"{s.src_id}: {s.name}" for s in self.scenarios])) - self._init_randomiser(seed) - self._visualiser = self._init_visualiser(in_suite.name) if graph or export_graph_data else None try: # a short trace without the need for repeating scenarios is preferred - tracestate = self._search_direct_trace() - if not tracestate.coverage_reached(): - logger.debug("Direct trace not discovered. Now exploring with loops, allowing repetition of scenarios.") - tracestate = self._try_to_reach_full_coverage(allow_duplicate_scenarios=True, randomise=True, - unreached_scenarios=tracestate.unreached) - else: + direct_tracestate = self._search_direct_trace() + if self._discovery_ready(direct_tracestate): + self.tracestate = direct_tracestate + n = len(direct_tracestate.covered_ids) + logger.debug(f"Using one of the discovered traces ({n} scenario{'s' if n != 1 else ''})") + self._report_tracestate_to_user(direct_tracestate) # The visualiser assumes that the last trace is the final selected trace, which is not always - # the case. Re-adding the selected trace to prevent the wrong path from being highlighted. - self._update_visualisation(TraceState(tracestate.prio_order)) - self._update_visualisation(tracestate) + # the case. Re-initialising and then adding the selected trace again prevents the wrong path + # from being highlighted. + self._update_visualisation(TraceState(direct_tracestate.prio_order)) + self._update_visualisation(self.tracestate) + else: + self.tracestate = TraceState([s.src_id for s in self.scenarios]) + self.tracestate.unreached = direct_tracestate.unreached + logger.debug("Direct trace not discovered. Now exploring with loops, allowing repetition of scenarios.") + self._generate_next_batch(self.batch_size) finally: # Draw the graph even when a timeout or user interrupt occurs if graph: self._write_visualisation(graph) if export_graph_data: self._export_graph_data(export_graph_data) - if not tracestate.coverage_reached(): + if len(self.tracestate) == 0: raise Exception("Unable to compose a consistent suite") - self._report_tracestate_wrapup(tracestate) - self.out_suite.scenarios = tracestate.get_trace() + self._report_tracestate_wrapup() return self.out_suite + def next_scenario_request(self): + if len(self.tracestate) <= self.out_suite.scenario_count(): + self._generate_next_batch(self.batch_size) + if len(self.tracestate) > self.out_suite.scenario_count(): + self.out_suite.scenarios.append(self.tracestate[self.out_suite.scenario_count()].scenario) + self.commit_count += 1 + self.tracestate.rewind_limit += 1 + + @property + def scenarios_committed(self) -> int: + return self.out_suite.scenario_count() + + @property + def scenarios_pending(self) -> int: + return len(self.tracestate) - self.out_suite.scenario_count() + + def are_all_targets_reached(self, tracestate: TraceState | None = None, committed_only: bool = True) -> bool: + if tracestate is None: + tracestate = self.tracestate + if self.time_target and time.time() < self.time_target: + return False + if committed_only: + if self.coverage_target and not tracestate[self.commit_count-1].coverage_reached: + return False + if self.scenario_target and self.commit_count < self.scenario_target: + return False + else: + if self.coverage_target and not tracestate.coverage_reached(): + return False + if self.scenario_target and len(tracestate) < self.scenario_target: + return False + return True + def draw_graph_from_export_file(self, file_path: str, graph_style: str): self._visualiser = self._init_visualiser() if self._visualiser: @@ -156,23 +281,26 @@ def _search_direct_trace(self) -> TraceState: if self._is_duplicate_prio_order(tracestates, prio_order): continue tracestates.append(self._one_shot_trace(prio_order)) - if tracestates[-1].coverage_reached() and not self._visualiser: + if self._discovery_ready(tracestates[-1]) and not self._visualiser: + tracestates[-1].unreached = self._unreached_scenarios(tracestates) return tracestates[-1] suggestion = self._create_suggestion_by_experience(tracestates) if self._is_duplicate_prio_order(tracestates, suggestion): continue tracestates.append(self._one_shot_trace(suggestion)) - if tracestates[-1].coverage_reached() and not self._visualiser: + if self._discovery_ready(tracestates[-1]) and not self._visualiser: + tracestates[-1].unreached = self._unreached_scenarios(tracestates) return tracestates[-1] index_longest = self._longest_trace(tracestates) - if tracestates[index_longest].coverage_reached(): + if self._discovery_ready(tracestates[index_longest]): + tracestates[index_longest].unreached = self._unreached_scenarios(tracestates) return tracestates[index_longest] logger.debug("Trying to extend most promising traces") prio_order = self._create_suggestion_by_experience(tracestates, index_longest) if not self._is_duplicate_prio_order(tracestates, prio_order): tracestates.append(self._one_shot_trace(prio_order)) - if tracestates[-1].coverage_reached(): + if self._discovery_ready(tracestates[-1]): return tracestates[-1] last_new = self._last_new_coverage(tracestates) while True: # while still discovering new coverage @@ -180,7 +308,7 @@ def _search_direct_trace(self) -> TraceState: if self._is_duplicate_prio_order(tracestates, prio_order): break tracestates.append(self._one_shot_trace(prio_order)) - if tracestates[-1].coverage_reached(): + if self._discovery_ready(tracestates[-1]): return tracestates[-1] last_new = self._last_new_coverage(tracestates) if last_new != len(tracestates)-1: @@ -188,7 +316,7 @@ def _search_direct_trace(self) -> TraceState: longest = tracestates[self._longest_trace(tracestates)] not_in_trace = sorted(longest.not_in_trace) - longest.unreached = sorted(self._unreached_scenarios(tracestates)) + longest.unreached = self._unreached_scenarios(tracestates) logger.debug( f"Longest trace so far ({len(longest.covered_ids)} scenario{'s' if len(longest.covered_ids) != 1 else ''})" f": [{', '.join(longest.id_trace)}]\n" @@ -197,6 +325,9 @@ def _search_direct_trace(self) -> TraceState: "(Scenarios marked with * are not part of any trace)\n\n") return longest + def _discovery_ready(self, tracestate): + return self.are_all_targets_reached(tracestate, committed_only=False) or len(tracestate) >= self.batch_size + def _longest_trace(self, tracestate_list: list[TraceState]) -> int: """returns the index of the trace that covers the most scenarios""" lengths = [len(ts.covered_ids) for ts in tracestate_list] @@ -236,7 +367,7 @@ def _one_shot_trace(self, scenarios: list[int]) -> TraceState: tracestate = TraceState(scenarios) self._update_visualisation(tracestate) candidate_id = tracestate.next_candidate(retry=False, randomise=False) - while candidate_id is not None: + while candidate_id is not None and not self._discovery_ready(tracestate): candidate = self._select_scenario_variant(candidate_id, tracestate) if candidate: # No valid variant available in the current state modeller.try_to_fit_in_scenario(candidate, tracestate) @@ -263,15 +394,14 @@ def _create_suggestion_by_experience(self, tracestate_list, target_index=-1) -> not_in_target = [id for id in tracestate_list[target_index].not_in_trace if id not in never_reached] return never_reached + not_in_target + tracestate_list[target_index].covered_ids - def _try_to_reach_full_coverage(self, allow_duplicate_scenarios: bool, randomise: bool = False, - unreached_scenarios: list[int] = None) -> TraceState: - tracestate = TraceState([s.src_id for s in self.scenarios]) - if unreached_scenarios: - tracestate.unreached = unreached_scenarios + def _generate_next_batch(self, batchsize): + tracestate = self.tracestate + old_len = len(tracestate) self._update_visualisation(tracestate) - while not tracestate.coverage_reached(): - candidate_id = tracestate.next_candidate(retry=allow_duplicate_scenarios, randomise=randomise) - if candidate_id is None: # No more candidates remaining for this level + while len(tracestate) < old_len + batchsize and not self.are_all_targets_reached(committed_only=False): + candidate_id = tracestate.next_candidate(retry=True, randomise=True) + if candidate_id is None: + logger.debug("No more candidates remaining at this position.") if not tracestate.can_rewind(): break tail = modeller.rewind(tracestate) @@ -295,7 +425,7 @@ def _try_to_reach_full_coverage(self, allow_duplicate_scenarios: bool, randomise if self.__last_candidate_changed_nothing(tracestate): logger.debug("Repeated scenario did not change the model's state. Stop trying.") modeller.rewind(tracestate) - elif tracestate.coverage_drought > self.DROUGHT_LIMIT: + elif self.coverage_target and not self.tracestate.coverage_reached() and tracestate.coverage_drought > self.DROUGHT_LIMIT: logger.debug(f"Went too long without new coverage (>{self.DROUGHT_LIMIT}x). " "Roll back to last coverage increase and try something else.") modeller.rewind(tracestate, drought_recovery=True) @@ -303,7 +433,6 @@ def _try_to_reach_full_coverage(self, allow_duplicate_scenarios: bool, randomise logger.debug(f"last state:\n{tracestate.model.get_status_text()}") self._update_visualisation(tracestate) self._update_visualisation(tracestate) - return tracestate @staticmethod def __last_candidate_changed_nothing(tracestate: TraceState) -> bool: @@ -342,14 +471,17 @@ def _fail_on_step_errors(suite: Suite): @staticmethod def _report_tracestate_to_user(tracestate: TraceState): - pending = ', '.join([str(i) + '*' if i in tracestate.unreached else str(i) for i in tracestate.not_in_trace]) + pending = ', '.join([str(i) + '*' if i in tracestate.unreached else str(i) + for i in sorted(tracestate.not_in_trace)]) logger.debug(f"Trace: [{', '.join(tracestate.id_trace)}] Pending: [{pending}]" f"{' Rejected: ' + str(tracestate.tried) if tracestate.tried else ''}") - @staticmethod - def _report_tracestate_wrapup(tracestate: TraceState): - logger.info("Trace composed:") - for progression in tracestate: + def _report_tracestate_wrapup(self): + if self.are_all_targets_reached(committed_only=False): + logger.info("Trace composed:") + else: + logger.info("First part of trace composed: (Check scenarios tagged with `mbt trace extension` for continued generation)") + for progression in self.tracestate: logger.info(progression.scenario.name) logger.debug(f"model\n{progression.model.get_status_text()}\n") @@ -363,7 +495,7 @@ def _init_randomiser(seed: str | int | bytes | bytearray | None): "Using system's random seed for trace generation. This trace cannot be rerun. Use `seed=new` to generate a reusable seed.") elif str(seed).lower() == 'new': random.seed() - new_seed = SuiteProcessors._generate_seed() + new_seed = ModelBased._generate_seed() logger.info(f"seed={new_seed} (use seed to rerun this trace)") random.seed(new_seed) else: diff --git a/robotmbt/suitereplacer.py b/robotmbt/suitereplacer.py index bc55e95a..843696d9 100644 --- a/robotmbt/suitereplacer.py +++ b/robotmbt/suitereplacer.py @@ -30,49 +30,43 @@ # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -from collections.abc import Callable, Iterator +from collections.abc import Iterator from typing import Any import robot.model +import robot.result import robot.running.model as rmodel from robot.api import logger from robot.api.deco import library, keyword from robot.libraries.BuiltIn import BuiltIn from .suitedata import Suite, Scenario, Step -from .suiteprocessors import SuiteProcessors +from .suiteprocessors import SuiteProcessor, ModelBased, Echo, Flatten Robot = BuiltIn() @library(scope="GLOBAL", listener='SELF') class SuiteReplacer: - def __init__(self, processor: str = 'process_test_suite', processor_lib: str | None = None): + def __init__(self, processor: str = 'robotmbt'): self.current_suite: robot.model.TestSuite | None = None self.mbt_anchor_suite: robot.model.TestSuite | None = None - self.processor_lib_name: str | None = processor_lib self.processor_name: str = processor - self._processor_lib: SuiteProcessors | None | object = None - self._processor_method: Callable[..., Suite] | None = None + self.processor: SuiteProcessor | None = None self.suite_gen: list[Iterator[Suite]] = [] # Generator for on-the-fly suite insertion self.test_case_gen: list[Iterator[Scenario]] = [] # Generator for on-the-fly test case insertion self.processor_options: dict[str, Any] = {} - @property - def processor_lib(self) -> SuiteProcessors: - if self._processor_lib is None: - self._processor_lib = SuiteProcessors() if self.processor_lib_name is None \ - else Robot.get_library_instance(self.processor_lib_name) - return self._processor_lib - - @property - def processor_method(self): - if self._processor_method is None: - if not hasattr(self.processor_lib, self.processor_name): - Robot.fail( - f"Processor '{self.processor_name}' not available for model-based processor library {self.processor_lib_name}") - self._processor_method = getattr(self._processor_lib, self.processor_name) - return self._processor_method + def load_processor(self): + if self.processor_name.lower() == 'robotmbt': + self.processor = ModelBased() + elif self.processor_name.lower() == 'echo': + self.processor = Echo() + elif self.processor_name.lower() == 'flatten': + self.processor = Flatten() + else: + self.processor = Robot.get_library_instance(self.processor_name) + return self.processor @keyword(name="Treat this test suite Model-based") def treat_model_based(self, **kwargs): @@ -93,12 +87,14 @@ def treat_model_based(self, **kwargs): local_settings = self.processor_options.copy() local_settings.update(kwargs) master_suite = self.__process_robot_suite(self.current_suite, parent=None) - modelbased_suite = self.processor_method(master_suite, **local_settings) + self.load_processor() + modelbased_suite = self.processor.process_test_suite(master_suite, **local_settings) self.suite_gen = [iter(modelbased_suite.suites)] self.test_case_gen = [iter(modelbased_suite.scenarios)] self.__clearTestSuite(self.current_suite) - self.add_next_new(self.current_suite) # add first test case only. Others are added at runtime by listeners. self.mbt_anchor_suite = self.current_suite + self.processor.next_scenario_request() + self.add_next_new(self.mbt_anchor_suite) @keyword("Set model-based options") def set_model_based_options(self, **kwargs): @@ -125,7 +121,7 @@ def show_graph(self, json_file_path: str, graph_style: str = 'scenario'): different graph style than was used during the test run. If no graph style is selected, then the scenario graph style is used. """ - SuiteProcessors().draw_graph_from_export_file(json_file_path, graph_style) + ModelBased().draw_graph_from_export_file(json_file_path, graph_style) def __process_robot_suite(self, in_suite: robot.model.TestSuite, parent: Suite | None) -> Suite: out_suite = Suite(in_suite.name, parent) @@ -222,10 +218,10 @@ def add_test(tc: Scenario, target_suite: robot.model.TestSuite): new_tc.body.create_keyword(name=step.keyword, assign=step.assign, args=step.posnom_args_str) target_suite.tests.append(new_tc) - def _start_suite(self, suite: robot.model.TestSuite, result): + def _start_suite(self, suite: rmodel.TestSuite, result: robot.result.model.TestSuite): self.current_suite = suite - def _end_suite(self, suite: robot.model.TestSuite, result): + def _end_suite(self, suite: rmodel.TestSuite, result: robot.result.model.TestSuite): if suite == self.mbt_anchor_suite: self.mbt_anchor_suite = None if not self.mbt_anchor_suite: @@ -235,9 +231,35 @@ def _end_suite(self, suite: robot.model.TestSuite, result): self.current_suite = self.current_suite.parent self.add_next_new(self.current_suite) - def _end_test(self, test_case: robot.model.TestCase, result): + def _end_test(self, test_case: rmodel.TestCase, result: robot.result.model.TestCase): if not self.mbt_anchor_suite: return + if not isinstance(self.processor, SuiteProcessor): + raise TypeError("processor must be of type SuiteProcessor") + if self.processor.are_all_targets_reached(): + logger.info(f"{self.processor.scenarios_committed} Scenarios completed for model. All targets achieved.") + return + + committed_old = self.processor.scenarios_committed + pending_old = self.processor.scenarios_pending + if not pending_old: + logger.info(f"{committed_old} Scenario{'s' if committed_old != 1 else ''} completed. Looking to extend trace.") + self.processor.next_scenario_request() + committed = self.processor.scenarios_committed + pending = self.processor.scenarios_pending + new_total = committed + pending + old_total = committed_old + pending_old + if not pending_old and new_total == old_total: + logger.info(f"Trace could not be extended.") + if not self.processor.are_all_targets_reached(): + new_tc = self.current_suite.tests.create(name='Confirm exit criteria') + new_tc.body.create_keyword(name='Fail', args=('Not all targets achieved',)) + self.mbt_anchor_suite = None + return + + if new_total > old_total: + result.tags.add('mbt trace extension') + logger.info(f"MBT trace generation prepared {new_total-old_total} new scenarios.") try: self.add_test(next(self.test_case_gen[-1]), self.current_suite) except StopIteration: diff --git a/robotmbt/tracestate.py b/robotmbt/tracestate.py index 77dc639e..970a5336 100644 --- a/robotmbt/tracestate.py +++ b/robotmbt/tracestate.py @@ -38,11 +38,12 @@ class TraceSnapShot: def __init__(self, id: str, inserted_scenario: Scenario, model_state: ModelSpace, - remainder: Scenario | None = None, drought: int = 0): + remainder: Scenario | None = None, coverage: int = 0, drought: int = 0): self.id: str = id self.scenario: Scenario = inserted_scenario self.remainder: Scenario | None = remainder self._model: ModelSpace = model_state.copy() + self.coverage_reached: int = coverage self.coverage_drought: int = drought @property @@ -59,6 +60,9 @@ def __init__(self, scenario_indexes: list[int]): self._tried: list[list[int]] = [[]] # Keeps track of the scenarios already tried at each step in the trace self._snapshots: list[TraceSnapShot] = [] # Keeps details for elements in trace self._open_refinements: list[int] = [] + # The rewind limit indicates a (soft) limit for scenarios that should not be rewound. E.g. because they were + # already scheduled for execution. It refers to the number of scenarios that should remain in the trace. + self.rewind_limit = 0 @property def model(self) -> ModelSpace | None: @@ -122,6 +126,7 @@ def copy(self): cp._tried = [triedlist[:] for triedlist in self._tried] cp._snapshots = self._snapshots[:] cp._open_refinements = self._open_refinements[:] + cp.rewind_limit = self.rewind_limit return cp def coverage_reached(self) -> bool: @@ -196,7 +201,8 @@ def confirm_full_scenario(self, index: int, scenario: Scenario, model: ModelSpac id = str(index) self._tried[-1].append(index) self._tried.append([]) - self._snapshots.append(TraceSnapShot(id, scenario, model, drought=c_drought)) + self._snapshots.append(TraceSnapShot(id, scenario, model, + coverage=min(self.c_pool.values()), drought=c_drought)) def push_partial_scenario(self, index: int, scenario: Scenario, model: ModelSpace, remainder=None): if self.is_refinement_active(index): @@ -206,16 +212,43 @@ def push_partial_scenario(self, index: int, scenario: Scenario, model: ModelSpac self._tried[-1].append(index) self._open_refinements.append(index) self._tried.append([]) - self._snapshots.append(TraceSnapShot(id, scenario, model, remainder, self.coverage_drought)) + self._snapshots.append(TraceSnapShot(id, scenario, model, remainder, + coverage=min(self.c_pool.values()), drought=self.coverage_drought)) def can_rewind(self) -> bool: - return len(self._snapshots) > 0 + rewind_margin = len(self._snapshots[self.rewind_limit:]) + if rewind_margin == 0: + return False + n = 1 + index, part = self.split_id(self._snapshots[-1].id) + if part and part > 1: + # When rewinding an 'in between' part, rewind both the part and the refinement + n += 1 + if part != 0: + return rewind_margin >= n + + # Refined scenarios that are already closed will be rewound in full. + # Check if the scenario's opening part is within the rewind margin. + for i in range(1, rewind_margin + 1): + if self._snapshots[-i].id == f'{index}.1': + n = i + return True + return False # went beyond rewind limit def rewind(self) -> TraceSnapShot | None: + """ + Performs a single rewind action, removing the most recently completed scenario from the trace. + + If the most recently completed scenario contained refinements, then the complete scenario, including + its refinements is rewound. If multi-part refinement is ongoing and a refinement step just ended, + without completing the full scenario, then the trace is rewound to before the latest refinement. + Use can_rewind() to check if a rewind is possible. + """ id = self._snapshots[-1].id - index = int(id.split('.')[0]) + index, part = self.split_id(id) self._snapshots.pop() if id.endswith('.0'): + # refined scenarios are rewinded in full self.c_pool[index] -= 1 self._open_refinements.append(index) while self._snapshots[-1].id != f"{index}.1": @@ -223,12 +256,20 @@ def rewind(self) -> TraceSnapShot | None: return self.rewind() self._tried.pop() + if part and part > 1: + # When rewinding an 'in between' part, rewind both the part and the refinement + return self.rewind() + if '.' not in id: self.c_pool[index] -= 1 if id.endswith('.1'): self._open_refinements.pop() return self._snapshots[-1] if self._snapshots else None + @staticmethod + def split_id(id: str) -> tuple[int, int | int, None]: + return tuple(map(int, id.split('.'))) if '.' in id else (int(id), None) + def __iter__(self): return iter(self._snapshots) diff --git a/robotmbt/version.py b/robotmbt/version.py index 0987cba6..aab78c2a 100644 --- a/robotmbt/version.py +++ b/robotmbt/version.py @@ -1 +1 @@ -VERSION: str = '0.13.0' +VERSION: str = '0.14.0' diff --git a/robotmbt/visualise/models.py b/robotmbt/visualise/models.py index 658d49bf..f30c7cc9 100644 --- a/robotmbt/visualise/models.py +++ b/robotmbt/visualise/models.py @@ -230,7 +230,7 @@ def update_trace(self, scenario: ScenarioInfo | None, state: StateInfo, length: else: # No change - sanity check if len(self.current_trace) > 0: - self._sanity_check(scenario, state, 'nothing') + self._sanity_check(scenario, state, 'nothing changed') def _push(self, scenario: ScenarioInfo, state: StateInfo, n: int): if n > 1: diff --git a/utest/test_suitedata.py b/utest/test_suitedata.py index 691f339f..67726962 100644 --- a/utest/test_suitedata.py +++ b/utest/test_suitedata.py @@ -66,6 +66,9 @@ def test_longname_with_parent_includes_all_parent_names(self): self.assertEqual(self.topsuite.suites[-1].scenarios[-1].longname, 'topsuite.suite B.scenario BB') + def test_scenario_count(self): + self.assertEqual(self.topsuite.scenario_count(), 6) + def test_error_in_suite_setup_is_detected(self): step = Step('top setup', parent=self.topsuite) step.gherkin_kw = 'given' diff --git a/utest/test_suiteprocessors.py b/utest/test_suiteprocessors.py index 1c8970af..77be7c92 100644 --- a/utest/test_suiteprocessors.py +++ b/utest/test_suiteprocessors.py @@ -33,50 +33,51 @@ import unittest from unittest.mock import patch, call -from robotmbt.suiteprocessors import SuiteProcessors +from robotmbt.suiteprocessors import ModelBased +from robotmbt.suitedata import Suite, Scenario, Step @patch('robotmbt.suiteprocessors.random.seed') class TestRandomSeeding(unittest.TestCase): def test_provided_seed_is_used_as_is(self, mock): - SuiteProcessors._init_randomiser("specific seed") + ModelBased._init_randomiser("specific seed") mock.assert_called_with("specific seed") def test_provided_seed_is_stripped(self, mock): - SuiteProcessors._init_randomiser(" specific seed\t") + ModelBased._init_randomiser(" specific seed\t") mock.assert_called_with("specific seed") def test_seed_none_keeps_system_seed(self, mock): - SuiteProcessors._init_randomiser(None) + ModelBased._init_randomiser(None) mock.assert_called_with() def test_seed_none_as_string(self, mock): - SuiteProcessors._init_randomiser("None") + ModelBased._init_randomiser("None") mock.assert_called_with() def test_seed_none_as_string_is_stripped(self, mock): - SuiteProcessors._init_randomiser(" None\t") + ModelBased._init_randomiser(" None\t") mock.assert_called_with() def test_seed_none_as_string_is_case_insensitive(self, mock): - SuiteProcessors._init_randomiser("nOnE") + ModelBased._init_randomiser("nOnE") mock.assert_called_with() def test_seed_new_generates_reusable_seed(self, mock): - SuiteProcessors._init_randomiser("new") + ModelBased._init_randomiser("new") self._is_generated_seed(mock.call_args.args[0]) def test_seed_new_is_stripped(self, mock): - SuiteProcessors._init_randomiser(" new\t") + ModelBased._init_randomiser(" new\t") self._is_generated_seed(mock.call_args.args[0]) def test_seed_new_is_case_insensitive(self, mock): - SuiteProcessors._init_randomiser("NeW") + ModelBased._init_randomiser("NeW") self._is_generated_seed(mock.call_args.args[0]) def test_generated_seeds_have_max_2_consecutive_vowels_or_consonants(self, mock): for _ in range(20): - SuiteProcessors._init_randomiser("new") + ModelBased._init_randomiser("new") new_seed = mock.call_args.args[0] self._is_generated_seed(new_seed) self.assertNotIn('***', new_seed.translate({ord(c): '*' for c in 'aeiouy'})) @@ -87,8 +88,8 @@ def test_seed_is_reset_after_using_specific_seed(self, mock): added to cover the issue where, after having rerun a specific trace, the next generated seed was always the same. """ - SuiteProcessors._init_randomiser("specific seed") - SuiteProcessors._init_randomiser("new") + ModelBased._init_randomiser("specific seed") + ModelBased._init_randomiser("new") new_seed = mock.call_args.args[0] mock.assert_has_calls([call("specific seed"), call(), call(new_seed)]) @@ -104,5 +105,122 @@ def _is_generated_seed(self, arg): self.assertTrue(3 <= len(word) <= 6) +class TestBatchSize(unittest.TestCase): + def setUp(self): + self.suite = Suite('testsuite') + init_scenario = Scenario('init scenario', self.suite, RobotTestCaseStub()) + init_step = Step('init keyword', parent=init_scenario) + init_step.model_info = dict(IN=["new prop"], OUT=["prop.flag = True"]) + init_scenario.steps = [init_step] + body_scenario = Scenario('body scenario', self.suite, RobotTestCaseStub()) + self.scenario_name_without_rep_count = len('body scenario') + step = Step('action keyword', parent=body_scenario) + step.model_info = dict(IN=["prop.flag = not prop.flag"], OUT=[]) # force a change so retries are not rejected + body_scenario.steps = [step] + self.suite.scenarios = [init_scenario, body_scenario] + self.processor = ModelBased() + + def test_batch_size_1(self): + out_suite = self.processor.process_test_suite(self.suite, scenario_target=3, batch_size=1) + self.assertEqual(self.processor.scenarios_pending, 1) + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 1) + self.assertEqual(self.processor.scenarios_committed, 1) + self.assertEqual(self.processor.scenarios_pending, 0) + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 2) + self.assertEqual(self.processor.scenarios_committed, 2) + self.assertEqual(self.processor.scenarios_pending, 0) + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 3) + self.assertEqual(self.processor.scenarios_committed, 3) + self.assertEqual(self.processor.scenarios_pending, 0) + self.assertListEqual([s.name[:self.scenario_name_without_rep_count] for s in out_suite.scenarios], + ['init scenario'] + ['body scenario']*(out_suite.scenario_count()-1)) + + def test_batch_size_2_last_batch_not_full(self): + out_suite = self.processor.process_test_suite(self.suite, scenario_target=3, batch_size=2) + self.assertEqual(self.processor.scenarios_pending, 2) + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 1) + self.assertEqual(self.processor.scenarios_committed, 1) + self.assertEqual(self.processor.scenarios_pending, 1) + self.processor.next_scenario_request() + self.assertEqual(self.processor.scenarios_pending, 0) + self.processor.next_scenario_request() + self.assertEqual(self.processor.scenarios_pending, 0) + self.assertEqual(out_suite.scenario_count(), 3) + + def test_batch_size_2_last_batch_full(self): + out_suite = self.processor.process_test_suite(self.suite, scenario_target=4, batch_size=2) + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 1) + self.assertEqual(self.processor.scenarios_committed, 1) + self.assertEqual(self.processor.scenarios_pending, 1) + self.processor.next_scenario_request() + self.assertEqual(self.processor.scenarios_pending, 0) + self.processor.next_scenario_request() + self.assertEqual(self.processor.scenarios_pending, 1) + self.processor.next_scenario_request() + self.assertEqual(self.processor.scenarios_pending, 0) + self.assertEqual(out_suite.scenario_count(), 4) + + def test_batch_size_10(self): + out_suite = self.processor.process_test_suite(self.suite, scenario_target=15, batch_size=10) + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 1) + self.assertEqual(self.processor.scenarios_pending, 9) + for _ in range(9): + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 10) + self.assertEqual(self.processor.scenarios_pending, 0) + self.processor.next_scenario_request() + self.assertEqual(self.processor.scenarios_pending, 4) + + def test_batch_size_3_is_trace_length(self): + out_suite = self.processor.process_test_suite(self.suite, scenario_target=3, batch_size=3) + self.assertEqual(self.processor.scenarios_pending, 3) + self.processor.next_scenario_request() + self.assertEqual(self.processor.scenarios_pending, 2) + self.processor.next_scenario_request() + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 3) + self.assertListEqual([s.name[:self.scenario_name_without_rep_count] for s in out_suite.scenarios], + ['init scenario'] + ['body scenario']*(out_suite.scenario_count()-1)) + + def test_requesting_beyond_targets_has_no_effect(self): + out_suite = self.processor.process_test_suite(self.suite, scenario_target=3, batch_size=3) + self.processor.next_scenario_request() + self.processor.next_scenario_request() + self.assertFalse(self.processor.are_all_targets_reached()) + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), 3) + self.assertTrue(self.processor.are_all_targets_reached()) + self.processor.next_scenario_request() + self.assertTrue(self.processor.are_all_targets_reached()) + self.assertEqual(out_suite.scenario_count(), 3) + + def test_multi_batch(self): + """Check some variations in batch size versus target size""" + for target, batch in [(1, 1), # Smallest target and batch + (23, 3), # Multiple batches needed te completer + (10, 24), # Batch size exceeds target size + (15, 14), # Batch size just not enough + (16, 16) # Batch size equals target size + ]: + out_suite = self.processor.process_test_suite(self.suite, coverage_target=0, + scenario_target=target, batch_size=batch) + while not self.processor.are_all_targets_reached(): + self.processor.next_scenario_request() + self.assertEqual(out_suite.scenario_count(), target) + self.assertListEqual([s.name[:self.scenario_name_without_rep_count] for s in out_suite.scenarios], + ['init scenario'] + ['body scenario']*(out_suite.scenario_count()-1)) + + +class RobotTestCaseStub: + def copy(self, **kwargs): + pass + + if __name__ == '__main__': unittest.main() diff --git a/utest/test_tracestate.py b/utest/test_tracestate.py index ed6c37a4..db2f241b 100644 --- a/utest/test_tracestate.py +++ b/utest/test_tracestate.py @@ -305,6 +305,30 @@ def test_can_index_tracestate_snapshots(self): self.assertEqual(ts[-1].scenario, 'three') self.assertEqual([s.id for s in ts[1:]], ['2', '3']) + def test_tracestate_snapshots_track_coverage(self): + """ + Coverage counter shows the number of times that full coverage is achieved. I.e., the + counter stays 0 until the last sceanrio is reached for the first time. Then it stays + at 1 until all scenarios have been executed at least a second time. + """ + ts = TraceState([1, 2, 3]) + ts.confirm_full_scenario(1, ScenarioStub('one A'), ModelStub()) + ts.confirm_full_scenario(2, ScenarioStub('two A'), ModelStub()) + ts.confirm_full_scenario(3, ScenarioStub('three A'), ModelStub()) + self.assertEqual(ts[-1].coverage_reached, 1) + self.assertEqual(ts[-2].coverage_reached, 0) + self.assertEqual(ts[0].coverage_reached, 0) + ts.confirm_full_scenario(1, ScenarioStub('one B'), ModelStub()) + ts.confirm_full_scenario(2, ScenarioStub('two B'), ModelStub()) + ts.confirm_full_scenario(3, ScenarioStub('three B'), ModelStub()) + self.assertEqual(ts[1].coverage_reached, 0) + self.assertEqual(ts[2].coverage_reached, 1) + self.assertEqual(ts[-1].coverage_reached, 2) + ts.confirm_full_scenario(3, ScenarioStub('three C'), ModelStub()) + self.assertEqual(ts[-1].coverage_reached, 2) + self.assertEqual(ts[-2].coverage_reached, 2) + self.assertEqual(ts[-3].coverage_reached, 1) + def test_adding_coverage_prevents_drought(self): ts = TraceState(range(3)) ts.confirm_full_scenario(ts.next_candidate(), ScenarioStub('one'), ModelStub()) @@ -345,6 +369,15 @@ def test_rewind_includes_drought_update(self): ts.rewind() self.assertEqual(ts.coverage_drought, 0) + def test_rewind_limit(self): + ts = TraceState([1, 2]) + ts.confirm_full_scenario(1, ScenarioStub('one'), ModelStub()) + ts.confirm_full_scenario(2, ScenarioStub('two'), ModelStub()) + ts.rewind_limit = 1 + self.assertTrue(ts.can_rewind()) + ts.rewind() + self.assertFalse(ts.can_rewind()) + def test_trace_id_properties(self): ts = TraceState([4, 1, 2, 3]) ts.confirm_full_scenario(3, ScenarioStub(), ModelStub()) @@ -394,19 +427,20 @@ def test_rewind_of_single_part(self): self.assertEqual(ts.get_trace(), []) def test_rewind_all_parts(self): - ts = TraceState([1]) + ts = TraceState([1, 2]) ts.push_partial_scenario(1, ScenarioStub('part1'), ModelStub()) + ts.confirm_full_scenario(2, ScenarioStub('two'), ModelStub()) self.assertIs(ts.coverage_reached(), False) ts.push_partial_scenario(1, ScenarioStub('part2'), ModelStub()) self.assertIs(ts.coverage_reached(), False) - self.assertEqual(ts.get_trace(), ['part1', 'part2']) + self.assertEqual(ts.get_trace(), ['part1', 'two', 'part2']) self.assertIs(ts.next_candidate(), None) ts.rewind() self.assertEqual(ts.get_trace(), ['part1']) self.assertIs(ts.next_candidate(), None) ts.rewind() self.assertEqual(ts.get_trace(), []) - self.assertIs(ts.next_candidate(), None) + self.assertIs(ts.next_candidate(), 2) self.assertIs(ts.can_rewind(), False) def test_partial_scenario_still_excluded_from_candidacy_after_rewind(self): @@ -416,16 +450,16 @@ def test_partial_scenario_still_excluded_from_candidacy_after_rewind(self): ts.rewind() self.assertIs(ts.next_candidate(), None) - def test_rewind_to_partial_scenario(self): - ts = TraceState([1]) + def test_rewind_partial_to_partial_scenario(self): + ts = TraceState([1, 2]) ts.push_partial_scenario(1, ScenarioStub('part1'), ModelStub(a=1)) - ts.push_partial_scenario(1, ScenarioStub('part2'), ModelStub(b=2)) + ts.push_partial_scenario(2, ScenarioStub('part1'), ModelStub(b=2)) snapshot = ts.rewind() self.assertEqual(snapshot.id, '1.1') self.assertEqual(snapshot.scenario, 'part1') self.assertEqual(snapshot.model, dict(a=1)) - def test_rewind_last_part(self): + def test_rewind_partial_to_full_scenario(self): ts = TraceState([1, 2]) ts.confirm_full_scenario(1, ScenarioStub('one'), ModelStub(a=1)) ts.push_partial_scenario(2, ScenarioStub('part1'), ModelStub(b=2)) @@ -435,14 +469,25 @@ def test_rewind_last_part(self): self.assertEqual(snapshot.scenario, 'one') self.assertEqual(snapshot.model, dict(a=1)) - def test_rewind_all_parts_of_completed_scenario_at_once(self): - ts = TraceState([1]) + def test_rewind_full_to_partial_scenario(self): + ts = TraceState([1, 2]) ts.push_partial_scenario(1, ScenarioStub('part1'), ModelStub(a=1)) - ts.push_partial_scenario(1, ScenarioStub('part2'), ModelStub(b=2)) + ts.confirm_full_scenario(2, ScenarioStub('two'), ModelStub(b=2)) + snapshot = ts.rewind() + self.assertEqual(snapshot.id, '1.1') + self.assertEqual(snapshot.scenario, 'part1') + self.assertEqual(snapshot.model, dict(a=1)) + + def test_rewind_all_parts_of_completed_scenario_at_once(self): + ts = TraceState([1, 2, 3]) + ts.push_partial_scenario(1, ScenarioStub('part1'), ModelStub()) + ts.confirm_full_scenario(2, ScenarioStub('two'), ModelStub()) + ts.push_partial_scenario(1, ScenarioStub('part2'), ModelStub()) + ts.confirm_full_scenario(3, ScenarioStub('three'), ModelStub()) ts.confirm_full_scenario(1, ScenarioStub('remainder'), ModelStub()) tail = ts.rewind() self.assertEqual(ts.get_trace(), []) - self.assertIs(ts.next_candidate(), None) + self.assertIs(ts.next_candidate(), 2) self.assertIs(tail, None) def test_tried_entries_after_rewind(self): @@ -456,8 +501,6 @@ def test_tried_entries_after_rewind(self): ts.reject_scenario(21) self.assertEqual(ts.tried, [20, 21]) ts.rewind() - self.assertEqual(ts.tried, []) - ts.rewind() self.assertEqual(ts.tried, [10, 11, 2]) ts.reject_scenario(12) self.assertEqual(ts.tried, [10, 11, 2, 12]) @@ -486,8 +529,9 @@ def test_highest_part_after_completing_multiple_parts(self): self.assertEqual(ts.highest_part(1), 0) def test_highest_part_after_partial_rewind(self): - ts = TraceState([1]) + ts = TraceState([1, 2]) ts.push_partial_scenario(1, ScenarioStub('part1'), ModelStub()) + ts.confirm_full_scenario(2, ScenarioStub('two'), ModelStub()) ts.push_partial_scenario(1, ScenarioStub('part2'), ModelStub()) self.assertEqual(ts.highest_part(1), 2) ts.rewind() @@ -496,10 +540,12 @@ def test_highest_part_after_partial_rewind(self): self.assertEqual(ts.highest_part(1), 0) def test_highest_part_is_0_when_no_refinement_is_ongoing(self): - ts = TraceState([1]) + ts = TraceState([1, 2, 3]) self.assertEqual(ts.highest_part(1), 0) ts.push_partial_scenario(1, ScenarioStub('part1'), ModelStub()) + ts.confirm_full_scenario(2, ScenarioStub('two'), ModelStub()) ts.push_partial_scenario(1, ScenarioStub('part2'), ModelStub()) + ts.confirm_full_scenario(3, ScenarioStub('three'), ModelStub()) ts.confirm_full_scenario(1, ScenarioStub('remainder'), ModelStub()) self.assertEqual(ts.highest_part(1), 0) ts.rewind() @@ -597,6 +643,34 @@ def test_trace_id_properties_for_partials(self): self.assertEqual(ts.not_in_trace, [1, 2, 5, 7]) self.assertEqual(ts.unreached, [5, 7]) + def test_rewind_limit_for_partials(self): + ts = TraceState([1, 2, 3]) + ts.push_partial_scenario(1, ScenarioStub(), ModelStub()) + ts.confirm_full_scenario(2, ScenarioStub(), ModelStub()) + self.assertTrue(ts.can_rewind()) + ts.rewind_limit = 2 + self.assertFalse(ts.can_rewind()) + + ts.push_partial_scenario(1, ScenarioStub(), ModelStub()) + self.assertFalse(ts.can_rewind()) + + ts.confirm_full_scenario(3, ScenarioStub(), ModelStub()) + self.assertTrue(ts.can_rewind()) + ts.confirm_full_scenario(1, ScenarioStub(), ModelStub()) + self.assertFalse(ts.can_rewind()) + + def test_rewind_limit_just_before_partial(self): + ts = TraceState([1, 2, 3]) + ts.confirm_full_scenario(1, 'one', ModelStub()) + ts.rewind_limit = 1 + ts.push_partial_scenario(2, 'part1', ModelStub()) + self.assertTrue(ts.can_rewind()) + ts.confirm_full_scenario(3, 'three', ModelStub()) + ts.confirm_full_scenario(2, 'final part', ModelStub()) + self.assertTrue(ts.can_rewind()) + ts.rewind_limit = 2 + self.assertFalse(ts.can_rewind()) + class ScenarioStub(str): """Stub for suitedata.Scenario""" diff --git a/utest/test_tracestate_refinement.py b/utest/test_tracestate_refinement.py index 49440dc7..1aa43daf 100644 --- a/utest/test_tracestate_refinement.py +++ b/utest/test_tracestate_refinement.py @@ -97,7 +97,6 @@ def test_rewind_to_swap_refinements(self): inner2 = ts.next_candidate() ts.reject_scenario(inner2) ts.rewind() - ts.rewind() self.assertEqual(ts.tried, [inner1]) self.assertEqual(ts.next_candidate(), inner2) ts.confirm_full_scenario(inner2, 'B2', {}) @@ -123,7 +122,6 @@ def test_rewind_partial_scenario_to_before_outer(self): inner2 = ts.next_candidate() ts.reject_scenario(inner2) ts.rewind() - ts.rewind() previous = ts.rewind() self.assertEqual(previous.scenario, 'HEAD') self.assertEqual(ts.get_trace(), ['HEAD']) @@ -394,15 +392,17 @@ def test_is_refinement_active_by_index(self): self.assertFalse(ts.is_refinement_active(2)) def test_remainder_can_be_set_and_retrieved(self): - ts = TraceState([1, 2]) + ts = TraceState([1, 2, 3]) ts.push_partial_scenario(1, 'one part1', {}, 'one part2') ts.push_partial_scenario(2, 'two part1', {}, 'two parts 2+3') self.assertEqual(ts.get_remainder(1), 'one part2') self.assertEqual(ts.get_remainder(2), 'two parts 2+3') + ts.confirm_full_scenario(3, 'three', {}) ts.push_partial_scenario(2, 'two part2', {}, 'two part3') self.assertEqual(ts.get_remainder(2), 'two part3') ts.rewind() self.assertEqual(ts.get_remainder(2), 'two parts 2+3') + ts.confirm_full_scenario(3, 'three', {}) ts.push_partial_scenario(2, 'two part2', {}, 'two part3B') self.assertEqual(ts.get_remainder(2), 'two part3B') ts.confirm_full_scenario(2, 'two', {})