Changes in RegressionTests
Editor Comment
New command line switches
Revision Differences of Revision 20
When you think your branch is ready for merging, there are two tests you can run to make sure that your code sticks to our coding style, and to test for a bunch of possible regression bugs. ¶¶
[TOC] ¶
¶
## Codecheck ¶
¶
We have codecheck rules that will help you stick to our coding style guidelines. These are run automatically whenever you compile a debug build, but the messages are hard to see sometimes. However, you can run the checks manually as well. ¶
¶
### Running Codecheck Manually ¶
¶
Depending on your build system, the following command will be available: ¶
¶
~~~~ ¶
cd build && make codecheck ¶
~~~~ ¶
¶
If you see lots of "see_if_codecheck_needs_to_run_" messages, or if the `codecheck` target is not available, you can manually run the Python scripts: ¶
¶
~~~~ ¶
utils/build_deps.py && cmake/codecheck/CodeCheck.py src/* | grep -v "src/third_party" ¶
~~~~ ¶
¶
### Testing Codecheck Rules ¶
¶
The codecheck rules themselves can also be tested. Run: ¶
¶
~~~~ ¶
./cmake/codecheck/run_tests.py ¶
~~~~ ¶
¶
## Test Scenarios ¶
¶
Some tests can be run automatically while compiling debug builds, but not all of them. We have some automated scenarios that will check for regression bugs. Running the complete test suite will keep opening and closing an instance of Widelands and grabbing your mouse, so make sure you're not doing anything else at the time. ¶
¶
### Full Test Suite ¶
¶
You can run the complete test suite with ¶
¶
~~~~ ¶
./regression_test.py -b ./widelands ¶
~~~~ ¶
¶
### Single Test Scenario ¶
¶
#### By Filename ¶
¶
If you only wish to run part of the testsuite, you can filter with regex for the filename, like this: ¶
¶
~~~~ ¶
./regression_test.py -b ./widelands -r=portdock ¶
~~~~ ¶
¶
This will run all test(s) (*lua files) with a keyword (portdock in this example) in their file path. You can of course use the entire filename to run just one test. Have a look at the `test` directory for which tests are available. ¶
¶
¶
#### With Direct Console Output ¶
¶
While debugging something specific or if you want to run a scenario from within a debugger, you can also use a different command to get the output directly on the console. Note that this won't set up the complete testing environment, so you will always need to use one of the above commands to verify when you're done. You will need to specify both the scenario map and the specific test script: ¶
¶
~~~~ ¶
./widelands --datadir=data --datadir_for_testing=. --fail-on-lua-error --scenario=test/maps/ship_transportation.wmf --script=test/maps/ship_transportation.wmf/scripting/test_rip_second_port_with_worker_in_portdock.lua ¶
~~~~ ¶
¶
Note that `lua_testsuite.wmf` needs special treatment - run it with: ¶
¶
~~~~ ¶
./widelands --scenario=test/maps/lua_testsuite.wmf --datadir_for_testing=. ¶
~~~~ ¶
¶
### Help ¶
¶
You can print help to the console with: ¶
¶
~~~~ ¶
./regression_test.py -h ¶
~~~~ ¶
~~~~ ¶
./regression_test.py --help ¶
~~~~ ¶
¶
### ASan ¶
¶
If Widelands was compiled with AddressSanitizer, it will always exit with error code `1` on success due to a third-party memory leak. In order to silence false-positives, it is possible to run `regression_test.py` with the `-i` switch to treat return code `1` as success. Only use this option if you are certain that you are not accidentally masking real errors though. ¶
¶
### Required Python Version ¶
¶
Also note that CodeCheck.py and regression_test.py **require python 2.7** or 3.4.x (tested with 3.4.4). So, if you get errors, ¶
check your version and then try to force the correct version like this: ¶
¶
~~~~ ¶
python2.7 ./regression_test.py -b ./widelands ¶
~~~~ ¶
~~~~ ¶
python3.4 ./regression_test.py -b ./widelands ¶
~~~~ ¶
¶
providing you are using **linux** or similar system. Chances are high that python2.7 is already installed on your box, if not use your package manager to install it.