Next: , Previous: How do the tests work?, Up: Black-box tests

2.2.4 When a test fails...

Don't panic!

If a test fails as a result of your work, remember that these are consistency tests, not correctness tests. Do you expect your work to produce any different output for that particular test?

For example, if you discover (and fix) a bug in the inverse FFT, the phasevocoder test will probably “fail”. This is to be expected: the previously-recorded output of the phasevocoder faithfully archived the buggy output, so the bug-free output is detected as different.

On the other hand, suppose you are adding a new classifier for machine learning, and the Windowing test breaks. This is not expected; a new feature should not impact basic functions like taking a Hanning window! In this case, you should investigate before committing your changes to svn.

Updating the test

If you are certain that your patch (and new output file) are good, then you should update the answer file. This is simply a matter of copying your new output file over the old answer file.

The new output file is found in BUILD_DIR/tests/black-box/. You may also create the file manually yourself; the exact command-line arguments used for each test can be seen with:

     make test ARGS="-V"

Please commit the changed src/tests/black-box/output/<> in a separate svn commit, and make sure the log message explains that your new output is superior to the old one.

Temporarily disabling a test

If you are planning on doing a lot of work on part of Marsyas (which would result in tests failing, but having no working output yet), tests may be commented out in the src/tests/black-box/CMakeLists.txt file. Again, please commit this change in a separate svn commit with a log message that explains this.