14 Use with emacs	Top	16 Test coverage support
15 Mutation testing

15 Mutation testing

The tool supports mutation testing with the SML translator and SML run-time system. Mutation testing is a technique to check the adequacy of test cases by seeing if a small change, a mutation, of the RSL source gives different test results. If the test results are different it shows that this mutation can be detected if made by error. If the test results are not different it suggests that more test cases might be needed. This is a useful technique when the test cases are developed during specification and later used on the final software. Mutation testing can increase the thoroughness of the testing, and hence the quality of the test cases.

Note that the support for mutation testing currently depends on all the RSL files needed being stored in one directory.

Before you start mutation testing you need to run the SML translator on the original files. Open the file with the test cases in it, translate to SML (RSL menu Translate to SML), run the SML translator (Run SML file) and save the results (Save results from SML run). See section 9.2.2 on saving results.

To make a mutation in an RSL module, open the module file, X.rsl, say, in emacs. Select the expression you want to change with the mouse by dragging, or by clicking with first the left at the beginning and then the right mouse button just past the end. Then select the item Make mutation in the RSL menu. (This menu item will not appear until you have selected something.)

You will first be prompted for the replacement text. For example, you might have selected < and want to try replacing it with <=. You type <= and hit Enter.

Then you are prompted for the file to translate. This might be X.rsl if the test cases are in the same file, or it might be a different file, such as TEST_X.rsl. The first time you will be offered X.rsl and can change it to TEST_X.rsl if you need to. If you make this change and later in the same session make another mutation to X.rsl you will be immediately offered TEST_X.rsl as the file to translate. Having corrected the offered file name if necessary, hit Enter. We will assume in the rest of this section that the file to be translated is TEST_X.rsl.

The RSL files in the current directory are copied to a subdirectory mutantn, where the final n is initially 0 (zero), and the mutation is applied to mutantn/X.rsl. Then the SML translator is run on mutantn/TEST_X.rsl, and provided there are no errors the SML run-time system is run on the output. Then you can see if there are any differences from the output of the original run before you made any mutations.

If there are many test cases and you want some automated support with the comparison, go back to the file you just translated, mutant0/TEST_X.rsl, say, or the original X.rsl, or TEXT_X.rsl, and select Compare with mutant in the RSL menu. This will save the results from the latest SML run, and offer two results files to compare, in this case TEST_X.sml.results and mutant0/TEST_X.sml.results. You can change these if you wish, or just hit Enter for each immediately if they are what you want. The "ediff" file comparison tool is then run on the two selected results files. This tool generates a small command window which shows how many differences there are, as well as showing in two main buffers the results files with differences highlighted. With the point in the command window, the main command keys are space bar to highlight the next difference, Delete to highlight the previous one, q to quit ediff (confirmed by y), and the toggle ? to show/hide all the commands available.

You can go back to X.rsl to make another mutation, which will be placed in subdirectory mutant1, starting by selecting an expression to change (perhaps the same expression) and using Make mutation again.

You can also tidy up by selecting the RSL menu item Delete mutant directories in the original X.rsl or TEST_X.rsl buffers. This will delete the subdirectories mutantn and their contents (including the results files: move these first if you need to preserve them).

15 Mutation testing
14 Use with emacs	Top	16 Test coverage support