SE-2832 Introduction to Software Verification

Dr. Mark Sebern -- Spring quarter 2013-2014


Lab 9

Revised: 7 May 2014

Acknowledgment

This lab was developed by Dr. Walt Schilling; all errors in its adaptation are the responsibility of the instructor.

Overview

In this lab, you will work in a small team (2-3 members) to perform mutation testing. You may not do this lab by yourself.

Mutation testing is an effective method that can be used to evaluate the effectiveness of JUnit test cases. The general idea is to modify the source code in small ways. These, so-called mutations, are based on well-defined mutation operators that either mimic typical user mistakes or force the creation of valuable tests. The process looks like this:

Mutation testing process

(Figure from sourceforge.net)

In this lab, you will use the PIT mutation tool to mutate the source code for several of your projects. In doing so, you will see how small changes to the source code can go unnoticed by test suites even when full coverage is achieved

Lab Assignment

Tutorial video

Watch the PitClipse mutation testing tutorial (PitTestDemo.mp4).

PitClipse installation

  1. In Eclipse, select Help->Install New Software...
  2. Add the PitClipse installation site. Click on the Add... button and enter the name “PitClipse” and the update site http://eclipse.pitest.org/release
  3. Install the plugin by following the dialog boxes.
  4. After the installation process is complete, allow Eclipse to restart.

Evaluating the tax code tests

Working with your partner(s), start by running the PIT tool on the tests you created for the tax calculator code in Lab 2. While you used code coverage to check the tests, it is very possible that mutation testing will reveal some tests problems with your testing.

As you run the tool, take a look at each mutation which is left live by your test suite. Determine if it is an equivalent mutation or not.

Calculate the initial mutation score using the formula:

\[MutationScore = {KilledMutants \over {TotalMutations - EquivalentMutations}} \]

Once you have removed all of the live mutations, recalculate the mutation score. The only remaining mutations at this point should be stubborn mutants.

Evaluating the dental calculator and stock market analyzer

Once you have calculated the mutation score for your tests on the tax code, run the PIT tool on your dental calculator (Lab 5) and stock market analyzer code (Lab 6). (Hint: You may split this work between/among your team members, working on both at the same time.) Determine the mutation score for each.

Once you have determined the mutation score, add JUnit (or TestNG?) tests to your test suite until the mutation score is as close to 100% as you can make it. To do this, write new test methods and place them at the bottom of the file until you have exhausted the ability to create test cases. See if mutation score has changed, and how much it has improved (optimistic bias here).

Assignment submission (due by 11:59PM CDT, Tuesday, May 13)

Prepare a PDF report (named "SE2832-username1-username2-username3-Lab9.pdf" (using dashes, not underscores, with the specified capitalization), where each "usernameN" is the lowercase MSOE email username of a team member) on your lab experience, including:

If you have any questions, consult the instructor.