Throw and Catch Coverage

When software applications are tested with an automated test suite, it is common to measure the statement or branch coverage. However, many other kinds of coverage may be defined. In this post, we focus on two kinds of coverage dedicated to exception handling.

From the viewpoint of testing, the throw coverage'' $\mbox{Cov}_{throw}$ is the ratio of executedthrow’’ statements T_ex over the total number of ``throw’’ statements T in the system under study. The throw coverage measures to what extent the test suite simulates the conditions of throwing exceptions. If it is low (e.g.:  ≤ 5%), it means that the test suite focuses on nominal usage testing. If it is high (e.g.:  ≥ 90%), the test suite covers most exceptional cases envisioned by the application developers.

T is obtained by a simple static analysis of the abstract syntax tree of the system under study. T_ex is obtained by wrapping every throw statement in a try/catch block that simply logs and counts the execution of throw statements.

Table gives the throw coverage on all 11 test suites of our dataset. It also presents the absolute metrics T and T_ex and, for comparison, the method coverage. %WORD toot many and

For the well-tested applications (method coverage > 50%), one sees that the throw coverage range from 15% to 97%. Well-tested applications also take care of testing exceptional conditions. The two control test suites, bluej and jedit, validate this, their ``throw’’ statements are almost not tested at all (e.g.; 2/213 for bluej). When comparing throw coverage and method coverage, one sees that throw statements are less thoroughly tested than the rest of the application. For instance, Shindig Gadgets has a throw coverage of 38% and a method coverage of 70%.

The catch coverage'' $\mbox{Cov}_{catch}$ is the ratio of executedcatch’’ blocks C_ex over the total number of catch blocks C in the system under study. The catch coverage measures to what extent the test suite exercises the catch blocks, i.e. to what extent the test suite specifies error-recovery. If a test case remains green after the execution of a catch block, it does mean that the recovery code in the catch block successfully repairs the state of the program. Note that catch coverage is close but slightly different of the ``fault-catch coverage’’ metric of Fu et al. , which focuses on measuring whether all possible caught exceptions have been thrown.

C is obtained by a simple static analysis of the abstract syntax tree of the system under study. C_ex is obtained by inserting a log statement at the beginning of every catch block.

Table gives the catch coverage on all 11 test suites of our dataset. It also presents the absolute metrics C and C_ex. The interpretation of this data is similar as for the throw coverage. Developers of well-tested applications also take care of testing the error-recovery of the application. For well-tested applications, catch blocks are indeed tested but less than the rest of the code.