Chapter 7. Generating unit tests – Code Generation in Action

Chapter 7. Generating unit tests

7.1 The big picture
7.2 Preparing for common concerns
7.3 A case study: augmented C code
7.4 Technique: the ordered test generator
7.5 Technique: the test data generator
7.6 Technique: the test robot generator
7.7 Finding a tool to do it for you
7.8 Design tips
7.9 Summary

There is nothing like the feeling of editing your code with confidence. And that kind of confidence can only come from knowing you have tests that check every aspect of the code for defects. Unit tests are invaluable for performing this role and ensuring overall code quality. They run against an API, which tests both valid and invalid use cases. A robust unit test will invoke the system with a variety of methods and data, and then report back as to the success or failure of the system.

The downside of unit tests is that it takes a lot of work to create and maintain them. The upside is that you can apply generation to solve that problem.

In this chapter, we present a number of generators that make it easy to create and maintain unit tests that ensure code quality. Because each unit test application has different requirements, you will need to customize the generators in this chapter to use them effectively in your environment.

7.1. The big picture

The case study from chapter 1 used several generated unit tests. The first of these tested the database access interface through the EJB layer. The database unit test system uses test data from the database to build a test data loader. This data loader uses the EJB layer to set the database and to determine whether the data was properly written. Figure 7.1 shows the unit test from chapter 1.

Figure 7.1. The database access layer unit test from the case study in chapter 1

The second set of unit tests we showed was a set of web robots that comes in the front door of the application through the web interface. Figure 7.2 shows how the user agents are generated and how they relate to the production pages.

Figure 7.2. Testing the original case study system by loading data through the user interface

The web robots use the same data as the database unit test but load that data in through the front end of the application. This tests the JSP pages, the database access layer, and the database schema. Using the web robots gives you an end-to-end test of your entire technology stack.

This chapter presents a case study that tests APIs in a manner similar to the database loader from the case study. In addition, we provide an architecture for the web robot unit test that was used in the study.

But before we start building our case study, it’s useful to think about some of the common concerns that you might come up against when implementing unit tests with code generation.

7.2. Preparing for common concerns

Who can argue with code quality? Because unit tests create higher quality, more reliable code, it’s tough to argue against them, but the specter of time and money can cast a shadow on the value of code meant for testing instead of production. Here are some of the more common concerns about unit testing.

7.2.1. Unit tests won’t be maintained

Maintaining production code is difficult enough without having to maintain the supporting unit tests. That may be true, but there are ways to ensure that your company’s investment in unit tests is maintained:

  • Keep the tests simple to create and maintain. One strategy is to keep your tests close to the code being tested. Another is to use code generation to make building the tests simple.
  • Make the test framework visible and easy to run. Ideally it should also leverage your existing development tools, or the test framework tools should be installed as part of the build environment install. No barriers should exist between the intention to test the code and the actual running of the tests and receiving an understandable report.
  • Support multiple levels of reporting. You should be able to test the overall system and get back a high-level report telling you what works and where you might run into problems. The user should be able to drill down on issues or rerun tests in specific areas to get more information about where issues are located.
  • Make unit testing part of the culture. Ensure that time is allocated for unit test development. Run the unit tests automatically every night with email notification of failures so that everyone knows when and where unit tests are failing.

7.2.2. It’s too hard to make unit tests

This is sometimes rephrased as “Unit tests won’t work on my code.” If you can write the code, tools are out there to test the code. It’s true that some types of code are easier to test than others. You need to analyze how to spend your QA money and balance the importance of the code being tested against the difficulty and expense of testing it.

For all of the code domains (e.g., database access, interface, RPC) presented in this book, effective unit test tools are available both in the public domain or for purchase. If you are willing to spend the money and the time, tools are available that make building tests for your application easier. APIs can be tested through white box unit tests, web interfaces can be tested through robots, and client/server applications can be tested through front-end testing tools, such as those from Segue (www.segue.com).

7.2.3. Unit tests will limit our flexibility

It seems counterintuitive, but unit tests actually increase your flexibility by allowing you to accept more risks in refactoring code. If you have thorough unit tests for your module, then you can refactor the code any way you please; so long as the unit tests pass, you can be reasonably sure that the module is still running properly.

Working with unit tests is an immensely gratifying experience. If you have not worked with a solid unit test framework before, you are missing out on valuable peace of mind. It will be the end of sleepless nights spent worrying about application stability or reliability.

7.2.4. The interface isn’t ready yet

When fully embraced, unit testing inverts the development process. You start by developing the unit tests, and then implement the interfaces to match the unit test cases. These tests should cover both normal and error conditions. Once the tests pass, you can feel confident that the interface is implemented completely.

When you invert the development process to create the unit tests before doing anything else, the question of whether the interfaces are sufficient for tests becomes a non-issue.

7.2.5. Unit tests are not worth the time

To understand the value of unit tests, you have to try using them at least once. I can guarantee that after you have experienced the confidence that comes with writing code with a unit test running as a bug-catching safety net, you’ll never have to be convinced again.

7.2.6. Adding unit tests to a legacy system is painful

Let’s face it: Doing almost anything with legacy code is painful. And that pain is multilayered. First, you have to try to understand the code. Then comes figuring out what changes need to be made. Finally, there is the fear that new changes will break the code.

This last part is where unit tests pay off. If you had unit tests in the first place, you wouldn’t be worrying about the ramifications of code changes.

When you are modifying core components of a legacy system, it is worth the time to integrate unit tests before making the changes. If you have thorough unit tests and they pass after the modifications, you can be reasonably satisfied that the system is stable.

7.2.7. Unit tests can only prove the existence of errors

In 1968 Edsger Dijkstra wrote that “Testing can demonstrate the existence of errors, but never their absence.” The idea is that you should be able to mathematically prove the validity of an algorithm and thus prove every possible test case. This is both fascinating and true. If you have algorithmic tests, you may not need unit tests. However, if your system does not have algorithmic tests you should consider using unit tests as a starting point to ensure code quality.

Our case study on unit test generation uses a generator that builds unit tests by parsing the test data directly from a C implementation file.

7.3. A case study: augmented C code

The case study generator allows an engineer to add test cases to C code using specifically formatted comments. What follows is some C code with test data embedded into the comments. The placement of the test data is critical. The test data should be directly in front of the function to which the data will be sent. In this way, you keep the test data close to the function and obviate the need to specify to which function the test data applies.

The output of the generator goes back into the input file. The following code fragment is the output of the generator using the input file:

In this code, we create a new function called run_tests, which is local to this module. This run_tests function calls add twice with the test data specified in the comments. If the add function does not respond properly, the system prints an error message and returns zero for failure. If all of the tests run properly, then run_tests returns 1 for success.

7.3.1. Roles of the generator

Before you architect a generator for the unit tests, you need to clearly define the role of the generator in your overall system. First, define which requirements are covered by the generator and which are not. Let’s start with what is covered:

  • Translation of technical unit test requirements into running unit tests (by technical requirements, we mean the input data for each test as well as the valid output for each set of input data).
  • The framework within which the unit tests will be run.
  • The assessment of the success or failure of the unit tests.

The generator does not take responsibility for:

  • Designing the tests.
  • Generating test data sets to test an arbitrary interface.
  • Testing interfaces where the functions need to be called in a specific order.
  • Testing object-oriented interfaces.

The limitation of the order dependency in the interface and the testing of object-oriented interfaces are addressed by other unit test generator designs presented in the sequential test case generator described later in section 7.4.

7.3.2. Laying out the generator architecture

The generator for the augmented C unit test system is based on the mixed-code generation model (see chapters 2 and 4 for more on mixed-code generation). It takes the C code as input, adds the test implementations using the test templates, and writes the new code back into the original file (after making a backup). Figure 7.3 is a block diagram of the generator.

Figure 7.3. This generator takes C code, which embeds tests, implements the tests, and injects them back into the C.

In the next section, we’ll look at a step-by-step view of how this generator operates as it executes.

7.3.3. Processing flow

C unit test generator

The C unit test generator follows these processing steps:

  • Reads the C file.
  • Tokenizes the C.
  • Parses the C tokens.
  • Prepares an array to act as a list of tests.
  • Inspects each function and follow these steps:

    • Looks at test data in the comments.
    • Adds the test data to the list of tests.
    • Uses a template to create the test function with all of the tests.
    • Adds the function into the original file text.
  • Makes a backup of the original file.
  • Writes the output into the original file.

7.3.4. Building the code for the unit test generator

We’re ready to start writing the code for our generator. Our generator uses the CTokenizer and CLanguageScanner from the language parsing toolkit discussed in chapter 3. Once the C comments and function prototypes are parsed and stored in memory, the generator uses the ERb text-template system to build the new test function. Once the construction of the function is complete, the generator injects the function back into the original file using a regular expression. The source code for the C unit test generator is shown in Listing 7.1.

Listing 7.1. testgen.rb

  1. PrototypeWithTests derives from the Prototype class defined in the language parser toolkit. The class adds an array of tests, which are TestData objects. Adding the tests to this class allows you to keep the test data in the same object as the function name and argument information.
  2. The TestData class wraps the name of the test, the expected result value, and the arguments for the function. There is one TestData object for each test, and it is attached to the PrototypeWithTests object to which the test is applied.
  3. parse_data uses the Rexml package to read the XML that defines a single set of test data. It takes the XML test as input and returns a TestData object.
  4. find_tests scans a comment string for all of the XML blocks of test data. The tests are returned as an array of strings, with each string containing one <test>...</test> block.
  5. generate_tests takes as input the name of the C file and processes the file, looking for tests and rebuilding the file with the test cases integrated. This function is the main entry point of the generator.
  6. By setting the prototypeClass member variable of the LanguageScanner, you are telling the LanguageScanner to use your class instead of Prototype when building new prototype objects. Your prototype objects have the tests member, which stores the array of tests associated with the prototype.
  7. After you have scanned the C file, you need to take each prototype and scan the comments for tests. Start by iterating over all the prototypes in the scanner by using the each method on the array of prototypes.
  8. If tests don’t have names, you create pseudo-random test names using the name of the function and an ordinal value.
  9. Setting prototypes in the local scope to the prototypes array within the LanguageScanner means the template can get to the variable using just prototypes. The template can get to your local variables because you use the binding method to give the template access to your locals.
  10. This creates the string that contains all the test comment markers and the result of the template.
  11. This code either finds the comments that bracket the test code and replaces them with the updated code, or if the comments cannot be found, the comments and the tests are appended to the end of the file.

Next we turn our attention to the template that both builds the tests and the function that holds the tests. The main body of the template is the run_tests C function. The template iterates over the prototypes and, within each prototype, it iterates over all the test data associated with that prototype. It’s within this inner loop that we create the if statement that invokes the function with the test data and checks the result.

7.3.5. Finding tools for test cases

Unit testing is becoming increasingly popular, thanks to the advent of the XP programming style and its reliance on unit testing. As such, few unit test generators are available. At the time of this writing, the only one we could find was the JUnitDoclet open source generator for Java’s JUnit testing framework (see www.junitdoclet.org/features.html).

In the next section, we’ll look at a version of this generator that pays attention to sequence.

7.4. Technique: the ordered test generator

The case study generator illustrates a test system which works well against an API that does not require sequential ordering for the tests. But what about testing an API which requires that the functions or methods be called in a particular sequence—where values from one call are passed on to subsequent calls? Sequence-based test systems can also be built using a code generator.

One approach is to use a test data file with an ordered set of data, in which special mark-up can be used to show where data should be stored in variables and passed on to subsequent routines. Here is an example of a test data set for a sequential test generator:

:ret1,add,1,2
5,add,:ret1,2

The file format is a comma-separated file format. The first field is the return value. If the return value is a number (e.g., 5), then the function should return that value. If the return value is specified using a :name, then the return value should be stored in a variable with the specified name.

The second field is the function name. In both cases in the example, the function name is add. The remaining fields are the arguments to the function.

Here is the code generated for the input file:

// Test 1
int ret1=add(1,2);

// Test 2
if ( add( ret1, 2 ) != 5 ) { return 0; }

To implement the first line of the file, we have run the add function with the arguments 1 and 2. The output of the function is stored in the ret1 variable. The second line also calls add, but this time it passes in ret1 and the value 2. We then check the output against the value 5 to make sure we are adding properly.

This is a simplistic example, but it demonstrates the basic technique for building a sequential test data set and lets you see what the resulting code will look like.

7.4.1. Roles of the generator

The roles of the ordered unit test generator are the same as the roles of the unit test case study generator, as described in section 7.3.1.

7.4.2. Laying out the generator architecture

To build the test code that fits within the test framework, you’ll use a tier generator model. We chose the tier model because you are building fully self-reliant code from an abstract model—the test data set. You take the ordered test data set as input. Then, using a set of invocation templates, you build the test code, which sits inside a test framework (e.g., JUnit, CPPUnit, Ruby/Unit). Figure 7.4 shows the block I/O architecture.

Figure 7.4. The I/O flow for a sequential test generator

In the next section, we will walk through the steps of this ordered generator as it executes.

7.4.3. Processing flow

Ordered test generator

Here is the process flow for the ordered generator:

  • Reads the test data input file and stores it locally.
  • Creates a buffer that will hold the test code.
  • Iterates through each test in order and follows these steps:

    • Inspects the return value to see if you are going to check against the return value or store the return value. If you are storing the return value, it runs the template to create the variable and selects the proper template to handle the invocation. It stores the variable creation code, if you created it, in the test code buffer.
    • Runs the invocation handler that builds the call to the function with the right arguments and either stores or inspects the output.
    • Stores the result of the template in the test code buffer.
  • Uses a template that provides a framework for the test code and adds the test code to the framework.
  • Stores the final product in the output file.

At the end of the process, you will have test code that allows for sequential API access.

7.5. Technique: the test data generator

Testing a function or class well means running a lot of tests on it with a variety of data. Ideally you would use data that stresses the target code in a number of directions. The data would test the positive function of the API by sending in valid data. In addition, it would test negative conditions by sending in bad data or by calling the API out of sequence. The mechanism should also test the API under load by sending in large amounts of data.

The test data generator randomly generates test data that can then be used to test against the target class or function. Ideally, you would create the random output once and store it so that it could be predictably used over and over again against the same target. Randomly generating data for each test is troublesome for two reasons. First, you will not get the same coverage of test cases each time. Second, you will get unpredictable errors as a result of differing test data each time the test is run.

7.5.1. Roles of the generator

In order to define the physical architecture of the test data generator, you have to start by defining the role of the generator and declaring with certainty what the generator is responsible for—and what it is not responsible for. It has one main responsibility: creating test data sets for the application.

And here are the tasks this generator is not responsible for:

  • Building the test data loader.
  • Building I/O functions or libraries to read the test data.
  • Validating test data.

With these guidelines in mind, you can now design the architecture of the generator.

7.5.2. Laying out the generator architecture

You can define the test data generator as a tier generator because it uses an abstract model of the test data and the test definition, along with some data and data templates to create the finished test data.

The generator takes as input a test definition file. This file contains information about how much test data should be created, how many test data types there are, and what each test data type looks like.

To create the data, the generator uses a set of data pools, from which it randomly selects data elements. The generator then merges the input data with data templates, which are used to format the data for output.

The data pools can be sets of names or addresses, or just different types of strings or numeric values—any set of data from which you could derive some interesting tests. The block I/O flow for the test data generator is shown in Figure 7.5.

Figure 7.5. The I/O flow for the generator, which builds data for test sets

The next section shows how this generator works.

7.5.3. Processing flow

Test data generator

Here are the process steps for the test generator:

  • Reads the test definition and stores it locally
  • For each test data type:

    • Builds enough data for the amount of tests specified by selecting data randomly from the data pools for each of the fields
  • Sends the data to the data templates for output formatting
  • Stores the output in the test data file

7.5.4. Using the test data generator and the sequential test generator together

Figure 7.6 shows how the test data generator can be used in conjunction with the sequential test generator. The test data generator builds test data sets, which are then fed to the sequential test generator to create test code within the test framework.

Figure 7.6. How the test data generator feeds the sequential test generator

You could argue that the generators should be merged, but by separating the data generation system from the test implementation system you extend the value of both systems. The test data generator can be used to build data for other systems, such as database loaders, while the sequential test generator can be used to find particular edge cases in either random data sets or test sets that are hand-coded.

7.6. Technique: the test robot generator

When it comes to user interfaces, web applications are easier to test than desktop applications. To test a desktop application through the user interface, you need a framework that can shield your test code from the system-level keyboard and mouse event generation used to implement the test. To test a web application through the user interface, all you need is an HTTP access library.

Applications that walk through a web user interface are called robots. An example robot for an online auction system may follow these steps:

  • Fetches the main page and checks for the login form.
  • Sends the login information via a POST.
  • Gets and stores the session cookie.
  • Requests the main page and parses the HTML to find the available items.
  • Iterates through each item and fetches its page. It checks each page for the proper data values against the database.

Each of these is a complete system test that makes use of the entire technology stack, from the database access layer through the persistence layer and the user interface. If a robot says your system is responding properly, then you are likely in good shape.

Robots have other uses as well:

  • Stress-testing an application by sending lots of requests in a short period of time
  • Random “monkey-testing” an application by fetching pages out of order
  • Argument-testing an application by sending invalid or missing arguments from the POST or GET

When building robots you should insulate your test code from the HTTP transit code required to get the page, post requests, or parse responses. The generator in section 7.5.1 builds interface wrappers that can be used by robots to post data to the web application or to fetch pages and parse their data.

7.6.1. Roles of the generator

Test systems become a source of infinite features once engineers understand that they no longer need to tediously hand-test their code. To keep that from happening, it’s ideal to lay down a set of responsibilities for the test system and its generator. The test robot generator has one main responsibility: building automated tests for the pages built to the predictable standards of the web user interface for the target application. Here we’ve listed those tasks the generator does not take responsibility for:

  • Documenting the tests.
  • Testing interfaces that extensively use client-side functionality, such as JavaScript or Flash; the automated testing agents cannot access or test that code because the test agent is not a complete web browser.

With these guidelines in hand, you can continue on to designing the architecture of the generator.

7.6.2. Laying out the generator architecture

The web generator in Figure 7.7 builds interface wrappers for your robots. The wrapper hides the web transit logic from the robot, leaving the robot free to simply specify which pages are to be called, with what arguments, and in what order. Multiple robots, serving different purposes, can use these wrappers. The robots may load data, stress test, or bridge between the application and other technologies, such as instant messaging.

Figure 7.7. A generator that builds robots for testing

Using page definitions, the generator builds wrappers that handle posting arguments and then parsing the returned HTML from the server. The page definition file must have this data about each page:

  • The page URL
  • The names and types GET or POST arguments
  • The structure of the data returned on the page

You can make it easy for the robots to find what they need by adding comments to the pages they will read. You can also bracket the important data on the page in DIV or SPAN tags, which are invisible to the user but are easily found using regular expressions. By targeting the page parser on specific DIV or SPAN tags within the page, you ensure that the robot will not break if the page structure changes for aesthetic reasons.

Let’s go back to the online auction system we discussed earlier. The first thing the robot does after logging in is to see what is on sale. Here is some example HTML for the “What is on sale?” page:

<html><head><title>What is on sale</title><head>
<body>
Here is what is on sale today (<div id="today">1/23/03</div>):<br><br>
<table>
<tr><td>Name</td><td>Price</td></tr>
<tr>
<td><a href="item.cgi?id=2601"><div id="name_2601">Java In A Nutshell</
div></a></td>
<td><div id="price_2601">$26.95</div></td></tr>
<tr>
<tr>
<td><a href="item.cgi?id=2602"><div id="name_2602">Perl In A Nutshell</
div></a></td>
<td><div id="price_2602">$35.00</div></td></tr>
<tr>
<td>

If we use a regular expression that reads the DIV tags and gets the ID and contents, we get the following ordered pairs of information:

today   1/23/02
name_2601    Java In A Nutshell
cost_2601    $26.95
name_2602    Perl In A Nutshell
cost_2602    $35.00

This is the data that is important to the robot. The rest of the page is aesthetic information, which is important but not for this type of testing. The DIV tags insulate our robot from any changes to the aesthetic look of the page. The graphic designers can make any changes they like, so long as the DIV tags remain.

Next we’ll look at how the robot goes about its work.

7.6.3. Processing flow

Test robot generator

The web robot page wrapper generator follows these steps:

  • Reads in the page definitions and stores them locally.
  • Goes through each page and follows these steps:

    • For a form page, builds a POST or GET wrapper to fill in the form variables.
    • For a report page, builds a page-parsing wrapper that reads and returns the contents of the SPAN or DIV tags.

Once this process is completed, you will have a set of wrappers that your robot can use to walk the site and that provides a level of abstraction between your robot logic and the layout of your site.

7.6.4. Generators for robots

Our wrapper generator merely built the interface wrappers for web pages that allows robots to walk our application with ease. Another type of generator can generate the robots themselves, using a dictionary of user actions. Here are two open source generators that build web robots:

7.7. Finding a tool to do it for you

Unit tests fit within test harnesses. These harnesses provide reporting capabilities, test data generation capabilities, as well as other utilities useful in building unit tests. The following list shows some off-the-shelf testing tools for unit tests:

7.8. Design tips

Here are some suggestions for designing your code so that you can build testing tools using generators:

  • For interface code, consistency is king. Preferably all of your pages will be generated to make sure that GET and POST arguments are named consistently and processed the same way across the interface.
  • For APIs, make sure your accessors and instance variables all use the same naming conventions. Also, your return types should be standardized across the interface.
  • Use JavaDoc or special comments that the test case builder can use to configure itself to handle special API cases such as Singletons.

7.9. Summary

The code base of any application is a valuable asset to the individual or company who owns or maintains the application. To maintain the code base long-term, software engineers have started to refactor code continuously. The term refactoring refers to the discipline of reworking the architecture of the application to match the current needs. As applications progress, you often find that the design decisions made in the beginning of the project are no longer valid, which means you have to refactor the code so that it will elegantly solve the current problem.

Unit tests are central to refactoring. If you have unit tests that test your component using test data and responses that match the current requirements, you can change the implementation of the component at will. As long as the output of the component still passes the unit tests, you can be reasonably sure the code is correct.

The key is having the unit tests in the first place. Having unit tests means that you can refactor. Refactoring means that you can keep the code base fresh with constant maintenance.

Using code generation for unit tests makes it easy to maintain and extend your unit tests. In this way, you avoid the common pitfall of not maintaining the unit tests or even failing to create them in the first place.

In the next chapter, we’ll tackle a generator that embeds SQL into another language.