I rarely post any articles on tech-topics but this one is good. The computer industry has a long way to go before it can achieve the quality standards of the automotive industry. Till that time, it will be good if Gates keeps his trap shut.
When You Test My Code, Be Gentle
By Rob Keefer/30-January-2001
At a client site recently, I was in a meeting where the facilitator was describing a new defect management tool. To make the meeting tolerable, he brought in a familiar story about when Bill Gates stated: "If GM had kept up with technology like the computer industry has, we would all be driving a $25 car that got 1000 miles to the gallon." The story goes on to give the response General Motors supposedly released comparing the auto industry with the computer industry. What if the auto industry had kept in step with the computer industry? For no reason whatsoever, your car would crash twice a day. Every time the lines on the road were repainted you would have to buy a new car. New seats would force everyone to have the same size butt.
A day or two later I was talking with a software developer from another department. During our conversation he said, "Programming is a lot of fun until testing gets involved." Then he realized I would be responsible for testing his software. As our discussion wound down and I turned to go, he said, with a timid voice, "When you test my code, be gentle."
Be gentle?
Have you ever paid much attention to the car commercials on television? They don't talk about their engineering or manufacturing process. They tell you that their cars are crash tested to ensure that their design is good enough for you. That is not gentle. The advertisements send a strong message: "We put our vehicles through grueling tests to provide a quality product for you."
Now think about the common television commercials of the computer industry. They essentially say, "We are doing cool stuff with computers." Implying that you can do `cool stuff', too. No software company is going to talk about the reliability of their system because, generally, the reliability isn't anything to brag about. In the November 2000 issue of Scientific American, Peter Forman and Robert Saint John point out that "even a cheap TV doesn't crash or freeze. The best computers still do."
What the automotive industry knows, and the software industry needs to learn, is how to do rigorous unit testing. Unit testing involves testing each function individually to verify that it works under many different circumstances. When a prototype for a new pressure hose for use in an automobile is tested, it is connect to an input-valve and an output-valve. The input-valve allows adjustment to the pressure going in; the output valve measures the pressure coming out of the hose. Other devices measure the stress, heat, etc. on the hose itself. Only after subjecting the prototype to harsh tests under extreme pressure is it integrated with the rest of the vehicle for further tests.
Similarly, a software product can have each component tested to ensure that it is working correctly and is reliable. Some suggest that these unit tests be written before the actual code is written to ensure that all the different circumstances which could break the function are, indeed, covered.
When a pressure hose it is tested, it is connected to a device that can generate high pressures to ensure a thorough test. Similarly, in order to test an isolated function in a program, you will need to create a driver program. This driver program will exercise all of the features of the function. If a function calls another function that does not yet exist, you will need to write a stub, an empty function, to fill in for the called function. When the stub function is flushed out, you may be able to use the original driver program to test this function as well.
For example, a driver program for testing a function that returns the sum of two positive integers will call the function with positive and negative numbers and zero. The driver program should also account for the specifics of the computer. If you know that the positive integers were declared with 8 bits, you should test with a large value to ensure the function handles these values gracefully. When you are finished testing the function, do not throw away the driver program. You may want to use it again on modified versions of the function. A good collection of drivers is a great asset when it comes to testing.
Here are a few tips for creating a good unit test.
How do you eat an elephant? One piece at a time.
If your code is structured properly, it should be fairly easy to isolate individual function calls. To isolate a function, it should only contain calls to other functions that have been unit tested or to stubs that return a known value. Your stub should return valid and invalid values so you can see how your function will respond to different stimuli. This way, your testing is focused, and you can be confident in your test and in the code.
See the world!
Programs can become complex quickly, and you need to have a plan to handle the complexity. When unit testing a function, map out the different paths that the function may follow, and test each path. At times, this may not be a reasonable task; it is hard to test all types of error handling. You should, however, be able to check most of the paths through your function with just a few tests. If your function requires many tests to ensure that each path is checked, the function may be too complex and you should rewrite it.
Push it to the limit.
I'm sure you have written code that works for a specific scenario you had in mind. A short time later you found out that the function didn't account for all of the possible values that it should have. You fixed the code to accommodate for the newly discovered values, but probably had a haunting feeling as you wondered what else you might have missed.
When you test a function you need to see how your function responds to many different values that could be passed as parameters. For example, if you have a function that takes one integer as a parameter, and your program assumes that the integer is between 0 and 9, you should also pass -1 and 10 to see how the function responds to values outside of the given boundary.
Unit tests, which are focused on the individual tasks of a function, test every path through a function, the extremes of the function, and can take some time to develop. By spending a little extra time developing thorough unit tests to test your code as you develop it, you can be more confident that your code will handle the tests that the testing group will run. Instead of building a tower of cards, you will build a structure that will stand up to the rigor actual use requires.
Automobiles contain very complex systems that have been developed independently of each other. The key behind much of the improvement we have seen in the auto industry over the past 20 years is that the engineers have begun to test each component's design as the component is built. When each of these high quality components are assembled together, you have a high quality automobile.
Once our industry begins to build more robust software, we may be able to throw our own jabs at the automotive industry. Until that time comes, however, I will be happy to let GM continue to build "low tech" cars, while the software industry tries to catch up with the auto industry's higher standards of quality.
Copyright 2001 Specialized Systems Consultants, Inc.
