The sink is full of washing, I am wearing odd socks and I haven’t been out of the house in days. It must be time to put out that new release. But how can I be sure my testing hasn’t missed a hideously embarrassing bug? Maybe I introduced a major bug when I made that ‘cosmetic’ change at 2am?
In an ideal world I would just run a comprehensive automated regression test suite. Unfortunately it is difficult to automate graphical user interface (GUI) testing and the majority of lines of code in most applications are GUI. I estimate that the code for my own table planner software is at least 75% GUI code (not including generated code, which would push it even higher).
So I try to manually execute every line of my application before I release it. If I have to make any changes to the code, I start over again. This is very dull, but at least I have a tool to help me: Coverage Validator. Coverage Validator instruments code and shows, in real time, which lines have been executed. Click a few buttons on your application and watch the executed lines of code change colour from pink to yellow. Execute every line in the file and all the lines change colour to cyan. No recompilation or relinking is required and it doesn’t slow down the tested application too much. This real-time feedback is incredibly powerful for testing.
Unfortunately it also has a lot of shortcomings:
- The usability isn’t great. There is a confusing plethora of options for instrumenting your code that I would rather not have to know about.
- It isn’t able to ‘hook’ (instrument) all the lines of code. Whole blocks get missed out for reasons I don’t fully understand. Single line branches are particularly likely to be missed.
- The GUI isn’t great. For example, the display flashes horribly if you resize it.
- The automatic results merging is just plain weird. At the end of a session it can merge your coverage results into a previous session. This information isn’t much use to me at the end of a session. I want to merge previous results at the start of a session so I know which lines I haven’t tested.
- The GUI is quite ugly. They really need to update those tired old icons.
However being able to see line coverage information in real time is just so incredibly useful that I am prepared to put up with the many shortcomings. I just run my application alongside Coverage Validator and, file-by-file and function-by-function, I try to turn the lines of code yellow (or, better still, cyan). Every time I have used Coverage Validator I have found at least one potentially embarrassing bug that I hadn’t discovered by any other means. The support has also been responsive. It is just a pity about the flaws, without them this would be a ‘killer app’ for testing.
Coverage Validator works with C++, Delphi and VB on Windows NT4, 2000, 2003 and XP. A single licence costs $199. A free 30-day evaluation licence is available.
I am using it on Vista currently, and it seems to work fine.
Good article.Thanks Andy, will check the product. I have only unit tests that cover main invisible part of program. Now it is good temp solution to test GUI part.
Interesting post Andy. Not only am I wearing odd socks, they’re inside out and I’m pretty sure they’re my wife’s.
On the GUI testing front, seperating your code into a Model-View-Presenter pattern and using mocks often helps with those hard-to-reach spots :)
I separate out the presentation and logic using an MVC type approach (using Qt signals and slots), but this doesn’t get around the fact that there is still a lot of code (and potentially bugs) in the presentation layer.
Ps/ Careful about those socks – it might be the thin end of the wedge. ;0)
Make the photo of odd socks of your wife and send it to me :)
RE: “Whole blocks get missed out for reasons I don’t fully understand. Single line branches are particularly likely to be missed.”… Are you running the tool on a debug or release build? Is it possible your compiler is inlining or optimizing away those sections of code, or in some other fashion reordering them so coverage validator doesn’t catch it?
>Are you running the tool on a debug or release build?
Usually release. IIRC their support people did tell me it was something to do with the way VC++ generated object code, but I can’t remember the details.
Are you running the tool on a debug or release build?
Pingback: Using defence in depth to produce high quality software « Successful Software
Pingback: Using defence in depth to produce high quality software « iParag.com
Pingback: CoverageValidator v3 « Successful Software
Pingback: Outsourcing software testing « Successful Software