Every time I write a post for this blog I carefully check it for typos. I then get my wife to proof-read it. She always finds at least one typo. Often there will be whole words missing that my brain must have interpolated when I checked it. I read what I thought I had written. She is unencumbered by such preconceptions.

Similarly, it isn’t sufficient to do all your own testing on software you wrote, no matter how hard you try. You will tend to see what you intended to program, not what you actually programmed. Furthermore your users have different experiences, assumptions, and patterns of usage to you. Even in the unlikely event that you manage 100% code coverage in your testing, those pesky users won’t execute those lines of code in the same order you did. I have spent hours testing a program without finding a bug, only to see someone else break it within minutes or even seconds.

So it is essential to involve people other than the original programmer in testing, in addition to (but not instead of) the testing programmers do on their own code. This poses something of a challenge to one-man-bands such as my own. I don’t have other programmers, let alone QA staff, to call on. I can, and do, use volunteer customers for beta testing. But, in my experience, beta testing is not an effective substitute for professional testing:

• It is haphazard. I never hear from ~90% of my beta testers.
• You can’t control beta testers sufficiently, for example you can’t set them tight deadlines, make them concentrate on a particular feature or do their testing on a particular operating system
• The quality of bug reports from customers is often poor. Customers often don’t understand (or don’t have the patience) to describe a bug in enough detail for you to reproduce it.
• Professional testers know how to break software.
• The new release should be as polished as possible before any customers see it. Your beta testers will be some of your most enthusiastic customers. You don’t want to use up that goodwill by sending them buggy software.

Consequently I like to pay third party testers to test my own PerfectTablePlan product after I have finished my own testing and before I do any beta testing. Previously I have used softwareexaminer.com, but they are no longer in business. So I decided to try a couple of other offshore testing companies I had heard about:

	testlab2.com
	qsgsoft.com

The problem with paying a testing company is that it is hard to assess the quality of their work until it is too late. If they report few bugs it could because there are few bugs or because they didn’t do a very good job of testing. By using 2 companies to test the same software release I was also testing the testers (I didn’t tell them this).

I paid each company to do approximately 3 days testing on the Windows and Mac versions of PerfectTablePlan. I was very pleased with the results. Both companies found a useful number of bugs in the software. They were also able to test on platforms that I didn’t have access to at the time (64 bit Windows 7 and Mac OS X 10.6). I didn’t keep an exact score, but I would say that QSG found more bugs, while TestLab2 was more responsive.

QSG found some quite obscure bugs. They were even able to tell me how to reproduce a very rare and obscure bug that I had been trying to track down for months without success. Communications were sometimes a little slow (at least partly due to us being in different time zones) but it wasn’t a huge issue. My only real grumble is their billing. Despite several reminder emails from me I am still waiting to be invoiced for the work several months later. I like to pay my bills promptly and then forget about them.

TestLab2 didn’t find quite as many bugs, but I was impressed with their responsiveness. They installed Mac OS X 10.6 within a few days of it being released, so they could test PerfectTablePlan on it. When I emailed them on  a Saturday about a last minute bug fix for Mac OS X 10.6 they tested the fix the same day. That is great service.

TestLab2 and QSG are based in Ukraine and India, respectively. At around $15/hour they are about a third the price of equivalent US/European companies I contacted (who might also outsource the work to Eastern Europe and India, for all I know). Some people believe outsourcing work to countries with lower costs of living is evil. I’m not one of them. I sell my software worldwide and I am also happy to buy my services worldwide, especially if I can get significantly better value for money by doing so. While there are rational arguments to be made about problems caused by differences in culture, language and time zone caused by outsourcing to other countries, I didn’t find any of these to be a major issue in this case. Most of the other arguments I have heard boil down to the simple ugly fact that some westerners feel they are entitled to a disproportionate share of the global pie. But I don’t see any reason why someone in Europe or North America is any more deserving of a job than someone in Ukraine or India.

With the help of these two companies I was able to put out a really solid PerfectTablePlan v4.1.0 release, despite the large number of new features. In fact, I am only just putting out a v4.1.1 with some bugs fixes several months later. I plan to use both companies again. I hope readers of this blog will give them some additional work to ensure they stay in business. But not so much that they don’t have time to do my next round of testing!