Outsourcing software testing

Every time I write a post for this blog I carefully check it for typos. I then get my wife to proof-read it. She always finds at least one typo. Often there will be whole words missing that my brain must have interpolated when I checked it. I read what I thought I had written. She is unencumbered by such preconceptions.

Similarly, it isn’t sufficient to do all your own testing on software you wrote, no matter how hard you try. You will tend to see what you intended to program, not what you actually programmed. Furthermore your users have different experiences, assumptions, and patterns of usage to you. Even in the unlikely event that you manage 100% code coverage in your testing, those pesky users won’t execute those lines of code in the same order you did. I have spent hours testing a program without finding a bug, only to see someone else break it within minutes or even seconds.

So it is essential to involve people other than the original programmer in testing, in addition to (but not instead of) the testing programmers do on their own code. This poses something of a challenge to one-man-bands such as my own. I don’t have other programmers, let alone QA staff, to call on. I can, and do, use volunteer customers for beta testing. But, in my experience, beta testing is not an effective substitute for professional testing:

  • It is haphazard. I never hear from ~90% of my beta testers.
  • You can’t control beta testers sufficiently, for example you can’t set them tight deadlines, make them concentrate on a particular feature or do their testing on a particular operating system
  • The quality of bug reports from customers is often poor. Customers often don’t understand (or don’t have the patience) to describe a bug in enough detail for you to reproduce it.
  • Professional testers know how to break software.
  • The new release should be as polished as possible before any customers see it. Your beta testers will be some of your most enthusiastic customers. You don’t want to use up that goodwill by sending them buggy software.

Consequently I like to pay third party testers to test my own PerfectTablePlan product after I have finished my own testing and before I do any beta testing. Previously I have used softwareexaminer.com, but they are no longer in business. So I decided to try a couple of other offshore testing companies I had heard about:

testlab2.com
qsgsoft.com

The problem with paying a testing company is that it is hard to assess the quality of their work until it is too late. If they report few bugs it could because there are few bugs or because they didn’t do a very good job of testing. By using 2 companies to test the same software release I was also testing the testers (I didn’t tell them this).

I paid each company to do approximately 3 days testing on the Windows and Mac versions of PerfectTablePlan. I was very pleased with the results. Both companies found a useful number of bugs in the software. They were also able to test on platforms that I didn’t have access to at the time (64 bit Windows 7 and Mac OS X 10.6). I didn’t keep an exact score, but I would say that QSG found more bugs, while TestLab2 was more responsive.

QSG found some quite obscure bugs. They were even able to tell me how to reproduce a very rare and obscure bug that I had been trying to track down for months without success. Communications were sometimes a little slow (at least partly due to us being in different time zones) but it wasn’t a huge issue. My only real grumble is their billing. Despite several reminder emails from me I am still waiting to be invoiced for the work several months later. I like to pay my bills promptly and then forget about them.

TestLab2 didn’t find quite as many bugs, but I was impressed with their responsiveness. They installed Mac OS X 10.6 within a few days of it being released, so they could test PerfectTablePlan on it. When I emailed them on  a Saturday about a last minute bug fix for Mac OS X 10.6 they tested the fix the same day. That is great service.

TestLab2 and QSG are based in Ukraine and India, respectively. At around $15/hour they are about a third the price of equivalent US/European companies I contacted (who might also outsource the work to Eastern Europe and India, for all I know). Some people believe outsourcing work to countries with lower costs of living is evil. I’m not one of them. I sell my software worldwide and I am also happy to buy my services worldwide, especially if I can get significantly better value for money by doing so. While there are rational arguments to be made about problems caused by differences in culture, language and time zone caused by outsourcing to other countries, I didn’t find any of these to be a major issue in this case. Most of the other arguments I have heard boil down to the simple ugly fact that some westerners feel they are entitled to a disproportionate share of the global pie. But I don’t see any reason why someone in Europe or North America is any more deserving of a job than someone in Ukraine or India.

With the help of these two companies I was able to put out a really solid PerfectTablePlan v4.1.0 release, despite the large number of new features. In fact, I am only just putting out a v4.1.1 with some bugs fixes several months later. I plan to use both companies again. I hope readers of this blog will give them some additional work to ensure they stay in business. But not so much that they don’t have time to do my next round of testing!

19 thoughts on “Outsourcing software testing

  1. Manfred Ekblad

    What a great idea to test two different software testing companies at the same time with the same software.

    There is definitely a market for this kind of services and the outsourcing industry keeps growing. I agree that there is no “evil” in outsourcing, as long as the staff is treated well and get proper salaries.

    Currently I’m in Indonesia, managing a small team of extremely talented web designers and developers. It’s a great country to run a company in but there are a lot of things you need to know to be successful.

  2. Bob Tipton

    I’ve in total agreement with Andy on this idea. I’ve used both of the companies mentioned and been pleased with the results every time. In fact, I just finished a round of testing with TestLab2 on a program that’s scheduled to be released for the first time in the next couple of weeks. Even though I had tested all of the use cases myself before I sent the program to them, they still found 1 major bug and several minor ones. The money I’ve paid to have these bugs found *now* is a bargain compared to trying to identify them from angry user reports later.

    I also agree with Andy’s assessment of beta testing – while you will often get good results, it takes a lot of time. As a 1 or 2-person company, time is the one thing you really don’t have enough of. I really believe you can get a quality product launched faster by using paid testers, rather than volunteers, at least for a new product. If you’ve got a pool of loyal users that are willing to test your next release, then by all means use them, but if you’re launching a brand new product, I think your time is better spent dealing with a paid testing group rather than trying to recruit and manage volunteer testers.

  3. Pradeep

    “.. some westerners feel they are entitled to a disproportionate share of the global pie.” – Very true.

    If a computer can come from China.. clothes can come from India.. why not software? not a big difference

  4. Tim

    Hi,

    Nice article!

    If you don’t mind would you mind telling me how you found these guys and then ultimately the reasons as to why you chose them?

    The reason I ask is I’m an UK expat living in NZ, 12 years industry Test and BA experience and could compete (almost) with their rates as I’m working part-time at home currently anyway.

    Many thanks,
    Tim

  5. sensei

    I agree with you on that putting some 3rd party testers who are not familiar to your software will add some value to your software , but you said you do not believe the beta phase in testing contributes enough and only 10 % of the beta testers work on it, so it is a percentage , the problem might be your not reaching sufficient number of beta testers which will make that 10% a huge number of people. my view after alpha testing , you need to do the beta phase then if you are suspicious about the software before releasing , then go and pay extra money for the 3rd party testers.
    Best Regards

  6. Andy Brice Post author

    >you said you do not believe the beta phase in testing contributes enough

    On the contrary, I think beta testing is very useful. I said “beta testing is not an effective substitute for professional testing”. Ideally you should do both (I did for PerfectTablePlan v4.1).

  7. chillbyte

    Andy,That was an excellent post I stumbled upon, and they are surely getting my business. Woudl you be able to provide a ball park of how much it cost you in total, and did you ask for load testing as well where they bulk load into your app and such to see if it break? Appreciate it mate!!

  8. simi

    First of all. Thanks very much for your useful post.

    I just came across your blog and wanted to drop you a note telling you how impressed I was with the information you have posted here.

    Please let me introduce you some info related to this post and I hope that it is useful for software testing community.

    There is a good Software Testing resource site, Have alook

    http://SoftwareTestingNet.com

    Thanks again
    simi

  9. James Clark

    Great article!

    I have just setup my own outsource software testing company http://www.testing4success.com and would love it if anybody would care to test my services.

    Being an experienced tester of over 12 years, I know the importance of good quality testing, value for money and efficiency. Feel free to drop me a line if you need a free quote.

    Maybe one day someone will write an article based on my company :)

    James

  10. Alexander

    Interesting article!

    Not so far I have gain a group of QA engineer and we are providing outsourcing software testing as a part of http://www.massqa.com company. We provide free testing for interesting projects.

    I would be very appreciate if somebody write article about our company sometimes. Please let me know if your application need to be tested :)

    Alexander

  11. Marc

    We have been outsourcing our development services to people in south america.
    The first thing you approach is that it is cheaper than hiring people here but many times we had problems with the testing face of the development teams.
    We decided to hire both, development and testing outsourcing services.

    http://www.testcrowd.com (for testing)
    http://www.bairesdev.com (for development)

    Marc

  12. Kate

    The trend nowadays is to outsource software testing. Offsite software testing companies are everywhere; they’re headquartered in the United States and in technology-rich countries such as India, Singapore and Japan. Company literature boasts the tremendous time and money saving features of outsourcing software testing while stressing high quality work you can count on. But is it really beneficial to outsource software testing? There’s no standard answer to this potentially expensive proposition.

Comments are closed.