We all like to think that our software is easy to use. But is it really? How do you know? Have you ever watched anyone use it? When I asked this questions to a room full of developers last year I was surprised at how many hadn’t.
Other people don’t see the world the way you do. Their weltanschauung (view on the world) is influenced by their culture, education, expectations, age, gender and many other factors. Below is a copy of a card I received for my birthday a few weeks ago (click for a larger image) which I think illustrates the gulf between how developers and their customers see the world rather well.
If your customers are also developers the difference in backgrounds may not be so large. But the difference in how they see your software and how you see it is still huge. You have been working on your software for months or years. You know everything worth knowing about it down to the last checkbox and command line argument. But your potential customer is probably going to download it and play with it for just a few minutes, or a few hours if you are lucky, before they decide if it is the right tool for the job. If they aren’t convinced, your competitors are only a few clicks away. To maximise your chances of making a sale you need to see your software afresh through your customer’s eyes. You can get some useful feedback from support emails, but the best way to improve the ease of use of your software is to watch other people using it. This is usually known as usability testing.
The basic idea of usability testing is that you take someone with a similar background to your target audience, who hasn’t seen your software before and ask them to perform a typical series of tasks. Ideally they should try to speak out loud what they are thinking to give you more insight into their thought processes. You then watch what they do. Critically, you do not assist them, no matter how irresistible the urge. The results can be quite surprising and highly revealing. Usability testing can be very fancy with one way mirrors, video cameras etc, but that really isn’t necessary to get most of the benefits. There is a good description of how to carry out usability tests in Krug’s excellent book Don’t make me think: a common sense guide to web usability. Most of his advice is equally applicable to testing desktop applications.
The main problems with usability testing are logistical. You need to find the right test subjects and arrange the time and location for testing. You also need to decide how you are going to induce them to give up an hour of their time. Worst of all, once you have used someone they are ‘tainted’ and can’t be used again (except perhaps to test changes in the new versions). It’s a hassle. Or at least it was. Much of this hassle is now taken care of for you by new web-based service www.usertesting.com .
The idea behind usertesting.com is very simple. You buy a number of tests for your website and specify your website url, the tasks you want carried out and the demographics (e.g. preferred age, gender and expertise of testers). Testers are then selected for you and carry out the testing. Once tests have been completed a flash audio+video recording of the session and a brief written report is uploaded for you. Finally you rate the testers on a 5-star scale. Presumably testers who score well will get more work in future. Ideally you should re-run your usability testing after any changes to verify that they are an improvement. I don’t know if usertesting.com allows for the fact that you probably won’t want the same tester a second time for the same project.
I paid $57 for 3 tests on perfecttableplan.com. I was happy with the tests, which pointed out a number of areas I can improve on. There was a problem which meant one of the tests still hadn’t been completed 4 days later. I emailed support and they sorted this out in a timely fashion. It is a new service and they are still ironing out a few glitches. Given the low costs and the 30 day money back guarantee I think it is definitely worth a try. It won’t take many extra conversions to repay your investment. usertesting.com is probably more useful to those of us selling to the wider consumer market. If you are selling to specialised niches (e.g. developers, actuaries, llama breeders) they might have difficulty finding suitable testers.
Unfortunately usertesting.com is currently only available for website usability testing. When I emailed them to suggest they extend the service to desktop apps they told me that it this might be a possibility if there was sufficient interest. I will be first in-line if such a service becomes available. Until then I am left with the hassle of organising my own usability tests. It occurs to me that I could do this remotely using a service such as copilot.com (now free at weekends)+Skype. This might be a good workaround for the fact that my office isn’t really big enough for two people (especially if they don’t know me very well!). It would also allow me to do testing with customers outside the UK, e.g. professional wedding planners in the USA. If I do try this I will report back on how I get on.
Have you looked at http://www.techsmith.com/morae.asp?
It would seem to help with the logistical side, and looks like the CoPilot based solution but with everything packaged up for you.
I don’t know how much it costs, but all the other TechSmith products are excellent.
(Great blog and product BTW)
Great article, as usual, Andy! Usability testing is very hard indeed. And as you say, finding quality subjects is even harder.
What we’ve done in the past at LandlordMax is tried to single out customers that gave us the best feedback on how to improve the software (not necessarily quantity, but rather quality). And those that we did single out, we treated well. Sometimes it was free upgrades, sometimes other perks. Whatever it was, we made sure to thank them appropriately. And they were also happy because they got to directly influence the software, which made their jobs that much easier (not all suggestions are weighed evenly).
Of course this is on top of continually compiling and listing all the suggestions we get. For Perfect Table Plan it’s probably a little different, but for LandlordMax we already have about 20 versions worth of features we can add. That’s just the nature of our product. It of course doesn’t apply to every product, and in some circumstances too many feature is a very bad thing. But ignoring that debate, I think I will check out this service. And hopefully they’ll offer desktop usability testing. It would be great to get some specific testing done for new users.
Ah what a sad ending to such a good story, I was all ready to dish up my $60 until you said it was websites only :(. Might not be a bad mISV idea for someone to come up with, I’d gladly pay too!
I think Morae is designed for face-to-face usability testing and it isn’t cheap at >$1000 for the suite.
TechSmith’s SnagIt is one of my favourite software tools.
Feedback fromn ‘power’ users is very useful. But I think you need ‘virgin’ users for what most people would consider usability testing.
Email them. They might bump it up the priority list if they get a few more emails. It can’t be that difficult for them to do (easy for me to say!) – they just need to include a VM in their test environment.
Pingback: Getting customer feedback « Successful Software
Thanks. We had usertesting carry it out for our website also and the results were incredibly useful. Especially, the user videos which shows the users testing your website.
Pingback: Easy screen sharing with Skype « Successful Software
“When I emailed them to suggest they extend the service to desktop apps they told me that it this might be a possibility if there was sufficient interest.”
This article has 2 years old now. Would you think that they have some plans for desktop app testing?
Outsourcing this service would be nice. :)
Pingback: 10 things non-technical users don’t understand about your software « Successful Software