Category Archives: article

Lessons learned from 13 failed software products

“No physician is really good before he has killed one or two patients.” – Proverb

Software entrepreneur culture is full of stories of the products that succeeded. But what about the products that failed? We rarely hear much about them. This can lead to a very skewed perspective on what works and what doesn’t (survivor bias). But I believe that failure can teach us as much as success. So I asked other software entrepreneurs to share their stories of failure in the hope that we might save others from making the same mistakes. To my surprise I got 12 excellent responses, which I include below along with one of my own. It is a small sample and biased by self selection, but I think it contains a lot of useful insights. It is an unashamedly a long post, as I didn’t want to lose any of these insights by editing it down.

Case #1: DRAMA

Contributor

Andy Brice.

The product

DRAMA (Design RAtionale MAnagement) was a commercialization of a University prototype for recording the decision-making process during the design of complex and long-lived artefacts, for example nuclear reactors and chemical plants. By recording it in a structured database this information would still be available long after the original engineers had forgotten it, retired or been run over by buses. This information was believed to be incredibly valuable to later maintainers of the system, engineers creating similar designs and industry regulators. The development was part funded by 4 big process engineering companies.

Why it was judged a commercial failure

Everyone told us what a great idea it was, but no-one bought it. despite some early funding from some big process engineering companies, none of them put it into use properly and we never sold any licences to anyone else.

What went wrong

Lack of support from the people who would actually have to use it. There are lots of social factors that work against engineers wanting to record their design rationale, including:
- The person taking the time to record the rationale probably isn’t the person getting the benefit from it.
- Extra work for people who are already under a lot of time pressure.
- It might make it easier for others to question decisions and hold companies and engineers accountable for mistakes.
- Engineers may see giving away this knowledge as undermining their job security.
Problems integrating with the other software tools that engineers spend most of their time in (e.g. CAD packages). This would probably be easier with modern web-based technology.
It is difficult to capture the subtleties of the design process in a structured form.
A bad hire. If you hire the wrong person, you should face up to it and get rid of them. Rather than keep moving them around in a vain attempt to find something they are good at.
We took a phased approach, starting with a single-user proof of concept and then creating a client-server version. In hindsight it should have been obvious that not enough people were actively using the single-user system and we should have killed it then.

Time/money invested

At least 3 man years of work went into this product, with me doing most of it. Thankfully I was a salaried employee. But the lack of success of this product contributed to the demise of the part of the company I was in.

Current product status

The product is long dead.

Any regrets?

It was a fairly painful experience. I would rather have spent all that money, time and energy on something that someone actually used. But at least I learnt some expensive lessons without using my own money.

Lessons learned

Creating a new market is difficult and risky.
Changing people’s working habits is hard.
Social factors can make or break a product. The end-users didn’t see anything in it for them.
If the end-users don’t like a product, they will find a way not to use it, even if their bosses appear to be enthusiastic about it.
Talk is cheap. Lots of people telling you how great your product is doesn’t mean much. You only really find out if your product is commercially viable when you start asking people to buy it.

Case #2: CleanChief

Contributor

Sam Howley.

The product

CleanChief was to be ‘The easy management solution for cleaning organisations’. Managing assets, employee schedules, ordering supplies, you name it CleanChief handled it. Essentially it was light weight accounting software for cleaning companies.

Why it was judged a commercial failure

A small number of copies were sold. No one is actively using it at present. Once I realised that it wasn’t a complete product and that additional development was required I moved on to other product ideas. I had basically run out of enthusiasm for the product.

What went wrong

I am not an accountant.
I have never run a cleaning company.
I developed it for more than two years without getting feedback from real cleaning companies. I was arrogant enough to think that I knew what they wanted (or could work it out on my own). Or maybe it was that I was just where I was most happy and comfortable – writing software. Talking to real users was new and to be honest a bit scary for me.
A successful cleaning company operator, a friend of a friend, offered to become involved for a 30% share. This was a gift from the heavens, exactly what I needed. I refused.
In a way, even though I spent so long on the product, I gave in too soon, I was just getting feedback from real users, just getting my first batch of sales when I decided to move on.
I developed the application in VB6 even though I knew it was outdated technology when I started the project.This meant there was no ‘cool factor’ when discussing it with other developers, I told myself it didn’t bother me, but it probably did.

Time/money invested

I worked on it at night and weekends for about 2 1/2 years. I paid for graphic design work, purchased stock icons and images. I probably spent a couple of thousand Australian dollars in total and an awful lot of time.

Current product status

I moved on to other products that have gone much better. My newer products were released in months rather than years and I looked for real feedback from real users from day one. they are:

QueryCell – an Excel add-in making SQL in Excel easy.
QuizNightChief – the easy way to organise a quiz Night.
CustomerCradle – The easiest way to record and report on where your customers come from.

I do occasionally ponder returning to CleanChief and trying to raise it from the ashes.

Any regrets?

No. Looking back I learned a few lessons from a huge amount of time and work, it was a very inefficient way to learn those lessons. But when you are new to something like starting a business or creating useful software being inefficient at learning lessons is the best you can do, it’s a thousand times better than not learning lessons at all.

I learned so much more in my two and a half years of trying to develop CleanChief than I did in the two and a half years prior to that, during which time I really wanted to start a software business but didn’t take any action.

Lessons learned

Hearing or reading some piece of advice is totally different to living it. Here are some of the ideas that I always agreed were true but didn’t fully understand the implications of until I had lived them out:

Force yourself to get out and talk to people. Ask their advice. Almost everyone will help if you ask them for feedback.
Force yourself to cold call a few businesses in your target market.
Create a plan of how to market your product.
Try and use your product as much as possible as you build it.
Get out of your comfort zone from day one
Do not have the mind set that the day you release version 1.0 is the finish line, it’s the starting line, so hurry up and get there.

Case #3: Chimsoft

Contributor

Phil Anderson.

The product

ChimSoft – Software for Chimney Sweeps.

Why it was judged a commercial failure

I believe this failed for two reasons:

Focusing on too small of a niche
Me not being able to work full time on it.

I don’t consider it a complete failure because I sold two copies when it retailed for $2k, and maybe 10-15 more copies when I lowered the price to $200. Those sales proved that I wasn’t completely off base in thinking there was a market for the software, but the cost of customer acquisition and the size of the market were too small. Customers wanted to have a bunch of phone calls, face-to-face etc… the type of stuff you only see with much more expensive software. The problem was that for a niche this small we had to charge a lot of money to make it worthwhile for us, but the customers were small businesses where this is a major investment, so the fit was never right. The other issue was the people that did buy it were not super tech savvy, so there was a high cost of support that made even a $200 product not worth it.

What went wrong

Having all partners who were not full-time, and had equal equity. I ended up doing most of the work and this is the main reason I didn’t force success is I felt I was in it alone.
Focusing on too narrow of a niche. The plan all along was to expand for all service industries, but it was much harder to make that move than we expected.
Not researching pricing more, we knew small businesses made major purchases for things that really helped their business, but I think it would have been better to have a cheaper product with wider appeal than an expensive product with narrow appeal.

Time/money invested

I invested maybe a year of time and $3k into the company. I did not take any huge risks on it, so there were no big negative outcomes.

Current product status

The company folded in 2007, I refocused my efforts on my existing companies (AUsedCar.com and BudgetSimple.com) and both have been doing well enough that I quit my day job.

Any regrets?

I don’t regret it entirely, I think I learned several valuable lessons about working with other people, small business sales, trade-shows and software development.

Lessons learned

Pick partners wisely. Don’t try to be even-steven with equity. Use restricted stock to ensure everyone does their part.
Know what your customers expect (24/7 phone support?) to determine if you can do this while working a day job.

Case #4: PC Desktop Cleaner

Contributor

Javier Rojas Goñi.

The product

PC Desktop Cleaner. Simple software that cleans your desktop and archives your files.

Why it was judged a commercial failure

My goal was to sell 10 units per month. I’ve sold less than 1 unit per month.

What went wrong

I think that the product concept is not useful enough. It’s not a thing that people would pay for.
The market exists (some people buy) but it’s too little or difficult to reach.
I didn’t do any market research. I just got in love with the idea and did it. Later, I’ve learnt to use “lazy instantiation marketing” and have trashed a lot of embryo projects. :-)

Time/money invested

I think I wasted near $500 in development tools and some freelancers. Not too much.

Current product status

I’m still selling it. I’ve thought about others products, but not really decided yet.

Any regrets?

No, it was a lot of fun and I learnt lot of things. In my “day job” I own a small firm that sells software for production scheduling. I’ve learn a lot of SEO and AdWords in the DesktopCleaner project that now I’m using with great results.

Lessons learned

Go for it, maybe you win, maybe you fail, but you will grow and get tons of useful knowledge on the way.

Case #5: Smart Diary Suite

Contributor

Dennis Volodomanov.

The product

Smart Diary Suite.

Why it was judged a commercial failure

It sells and the profits cover current investments in the product, but there is little left over on top of that.

What went wrong

If I had a chance to do anything differently:

Take it seriously from day one.
Never stop developing and supporting.
Invest as much as possible in marketing early on.
Don’t stop believing in your creation.

Time/money invested

Up to this point, I have spent 13 years on Smart Diary Suite and a lot of money went into buying hardware, software, hosting, marketing, etc… All of that money came from my day job, but at this point SDS has recovered all of that back and is now making a small profit. The actual amount is hard to calculate (over the 13 year span), but we would be talking in tens of thousands of US dollars.

Current product status

For a while it may have seemed like SDS is not going to be successful, but that’s probably my fault – I stopped believing for a little while. Now I am back, starting again and this time I’ll make sure it doesn’t fail.

Any regrets?

I do not regret doing it. I regret allowing myself to stop working on it, basically bailing out on it for a while – that is my biggest mistake.

Lessons learned

If you want a successful product – believe in it and let others know that you believe in it.

Case #6: Highlighter

Contributor

Mike Sutton.

The product

Highlighter. A utility to print neatly formatted, syntax highlighted source code listings.

Why it was judged a commercial failure

I earnt a grand total of £442.52 (about $700 in todays money) in just over two years, so I guess it paid for itself if you exclude my time.

What went wrong

Since it was my first product and I was very green about both marketing and product development. I would suggest the following would have made things better:

Get feedback from potential users about the product (eg from the ASP forums). Some parts of the program where probably too option heavy and geeky.
Diversify. If people didn’t want to print fancy listings, maybe they would have wanted them formatted in HTML.
Better marketing. I’m not sure this would have saved it, but all I knew in those days was uploading to shareware sites. I never even sent a press release.

I figure it failed simply because it was a product nobody wanted. Actually, more importantly than that,, it was a product *I* didn’t want to use, but it developed from a larger product I was working on, on the assumption I could earn some money on the side from part of the code. Since then I’ve stuck to products which I’ve actually wanted to use myself. There’s a lot to be said for dogfooding, not just for debugging, but for knowing where the pain points are and what extra features could be added.

Time/money invested

I would guess a couple of months of evening/weekend development time. Financially there was little spent, except that I offered the option of a printed manual and CD for an extra charge. One customer took me up on the offer, so I had to get 100 manuals printed and 99 of them went in the bin.

Current product status

I moved on to another product which has sold over £50,000 and a third which has earnt even more than that. Not enough to retire on but considering I only do this part time it must work out at a great hourly rate. There’s a lot to be said for not giving up…

Any regrets?

Nope. I figure every failure in life teaches you valuable lessons. Of course if I’d made a large financial investment I may feel differently, but that’s one of the big advantages of software over physical product sales.

Lessons learned

Just to reiterate – develop something which you find useful, instead of second guessing others.

Case #7: R10Clean

Contributor

Steve Cholerton.

The product

R10Clean. A data cleaning and manipulation tool.

Why it was judged a commercial failure

In the 18 months or so it’s been on the market I have sold 6. It has been £199, £99 and £19 – with no effect on sales !

What went wrong

Not sure what I did wrong ? The product is maybe too techie ?

Time/money invested

No effect financially as at the time I was in a strong financial position.

Current product status

I still have it for sale but do not market it at all. I have other products.

Any regrets?

I don’t regret it as it saved me a ton of time when I was working with legacy databases a lot, as a commercial product it has been raved about (once!) and received a good review from the Kleper report, but has failed totally.

Lessons learned

Advice to others ? Just because you need it personally, don’t assume the rest of the world does too. :-)

Case #8: nBinder

Contributor

Boghiu Andrei.

The product

nBinder, packs multiple files into a stand alone executable with over 50 advanced output and file unpack options, conditional run and commands.

Why it was judged a commercial failure

It was the first product I began selling. It sold to 300+ customers in 4 years. But for about a year the sales began to go down and have finally stopped completely.

What went wrong

The biggest problem was that because it was a packer intended for people that wanted to pack their products (software or games) into a single package (compressed and encrypted) many have used it for creating malware by binding malware files to legit files and then distributing the output so it isn’t detected by antivrus software (although it would be detected at runtime). Because of this I had lots of problems with antivirus companies that flagged files create with nBinder as malware. This was of course affecting legit users as their files would be falsely marked as malware. I used virustotal.com to see which antivirus detected it and contacted the antivirus manufacturer as soon as I detected the problem. In most cases they would remove it from their definitions. But it was an uphill battle because it would appear again in a matter of weeks. Some small AV companies didn’t event bother to reply to my emails to fix the problem. Others were using heuristics to flag files create with my applications and AV developers were reluctant to whitelist files created with nBinder. You can imagine it that it was enough for an AV such as Kaspersky or Norton to pick my files as malware for a day and customers would be affected and not use my product any more, especially that it took about 3 days for AVs to remove the false positive.
Infrequent updates. Due to lack of time I only updated the product once or twice a year and this affected the product a lot.
No marketing. I decided that I didn’t want to invest money in marketing so, except for a short AdWords campaign, I invested no money in marketing.
My decision to develop 3 products instead of concentrating on one or two affected development time and quality. I have worked on 3 products simultaneously instead of concentrating on making a single good one. The reason I worked on 3 is because I enjoyed developing different software in different categories. I didn’t start this for money but for the fun of development.

Time/money invested

I invested almost no money (except for hosting costs). Time invested I can’t really say exactly, but not too much as I only worked on nBinder in short bursts like 6 hours a day for a week or so before releases.

Current product status

Still for sale. My other products are:

nCleaner – a free system cleaner that has gone quite well (over 2 million downloads).
nMacro – an automation tool that has seen some limited success (bought by over 100 customers in a year or so).

Any regrets?

It’s not a total failure as I did make some money out of it with no investment, so I don’t regret starting it, but it could have been much better.

Lessons learned

Words of advice for others trying to make money from software development:

Study the market and the current trends very well.
Before deciding to take on large competition make sure you have something better (at least from one point of view) than the competition ( for example you might not have the same features but you have a better GUI and general presentation).
Do not get scared of an overly populated market segment. For example with nBinder I picked a segment with very little competition but also few possible users and the results were not so great (I didn’t have many users). With nCleaner I went head-to-head with lots of already established products but also the market is very big. Although nCleaner is free it has had the most success because there are so many potential users (anyone with a PC actually), so it had over 2 millions downloads and I still receive lots of mails regarding it, even if the last update was in 2007. So it is possible to have success in a market with lots of competition with no investment but it’s hard to reach the level of more established products.

Case #9: Net-Herald

Contributor

Torsten Uhlmann.

The product

Net-Herald – a monitoring application for water supply companies. It was a complex client server application that would receive monitoring data from specialized hardware and store that data inside a SQL database. The client displays that data in different graphs, provides printable reports or sends alarm messages via SMS if a monitored value is not within its specified limits.

I developed Net-Herald as a perfect fit for that specialized hardware that is provided by a local manufacturer. That way, so I hoped, I could profit from their sales leads and would find a smoother way into these water supply companies. The downside of course, was that my software would only work with their hardware.

Why it was judged a commercial failure

I sold a first license fairly soon after I had a sellable product, although it took the customer nearly a year until they finally bought. But since then I sold only one more license within the last 4 years or so.

What went wrong

I didn’t do my own marketing and the hardware guys weren’t really concerned with selling my software.
Water management companies have a terribly long sales cycle. Other vendors monitoring applications usually cost tens of thousands and are geared toward large suppliers. Whenever a supplier buys into such a product he is unlikely to change within the next decade or more. I tried to position my software towards small suppliers but even then most of them were already locked into another vendor’s solution.
My software only worked with a specific hardware. That narrowed the marked down substantially.
In the end the software became too complex for one poor mortal to maintain. Because the software didn’t produce any substantial income I had to stop adding new features which would make it attractive for more prospective clients.
This kind of software is not sold over the Internet. Rather it needs very active sales people that nurture clients over a rather long period of time.
All these facts indicate that software like this should not be developed by a one man show.

Time/money invested

The development time for the first sellable version was maybe about 9 months. I didn’t have a job income at that time, but got funding due to government support for small start-up businesses. So I didn’t drain our family’s personal finances. But I did of course invest a great deal of time and sweat.

Current product status

Now, I have drawn a line and stopped active development of Net-Herald. I still do some custom extensions for my first clients. But I no longer market the software. I have instead focused on my consulting services. I also try to learn developing and selling software with my cross-platform drag and drop product Simidude.

Any regrets?

I didn’t succeed yet selling my own software (which is still my goal) but I do not regret doing it. I developed Net-Herald using (Java) technologies that now give me leverage at my consulting gigs. All in all it was a heavy ride. But it was fun and I would do it again.

Lessons learned

My biggest mistake was the lack of market analysis. I trusted the word of the hardware manufacturer without verification.
I have written more about the above and some other failures on my blog.

Case #10: HabitShaper

Contributor

Adriano Ferrari.

The product

HabitShaper – set and track daily targets for your goals (weight loss, quit smoking, jogging, writing, etc…).

Why it was judged a commercial failure

I sold a few copies, but not enough to make back the time I invested in it and my conversion numbers and traffic are below average.

What went wrong

Did not do enough pre-production research (talking to customers, etc).
Did not do a large enough beta to make up for lack of initial research.
Ignored gut-feeling that my product is better suited to being web-based and multi-platform (incl. mobile).
Did EVERYTHING myself (logo, web design, video, software, AdWords, etc).

Time/money invested

I worked on it two years, part-time, while doing Masters/PhD in Physics. It had no impact on my finances (very little money invested) or circumstances.

Current product status

I am relaunching as a web-based product this summer.

Any regrets?

Not in the least! I learned about as much from making HabitShaper as I have from my MSc thesis and PhD work.

Lessons learned

Most important: PAPER prototypes, minimum viable product, and iterate.
Don’t be afraid to launch early.
Launch a little bigger than you’d expect (it’s harder to find those initial customers than you think).
Don’t be afraid to change directions, especially early on.
Doing things yourself is a great learning experience, but if you want to get your product out to customers as fast as possible, don’t be afraid to invest money and outsource your weaknesses.

Case #11: BPL

Contributor

Jim Lawless.

The product

BPL – Batch Programming Language Interpreter.

Why it was judged a commercial failure

I sold about 10 copies.

What went wrong

I didn’t really do enough research to find out if the target market was in existence. I was hoping that network admins and support staff members would find it easier to use than batch files and less complicated than any of the free scripting language options available. So, I just rushed to get the MVP (Minimum Viable Product) out the door.
I never did provide a compiler that would build a stand-alone EXE. I think that might have met with more success.
I didn’t do much as far as advertising the existence of the product.

Time/money invested

I only spent a few weeks coding and documenting it in my spare time. Support issues sometimes took a whole evening, but nothing major. It did not have any impact on my finances as I had invested nothing but my time.

Current product status

I will still address support issues with this product for registered users, but I don’t actively sell it. I’ve open-sourced the program and it still really isn’t seeing heavy use.

I was more successful with other products. I have a few retired products that saw some good bulk-purchase deals ( command-line DUN HangUp, command-line scheduler ) and I still sell the following (for Windows):

All of the above still bring in a modest passive income.

Any regrets?

Not at all. “Nothing ventured,…”.

Lessons learned

Had I not attempted to bring the BPL product to life, I might still be sitting here wondering “what if?” I think it was very beneficial for me to invest the time to try out this idea.

Case #12: Anonymous

Contributor

Anonymous.

The product

A time tracker.

Why it was judged a commercial failure

Because it is not my primary income. I have about 150 customers in one year.

What went wrong

No marketing.
No real thought into features.
I don’t spend any time on it.

In my defense, the reason I do not spend much time on it is that the market became saturated with ‘me toos’ right after I released, which was quite expected. In fact, as I was looking for users, I got an email from a competitor suggesting that I don’t enter the market because they are working on the same thing! I don’t know what I would do differently. Maybe spend more time on it? I think the law of diminishing returns applies quite early in this space so I am not sure.

Time/money invested

Since inception (Nov 2008), I’ve spent close to 250 hours total. Total cash outlay was something like $500.

Current product status

I never tried to make it succeed, to be honest. It was only a learning experience for me. What I probably need now is to go all in. Quite frankly, if I double the sales for this product, I can quit all consulting work. But I really do not think it is a good idea to work on this app full time as it is too simple.

Any regrets?

Definitely not.

Lessons learned

Do it!
Solve a problem people know they have.
Don’t invest too much time and money at the beginning.
Don’t be wedded to a particular idea.
Don’t only listen to your customers. Listen to yourself. After all, you created the idea which attracted the customers.
Never promise a feature for a sale. I’ve never done it but the pressure is really great. My stock response is always: “While such a feature may be available in the future, I recommend that you only use current features when deciding on your purchase.”
Do use Google to your advantage.

Case #13: ScreenRest

Contributor

Derek Pollard.

The product

ScreenRest – a consumer software product that reminds users to take regular rest breaks while using their computer.

Why it was judged a commercial failure

ScreenRest failed commercially because we built a product without having a clearly defined market. This was compounded by it offering prevention, not a solution. ScreenRest continues to regularly sell a small number of licences but not in sufficient quantity to justify further enhancements. The conversion rates are good, but there are simply not enough visitors to the website.

What went wrong

Not doing market research first.
Creating a prevention rather than solution product – people generally wait until they have a problem and then look for a solution.
Creating a product with medical associations – the SEO and PPC competition for related keywords is prohibitive for a product with a low purchase price.

Time/money invested

At least £2000 was spent on the project, including software licences and additional hardware. The product and website were created over roughly 12 months by myself and my wife Lindsay, some during spare time, then part-time and finally full-time so it is difficult to determine the total number of hours. Working part-time and then full-time on ScreenRest caused a significant impact on our finances. Although right from the beginning we saw this as in investment for building a business.

Current product status

Once the product was complete and we started learning SEO it became all too apparent that organic search traffic for related keywords was going to be insufficient. Research into PPC then revealed that the price point was too low to support purchasing medical terms. Planned features for ScreenRest have been put on hold and no further marketing is planned. We continue to support new and existing ScreenRest customers and plan to do so for the foreseeable future. Rather than create another software product we chose to use what we had learned about marketing, copywriting and SEO to create a series of websites targeting a range of topics (often known as niche sites). The most successful of these sites we are expanding in value and functionality to fill gaps not serviced by the competition.

Any regrets?

No. ScreenRest succeeded in every way intended, other than commercially. Creating it was a rewarding learning exercise that started us down a path to finding the intersection of our skills, experience and market opportunities.

Lessons learned

Start with market research – creating a high-quality product you believe in is not enough on its own.
Make sure you can identify a specific target market, that you can reach that market and that it is large enough to support your financial goals.

Conclusion

Analysing the above (admittedly small and self-selected sample) it is clear that by far the commonest cause of failure were:

lack of market research
lack of marketing

With the benefitof 20/20 hindsight it seems blindingly obvious that we should:

spend a few days researching if a product is commercially viable before we spend months or years creating it
put considerable effort into letting people know about the products we create

Yet, by my count, a whopping 6 out of 13 of us admitted to failing to do each of these adequately. Probably we were too busy obsessing over the features and technical issues so beloved of developers, which actually contributed to far fewer failures.

It is also noticeable that, despite the failure of these products, there are few regrets. Important lessons were learned and no-one lost their house. Many of us have gone on to develop successful products and the others will be in a much stronger position if they do decide to try again.

A big thank you to everyone who ate a large slice of humble pie and submitted the above. I hope we can prevent other budding software entrepreneurs making the same mistakes. Even if you don’t succeed, you will learn a lot.

Feel free to add your own hard-won lessons from failure in the comments below.

“No physician is really good before he has killed one or two patients.” – Hindu proverb

Social factors can make or break a product.

Unskilled and unaware of it

Unskilled and unaware of it Have you ever noticed that you rarely (if ever) meet someone who admits to having below average driving skills? My Grandfather started driving a car before UK driving tests became compulsory in 1935. So he never had to take a driving test. This was lucky for him, because he was a terrible driver. He would get distracted and cross the line into oncoming traffic, veering back at the last second when his passenger started shouting. He claimed he had never been in accident, but I expect he would have seen quite a few if he had ever thought to look in his mirrors. Few people would accept a lift from him a second time. Even as a young boy, I realised that I was in mortal danger getting into his car. I would make an excuse and make my own way by bicycle. And yet, he considered himself a good driver. After having read the excellent book Bad science by Doctor, journalist and blogger Ben Goldacre, I think I know why.

The book contains this startling graph from the paper ‘Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments’ by Kruger and Dunning :

Graphics courtesy of explorativeapproach.com (click to enlarge)

The graph shows how good subjects were at logical reasoning as measured by a test (blue), how good they thought they were (green) and how well they thought they did at the test (red). Apart from the top quartile, there was actually an inverse relationship between how skilled people thought they were and how skilled they actually were. The study also showed that the least competent individuals were also the least capable at recognizing the skill levels of others. This is the abstract from the paper:

People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.

Or, more succinctly:

Ignorance more frequently begets confidence than does knowledge. Charles Darwin

The findings fit in with the concept of ‘unconsciously incompetent’ in Maslow’s four stages of learning:

Unconscious Incompetence: The individual neither understands nor knows how to do something, nor recognizes the deficit, nor has a desire to address it.
Conscious Incompetence: Though the individual does not understand or know how to do something, he or she does recognize the deficit, without yet addressing it.
Conscious Competence: The individual understands or knows how to do something. However, demonstrating the skill or knowledge requires a great deal of consciousness or concentration.
Unconscious Competence: The individual has had so much practice with a skill that it becomes “second nature” and can be performed easily (often without concentrating too deeply). He or she may or may not be able to teach it to others, depending upon how and when it was learned.

The Dunning-Kruger effect might explain why every crank and barstool scientist appears to think they understand global warming better than the world’s top climate scientists. I am also reminded of my own experiences learning ju-jitsu. After a year or two of training, having reached the exalted rank of green belt, I thought I was pretty good. It was only some years later, as a black belt, that I realised how much I still had to learn.

What has this got to do with software? Well, it might explain some of the very poor website design, GUI design, graphics and copywriting I see from time to time. The perpetrators may be sufficiently unskilled, that they don’t even realise how unskilled they really are. It is a thorny problem. Firstly you don’t realise you have the problem. Secondly, even if someone convinces you to delegate or outsource that type of work, you are unable to accurately assess the work of others. It is particularly worrying for one man software companies (such as myself) who have to perform or oversee a very wide range of skills including: website design, user interface design, programming, testing, copyrighting, marketing, PR, documentation, support and systems admin.

By the same token, many customers with poor IT skills might not have any insight into the extent of their deficit. Below are the results of a survey I did with some of my own customers a couple of years ago:

Note how skewed the results are and bear in mind that relatively few of my customers are IT professionals. Similarly skewed results were reported recently by Patrick McKenzie in his blog:

Surprisingly many of my customers self-evaluate as comfortable with computers. 50% were “very comfortable”, and 30% were “mostly comfortable”. These numbers are, candidly speaking, not what I would have assigned on the basis of reading support requests for three years.

I don’t have any easy solutions for this problem. All you can do is:

accept that you might not be the best judge of your own competence in all areas
actively solicit feedback from your customers and your peers and listen carefully
be a bit more tolerant of newb@aol.com when they blame your software for problems arising from their lack of basic IT knowledge
console yourself that, whatever your IT shortcomings, at least you are a good driver

5 great ways to waste money in Google Adwords

I have looked at quite a few Google Adwords accounts as part of a 1-day consulting package I do for other microISVs and small software vendors. I have also talked to a lot of people at conferences and on forums about Adwords. It clear that a lot of people are wasting a lot of money on Google Adwords, sometimes with really basic mistakes.

For example:

paying $1.50 per click to advertise a $20 utility
paying $1 per click for an expensive, English language only development tool in some of the poorest, non-English speaking countries in the world.

Ouch.

Below I list 5 great ways to waste money in Adwords. I have seen them far too often. I have considerable admiration for what Google has achieved. But I think Larry and Sergey are probably rich enough already. I would like to see a lot less people throwing money at them and getting nothing useful in return.

1. Don’t use conversion tracking

conversion tracking adwords

With conversion tracking you decide a goal to track (typically a customer buys your software) and put a small script provided by Google on the appropriate page (e.g. the ‘thank you for buying’ page). Google will then use cookie tracking to calculate the cost per conversion for your ads and keywords. It is that simple and you can set it up in a few minutes. I can’t understand why anyone wouldn’t use conversion tracking. With conversion tracking you will soon notice that some ads and keywords convert consistently better than others, often much better. Armed with this information you can optimise Adwords by changing bid prices and deleting under performing ads. Without conversion tracking it is pure guess work.

Conversion tracking isn’t perfect:

Adwords cookies time out after 30 days. If someone buys 31 days after they click your ad it won’t be tracked. And the cookie may be pushed out of the cache before 30 days.
If someone clicks your ad on one computer and then buys it on another computer (or even using a different browser on the same computer) the conversion won’t be tracked.
Some customers may have cookies disabled.

But imperfect data has to be better than no data. Obviously the 30 day limit on cookies is problematic if you are selling software with a sales cycle that is typically 30 days or longer (e.g. software with a 30 day trial). In that case you are probably better off tracking downloads, rather than sales. The fact that someone takes time to download your software, rather than bouncing straight out of your site, at least shows some interest. If you have stats which show your typical download to sale ratio (and you should) you can use this to work out what a download is worth, and set your bid price accordingly.

2. Don’t use negative keywords

adwords negative keywords

My own experiences with Adwords quickly showed me that people will click an ad, even if it isn’t at all relevant to what they are searching for. For example people searching for “747 seating plan” will click on an ad with the title “wedding seating plan”. It is the nature of the web that people are surfing rather than reading, and clicking on an irrelevant ad doesn’t cost them anything. You can avoid a lot of wasted clicks with carefully set-up negative keywords. For example, you can be sure that I have “747” set up as a negative keyword.

Ways to find negative keywords include:

Generating ‘Search Query Performance’ reports from Google Adwords reporting
looking through your web logs/analytics for the sort of terms people are typing into search engines to find your site
using Google’s keyword suggestion tool
using Google suggest

There is also a useful list of negative keyword suggestions on Alwin Hoogerdijk’s blog.

3. Advertise in developing countries

adwords country selection
Over a billion people have access to the Internet. Many of them are in developing countries and aren’t realistically going to buy your software due to a combination of: cost (even $20 is a lot of money to people in many developing countries), payment issues (they may not have access to credit cards), language issues (your software isn’t localised for them) and cultural issues (there just isn’t much respect for intellectual property and copyright in many parts of the developing world). But that certainly won’t stop them clicking on your ads and you still have to pay for the clicks.

Start with wealthy countries where plenty of people speak a language your software has been localised into. If you really think you might be able to make a return in developing countries, then test it by creating a separate campaign that only runs in these countries and set your bids much lower (it is very easy to duplicate a campaign with Adwords Editor).

4. Bid too much

adwords bids

Lets use an example:

Your software sells for $30, of which you get $20 after subtracting ecommerce fees and average support costs.
Your typical visit to sale conversion ratio is about 1%.

That means you will only break even if you pay $0.20 per click through Adwords. Personally I find it hard to justify paying more than 50% of my profit to Google. So I wouldn’t bid more than $0.10 per click. If I couldn”t get any impressions at $0.10 per click I would try to either improve my quality score (e.g. improve my ads or delete keywords with low clickthrough rates) or find cheaper ‘long tail’ keywords to bid on. Paying $0.20 or more just to ‘get on the first page’ of Google is crazy (unless perhaps, it is a loss leader for market research purposes). You can’t make up on volume what you lose on each sale!

5. Don’t monitor your results

adwords reporting

Leaving your adwords campaign running for months on ‘auto pilot’ is ill-advised. Adwords is a constantly changing landscape. Google is continually changing the system and your competitors are coming and going and changing their Adwords campaigns and their products. So you need to continuously monitor how you are doing. Google makes this very easy. For example, you can just set up Adwords reporting to email you a weekly summary of the number of conversions and the cost per conversion for each adgroup. A quick glance through this will let you know if things are going awry.

Conclusion

Adwords can be a very responsive, cost effective and well targeted form of advertising, if you take the time to learn the ropes and experiment. Below is a graph of my return on investment from Adwords for my table planning software over 5 years (almost certainly an underestimate due to the short-comings of conversion tracking, as discussed above). You can see that, after a few months finding my way, I was able to get a consistent ROI of around 4 or 5 to 1 and maintain this in the face of increasing competition.

ROI = number of dollars in sales for each dollar spent on Adwords (1=break even).

Adwords is a complex system and the defaults are weighted in the house’s favour. In this article I have only touched on a few of the biggest mistakes I see. Google will give you plenty of rope to hang yourself and there are lots of other, less obvious ways to lose money. You really need to take the time to learn the system and experiment if you are going to have any chance of getting a decent return.

When I started with Adwords 5 years ago I read the Perry Marshall e-book on Adwords (beware – long copy!). I found it quite helpful. I assume they have kept it up to date. If nothing else, you will learn what it is like to be relentlessly marketed and upsold to. Google also has lots of free Adwords documentation and videos. If you go to conferences such as SIC or ESWC it well worth listening to Adwords specialists such as Dave Collins of softwarepromotions.com (formerly sharewarepromotions.com) talk about Adwords. There is also lots of useful information in the blogosphere. Start with a small daily budget and gradually increase it as you learn what works for you.

If you haven’t got the time or inclination to learn the system and experiment, pay someone who knows what they are doing to do it for you or stay well away from Adwords. Also bear in mind that Adwords works better for some products than others. If I was selling a $20 Mac-only product in a market with lots of more expensive competitors, I probably wouldn’t even bother trying Adwords.

** Update **

I used a deliberately provocative headline for this post, because I wanted to emphasize the fact that a lot of people are wasting a lot of money on Adwords. It seems to have worked in terms of traffic. But, judging by comments here and on Hacker News it has also confused some people. Sorry about that. To clarify, the sections heading are telling you how to waste money. To maximize your ROI you should do the opposite:

use conversion tracking
use negative keywords
only advertise in richer countries
not bid too much
monitor your results

Should you offer a money back guarantee?

money back guarantee A few weeks ago I was going to buy a digitizer tablet for my PC. Then I noticed in the vendor’s terms and conditions that they wouldn’t accept a return once I had opened the packaging. But I couldn’t know if the tablet works until I open the packaging. Duh. I didn’t buy it. Similarly I look for a sensible money-back guarantee whenever I buy software. I don’t remember ever invoking such a guarantee for software, but it is nice to know that I could if I wanted to. Also, I see the lack of such a guarantee as a warning signal that the vendor isn’t confident about the quality of their product.

I offer a 14-day money back guarantee on my own Perfect Table Plan software. The only provisos are:

They have to tell me what they didn’t like about my software. This is very useful feedback for me.
They have to email me that they have uninstalled the software and won’t use it again. I have no way of checking this, but I want them to be clear in their own mind that they are a liar and a cheat if they carry on using it (if you have read Ariely’s excellent ‘Predictably irrational’ you will know that many people are prepared to be a little dishonest, but few will lie and cheat outright).
They have to return the CD, if they purchased one.
The guarantee is only valid for 14 days.

But I am fairly relaxed about about all of these. If it is clear that someone thinks they haven’t got their money’s worth out of my software, I will pretty much always give them their money back.

Note that is isn’t a ‘no questions asked’ money back guarantee. I haven’t been quite brave enough to try that yet and really want feedback on why they didn’t like my software. Also the 14 day guarantee is shorter than most. The reason is that a lot of people buy my software for a single use (e.g. their wedding reception) and I don’t want to make it too easy for them to use the software and claim a refund after they have finished with it. However I have heard vendors say that their refund rate actually dropped when they extended the length of their money back guarantee (due to increased procrastination, perhaps). I may test switching to a ‘no questions asked’ and/or longer guarantee period at some point.

The advantages of a money back guarantee to the vendor are:

More sales. If they customer is confident they can get their money back they are more likely to buy. I don’t have any numbers to back this up, as I have always had a money back guarantee. But I know I am considerably more likely to buy if there is a money back guarantee. Aren’t you?
Less chargebacks. If a customer buys with a credit card they can get their money back anyway. They just have to ring their credit card company and do a chargeback. Your payment processor will then take the payment back and, to add insult to injury, slap a chargeback fee on top. Too many chargebacks and they might even close your account. Better to refund and avoid the chargeback fee.
Less bad vibes. No matter how great your software is, some people aren’t going to like it. Maybe your software isn’t a good fit for what they want to do. Maybe they are just having a bad day. Better to give them their money back than to have them bad mouth your product on every forum they can find.
Less bad customers. Some customers (thankfully very few, in my experience) cost more in time and mental energy than their licence fee is worth. It is better for you to cheerfully refund them and focus your efforts on more financially and psychologically rewarding customers.
Staying legal. You may be legally obliged to give a refund in some circumstances.
Good karma. If you aren’t happy, I really don’t want your money.

Generally it costs you nothing to refund a software purchase, apart from a few seconds of your time (depending on your payment processor, some may not refund the processing fee). The only disadvantage of a money back guarantee is that it makes it easier for a customer to cheat you. A lot of vendors worry about this, but in my experience (and of others I have spoken to) this isn’t much of an issue in reality. My refund rate has been consistently around 0.5% (I am not including cases where I refunded because people bought the wrong type of licence, bought 2 licences instead of one etc.). I would be very surprised if dropping my prominent money back guarantee didn’t also drop my sales by a lot more than 0.5%. So, even if all the refunds are fraudulent (which I very much doubt) I am confident that the refund policy increases my profits overall. Sufficiently confident that I don’t intend to run an A/B test any time soon.

Interestingly, my refund rate is 10 times lower amongst customers who have purchased a CD. This could be because these customers are less price sensitive and so don’t see the refund as worth their precious time. Or it could be because of the extra hassle of having to send the CD back (I know a wily B2B vendor who includes a CD with every purchase for exactly this reason). Probably it is a combination of both.

Some vendors think that they don’t need a refund policy if they have a free trial. I don’t agree. When I buy software I want a free trial AND a money back guarantee in case I only discover a problem after purchasing. Also I know (from a survey) that some 25% of my customers don’t even try the free trial of my software before they buy. I expect I would lose a lot of these sales without a money back guarantee.

I think the case for a money back guarantee is even stronger for B2B software. Customers buying B2B software typically aren’t spending their own money, so they are probably less likely to ask for a refund. Especially as this would mean admitting to their boss and peers that they made a mistake buying your software in the first place. Certainly I have a lower refund rate to businesses than to consumers.

From a business point of view, I think the only case where you can justify a no refund policy is when you have a high cost of sale, e.g. enterprise software that requires a lot of configuration. In that case you could include a non-refundable set-up fee that covers your costs, but still have a money back guarantee on the remainder of the purchase.

No doubt refund rates vary according to product type, price range, customer demographics, geographic market and a range of other factors. But , reading forums and talking to other vendors, the typical refund rate seems to be in the range 0.1% to 1%. If your rate is much above 1%, perhaps there is a problem with your product you need to address? If your rate is much less than 0.1%, perhaps you aren’t marketing your product aggressively enough?

In the early days I found it hard not to see refund requests as an insult to my product. But now it really doesn’t bother me and I cheerfully make the refund. I just add the key to a ‘blacklist’ in the software so it won’t work in any future releases. I don’t attempt to disable it in the current release. I don’t see implementing a ‘phone home’ strategy to make this work as being a profitable use of my time.

In summary, by not giving a money back guarantee you might avoid a small number of customers cheating you. But I think you are very likely to be losing a lot more in chargebacks, missed sales, ill will and missed feedback than you save in fraudulent refunds. Try it. You can always revert back in the unlikely event that your refunds go up significantly more than your sales. And if you have a money back guarantee you should shout about it on your website. Having a money back guarantee and not advertising it prominently seems like the worst of all worlds to me.

Outsourcing software testing

Every time I write a post for this blog I carefully check it for typos. I then get my wife to proof-read it. She always finds at least one typo. Often there will be whole words missing that my brain must have interpolated when I checked it. I read what I thought I had written. She is unencumbered by such preconceptions.

Similarly, it isn’t sufficient to do all your own testing on software you wrote, no matter how hard you try. You will tend to see what you intended to program, not what you actually programmed. Furthermore your users have different experiences, assumptions, and patterns of usage to you. Even in the unlikely event that you manage 100% code coverage in your testing, those pesky users won’t execute those lines of code in the same order you did. I have spent hours testing a program without finding a bug, only to see someone else break it within minutes or even seconds.

So it is essential to involve people other than the original programmer in testing, in addition to (but not instead of) the testing programmers do on their own code. This poses something of a challenge to one-man-bands such as my own. I don’t have other programmers, let alone QA staff, to call on. I can, and do, use volunteer customers for beta testing. But, in my experience, beta testing is not an effective substitute for professional testing:

It is haphazard. I never hear from ~90% of my beta testers.
You can’t control beta testers sufficiently, for example you can’t set them tight deadlines, make them concentrate on a particular feature or do their testing on a particular operating system
The quality of bug reports from customers is often poor. Customers often don’t understand (or don’t have the patience) to describe a bug in enough detail for you to reproduce it.
Professional testers know how to break software.
The new release should be as polished as possible before any customers see it. Your beta testers will be some of your most enthusiastic customers. You don’t want to use up that goodwill by sending them buggy software.

Consequently I like to pay third party testers to test my own PerfectTablePlan product after I have finished my own testing and before I do any beta testing. Previously I have used softwareexaminer.com, but they are no longer in business. So I decided to try a couple of other offshore testing companies I had heard about:

	testlab2.com
	qsgsoft.com

The problem with paying a testing company is that it is hard to assess the quality of their work until it is too late. If they report few bugs it could because there are few bugs or because they didn’t do a very good job of testing. By using 2 companies to test the same software release I was also testing the testers (I didn’t tell them this).

I paid each company to do approximately 3 days testing on the Windows and Mac versions of PerfectTablePlan. I was very pleased with the results. Both companies found a useful number of bugs in the software. They were also able to test on platforms that I didn’t have access to at the time (64 bit Windows 7 and Mac OS X 10.6). I didn’t keep an exact score, but I would say that QSG found more bugs, while TestLab2 was more responsive.

QSG found some quite obscure bugs. They were even able to tell me how to reproduce a very rare and obscure bug that I had been trying to track down for months without success. Communications were sometimes a little slow (at least partly due to us being in different time zones) but it wasn’t a huge issue. My only real grumble is their billing. Despite several reminder emails from me I am still waiting to be invoiced for the work several months later. I like to pay my bills promptly and then forget about them.

TestLab2 didn’t find quite as many bugs, but I was impressed with their responsiveness. They installed Mac OS X 10.6 within a few days of it being released, so they could test PerfectTablePlan on it. When I emailed them on a Saturday about a last minute bug fix for Mac OS X 10.6 they tested the fix the same day. That is great service.

TestLab2 and QSG are based in Ukraine and India, respectively. At around $15/hour they are about a third the price of equivalent US/European companies I contacted (who might also outsource the work to Eastern Europe and India, for all I know). Some people believe outsourcing work to countries with lower costs of living is evil. I’m not one of them. I sell my software worldwide and I am also happy to buy my services worldwide, especially if I can get significantly better value for money by doing so. While there are rational arguments to be made about problems caused by differences in culture, language and time zone caused by outsourcing to other countries, I didn’t find any of these to be a major issue in this case. Most of the other arguments I have heard boil down to the simple ugly fact that some westerners feel they are entitled to a disproportionate share of the global pie. But I don’t see any reason why someone in Europe or North America is any more deserving of a job than someone in Ukraine or India.

With the help of these two companies I was able to put out a really solid PerfectTablePlan v4.1.0 release, despite the large number of new features. In fact, I am only just putting out a v4.1.1 with some bugs fixes several months later. I plan to use both companies again. I hope readers of this blog will give them some additional work to ensure they stay in business. But not so much that they don’t have time to do my next round of testing!

A survey of ecommerce providers for software vendors

Overview

The choice of ecommerce provider is probably one of the more important ones you make as a software vendor. It isn’t too hard to compare providers by feature set or price. But what about other vital attributes, such as support, reliability, ease of set-up and how they treat your customers? It isn’t realistic to try every provider, so this major decision is often made on the basis of haphazard anecdotal evidence from forums. I created a survey in an attempt to gather some systematic data on the ecommerce providers most commonly used by small software vendors. I present the results below without fear or favour. Skip ahead to ‘Overall ranking’ if you are in a hurry.

Methodology

I posted a request for survey responses on this blog and on a few forums frequented by microISVs and small software companies. Any vendor of software (desktop or web based) not directly affiliated with an ecommerce provider was eligible to take part. Software vendors were invited to fill out a survey form on wufoo.com for each ecommerce provider they had used in the last 2 years. They had to supply their product URL and an email address from the same domain so that I could verify their identity. They also had to check a box proclaiming:

I am a software vendor and I have used this Ecommerce provider in the last 2 years. I have no commercial interest beyond being a customer. (If you have affiliate links to the Ecommerce provider, that isn’t a problem.)

They then had to reply to an automated email from wufoo to the email supplied confirming it was them that had completed the form. If they didn’t reply to the automated email I followed up with a few more emails. Although tedious for me, I felt this was an important safeguard to avoid any possibility of fraudulent entries. I also checked for duplicate entries, duplicate IP addresses and other suspicious patterns. The survey was open from the 5th to the 8th October. Any responses not validated by 10th October were removed from the data.

The data

202 survey responses were received from 166 different software vendors. 9 responses were rejected as I could not verify their identity (they didn’t respond to several emails). 1 response was rejected due to a possible conflict of interest raised by the software vendor (they had done paid work for one of the providers). This left 192 valid responses. I saw no evidence of any attempt to rig the results.

You can download the raw data. It has been stripped of any personal identifying information. Feel free to do your own analysis or check my results.

Providers

The survey listed 14 of the major ecommerce providers, plus an ‘other’ box for providers not listed. Valid responses were received for 25 different ecommerce providers, as shown below:

Note that ‘e-junkie+PayPal/GoogleCheckout/2Checkout’ has been shortened to ‘e-junkie’ for brevity.

Questions

Below I show the average (mean) score per ecommerce provider by survey question. The providers are sorted by score. Providers with less than 3 responses weren’t considered statistically valid and are not shown here (see the raw data for all responses).

Features

“How do you rate the range of features offered, e.g. coupons, support for multiple currencies, CD shipping, affiliate tracking, handling of tax etc.”