article « Successful Software

Archive for the 'article' Category

Pivot Tables demystified

Online businesses generate a lot of data. Sales data, marketing data, traffic data etc. Pivot tables can be a useful tool for analysing and extracting useful information from this sea of data. A lot of people seem to have heard of Pivot Tables without understanding what they are or how to use them. Despite the slightly cryptic name they are really just a way of summarizing tables of data. Nothing to be intimidated by. I’m going to try to demystify them here with a simple example.

Let’s imagine you have got an Excel spreadsheet with a month of (fictitious) sales transaction. It looks like this:

You want to find out:

The total value of sales of each product
The number of sales of each product
The total value of sales of each product by country
The total value of sales of each product by day of the week

How would you do it? You can crunch the numbers using a calculator, but that is very tedious and error prone. If you are a programmer you can export the data to a .csv file and write a small program in your favourite language to read it in, crunch the data and write the results out. You probably think you can do it in 10 minutes, but I bet it will take you at least an hour to get it working and debugged. I did all the above in 1 minute and 20 seconds using Excel pivot tables.

Here is how you can do it (screenshots from Excel 2007). You can download the spreadsheet if you want to try it yourself.

Select all the data, including the header (A1 to D222).

Click on Insert in the Excel Ribbon bar.

Click on PivotTable.

The Create Pivot Table window will appear. Click OK.

The PivotTable Field List will appear. We want to find the number of sales for each product. Drag the product field to Column labels and the value field to Values. This gives us the total value of sales by product.

To change this to the number of sales pull down the Sum of value drop-down list and select Value Field Settings….

Change Summarize value field by from Sum to Count and click OK.

The number of sales of each product are now shown.

Now drag the country field to Row labels. Then click on Count of value, select Value Field Settings…, change it back to Sum and click OK. The total value of sales of each product by country is now displayed.

We now need to add a day of the week column. Click on Sheet 1 and add a new column as shown.

Now select all the data from A1 to E222 and insert a new pivot table, as before.

Drag the fields as shown to get the total value of sales of each product by day of the week.

(We’ll quietly ignore the fact that the number of days aren’t divisible by 7 in the data.)

We can even display by product, day of the week and country with one more drag.

Excel also supports pivot charts if you prefer your results in graphical form.

Obviously this is a very simple example, but pivot tables can be used to quickly analyse much larger and more complex data sets. Next time you have got some data to analyse consider whether you would be better using pivot tables or pivot charts.

Losing weight with a minimum of willpower

Published 30 October 2011 article , off topic 16 Comments
Tags: diet, fat loss, obesity, slow carb, weight loss

If you are happy with your weight you can skip this post. Normal programming will be resumed soon.

We all know how to lose weight: eat less and move more. So why do so few people succeed? You only have to spend a few minutes walking around any wealthy western country to see that obesity is a huge problem. The human body is truly an incredible thing, evolved over millions of years. It is so much more than just a mere transport and fuelling system for the brain. We only get one body, it is a tragedy to neglect it.

The past is always with us

Our evolutionary past is working against us. Agriculture is less than 10,000 years old. That is only about 400 generations. A blink of an eye in evolutionary terms. While our technology and culture may have changed beyond all recognition in that time, our bodies and brains are still very similar to those of our palaeolithic ancestors. Their innate drives to eat salty, fatty and carbohydrate rich food and minimize their energy consumption helped to keep them alive on the hostile African savannah. But these ancient drives do little to help urban humans with almost unlimited access to food and increasingly sedentary lifestyles. Hence the burgeoning waist lines we see all around us today.

Tightening trousers

Over the last few years I had put on a few kgs. I have been spending a lot of time in front of the computer; working on my software product, doing consulting and writing this blog. Having a young child also reduced the time available for exercise. The 32 inch waist trousers I have been wearing for the last 20+ years (not the same pair, I would like to point out) were starting to feel uncomfortably tight. I either had to lose weight or go clothes shopping. Surely anything is better than clothes shopping.

But how to lose the weight? Previous attempts to lose weight have never lasted:

I went trekking in the Himalayas, spending 16 days walking up to the top of Mera Peak (6,476 metres). Weight loss wasn’t the goal, but the extreme exertion at high altitude meant I had the beginning of a ‘six pack’ when I got back. It didn’t last though.
I wrote down everything I ate on a spreadsheet and worked out the calories by weighing the food and using the calorie information on the packet or the CalorieKing website. I tried to eat less than 1800 calories a day[1]. It worked for a while, but the bookkeeping was tedious and I got bored and hungry. I gave up after a few weeks.
I tried the Zone Diet. Again it worked for a while, but I found the constraints (maintaining certain proportions of protein, fat and carbohydrates) tedious. I was also less than convinced by the science. I gave up after a month or two.

Success!

I decided to try a different approach. It is based on the same approach I take to my business – learn the concepts and measure the results. 8 months later I have dropped from 17.5% body fat to 13.0% body fat and lost over 6 kg (13 lb) in weight. I will spare you the before and after photos, but suffice to say that I can now easily fit into 30 inch waist trousers (damn – I still had to go clothes shopping!). And I never went hungry. I wasn’t even particularly disciplined, taking a week off here and there when I went on holiday. I won’t be appearing on the cover of a men’s fitness magazine or slimmer of the year. But I think it’s not bad for a 45 year old with a sedentary job, a distinct lack of ‘thin genes’ and a weakness for most fattening things.

I don’t claim to be an expert nutritionist or physiologist. Even if I was, the knowledge in this field seems to be constantly changing. But here is the approach that worked for me. It might work for you.

The slow carb diet

The cornerstone of my new approach was inspired by (but didn’t follow the letter of) Tim Ferriss’ slow carb diet. You should go and read his article, but the key elements are:

diet 6 days per week – eat lots of lean protein, beans, pulses and veggies, no high energy carbs (e.g. sugar, bread, rice, pasta, potato, fruit, beer and sugary drinks)
binge 1 day per week – pig out and eat whatever you like, all-you-can-eat buffet here I come!

There is more detail in his Four Hour Body book [2].

This approach makes sense in terms of our evolutionary heritage. Our hunter-gatherer ancestors wouldn’t have had access to refined carbohydrates, such as bread, and fruit would only have been available in season. They would also have regularly alternated between feast and famine.

A typical diet day’s intake for me is:

2 eggs for breakfast with chilli sauce (fried with a tiny amount of oil sprayed into a non-stick pan)
a big salad for lunch
cajun style fried white fish with lots of veggies for dinner
several teas, coffees and glasses of water plus a glass of red wine

The binge day helps both physiologically (it presumably stops your metabolism slowing down in response to the reduced calories) and psychologically (you are never more than 6 days from eating what you want).

Pick an exercise regime you can stick with

I also increased the amount of exercise I do. The key to this was doing things I found enjoyable and a combination of routine (doing some exercises the same day every week) and variety (varying what I did on other days). I have found that a routine of capoeira, running and kickboxing pad work plus some swimming, weights and cycling for variety suits me. I average 3-4 30-60 minute exercise sessions per week.

I really enjoy capoeira, an Afro-Brazilian fusion of martial art, dance and music. It is very strenuous, but the fact that I am learning a skill (and trying to avoid getting kicked in the head) stops me thinking too much about the exertion. I also find running and weight training quite bearable when I am listening to an interesting podcast. I am lucky enough to live near a beautiful park with a lake. Running around the lake is even quite enjoyable on a sunny day. The fitter I get, the more enjoyable it becomes. Different things work for different people. People have even lost weight combining World of Warcraft with a treadmill or exercise bike. Don’t try to force yourself to do stuff you hate, you won’t stick with it.

Your body burns mostly glycogen stored in the muscles for the first 30 minutes of exercise. So you need to exercise for more than 30 minutes to start burning fat.

It takes a lot of exercise to burn off a small amount of food. About 15 minutes of running is required to burn the calories from a single slice of bread. I knew someone who was significantly overweight despite cycling 10 miles a day. It was only when he changed his diet as well that he started to lose weight.

Any form of exercise that builds muscle mass (e.g. weight training) is good, because extra muscle will increase your metabolic rate which will burn more calories.

Weigh yourself every day

If you weigh yourself every day, you will soon see a direct correlation between weight loss and how well you stick to your diet/exercise regime. I found this daily feedback very helpful for motivation. If you only weigh yourself once a week your progress may get lost in the noise, e.g. due to the varying content of your digestive tract. I weigh myself every morning, as soon as I get out of bed. Here is a graph of my daily weight over the last 8 months:

You can see the sawtooth pattern of the weekly binge day. It is easier to see overall progress using a 7 day moving average:

Each peak was a suspension of the diet and exercise regime for holidays, birthdays and conferences.

A bit of science

Different foodstuffs vary hugely by the amount of calories per gram. You could eat 10kgs (22lbs) of green vegetables per day and not exceed your daily calorie intake (you might get a divorce though). Look at the side of the packet when you buy food, you’ll be surprised. For example, 1 gram of:

spinach = 0.2 cal
onion = 0.4 cal
kidney beans = 1 cal
bread = 3 cal
french fries = 3 cal
sugar = 4 cal

Try to avoid food where you don’t know what goes into it. For example, a lot of home delivery pizza contains vast amount of hidden salt, fat and carbs.

All calories are not equal. One would think that eating fat makes you fat. But there is increasing evidence that carbohydrates, not fat, is enemy number one when it comes to losing weight. Particularly carbohydrates with a high GI (ones that are quickly absorbed into the blood stream). Carbohydrate intake affects insulin levels, which in turn affects fat deposition. I was moderately active before and I didn’t eat excessively, but I ate a fair amount of bread, rice and potatoes. Reducing the amount of carbs has been the biggest change for me and I believe the main cause of my weight loss. Note that many ‘low fat’ foods (e.g. low fat yoghurts) contain large amounts of added sugar.

Protein makes you feel full. Studies show that x calories of protein will make you feel less hungry than x calories of carbohydrates or x calories of fat. High GI carbohydrates can even make you feel hungry (as you will know if you have ever felt starving after drinking lots of carbohydrate rich beer).

Low carb doesn’t have to be boring. Pickles, chilli sauce and balsamic vinegar liven things up without adding calories. The amusingly named Ring of Fire (‘so hot it will burn you twice’) is my favourite chilli sauce. It’s not actually that hot.

It is easy to underestimate how much you are eating. Studies show that overweight people tend to be worse at estimating their calorie consumption. I watched a TV documentary in which they asked an overweight women to work out her daily calorie intake. She calculated 1000 calories per day. Precise measurements showed it was, in fact, 3000 calories per day. The large bowl of fruit she ate every morning was 1000 calories on its own.

Smaller plates=smaller portions. Studies have shown that simply reducing the size of your plate can reduce overall calorie intake.

Starving yourself doesn’t work. Not only does it require willpower that few of us possess, but also the body simply adapts to the lower calorie intake by reducing metabolic rate. This can then cause you to put on weight if you return to your old eating habits.

Studies show that most overweight people do not have low metabolic rates. In fact, they typically have higher than average metabolic rates due to their increased mass requiring more work to maintain. If you are fat, the chances are that you are eating too much and not doing enough exercise, rather than it being due to a slow metabolism.

Watch what you drink

Watch what you drink, as well as what you eat. Drinks can contain surprisingly high amounts of calories. A single small can of cola contains around 5% of your daily energy intake. Alcoholic drinks and milkshakes can also contain a lot of calories.

Drink more water. A lot of rubbish is talked about hydration. But water does make you feel more full. So it is a good idea to have a big glass of water between and/or during meals. I find it easier to drink water if it is chilled in the fridge with a squeeze of lemon. No need to waste money on bottled water. It works even better if the water is in the food. Soup is a good way to make yourself feel full without consuming too many calories.

Food supplements

Tim Ferriss also recommends PAGG (Policosanol, Alpha lipoic acid, Garlic extract and Green tea) food supplements for weight loss. But these are about £70/$100 per month and I haven’t been able to find any peer-reviewed, double-blind trials that shows that it actually works. I won’t be spending that sort of money without some proper scientific evidence. I am not aware of any other food supplements that are proven and without side effects.

Social factors

Get your partner or house mate to diet with you. It is much harder to stick to your diet if the fridge is full of things you can’t eat. Declaring your intention to lose weight publicly can also provide extra motivation. Feel free to post your current weight and target weight in the comments. ;0)

Be realistic

Have realistic expectations about how fast you will lose weight. I came nowhere near losing the ’20 lbs. of Fat in 30 Days’ that Tim Ferriss claims. I averaged around 0.3 kgs (0.7 lbs) lost per week over the initial 4 months, and less after that. If it has taken you years to put on the weight, you are unlikely to lose it in a month or two. Despite claims to the contrary by people with books to sell.

The long term

Losing weight and keeping it off requires a long term change of lifestyle. Only time will tell if I can improve on, or even maintain, my current weight loss. I expect I will put a bit of weight back on over the winter. But I think I have found a regime that works and that I can live with. Putting this article in the public domain gives me a bit of extra incentive!

A call to action

If you need to lose some weight, then today is as good a day to start as any. If you do, it would be interesting to get some additional data for a follow-up post in a few months. Email me (andy at oryxdigital dot com) if you are interested in taking part. You can remain anonymous, if you wish.

[1] Note that I am following the (ridiculous) convention of referring to kilo calories (kcal) of food energy as ‘calories’.

[2] It is an interesting and entertaining book. But beware that:

a lot of it is controversial, as you will see from the comments in Amazon
I find many of his claims of results of weight loss and weight gain hard to believe, and I think they certainly aren’t representative of what most people will achieve
experimentation is good, but self-experimentation is not as reliable as formally conducted, peer-reviewed, double-blind experimentation, especially when the author has a book to sell

Capoeira photo by HoveringDog.

What type of free trial should I offer for my software?

Published 19 September 2011 article , marketing , software 13 Comments
Tags: download, feature limited, free, nagware, software, time limited, trial, watermarked

Once upon a time, the idea that you would allow people to try your software before they bought it was revolutionary. But now, thanks to the shareware movement and the ease with which software can be downloaded from the Internet, free trials are the norm for most types of off-the-shelf software. Prospective customers no longer have to rely on reviews of questionable independence or reading the packaging in a shop. They can try the software for themselves before making any commitment. This has been overwhelmingly a good thing for software users. It has also been a boon for vendors of good software.

When I surveyed 92 small software product vendors in 2009, 100% of them offered a free trial.

Eric Sink says:

Every small ISV today should give its customers an opportunity to try before they buy. It is officially now absurd to do otherwise. Customers will come to your Web site and expect to find a demo download.

And that was in 2004.

So, for most software products, the question isn’t – should I have a free trial? The questions is – what sort of free trial should I have? As with everything related to marketing, it depends. There are many different approaches. Below I describe some of the more common ones.

Time-limited

Typically this takes the form of a fixed number of contiguous days of free and unrestricted use. The software then stops working and you need to buy a licence to continue using it. The time period is often 30 days. As you can see in the pie chart above, this was the most common type of trial in my survey.

The advantage of this approach is that it allows the user to try the full functionality of the software. But it does have a number of issues:

The trial might expire before they have finished their evaluation.
It isn’t suitable for software that might only be needed for a limited time. For example, a 30 day time-limited trial wouldn’t be a good idea for my wedding seating arrangement software as a wedding is a one-off event (we hope), and people could just start the trial 30 days before their wedding.
You have to find some way to hide the data about the date the trial starts.
It is relatively easy to circumvent. Even if you hide the install date well and check for changes to the system clock a potential customer can just keep reinstalling the software inside a new virtual machine each time the trial expires.
Cookie expiration is an problem. For example, Google Adwords conversion tracking cookies only last a maximum of 30 days. So Adwords conversion tracking won’t count a sale on day 31, which is probably where most of your sales will happen with a 30 day trial.
As most sales will only happen after your trial expires, you will have to wait longer to get your money.

I have sometimes downloaded software, started to evaluate it, got distracted and then returned only to find the trial has timed out. Very frustrating and unlikely to result in a sale. So I am generally not a fan of fixed duration trials. There are a couple of ways you can try to work around this issue:

Only limit the amount of time they are actually actively using the software. For example, allow them 8 hours of total active usage. This shouldn’t be too hard to program. For example, stop the timer if there hasn’t been a key press or mouse click event in 2 minutes.
Allow them to request an extension. Then at least you have got their email address and can follow them up.

Usage-limited

In this approach you limit the number of times a certain action can be performed. For example the number of pages they can print or the number of times they can start the software. This avoids the issue of a time-limited trial which expires before the user has finished their evaluation.

Feature-limited

In this approach the trial disables an important function of the software, for example printing or saving. The problem is that a user won’t know if this feature works properly until they buy the full version. This may put some people off.

It has the advantage that you can ship a separate trial version with features missing completely from the executable, which makes life a little harder for crackers (note that they can still get hold of the non-trial version to crack if they want to, e.g. with a stolen credit card number). But it also makes life harder for customers, as they have to install the software a second time after purchase. Personally I care more about making life easy for my paying customers.

Capacity-limited

A capacity-limited trial restricts the amount of data that can be entered. For example, a password manager might only allow you to enter 50 passwords into the trial version. This approach can be problematic when performance is important. For example, if you limit a database trial to one thousand records, how can the user test whether the search performance is adequate for a database with a million records?

Output-limited

Many products exist purely to produce some form of physical or electronic output, for example image editors and label printers. Adding a watermark, or altering the output in some other way, can be an effective way to limit a trial. But you need to make sure that the modification to the output can’t easily be removed or worked around (e.g. using screen capture). You also need to be sure the user doesn’t think the modifications in the output are due to a bug in the software.

Nagware

Nagware allows you to use the software without restrictions, but ‘nags’ you periodically to pay for it. Usually this takes the form of a window that pops up when you start or exit the software. But I once used some software that also nagged you in audio. A woman’s voice with a heavy Scottish accent no less. It got uninstalled very quickly! Nagware isn’t very effective in my experience. I never did buy WinZip. Did you? After a while you just click the ‘continue’ button without thinking about it. Little tricks, like moving the ‘continue’ button or greying out for a few seconds are just annoying. And annoying people doesn’t seem like a great start to a business relationship.

No trial

Some software has no trial, just a money back guarantee. If your software is an enterprise system that takes significant effort to configure, then this is entirely understandable. But if it is off-the-shelf, downloadable software, what are you trying to hide? On the plus side it avoids the issue of people downloading software and then never getting around to trying it. My own stats show that only some 40% of people who start a download of my software actually install and run it. Also many people won’t ask for their money back even if they don’t like your product. So you might get sales to people who wouldn’t have purchased with a trial. But do you really want these people as customers? Personally I am unlikely to buy a software with no trial, unless there is no real alternative. I assume you won’t let me try your software because it isn’t very good. I’m sure many other people feel the same way.

Hybrid trial

Hybrids of the above approaches are also possible. For example, the trial of my wedding seating arrangement software doesn’t allow you to save, print or export plans with more than 30 guests – a hybrid of the capacity-limited and feature-limited approaches. I figure that 30 guests is enough to show what the product does, but not enough to be useful for most events. Also no-one is likely to pay for event planning software for an event with 30 or fewer guests.

Conclusion

A good trial is a balancing act. You need to give prospective customers enough to show them your software could solve their problem, but not enough to actually solve their problem. But if you are too restrictive they might go to a competitor with a more relaxed trial policy. It can be tough to get the balance right or to know whether a different approach would get better results.

Obviously, the best type of trial depends very much on your product. If it is a product that is likely to be used a lot and is going to increase in value as it is used, then you might be best offering a generous time-limited or usage-limited trial. But if it is a product that is only needed for a one-off task or a limited period of time, then a feature-limited, capacity-limited or output-limited trial probably makes more sense.

For example, most consumers will (unless they are very unlucky) only want to use disk recovery software once. So it wouldn’t make sense to offer a 30 day free trial. It would probably make more sense to offer a feature-limited trial that allows them to see what data could be recovered, but not actually allow them to do the recovery until they pay up. But if you are selling to professional disk recoverers, then a time or usage-limited trial might be appropriate.

I asked Craig Peterson of the Beyond Compare file comparison tool about their very generous trial policy (30 non-contiguous days of use) in an interview and he said:

That goes back to competing with all the other products out there. If someone installs two programs to evaluate, and then doesn’t have a chance to really try them out until a month later, the one that works is more likely to get the sale. It also makes it more likely that potential customers will learn the application and start relying on it, so when it does come time to pay they’re less likely to throw out that investment and switch to another tool.

Data comparing different types of trial is hard to come by:

My 2009 survey didn’t show any clear difference in mean conversion rate between time and feature-limited trials (there wasn’t enough data for usage-limited trials to be worth counting):

Patrick McKenzie reports on an A/B test of nagware vs feature-limited versions of the same software, showing the feature limited version had a 5 times higher conversion rate.
A poster on the Business of Software forum reported that a bug which accidentally turned his trial from feature-limited after 30 days, to feature-limited as soon as it was installed, increased his conversion rate by 33%.

The nagware vs feature-limited result is fairly conclusive. But, apart from that, there doesn’t seem to be much hard data to go on. Even if there was more data, it wouldn’t necessarily apply for different products in different markets. So, unless you want to program multiple types of trial and run lots of split tests (trial and error?), you are going to have to ‘go with your gut’. It is tempting to pick the same trial model as your competitors. But remember that part of successful marketing is being different.

So there are no easy answers. But don’t just choose a 30 day time-limited trial because that is what everyone else is doing. Have a think about what fits best with your product, market and customers. Be creative.

Selling software vs selling eyeballs

Published 4 August 2011 article , Google , marketing , microISV , software 23 Comments
Tags: ads, adsense, advertising, conversion ratios, cpm, eyeballs, software

Lets say I’ve written some downloadable software and I want to make some money from it (‘monetize it’ in the ghastly common parlance). Should I charge people for using the software or should I give them the software for free and make my money from ads?

Lets look at some numbers.

The typical conversion rate for downloadable software is around 1%. That means that about 1% of the people that visit your site will typically buy your software. So, for each $1 of your sale price you will make around $0.01 per unique visit. Downloadable software is often priced around $30, so lets say $0.30 per unique visitor. Some software sells for less than $30, and some for a lot more. Also I haven’t taken account of the lifetime value of a customer (e.g. upgrades) – which will increase the value per customer; or payment processing and advertising costs – which will reduce the value per customer. It is just a ball park figure.

How much money could I make from advertising if I give the software away instead? I have been doing some research for a while on this. Based on various data I have gleaned from the BOS forum and blogs, advertisers typically pay per $1-$2 per 1000 impressions (CPM). Some data points:

A well known ad network offered me a $2 CPM (-19% commission) to put ads on this blog.
Dating site plentyoffish.com reported making $10k/day from Adsense off 200 million pages per month in 2006, which is a CPM of $1.5
A sample of 8 Facebook app developers were averaging less than $1 CPM.
“If a site like Stack Overflow, which does almost a million pageviews a day, can’t make enough to cover even one person at half time using Google AdSense, how does anyone make a living with AdSense? Does it even work?” (Stackoverflow blog)
“Charging your end user isn’t the only way of pricing software. You can choose to give it away for free and then make money by, for example, charging for consulting, installation and training; or selling advertising. The latter, although a common model for web sites, is extremely hard to make work. CPM – the cost per thousand impressions – can be as low as a dollar. In other words, to generate one thousand dollars of revenue you might need to serve up as many as a million pages. To generate enough revenue to support a team of three or four people, that means having ten million page views per month. Most web applications simply aren’t going to attract that sort of traffic.” (p57 of “Don’t just roll the dice”)

So, taking a ballpark CPM of $1.5, I would be making $0.0015 per page impression.

Obviously I am comparing apples (unique visitors) and pears (impressions) here. How many impressions does 1 unique visitors equal? My own table planning software averages around 2 impressions per unique visitor (many visitors bounce out after reading 1 page, even those that buy might only visit the home, download and purchase pages). So, assuming this is typical, the product based site described above should be making around $0.15 per page impression. Based on these (admittedly rough) numbers an ad driven site needs approximately 100 times as many page impressions per day to make the same money as a product driven site. To make around $100k per year the product site would need about 900 visitors/1,800 impressions per day. To make the same amount the ad driven site would need around 90,000 visitors/180,000 impressions per day. But it is worse than that because the ad driven site is going to have significant hosting fees and potentially many more users to provide support for. I know which business model I prefer.

So why not get the best of both worlds – sell the software AND put ads on the site? Because then you are sending out all sorts of bad vibes (“this software isn’t good enough that they can make a living off it”) for a measly 1% extra income from the ads. I’m confident the presence of ads will lose you >1% in product sales.

An ad supported model is only viable when you have lots of traffic. Most downloadable software (or web apps) won’t be able to generate that sort of traffic, even if it is good and you give it away for free. If you really want to run an ad supported business, you are probably better off basing it around forums and user generated content than free software.

In the final analysis if you are creating software I think it makes more sense to create something of value, grow some balls and charge for it. Rather than giving it away and selling eyeballs in the hope that someone else will take their money and throw you some scraps. Think balls, not eyeballs.

Setting an optimal bid price for Google CPA bids

Published 12 July 2011 adwords , article , Google , marketing , software 8 Comments
Tags: adwords, bid, conversion optimizer, cpa, cpc, CTR, Google, graphs, ppc, results

A couple of years ago I wrote up the results of an experiment comparing Cost Per Action vs Cost Per Click bidding in Google Adwords. At the end of the experiment I decided that I did trust Google CPA bidding, but the results from CPA bidding weren’t compelling enough for me to switch. So I stayed with my mature CPC campaign. Subsequently I spoke at length with Adwords guru David Rothwell and Adwords master practitioner Alwin Hoogerdijk. They convinced me that:

I hadn’t really given Google CPA enough learning time – the more data Google has the better it should be able to do. The mighty Google brain might even be able to spot and exploit patterns I would find very difficult to emulate (e.g. based on season, country, day or week or time of day).
I should switch from paying per sale to paying per download, as this would give Google an order of magnitude more data to work with.
CPA bidding would require a lot less of my precious time to manage.

So I switched back to CPA. This time measuring a conversion as a successful download and install (my table planner shows a help page in a browser on first run, this contains the Google conversion tracking script).

So now, instead of having to choose thousands of bid prices (one for each keyword and match type in each campaign), I had to choose a single bid price – what I am prepared to pay Google for a download. If I pay too little for a download: Google won’t show my ads much, I won’t make many sales and my profit will be low. If I pay too much for a download: Google will show my ads a lot, but the amount I pay for each conversion will be high and my profit will be low. In between their should be a ‘sweet spot’ that gives me optimal profit. But how to find that sweet spot?

Looking at analytics data I have a good idea at what rate Adwords traffic converts to sales. I chose a CPA bid based on this and then I randomly varied the bid up or down every 7 days (some days of the week perform consistently better than others for my product). The graphs below show the results. Each data point is 7 days of data. The black lines are linear trend lines. I deliberately haven’t put values on the axes, but the x and y axes are all linear, starting at 0.

The trends are pretty clear. Increasing CPA bid price:

increases the number of times your ads are shown
makes little difference to the click through rates
decreases the click to download and download to sale ratios

So higher bids means more sales, but also a higher cost per sale. But, of course, the really important metric is profit. So I worked out the average daily profit from Adwords traffic, which is the net sales income (gross sales minus sales costs, including payment processor fees and support costs) minus Adwords costs. Again each point is 7 days of data. The black line is a 2nd order polynomial trend line.

The data is quite noisy. But some data is a lot better than none and there does appear to be sweet spot about where the red arrow is. The curve is fairly flat meaning that I don’t have to be too precise in my bid price to get a near to optimal return. But if I bid twice the optimal price my profit will drop by about 35%.

In an ideal world I would have run all these different bid prices concurrently, instead of one after the other. But that just isn’t possible with Adwords at present (you can use Google Adwords experiments to split test bid prices, but only 2 at a time). Also I could have gathered a lot more data, used longer time intervals (7 days probably isn’t long enough for Google CPA to get into its stride) and bid a lot higher and lower, to make the trends clearer. But I wasn’t prepared to spend the extra time and money required.

If you are using CPA bidding you should be able to carry out a similar analysis on your own Adwords account to find your own CPA bidding sweet spot. If you are still using CPC, perhaps you should consider switching to CPA and let Google do some of the heavy lifting for you. You can switch any campaign that has 15 or more conversions per month to CPA bidding in the ‘Settings’ tab.

You can always switch back to CPC later. If you aren’t using Google conversion tracking, well you really should be.

A word of warning. Not all downloads are equal. You might think that download to sale rates would vary a lot less than impression to click and click to download rates (I did). But download to sale ratios can vary a lot between different campaigns, even for the same product. For example, my analytics data shows that downloads from Adwords display (=contents/adsense) traffic only convert to sales at around a quarter of the rate of Adwords search traffic. So display campaign downloads are worth a lot less to me than search campaign downloads and I set my CPA bids accordingly.

I showed a draft version of this post to Alwin Hoogerdijk of Collectorz.com collection database software, who first persuaded me to switch back to CPA and knows a lot more about CPA than I do. He had the following to add:

When using CPA bidding you should give Google more room to experiment. On search, this means using more broad match terms, or at least modified broad match. And less negative keywords (I removed a lot of my negatives lately). The idea is that Google will automatically find out what works and what doesn’t (again, this may take a lot of time).

On the Content Network it means being less trigger-happy with the site exclusions. Without CPA bidding, I would be more likely to exclude generic sites like Facebook, about.com, etc… But with CPA bidding, I tend to allow the optimizer to display on those sites and the find the right pages within those sites to show my ads on.

In my experience, the optimal CPA bid can vary (wildly) between products, campaigns, etc. . Content Network CPA’s in general tend to be much lower, for the same products. Strangely enough, content network visitor sign up (or downloads) are worth less than search traffic sign ups. Which wasn’t what I was expecting. Of course, content network traffic is less targeted in general so one would expect a lower sign up rate. But even if those visitors sign up, they convert less well to actual sales too. Tricky.

3 Low-Competition Niches In Retail Software

Published 8 June 2011 article , guest posts , marketing , software 18 Comments
Tags: analytics, economic order quantity, EOQ, forecasting, ideas, Lokad, marketing, product, retail, scorecard, software, stock, supplier

This is a guest post by Joannes Vermorel, founder of the Lokad Forecasting Service.

Software developers seem to be herd animals. They like to stay very close to each other. As a result, the marketplace ends up riddled with hundreds of ToDo lists while other segments are deserted, despite high financial stakes. During my routine browsing of software business forums, I have noticed that the most common answer to Why the heck are you producing yet another ToDo list? is the desperately annoying Because I can’t find a better idea.

This is desperately annoying because the world is full (saturated even) with problems so painful that people or companies would be very willing to pay to relieve the pain, even if only a little. A tiny fraction of these problems are addressed by the software industry (such as the need for ToDo lists), but most are just lacking any decent solution.

Hence, I detail below 3 low-competition software niches in retail. Indeed, after half a decade of running sales forecasting software company Lokad, I believe, despite the potential survivor bias, that I have acquired insights on a few B2B markets close to my own. Firstly I will address a few inevitable questions:

Q: If you have uncovered such profitable niches, why don’t you take over them yourself?
A: Mostly because running a growing business already takes about 100% of my management bandwidth.

Q: If these niches have little competition, entry barriers must be high?
A: Herding problems aside, I believe not.

Q: Now these niches have been disclosed, they will be swarmed over by competitors, right?
A: Odds are extremely low on that one. The herd instinct is just too strong.

Q. Do I have to pay you if I use one of your ideas?
A. No, I am releasing this into the public domain. I expect no payment if you get rich (unless you want to!) and accept no liability if you fail miserably. Execution is everything. And don’t trust a random stranger on the Internet – do your own market research.

Before digging into the specifics of those niches, here are a couple of signs that I have noticed to be indicators of desperate lack of competition:

No one bothers about doing even basic SEO.
No prices on display.
No one offers self-signup – you have to go through a sales rep.
Little in the way of online documentation or screenshots are available.

However, lack of competition does not mean lack of competitors. It’s just not the sort of competition that keeps you up at night. Through private one-to-one discussions with clients of those solutions, here is the typical feedback I get:

Licenses are hideously expensive.
Setup takes months.
Upgrade takes months (and is hideously expensive).
Every single feature feels half-baked.

By way of anecdotal evidence – during a manufacturer integration with our forecasting technology a few months ago at Lokad, we discovered that the client had been charged $2,000 by its primary software provider in order to activate Remote Desktop on the Windows Server where the software was installed. Apparently, this was well within the norm of their usual fees for the inventory management system in place.

Granted, just being cheaper is usually not a good place to be in the market. Yet, when a competitor’s software is designed in such a way that it takes a small army of consultants to get it up and running, they can’t just lower their license fees to match yours – assuming that your design is not half-baked too. The competition would have to redesign their solution from scratch, and give up on their consultingware revenues. So you are in a great position to drive competition crazy.

With a market managing over two-thirds of the US gross domestic product, one would expected retail be saturated by fantastic software products. It turns out this is not the case. Not by a long shot – except eCommerce (e.g. online shopping carts) which attracts a zillion developers for no good reason.

Some salient aspects of the retail software market:

Most retailers are already equipped in basic stuff such as point-of-sale, inventory management and order management systems. So you don’t have to deliver that yourself. On the contrary, you should rely on the assumption that such software is already in place.
As far the Lokad experience goes with its online sales forecasting service, retailers are not unwilling to disclose their data to a 3rd party over the Internet. It takes trust and trust takes time. Interestingly enough, at Lokad we do sign NDAs, but rather infrequently. We are not unwilling, but most retailers (even top 100 worldwide ones) simply don’t even bother.
Retailers have a LOT of data, and yet unlike banks, they have little talented manpower to deal with it. Many retail businesses are highly profitable though and could afford to pay for this kind of manpower, but as far I can tell, it’s just not part of the usual Western retail culture. Talents go to management, not to the trenches.

Niche 1: EOQ (Economic Order Quantity) calculator

Retailers know they need to keep their stocks as low as possible, while preserving their service levels (aka rate of non stock-outs), see this safety stock tutorial for more details. If the marginal ordering cost for replenishment was zero, then retailers would produce myriads of incremental replenishment orders, precisely matching their own sales. This is not the case. One century ago, F. W. Harris introduced the economic order quantity (EOQ) which represents the optimal quantity to be ordered at once by the retailer, when friction factors such as the shipping cost are taken into account. Obviously, the Wilson Formula (see Wikipedia for details) is an extremely early attempt at addressing the question. It’s not too hard to see that many factors are not accounted for, such as non-flat shipping costs, volume discounts, obsolescence risks …etc.

Picking the right quantity to order is obviously a fundamental question for each retailer performing an inventory replenishment operation. Yet, AFAIK, there is no satisfying solution available on the market. ERP systems just graciously let the retailer manually enter the EOQ along with other product settings. Naturally, this process is extremely tedious, firstly because of the sheer number of products, secondly because whenever a supply parameter is changing, the retailer has to go through all the relevant products all over again.

The EOQ calculator would typically come as multi-tenant web app. Main features being:

Product and supplier data import from any remotely reachable SQL database[1].
Web UI for entering / editing EOQ settings.
EOQ calculation engine.
Optional EOQ export back to the ERP.

Pricing guestimate: Charge by the number of products rather than by the number of users. I would suggest to start around $50/month for small shops and go up to $10k/month for large retail networks.

Gut feeling: EOQ seemingly involves a lot of expert knowledge (my take: acquiring this knowledge is a matter of months, not years). So there is an opportunity to position yourself as an expert here, which is a good place to be as it facilitates inbound marketing and PR with specialized press. Also, EOQ can be narrowed down to sub-verticals in retail (e.g. textiles) in case competition grows stronger.

Niche 2: Supplier scorecard manager

For a retailer, there are about 3 qualities that define a good supplier: lowest prices, shortest shipment delays, best availability levels (aka no items out-of-stock delaying the shipments). Better, sometime exclusive, suppliers give a strong competitive edge to a retailer. Setting aside payment terms and complicated discounts, comparing supplier prices is simple, yet, this is only the tip of the iceberg. If the cheapest supplier doesn’t deliver half of the time, “savings” will turn into very expensive lost sales. As far I can observe, beyond pricing, assessing quality of the suppliers is hard, and most retailers suffer an ongoing struggle with this issue.

An idea that frequently comes to the mind of retailers is to establish contracts with suppliers that involve financial penalties if delays or availability levels are not enforced. In practice, the idea is often impractical. Firstly, you need to be Walmart-strong to inflict any punitive damage on your suppliers without simply losing them. Secondly, shipping delays and availabilities needs to be accurately monitored, which is typically not the case.

A much better alternative, yet infrequently implemented outside the large retail networks, consists of establishing a supplier scorecard based on the precise measuring of both lead times (i.e. the duration between the initial order and the final delivery) and of the item availability. The scorecard is a synthetic, typically 1-page, document refreshed every week or every month that provides the overall performance of each supplier. The scorecard includes a synthetic score like A (10% best performing suppliers), B and C (10% worst performing suppliers). Scorecards are shared with the suppliers themselves.

Instead of punishing bad suppliers, the scorecard helps them in realizing there is a problem in the first place. Then, if the situation doesn’t improve after a couple of months, it helps the retailer itself to realize the need for switching to another supplier…

The scorecard manager web app would feature:

Import of both purchase orders and delivery receipts (this might be 2 distinct systems). [2]
Consolidation of per-supplier lead time and availability statistics.
One-page scorecard reports with 3rd party access offered to the suppliers.

Pricing guestimate: Charge based on the number of suppliers and the numbers of orders to be processed. Again, the number of users having access to the system might not be a reliable indicator. Starting at $50/month for small shops up to $10k/month for large retail networks.

Gut feeling: By positioning your company as intermediate between retailers and their suppliers, you benefit from a built-in viral marketing effect, which is rather unusual in B2B. On the other hand, there isn’t that much expert knowledge (real or assumed) in the software itself.

Niche 3: Dead simple sales analytics

Retail is a fast-paced business, and a retailer needs to keep a really close eye on its sales figures in order to stay clear of bankruptcy. Globally, the software market is swarming with hundreds of sales analytics tools, most of them being distant competitors of Business Objects acquired by SAP years ago. However, the business model of most retailers is extremely simple and straightforward, making all those Business Intelligence tools vast overkill for small and medium retail networks.

Concepts that matter in retail are: sales per product, product categories and points of sale. That’s about it. Hence, all it should take to have a powerful sales visualization tool setup for a retailer should be access to the 2 or 3 SQL tables of the ERP defining products and transactions; and the rest being hard-coded defaults.

Google Analytics would be an inspiring model. Indeed, Google does not offer to webmasters any flexibility whatsoever in the way the web traffic is reported; but in exchange, setting-up Google Analytics requires no more than merely cutting-and-pasting a block of JavaScript into your web page footer.

Naturally it would be a web app, with the main features being:

Product and sales data import from any remotely reachable SQL database.[2]
Aggregate sales per day/week/month.
Aggregate per product/product category/point-of-sales.
A Web UI ala Google Analytics, with a single time-series graph per page.

Pricing guestimate: Regular per-usage fee, a la Salesforce.com. Starting at $5/user/month basic features to $100/user/month for more fancy stuff.

Gut feeling: probably the weakest of the 3 niches, precisely because it has too much potential and is therefore doomed to attract significant attention later on. Also, achieving a wow effect on first contact with the product will probably be critical to turn prospects into clients.

Market entry points

Worldwide, there are plenty of competitors already for these niches. Yet, again, this does not mean much. Firstly because retail is so huge, secondly because it’s a heavily fragmented market anyway. First, there are big guys like SAP, JDA or RedPrairie, typically way too expensive for anything but large retail networks. Second, there are hundreds of mid-market ERPs, typically with a strong national (or even regional) focus. However, those ERPs don’t delve into fine-grained specifics of retail, as they are too busy already dealing with a myriad of feature requests for their +20 modules (accounting, billing, HR, payments, shipping … etc). Hence, there is a lot of space for razor-sharp web apps that focus on one, and only one, aspect of the retail business. Basically single-minded, uncompromising obsession with one thing, leaving aside all other stuff to either ERPs or other web apps.

In order to enter the market, the good news is that mid-size retailers are pretty much everywhere. So you can just use a tiny bit of networking to get in touch with a couple of neighbouring businesses, even if you don’t have that much of a network in the first place. Then, being razor-sharp in a market where very little online content is available, offers you a cheap opportunity at doing some basic SEO based on the very specific questions your software is addressing.

Q: I am interested, I have questions, can I ask you those questions?
A: Naturally, my rate is 200€/h (no just kidding). Yes, email me.

[1] Don’t even bother about providing a super-complicated setup wizard. Just offer a $2k to $5k setup package that includes the ad-hoc handful of SQL lines to match the existing data of the retailer. We are already using this approach at Lokad with Salescast. Alternatively, we also offer an intermediate SQL schema, if the retailer is willing to deal with the data formatting on its own.

[2] Again, I suggest an approach similar to the one of Salescast by Lokad: don’t even try to robotize data import, just design the software in such a way that adding a custom adapter is cheap.

Joannes Vermorel is the founder of Lokad, company motto “You send data, we return forecasts”. Lokad won the first Windows Azure award from Microsoft in 2010, out of 3000 companies applying worldwide. He has a personal blog that mostly deals with cloud computing matters.

15 criteria for evaluating software product ideas

Published 26 May 2011 article , marketing , microISV , software 11 Comments
Tags: evaluate, ideas, market, products, software

Choosing the right product to develop is crucial. Great execution is also very important. But if you develop a product that no-one wants or no-one is prepared to pay for, then you are going to fail, no matter how well you execute it. You can often tweak a product or its marketing to make it more successful based on market feedback (‘pivot’) . But the less pivoting you have to do, the better. Below I list some of the criteria I think are important for evaluating the potential of new commercial software products.

1. Are you solving a real problem?

Has your customer got a ‘bleeding neck’? Is your software solving a problem compelling enough that someone is going to download it, install it, evaluate it, buy it and then learn to use it, with the accompanying risks of credit card fraud and malware? It is hard to change people’s habits. They are going to keep doing what they are doing now (e.g. pen and paper or Excel) unless you can convince them your software offers them very significant advantages.

2. How much will people pay for this product?

This is a complex question and depends on many factors. You should be able to get a rough idea by looking at your closest potential competitors. But there are some types of software that people don’t expect to pay for, no matter how difficult or expensive it is to develop – for example web browsers and media players. There are some users who can’t pay – for example children and people in some developing countries. And there are some people who won’t pay – for example many Linux users. So good luck selling a media player aimed at teenage Linux users in China.

3. Is the market big enough?

Is the market big enough for you to make a living? How many people are looking for solutions to this sort of problem? This is less of a problem than most people think. Given the huge number of people with Internet access and credit cards it is possible for a small company to make a decent living from a market that appears very narrow. Narrowing your market also allows you to be much more focussed in your marketing.

4. Can you promote it cost effectively?

How are you going to reach customers: Adwords, SEO, partners, magazine ads, direct mail, social media, affiliates, resellers or other methods? Can you do it cost effectively? How much is each sale from Adwords going to cost you assuming a 1% conversion rate? If it costs you $31 in advertising for each sale of a $30 product, you aren’t going to be in business long. But if you can cross-sell it to customers you already have a relationship with, that is a huge plus.

5. How much competition is there?

If there are lots of established competitors, you may have a hard time getting noticed. Personally I wouldn’t want to go into any market where I didn’t have a reasonable shot of getting to the first page on Google for at least some of the important search terms. For example, I think it would be incredibly tough to succeed with yet another Twitter, RSS, todo list or backup application. Conversely, if there are no competitors, that means that there may be no market. Creating a new market is tough, especially for a small company. Ideally you want a market where there are competitors making a decent living, but you think you can do a better job than them, or at least be different to them in some important way.

6. How is your product different?

Many vendors try very hard to reach feature parity with their competitors. But successful marketing means being different to your competitors. How is your product going to be different? What is your positioning? Note that just being cheaper than your competitors is not enough.

7. How high is the barrier to entry?

How long will it take you to create a minimum sellable product? If the barrier to entry is too high, you may never have the time, cashflow and energy to reach v1.0. As a self-funded microISV I wouldn’t want to work on any product where I couldn’t deliver something sellable (a minimum viable product) within 6 months. Conversely if the barrier to entry is too low, then it will be easy for others to copy your idea if it is successful.

8. Can you reach critical mass?

Some types of applications need a certain number of users before they can take off (network effect). For example, a massively multi-player game, dating site or auction site isn’t going to be very interesting until the number of users reaches a certain threshold. Do you have the contacts and financial resources to reach this threshold?

9. Do you have the technical skills and domain knowledge to create this product?

If not, how long will it take to learn them? Different technologies suit different types of problems. Using an inappropriate technology, just because it is one you have experience in, is unlikely to end well.

10. Are you scratching your own itch?

If you can be your own customer, then this can be very helpful in coming up with a good solution. But be wary about assuming that your needs are the same as everyone elses.

11. What is the lifetime of the product?

Is the technology is going to be obsolete or will the market disappear within a couple of years? Are customers likely to buy upgrades to new versions? The longer you can sell a product for, the more profitable it is likely to be.

12. Is a good domain available?

Can you get a good domain for your product? Domains that contain keywords that people are likely to search on will help with SEO.

13. What are the risk factors?

Every dependency is a risk factor. If the platform your products runs on dies, then your product dies. If you are writing an add-on for another product, then you can be put out of business pretty much overnight if the core product dies or if the functionality of your add-on is incorporated into the core product. Can you get source code for third party libraries?

14. Is the passion there?

Good software takes a lot of time and effort. Don’t believe the hype about 4 hour work weeks. Is it going to be interesting and fun? Do you have the passion and commitment to still be working on this product in 10 years time?

15. Will it make the world a better place?

Software products can be an enormous force for good in the world, increasing productivity and allowing people to do things they couldn’t do otherwise. You don’t have to be the next Google to be doing something worthwhile. But creating a “me too” clone of an existing software package or a product that encourages anti-social behaviour (e.g. spamming) isn’t going to make the world a better place.

Making a decision

You need to look at all these criteria before you make a decision. For example, a short lifespan or a small market might be compensated for by a high ticket price. If you are evaluating several products, create a simple table with a row for each criteria and a column for each product and compare them side by side.

Did I miss any important criteria?

A small experiment with LinkedIn ads

Published 23 February 2011 article , marketing , reviews , software 15 Comments
Tags: advertising, cpm, linkedin, linkedin.com, marketing, metrics, pay per click, ppc, promotion

LinkedIn.com (the B2B equivalent of Facebook) supports Google style pay per click ads. So I decided to run some ads for my seating planner software as an experiment. Here is a brief summary of my (very brief) experiences.

The good news

LinkedIn ads can be laser targeted. You can specify who you want to see your ad based on their job function, company, gender, age group, country and (best of all) the LinkedIn groups they belong to. I targeted 10,102 LinkedIn members who live in wealthy English speaking countries, belong to various LinkedIn groups related to event planning and have appropriate job titles. The campaign was quite painless to set up. It probably took me less than 10 minutes in total and I started getting impressions within an hour or so.

The bad news

The minimum allowed CPC (cost per click) was $2. Ouch. I know from extensive experience with Google Adwords that there is no way I can get a return on that.

The minimum allow CPM (cost per thousand impressions) was $3. If the CTR (click through rate) is around 1% (about what you might expect from Google search ads) this is $0.30 per click. Possibly profitable. If the CTR is around 0.1% (about what you might expect from Facebook ads) this is $3 per click. No better than the CPC bidding. Given that LinkedIn is more similar to Facebook than Google search, I expected the latter. I decided to spend a few dollars to find out. The results are below (click to enlarge):

So, with an average 0.17% CTR, I ended up spending $1.76 per click. Given my average transaction value and a realistic conversion rate I know that I can’t make any return on this. Also the CTR is likely to drop the more often people see the ad. So I stopped the experiment after less than 24 hours, before I wasted any more time or money. As far as I can tell (based on my own cookie tracking - LinkedIn ads don’t have their own conversion tracking) I didn’t make any sales. But that is hardly surprising given the small number of clicks.

Summary

Obviously $19.38 is a tiny amount to spend, but I think it told me what I needed to know about LinkedIn ads. Unless they reduce their CPC or CPM bid prices by an order of magnitude there is no way I can make a return. Of course, if you are selling a product where the average lifetime value of a customer is hundreds or thousands of dollars, the numbers might work out quite differently for you.

Advertising your software on Facebook (=Fail)

Does the world really need yet another Twitter client, RSS reader, ToDo list or backup application?

Published 1 February 2011 article , marketing , microISV , software 26 Comments
Tags: backup, competition, marketing, markets, microISV, niches, rss, software, todo, twitter

My heart sinks every time I hear a would-be-entrepreneur announcing they have written yet another Twitter client, RSS reader, ToDo list or backup application. Haven’t we got enough of those already? There are more than 1,900 Twitter apps already (possibly a lot more). Somebody probably released another one while I was writing this post. We have passed the Twitter app event horizon, where it is probably quicker to write your own custom app than it is to try and work out if any of the existing apps fulfils your requirements.

Even if you have done something radically new, interesting and different in one of these markets, how are you ever going to get noticed amongst thousands of more established competitors? Wouldn’t it be better to find a market that is currently under-served by software? It may be less fashionable than writing software for other techies, but it will probably contribute more to the sum of human happiness and be a lot more profitable.

There must be thousands of niches where there is a real need for software, but limited competition. You just need to open your eyes to the bigger world around you. It may mean having to learn about an unfamiliar domain. But it is generally much easier for a software developer to learn some domain knowledge about, say, butterfly collecting, than it is for the average butterfly collector to learn to create a software product. Next time you are talking to a non-techie about their job or hobbies, just ask them “Do you use software for that?” and “Is it any good?”. The ideal answers you are looking for are “Yes” (if there are existing software packages, there is probably a market) and “No” (maybe you can do better).

How to remove software cracks and keygens from file hosting sites

Published 25 January 2011 article , guest posts , microISV , piracy , software , tools 32 Comments
Tags: crack tracker, cracks, cracktracker, DMCA, keygens, microISV, Nikos Bozinis, piracy, rapidshare, software, warez, xplorer2, zabkat

Software piracy is a real issue for every software company, large and small, and it isn’t going away any time soon. So when I heard that fellow microISV owner Nikos Bozinis had created a tool to help software vendors fight piracy, I asked him to write a guest post. He kindly agreed to write this post about software piracy, the Digital Millennium Copyright Act and his CrackTracker product.

Why buy something when you can download it ‘for free’? Billions of dollars are lost every year from illegal downloads of music, movies and software. People around the world seem to have very lax morals when it comes to abusing digital content. Downloading the latest movie or windows software from rapidshare.com somehow doesn’t strike them as theft — it’s not like stealing a loaf of bread! The traditional music industry is already down on its knees as a result, and software may be the next to follow.

Software authors and music enterprises are fighting back by tightening the DRM (Digital Rights Management) of their products in a futile effort to stop online piracy. But usually crackers have no problem circumventing any protection system that we can dream up. To add insult to injury legitimate customers are usually hurt by such reinforced software protection and activation systems. A little bit like the war on terror, isn’t it?

A different line of defense for ailing copyright owners is the Digital Millennium Copyright Act (DMCA), a US law with global reach for copyright protection (the european EUCD equivalent is not as broadly known). This law is very broad, and not without controversy, but it works – closing down websites that distribute illegal content and removing copyright infringing downloads from file-hosting websites with summary procedures, among other things. So if you discover your software illegally distributed in some warez website, you can send a so called “DMCA section 512 takedown notice” to the website host and they are expected to remove that particular file from circulation — or risk the wrath of the law.

Software Piracy

I have been a microISV for over 10 years so lets forget about the entertainment industry and concentrate on my field, software. There are over 200,000 programs listed on download.com and that’s just for Windows. Many are created by very small to medium sized companies — many even run by a single programmer/webmaster/marketer/entrepreneur. I bet that all these programs are cracked in one way or another — at least those popular enough for crackers to care about them. If you search for warez or torrents you will find the software you want for free, either the latest or an older working version.

Piracy statistics from Business Software Alliance report 2009 (click image to enlarge).

I sell a file manager called xplorer². I track how many people install the program every day and also I have a good guesstimate for the number of people using cracked versions of xplorer². I estimate over 70% of the regular users use one of the known keygens. Imagine if this 70% didn’t exist or it was converted to regular paying customers!

How is it done?

Downloadable software falls into 2 categories: those that run in trial mode until you buy a key to unlock the full functionality; and those that are special downloads for customers that pay the registration fee. In all cases some sort of unlocking takes place using a plain key, or a license file, or online activation, or some combination thereof. Many ISVs write their own licensing code, while others rely on off-the-shelf protection and licensing products (Armadillo, WinLicense etc).

Imagine you shipped your source code along with your program, then it would be trivial for even amateur crackers to bypass your protection and run the program without paying. Very few vendors supply source code, but people in the know can read off your licensing logic like an open book using specialized reverse engineering tools (softICE, IDA and other debuggers and disassemblers). Then they can create a ‘patch’ or modification to your executable that bypasses the protection.

An even worse type of compromise is a keygen. When the cracker uncovers the logic of your unlock keys, he can create a program to generate such keys which look and behave exactly like the legitimate ones you sell to your customers. Then he doesn’t need to patch your program, he just supplies this keygen to the warez community and everyone can help themselves to your program. You can guard yourself against such attacks using asymmetric encryption algorithms for your keys.

Is there a perfect protection system?

In short, no. If you consider that your program is presenting its logic to anyone with moderate experience in machine language, then sooner or later any protection can be circumvented. Professional protection schemes utilize encryption to protect sensitive parts of your code, but even they won’t withstand the cracker test. And remember the harder your DRM the more likely your program will be mistaken for malware (!) as many viruses and trojans use encryption tricks.

Even if there was a perfect system, your sales would still be at risk. All that’s required is some of your customers to post their unlock key in a warez site, and the game is lost. You would then blacklist that serial, until another one was leaked and so on.

The warez scene

There are people who don’t spend any time in Facebook or YouTube. They surf the internet for free stuff. Cracked versions of commercial software (aka warez) circulate in some shady forums that bring together the crackers with the downloaders e.g. www.warez-bb.org. Browse a warez site and you will find any software, movie or music you fancy, with an assortment of popups and dodgy advertisements of the usual internet 3P products (Pills, Poker and Girls [sic]). For your convenience there are even specialized search engines that search a number of such forums simultaneously, e.g. www.warez.com.

These forums do not host the actual files. They refer the traffic to specialized file hosting services like rapidshare.com. To make the most of warez you need to buy a subscription to access such file hosting sites (e.g. unlimited downloads from $9/month). Incurable cheapskates could get away without paying anything though, as you can download for free after a forced (nag) waiting of a minute or two.

A bit more up-market are download sites where to gain access you need to purchase a subscription, e.g. www.nowdownloadall.com. I have never paid to enter such a site, but they promise access to any download you can imagine. So you pay a monthly fee to download as much as you like. Note that this is different from paid-for hosting mentioned above. I suppose that you need a file hosting subscription on top to get the actual files downloaded. With so much stuff available for free I don’t know if this approach makes economic sense.

Finally there are traditional peer-to-peer file sharing networks, where people share their software music and video through torrents. After the demise of Napster torrents are still strong, with completely decentralized databases immune to legal intervention. The downside of torrents is their inherent unreliability, so people in a hurry will prefer the immediate gratification of a full download from rapidshare.com and the like.

Why do they do it?

It is easy to understand why someone will prefer ‘free’ software instead of paying up. But what about the crackers, the people who circumvent the DRM and distribute these warez. Why do they do it? Here are a few plausible motives:

For kicks. The traditional hacker stereotype is a geeky person whose pastime is breaking into computer networks. Cracking into a software’s protection and stripping it clean must be a pleasure in itself, a ritual destruction of the evil Death Star.
For glory. Marxist theory claims that private property is theft. This concept has struggled with real tangible property, but digital property is the ideal trophy. Many groups feel that software and music should be free (!) so taking down the big media and software corporations is a noble cause for them. But many small ISVs fall victims too, and the real motives are far less revolutionary…
For profit. Marx is dead; long live Das Kapital. Warez downloads are big business in a number of ways:
- Direct subscriptions charges to access the downloads
- Selling password unlockers (e.g. you download something in a ZIP archive which is locked and you need to buy some software to unlock it)
- Distributing malware. Many downloads are packed with malware (sample report for a keygen), from straightforward scams and ransomware to trojans that turn your computer to a zombie, waiting for instructions to launch a DDoS attack or send spam.

You can remove illegal downloads

If your software is available to download from warez sites, either compromised (patched or keygened) or simply accompanied by a simple serial number to unlock it, you will definitely lose sales. The good news is that, using DMCA provisions, you can have these unauthorized downloads removed. Without these downloads prospective users will have no choice but to buy your software — or move on to your competitor’s cracked software.

Here is how to remove illegal downloads:

Find your download links. All illegal downloads end up in a host like rapidshare.com or megaupload.com (I know of more than 100, but there are 10-20 big player websites). A standard Google search for your software name plus ‘crack’, ‘keygen’ or ‘rapidshare’ will find some hits, especially if you search in groups or blogs. Even better use specialized warez search engines like www.filestube.com with just your software name as a keyword — the results will be just downloads.
Validate download URLs. Some of the download links you discover may be dead (e.g. very old). Click on each one to see if they are valid or 404.
Send DCMA notices. Group the download links by provider (rapidshare, hotfile, etc), and send a DMCA notice to the abuse email address of each website. Usually this is abuse@website.com (e.g. abuse@rapidshare.com). Each website lists the steps for filing DMCA notices for file removal.

This sounds like a lot of hard work, and it can be, but it works. File sharing websites like rapidshare.com run a legitimate business — they are not responsible for cracks — so if you send them a polite DMCA takedown notice they will remove the copyright infringing downloads.

The DCMA takedown notice

Strictly speaking when you send a DMCA notice you are making allegations of copyright infringement, which is a serious crime. You would imagine that a formal complaint should be launched under the guidance of a solicitor/lawyer. Given the amount of copyright infringement that goes on, the red tape would bring everything to a standstill. The beauty of the DMCA law is that it simplifies the procedure. Sometimes a plain English email explaining the situation to the download site, along with a list of your download locations is all that’s required to have the links removed.

A few websites require a more formal DMCA email including details such as your company address, contact telephone numbers, and some boilerplate statements like “I swear, under penalty of perjury, that the information in the notification is accurate…”. You can find many sample DMCA notices online so I won’t repeat them here. The general idea is that you present yourself as the copyright owner and declare the download URLs as unauthorized, and therefore infringing your copyright.

Torrents slip by

DMCA is very good for removing illegal downloads hosted in popular file sharing websites, but it is powerless against torrents. There is no single source for the download, as the files are kept in many computers. You would have to contact each and every person who shares illegal copies of your software in the peer-to-peer network. This would be hopeless and a waste of effort. Thankfully for the ISV, torrent use is on the decline. People prefer direct downloads of the full package instead of slower peer-to-peer downloads.

The sales pitch

Anyone can search and remove illegal downloads manually. I was doing it the hard way for quite some time, each time I released a new version of my software tool (there’s a lot of cracker activity for each release as they need to update their patches and keygens). However this is very tedious, as you must:

enter shady warez forums to search for your keyword, facing annoying popups and adverts you wouldn’t want your wife to see
search many locations to ensure you get as many download URLs as possible
validate each download URL to see if it is still alive or dead
organize download URLs and write DMCA takedown emails for each file hosting website

Even if one wipes all the illegal downloads, new ones will appear over time. So the locate-report-remove cycle must be repeated regularly. This was the motivation for writing Crack Tracker, a tool that simplifies the removal of illegal downloads.

Crack Tracker is a desktop tool, with a meta search engine that securely scans warez databases for your downloads. You supply the search keyword (e.g. your software title or company name) then crack tracker will do an exhaustive search, collect a list of suspect download locations and verify the links with robotic efficiency. After you examine the results you just hit a button and the relevant DMCA emails are sent automatically. It doesn’t get any easier than that.

Crack Tracker doesn’t have a fancy user interface but it is very easy to use. It knows of more than 120 file hosting websites and works with 6 major warez search engines (the list is expanding). It is free to try as a search engine; to send the actual DMCA emails you need a registration, but I believe the price is very reasonable, especially if you consider the money you lose in pirated versions of your software.

Why don’t you try it for free and see how many cracks of your software it finds?

Download CrackTracker for Windows (318KB)

Nikos Bozinis ditched his Process Systems Engineering PhD to run his own microISV ZABKAT since 1999. He also writes a weekly blog focusing on file management and occasionally on programming, debugging and running a software business.

Archive for the 'article' Category

Share this:

The past is always with us

Tightening trousers

Success!

The slow carb diet

Pick an exercise regime you can stick with

Weigh yourself every day

A bit of science

Watch what you drink

Food supplements

Social factors

Be realistic

The long term

A call to action

Share this:

Time-limited

Usage-limited

Feature-limited

Capacity-limited

Output-limited

Nagware

No trial

Hybrid trial

Conclusion

Share this:

Share this:

Share this:

Niche 1: EOQ (Economic Order Quantity) calculator

Niche 2: Supplier scorecard manager

Niche 3: Dead simple sales analytics

Market entry points

Share this:

1. Are you solving a real problem?

2. How much will people pay for this product?

3. Is the market big enough?

4. Can you promote it cost effectively?

5. How much competition is there?

6. How is your product different?

7. How high is the barrier to entry?

8. Can you reach critical mass?

9. Do you have the technical skills and domain knowledge to create this product?

10. Are you scratching your own itch?

11. What is the lifetime of the product?

12. Is a good domain available?

13. What are the risk factors?

14. Is the passion there?

15. Will it make the world a better place?

Making a decision

Share this:

The good news

The bad news

Summary

Share this:

Share this:

Software Piracy

How is it done?

Is there a perfect protection system?

The warez scene

Why do they do it?

You *can* remove illegal downloads

The DCMA takedown notice

Torrents slip by

The sales pitch

Share this:

Follow Blog via Email

Blog Stats

Top Posts

Recent posts

Categories

Archives

Blogroll

Follow “Successful Software”

You can remove illegal downloads