Tag Archives: experiment

Remarketing – does it work?

remarketingIf you go to Amazon and browse watches, you will suddenly notice a preponderance of watch ads everywhere you go on the Internet. This is ‘remarketing’ (also known as ‘retargeting’ or, more colloquially, ‘cyber stalking’). Wikipedia defines it as:

a form of online targeted advertising by which online advertising is targeted to consumers based on their previous Internet actions, in situations where these actions did not result in a sale or conversion.

Given that:

a) It is hard work to get potential purchasers to visit your website

b) some 99% of visitors to a typical website leave without buying anything

It seems to make sense to spend some time and money reminding non-purchasers to come back to your website in the hope that they will eventually purchase.

The basic mechanics of remarketing are:

  1. Sign up with a remarketing publisher such as Google, Perfect Audience or Adroll.
  2. Upload some graphical ads.
  3. Bid to show these ads on other sites.
  4. Add a script on your site which cookies visitors for remarketing.
  5. When your visitor leaves your site and goes to another site in the remarketing publisher’s network, the remarketing cookie is read and an ad is shown (or not, depending on how much you and other advertisers are bidding per impression).
  6. Hopefully people will see your ad, click through and buy your product. Or they may just be reminded to continue the trial, without clicking the ad.

A number of people I have spoken to told me it was very cost-effective. But when I asked how they knew that these remarketing conversions wouldn’t have purchased anyway, I didn’t get a satisfactory answer. It seems straightforward enough to test this: run an A/B test, showing remarketing ads to 50% of your visitors and see what difference it makes to conversions. But an online search and some asking around turned up very little data. The one decent study I found quoted an 18% increase in conversions (yes please!), but was for an ‘e-commerce website’. So, dear reader, I have done the experiment using Perfect Audience for remarketing, Visual Website Optimizer for A/B testing and my table planner software as the subject. Here are my results:

visits sales conversion
Control (No
remarketing )
10,539 162 1.54% (±0.15)
Remarketing 11,137 174 1.56% (±0.15)

So the remarketing showed an increase in sales of 1.6% over 21,676 visits and 336 sales. However it is noticeable that the 95 percentile error bars are rather large compared with the conversion rates. I am only 95% sure that the conversion rates are in the range 1.69% to 1.39% (control) and 1.71% to 1.41% (remarketing). Which means the change in conversion rate could be anything from +23% to -19% (but given normal distribution curves, most likely somewhere in the middle). According to Visual Website Optimizer, we can only be 56% sure that the increase in conversions is a real effect, and not just a statistical anomaly. The graph below shows the convergence of the conversion rates over time (blue is the control, orange is with remarketing).

remarketing experimentIt gets worse when you factor in the cost of the remarketing. I know the value of the sales and the cost of the ads over the period of the experiment. So I can work out that for every $1 I spent on remarketing I was getting around $0.95 back in extra sales. It isn’t looking like a winner for me, especially when you factor in the time taken to set-up and administer it.

Some points to note:

  • Remarketing resulted in 1.8% less installs than the control. This is probably just a statistical anomaly (67% chance of being statistically significant).
  • I choose Perfect Audience based on the recommendation of Rob Walling, who has experimented with Google, Adroll, remarketer.com and Perfect Audience. Unlike Google, Perfect Audience allows remarketing across a wide range of platforms and websites, including Facebook and Twitter. I found their system to be relatively flexible and easy to set-up. But being billed weekly is a bit tedious for my bookkeeper.
  • I showed my ads on Facebook and various websites. I didn’t show them on Twitter as my previous experiences with advertising on Twitter haven’t been great.
  • With remarketing you pay per impression, not per click. I set my CPM (cost per thousand impressions) relatively low. I ended up averaging $1.55 for web ads and $1.14 for Facebook ads.
  • Click through rates were miserable, averaging just 0.051% for both web and Facebook ads.
  • The average cost per click was $2.58. This is a lot more than I pay per click on Adwords.
  • I remarketed to people that arrived on my home page. I stopped targeting them after 30 days or after they had purchased.
  • I didn’t remarket to visitors from developing countries, as they very rarely buy my software. Had I remarketed to visitors from every country the remarketing conversion rate would probably have been slightly higher, but the ad costs would have been significantly higher.
  • I didn’t get any complaints from customers about being ‘stalked’.
  • I just knocked up some ad graphics myself (examples below). I got the idea for an attention-grabbing ugly ad here. It didn’t perform well though.


ugly handwriting 300x250 3I probably could probably improve the ROI on remarketing with some experimentation. E.g.:

  • Trying professionally designed ads.
  • Trying different bids.
  • Experimenting with only showing ads to people who have installed the trial vs only showing ads to people who haven’t installed the trial.

But it doesn’t really seem worth the opportunity cost given the results to date.

Of course, my experiment is just one data point. Remarketing might work better for you if you have a higher average lifetime value for a customer (many of my customers buy the $30 version of PerfectTablePlan for their wedding and never purchase from me again). If you have a B2B product with an average lifetime value in the hundreds or thousands of dollars, it is probably worth a try. You will have to run your own A/B test to find out. If you do, please let me know what the results are.

An experiment with Pinterest

Pinterest is the latest darling on the social media scene. Pinterest reportedly hit ten million monthly unique visitors faster than any other website. It is currently claimed to have some 15 million users. Basically it allows you assemble (‘pin’) images from anywhere on the web into themed folders (‘pinboards’). You can download browser plug-ins that allow you to pin an image from a website to one of your pinboards in a few clicks. The social element comes from ‘following’ other Pinterest users and commenting on and re-pinning their images. It is a simple idea slickly executed. The emphasis on images makes it rather different to Facebook, Twitter and other social media brands.

83% of US users of Pinterest are women and typical pinboard themes include:  fashion, funny images of cats, interior design, places to visit, food and wedding ideas. Check the Pinterest home page to see a sample of images currently being pinned. The interest in weddings is particularly relevant to me as I sell wedding seating planner software. So I started to create some wedding reception themed pinboards as an experiment. I quickly decided that scouring the web for pictures of pretty seating charts, place cards  and wedding cakes wasn’t a) a good use of my time b) the right thing for a 46 year old heterosexual man to be doing (GRRR!). So, using outsourcing site odesk.com, I found a nice lady in the Philippines to do it for me for a very reasonable hourly rate. She also (unsurprisingly) proved to have much better taste than me. I think she also really quite enjoyed herself! The resulting Pinterest pinboard is at http://pinterest.com/tableplan/.

Here is some data from my little experiment:

Cost: approx $50, plus a few hours of my time
Total ‘pins’ to date: 551
Clickthroughs to http://www.perfecttableplan.com/: 154
Avg time on site: 0:42
Bounce rate: 75%
Traceable sales (from analytics and cookie tracking): 0

So that works out >$0.30 per click for not very targeted traffic (as shown by the bounce rate and avg time on site) and not a single sale. Not very encouraging. What’s worse, it only generated traffic while new pins were being added. As soon as new pins were no longer being added the clickthroughs fell off a cliff:

I may get some SEO benefit. But Pinterest isn’t looking a like a win for me. Also there are issues with Pinterest terms and general IP issues. Pinterest’s terms originally claimed that they could sell anything you pinned. They have since amended that to something more sensible. But pinning images you don’t own the copyright to is still problematic. I think most sites will be happy for people to pin their images, as long as they are correctly attributed – it is free advertising for them.  And it is obvious that most Pinterest users are happily pinning images without any thought about copyright. But it could conceivably get you into trouble for copyright infringement.

If you are a big name company with a big marketing budget, it may be worth putting some effort into Pinterest. Especially if you already have a big catalogue of images you own. But, based on my experiences, I think most small software companies can find better ways to spend their time.

A test of Cost Per Action (CPA) vs Cost Per Click (CPC) in Google Adwords

CPA vs CPCThe traditional approach to Google Adwords is to set a bid price for each keyword. This is known as Cost Per Click (CPC). Google then then uses the bid prices in conjunction with a secret formula (the quality score) to decide how high to rank your ad in the Adwords results. If you bid more, your ad will appear higher and typically get more clicks, but your cost per click will increase. So setting an optimal bid price is important. Bid too little and you won’t rank high enough to get a decent number of clickthroughs. Bid too much and you will potentially end paying more to Google than you recoup in sales.

An alternative approach is to tell Google Adwords how much you are prepared to pay for a particular action, e.g. a sign-up, download or sale. This is known as Cost Per Action (CPA) or Conversion Optimizer. Google will then automatically calculate your bid prices and attempt not to exceed the CPA you set (although this isn’t guaranteed).

CPA sounds great. I can stay in bed a bit longer while the mighty Google brain does the bid tweaking for me. Unfortunately I wasn’t able to use CPA. I  count sales as conversions (not downloads) and I have my adwords account split into a number of campaigns by geographic region and by type (e.g. search vs content). Having my campaigns structured like this, rather than one monolithic campaign, makes for more flexibility (e.g. different ads, phrases and bid prices for different geographical areas) and more useful reports (e.g. separate reports for search and content). But it also meant none of my Adwords campaigns made the minimum threshold for conversions per month.

When Google dropped the minimum threshold for CPA to 30 conversions per campaign per month, one of my Perfect Table Plan search campaigns became eligible. So I did an experiment. I ran a campaign for 4 weeks using CPC, then 9 weeks using CPA, then another 4 weeks using CPC. I set the CPA bid to roughly the average cost per conversion I got for CPC. I was curious to see if Google would find sweet spots that I had been missing or whether they would bid as high as they could to take as much money off me as possible. Summary: CPC outperformed CPA on all key metrics, including: 4.4% higher conversions, 9.4% lower cost per conversions and 8.0% higher profit.

The detailed results are as follows:

metric CPC  (vs CPA)
impressions/day +13.9%
clicks/day +1.3%
conversions/day +4.4%
CTR -11.1%
conv rate +3.1%
income/day +4.4%
cost/day -5.5%
CPC -6.6%
profit/day +8.0%
PKI -5.2%
ROI +10.4%
cost per conversion -9.4%

In graphical form (click to enlarge):

CPA vs CPC graph 50pc


  • The values given are taken by computing (CPC metric – CPA metric)/(CPA metric). E.g. ROI of +10.4% means that CPC had a 10.4% higher ROI than CPA.
  • Only a single (geographically based) search campaign was measured. The total number of conversions during the time period of the test was in 3 figures.
  • I only measured sale conversions. This gives me less data than measuring downloads, but I think it is unsafe to assume the number of downloads correlates closely to the number of sales.
  • The PerfectTablePlan sale price is £19.95/$29.95. To calculate profit I only counted 75% of the price of a sale (the other 25% was assumed to cover the cost of support, ecommerce fees and other overheads associated with the sale).
  • Each of the time periods was a multiple of 7 days to avoid any issues with different results on different days of the week.
  • I ran CPC for an equal amount of time either side of the CPA test to try to balance out any seasonal factors.
  • Google conversion tracking uses Cookies and is therefore not 100% accurate.
  • PKI is Profit Per Thousand Impressions.
  • ROI is Return On Investment.

It wouldn’t be wise to draw any sweeping conclusions from one test with a limited amount of data. However I believe the results show:

  • A CPA campaign running for 9 weeks wasn’t able to outperform a mature CPC campaign. The CPC campaign had been running for over 4 years, but one would have thought CPA would have been able to use that pre-existing  data. CPA might have performed better if given longer. It would probably also have done better against a less mature CPC campaign.
  • Google didn’t rob me blind using CPA bidding. The CPA cost per day was only 5.5% higher.
  • The results weren’t hugely different. On the basis of the above results one might still conclude that CPA is superior to CPC as it requires less time to manage.