Tag Archives: software

Support is wild in an age of AI

2 Replies

Recently a customer emailed me:

Please consider re adding the option to convert from one time zone to another as was available in [Easy Data Transform] version 1.x. See the attached screen dump.

Easy Data Transform has never had a time zone conversion feature, so that is a bit strange. Although . But the screenshot really set off alarms bells as, despite saying “Easy Data Transform v1.11.2” in the title bar, that is not our software!

I thought that someone was trying to pass of their product as ours and did a search. But I couldn’t find any reference to another piece of software called “Easy Data Transform”. I emailed the customer to ask where they had got the software from. The reply came back:

The screenshot was from a ChatGPT output – maybe hallucinating.

Wow. Not only had ChatGPT hallucinated the feature, but also a fever dream screenshot of the user interface, with the non-existent feature. It looks like a real screenshot at first glance, but the icons are a giveaway if you look closer.

For reference, this is what the actual user interface looks like:

ChatGPT has got some of the input types, transforms and menus correct. But otherwise it looks quite different.

I had another brush with AI hallucinations when I asked MS Copilot how to perform ‘one hot encoding’ in Easy Data Transform. It came back with a very plausible and confident sounding answer, including this summary of the transforms required:

Just one problem – only 1 of these 4 transforms actually exists (Split Col). It hallucinated the other 3!

Customers are increasingly typing questions into AIs, rather than reading documentation or asking on a forum. That is good news if it means that the customer gets a quick and accurate answer without troubling busy developers. But it is very bad news if they are getting incorrect answers, especially when these answers look plausible and are confidently presented. It is galling enough that AIs are stealing all our web traffic, without them giving our customers bad support advice as well! Wild times are ahead.

Ps/ Time zone conversion and one hot encoding are now available in the latest version of Easy Data Transform.

Stop Publishing Garbage Data, It’s Embarrassing

3 Replies

Twice this week, I have come across embarrassingly bad data.

The first instance is the UK government’s fuel finder data. This is a downloadable CSV file of fuel station locations and prices from around the UK. A potentially very useful database, especially during the current conflict in the Middle East. A customer suggested it as a possible practice dataset for my data wrangling and visualization software, Easy Data Transform . So I had a quick look and spotted some glaring errors within a few minutes.

A quick plot of the latitude and longitude shows some clear outliers:

On further investigation, some of these UK fuel stations are apparently located in the Indian and South Atlantic oceans. In at least one case, it looks like they got the latitude and longitude the wrong way around.

A quick look at the fuel price columns also shows some major issues:

The ratio between the most expensive and cheapest fuel (per litre) is 1538:1. Clearly wrong.

Shown as a histogram with a logarithmic Y axis:

I am guessing that the reason for this bad data is that the fuel stations are submitting their own data and, humans being humans, they make mistakes. But then the government is publishing the data without even the most basic checks. That just isn’t good enough.

I reported the problem on 22-Mar-2026. They acknowledge my email on 24-Mar-2026 (“Thank you for sharing this, we have passed this on to the technical team to have a look at.”). The CSV file published on 29-Mar-2026 still has the garbage data.

The second instance is a report on electric cars from UK motoring organization, the RAC. The first graph in the article is this:

Did the number of Battery Electric Vehicles on the UK’s roads suddeny drop from ~1.4 million in 2024 to ~0.0017 million in 2025? What happened to those ~1.4 million vehicles? I’m guessing that someone got their thousands and millions mixed up. But then they published the report with this glaring error. Did anyone mathematically literate even check this graph?

Lousy data undermines trust in institutions and can lead to bad decisions. I fear we are heading for a future where LLMs generate data, which people don’t bother to properly check. This data is then used train LLMs. The error is then much harder to spot once it is served back without the original source by LLMs. A slop-apocalypse.

Authors should have their work proof read, programmers should test their code and data people should do basic data validation. Let’s take some pride in our work.

Transform Messy Data into Insights with Easy Data Transform Advanced Edition

2 Replies

My data wrangling product, Easy Data Transform, got a major upgrade yesterday, with the addition of data visualization capabilities. Here are some examples of what it can produce in a few clicks:

You can see this new visualization feature in action in the video below:

(Likes and subscribes help me with the YouTube algorithm and are much appreciated)

Human brains are highly developed for visual processing. But data is often messy and in the wrong form for visualization. So data wrangling and visualization, tightly integrated together, is a powerful combination. It was a lot of work, but it makes Easy Data Transform a much more complete, end-to-end, solution. No more pasting data into Excel to get a chart!

At the same time, I have segmented Easy Data Transform into 2 products: Easy Data Transform Core Edition (data wrangling, one-time fee $99) and Easy Data Transform Advanced Edition (data wrangling+visualization, one-time fee $198). The Advanced Edition is a paid upgrade from the original product. Optional, of course. I sweetened the deal with a time-limited discount for existing customers who want to upgrade to Advanced Edition. The response from Easy Data Transform customers has been very positive.

Have you got some messy data, you want to turn into insights? Why not give Easy Data Transform Advanced Edition a try? It:

  • Runs natively on Windows and Mac.
  • Is drag and drop (no syntax to remember).
  • Does not store your precious data on someone else’s server.
  • Is deterministic and will not hallucinate!
  • Can process millions of rows in seconds.
  • Can create re-usable templates for repeatable processes.

You can download a free trial here. And you can get 15% off Easy Data Transform Advanced Edition until 17-Jul-2026 using this discount link.

Is the golden age of Indie software over?

The concept of shareware appeared in the 1980s. Developers would use relatively primitive tools to create their software, then promote it via fanzines, user groups and bulletin boards to a niche audience of shareware fans. If you wanted to try the software, you would have to get hold of a floppy disk with it on. And, if you wanted to buy a licence, you would generally have to post a physical cheque to the developer. This was being an Indie developer in hard mode. A few people made a lot of money, but most vendors made modest returns on their efforts.

I started selling my first software product in 2005. This was a good time to start up as an independent software vendor. High quality compilers, IDEs, debuggers, version control systems and web servers were widely available and mostly free. The market for software was growing, as more and more people purchased PCs and Macs. Payment processors were starting to streamline online payments. But the real revolution was being able to distribute your software worldwide via an increasingly ubiquitous Internet. And getting noticed by potential customers, while never easy, was generally achievable through writing content for search engines to find, paid online ads (such as Google Adwords pay per click), download sites or even ads in physical magazines. With a lot of hard work and a bit of luck, it was quite possible to make a decent living.

Things have continued evolving at a rapid pace over the 20 years I have been selling software. Development tools have continued to improve. Mobile and web-based software has become mainstream. App stores have appeared. Outsourcing became a thing. Subscription payment models are increasingly common. Mostly these changes haven’t affected my business too much. But recently things have begun to feel noticeably harder.

LLMs have made a major impact. While I don’t worry that LLMs will do a better job than my seating planner software, data wrangling software or visual planning software any time soon (my main competitor remains Excel), everyone is noticing that their web traffic is falling. People increasingly read LLM summaries rather than clicking on search engine links or the accompanying ads. Maybe the LLM will include a link to the website that they ripped off the content from, but probably they won’t. So writing content in the hope of traffic from search engines is becoming less and less of a viable strategy to get noticed.

Other promotional channels are getting squeezed as well. Online ads are increasingly expensive and rife with click fraud. This makes it hard to get any chance of a return, unless lifetime customer value is hundreds of dollars. Google Adwords is a case in point. In the early days, I could get lots of targeted clicks at an affordable price. But Google have done everything they can to raise bid prices and generally enshittify Adwords, so they can grab more and more of the value in every transaction. I now get barely any clicks at bid prices I am prepared to pay.

One of the few useful promotional channels left is YouTube. But it is very time-consuming to produce videos and the amount of competition is huge. I fully expect generative AI to erode its value over time, as AI slop floods the channel.

Typically promotional channels start off great for vendors and become less great over time (the law of shitty clickthrus). But then new promotional channels appear and the dance starts again. But there just doesn’t seem to be much in the way of viable new channels appearing for Indie vendors like myself. My experiment with advertising on Reddit did not go well.

LLMs potentially also make software easier to write, which is a double-edged sword. It might help you code features faster, but it also lowers the barrier, so that more people can compete. Even if your new competition is bug riddled garbage, ‘vibe coded’ by someone who doesn’t know what they are doing, it still makes it harder for your product to get noticed.

The general cost of living crisis hasn’t helped either. The super-rich are making out like bandits, but everyone else has less disposable income. And that is only going to get worse when the current AI funding circle-jerk implodes.

Each of the different software platforms also have their own issues.

  • Downloadable software has fallen out of fashion and the market is shrinking as increasingly people expect software to be web-based. People are also wary about downloading software onto their computers, in case it contains malware.
  • Web-based software is more of a service than a product and is expected to be available 24×7. Expect to get lots of very unhappy emails if your server falls over. And woe betide you if your customer data is hacked. Disappearing off somewhere for a few days without an Internet connection is not really viable, unless you have employees.
  • Mobile-based software is expected to be free or, at best, very cheap. So requires huge scale to make any decent return. And that is tough when there are some 2 million apps in the iPhone app store. You are also at the mercy of app store owners, who really don’t have your best interest at heart.

The new wave of AI tools must be creating new opportunities, but it seems these opportunities are mostly there for big companies, not for Indie developers. And it is very risky to build your product as a thin layer on top of someone else’s platform. Ask people who built tools and services on top of Twitter.

It feels that it is getting harder for small software vendors, like myself, to make a living. Of course, this could be just the ramblings of a 50-something-year-old, looking back through his rose-tinted varifocals. What do you think? Has it got harder?

If you want to show indie software vendors some love, check out all the great indie software for Mac and Windows (including my own Easy Data Transform and Hyper Plan) on sale at Winterfest.

Software upgrade economics: some real numbers

My seating planner software, PerfectTablePlan, is now at v7. Major upgrades are paid (discounted 60% compared to new licences), which means I have done 6 cycles of paid upgrades. I was curious about how long it took people to upgrade, and what percentage of sales are upgrades. So I took a few minutes to crunch the numbers direct from my licence key database, using my data wrangling software, Easy Data Transform.

Here are the number of upgrade licences I sold for each week after the major upgrade. Each release is in a different colour. The values are normalised so that the peak is the same height for each release:

Upgrade licences sold per week after a major upgrade, across 6 upgrades

That looks rather messy. So here it is with the values for the 6 upgrades summed:

Upgrade licences sold per week after a major upgrade, summed across 6 upgrades.

There is a long tail of upgrades. Even when the gap between releases was 6 years, I was still getting regular upgrade purchases.

With the v5 to v6 upgrade it took:

  • 23 weeks before 50% of the upgrades were sold.
  • 74 weeks before 75% of the upgrades were sold.

So it isn’t a neat exponential decay.

This table shows how many users actually upgraded from v5 to v6:

EditionUpgraded
Home edition12%
Advanced edition31%
Professional edition45%

Most of the Home edition purchasers are buying a licence for a one-off event, such as a wedding. So it is not surprising that they are much less likely to upgrade. But I think it also shows that less price-sensitive customers are significantly more likely to upgrade, even when the upgrade is more expensive.

This graph show the percentage of PerfectTablePlan licences sold each month that were upgrades, over the 20 year life of the product:

Percentage of sales that are upgrades per month.

You can see that upgrades are still increasingly important over time. Upgrades are worth less than new sales, so selling 80% upgrade licences in a month doesn’t mean 80% of revenue is from upgrades. However, upgrades are still an increasingly significant source of revenue for us. I’m glad I never agree to free upgrades for life.

Could I have made more sales with more frequent major upgrades? Definitely. But I was also working on other projects. And I am not out to squeeze every last penny out of my loyal customers.

Could I have made more sales with a subscription model? Possibly. But subscriptions weren’t really a thing for desktop software, when I started 20 years ago. And I never felt like making a major change to a licensing model that had worked well for me, so far.

The Online Safety Act for forum owners

Bearing in mind that I am a developer and NOT A LAWYER, it is my understanding that if:

  • You host a discussion forum.

and

  • You, or any of your users, are based in the UK.

Then the UK government considers you subject to the UK Online Safety Act. Even if your forum is about hamsters. Failing to comply could get you a fine of up to £18 million or 10% of the company’s global revenue, whichever is greater.

That got your attention, didn’t it?

How can you be subject to UK law if you don’t have a presence in the UK? Good question. Are they going to extradite you or grab you if you come to the UK on holiday? I have no idea. Isn’t it all a bit over the top for a small customer forum? I think so. What if every country starts trying to apply their laws to citizens in other countries? Bigger yachts for the partners in law companies, I guess.

If you are outside the UK, you will have to make your own decision about whether to care about this law. But I am based in the UK. So I definitely need to care about it – even though my Discourse forum is a highly moderated, technical forum about data cleaning, transformation and analysis, with no porn, violence or gambling content.

There are hundreds of pages of guidance about this new law, which covers massive companies, such as Google and PornHub, as well as my little company. But my understanding is that the minimum a forum owner needs to do is:

  • Have an online safety policy
  • Assess the likelihood of children accessing your site
  • Assess the risk of harmful content
  • Take appropriate measure based on the above
  • Regularly update your assessments

Depending on your assessment and the type of content, you may need age verification checks or other measures. I assess that my data wrangling forum is not attractive to children and very unlikely to have harmful content, so I have not gone through the massive ball-ache of adding age verification.

For good measure, I have also disabled the ability of customers to direct message each other. As that is something I can’t moderate and don’t want to be responsible for.

This is 2025, so I generated my online safety policy and assessments with some help from Microsoft CoPilot. Feel free to use them for some inspiration if you need to generate your own policy. But please don’t copy them word-for-word.

Incidentally, generating documents that none of my customers will ever read seems the perfect use case for LLMs.

I really hope I don’t get forced to add age verification. I would rather shut down the forum.

For more details, start with the official Online Safety Act Explainer.

I see Wikimedia is pushing back against the UK government. It will be interesting to see how that goes.

If you think I have got anything wrong here, please post in the comments.

** Update: 29-Jul-2025 **

The Online Safety Act probably also applies to this blog. Oh boy…

Pay to play: The ugly truth behind top 5 rankings

I received this unsolicited email a few days ago.

I develop a data wrangling tool, Easy Data Transform. It not really a ‘data extraction service’. Also, they are implying that they will put you in the top 5 for whatever category you want. If you pay, presumably. Sounds sketchy. I decided to email them back, to find out a bit more.

I got a long reply. But the key bits are:

So basically ‘Top 5’ is actually ‘Top 5 most willing to pay’. Ugh. I feel dirty just reading the email.

Bear this in mind next time you see a ‘Top X of Y’ article. They may not all be pay-to-play. But I suspect a lot of them are.

Reaching 3 million page views on my blog

My recent post, 20 years working on the same software product, finally got this blog past 3 million page views:

And it only took me 18 years! I know some people wouldn’t get out of bed for 3 million views, but that isn’t going to stop me bragging about it.

I haven’t really done much to promote the blog, apart from occasionally posting links to Hacker News.

The yearly hits have gone down over time. Mostly because I have been writing less often. These days I have 3 products to keep me busy. But also blogs are less of a ‘thing’ than they used to be.

Here are the 20 most visited posts:

Probably the high point for the blog was the software awards scam post getting a mention in the Guardian newspaper.

The low point was when WordPress accidentally shut down the blog.

Power laws are very much in evidence, with the top 1% of the posts accounting for 18% of the hits. I have been consistently wrong in guessing which posts would be popular.

Was all that effort, writing articles worth 35k (of untargetted) clicks throughs to my PerfectTablePlan website? Probably not directly. Even when people did click through to my product websites, the engagement was often very low. But I am guessing that the improved domain authority from links to my seating plan software website has been helpful in improving search rank (see what I did there?). Promoting my products was never the only aim of the blog anyway.

Some posts I have written were mostly notes to my future self. And there have been several cases where Googling for an answer sent me to an article on my own blog that I had fogotten having written.

I have accepted a few guest posts. But I have been extremely picky about which guest posts to accept. I have also turned down plenty of offers for paid links.

Here is where the traffic came from, by source:

I was quite suprised by how much traffic has come from stumbleupon.com.

Digg.com, remember them?

Google completely dominates the search engine results, with Bing managing a pitiful 2.6% of search engine hits. Presumably from people too lazy or ‘non technical’ to change their Windows defaults.

Here is the traffic, by country:

Very little traffic came from Africa, South America or Asia:

Of course, it is hard to know how much of the traffic is humans and how much is bots.

There have been some 37k non-spam comments:

Quite a lot of the comments are responses by me. I have also learnt a lot of useful stuff through feedback on the blog and discussions, when links were posted to places like Hacker News. But the number of comments on the blog has markedly decreased, even taking account of the overall decrease in traffic. On the plus side, I have a lot less comment spam to deal with. It was quite overwhelming at one point. This is a comment from the blog in 2008:

I have given up looking through the spam logs. There is just too much of it and one can only read so many spam comments about Viagra and bestiality without becoming profoundly depressed about the human condition.

Thankfully WordPress seem to have greatly upped their game on spam detection since then.

Here is the top 20 sites where the traffic went:

The ‘social capital’ from the blog has been useful for promoting my consulting services and the training course I ran. Also for promoting various charitable and other causes I felt worthwhile.

I have a vague idea that I might, one day, write a book about starting a small software company. If I do, I will certainly mine the blog for material.

PS/ No, tiresome ‘SEO experts’, I still don’t want to put your boring, crappy guest post ‘articles’ with their dodgy links on my blog. So please don’t waste both our time by asking.

A quick experiment with generative AI

WordPress offered to generate an image for my last blog post. Here is the prompt it suggested:

“Generate a high-resolution, highly detailed image capturing the essence of “20 Years of PerfectTablePlan Software.” The main subjects should be two screenshots side-by-side: one showcasing PerfectTablePlan version 1, reflecting a vintage desktop interface with a Windows aesthetic from 2005, and the other displaying version 7 with a sleek, modern design. The lighting should be bright and inviting, emphasizing the contrast between the older and newer software. The style should blend nostalgia with innovation, showcasing the journey of the product over two decades. Ensure the image has sharp focus and intricate details to attract the reader’s attention.”

And here 5 images it came up with, from that prompt:

They are simulateously very impressive and hilariously awful. Quite apart from the weird text (“sex 20”?), none of the screenshots look even slightly like PerfectTablePlan. I think I’ll pass!

20 years working on the same software product

I released version 1 of my table seating planning software, PerfectTablePlan, in February 2005. 20 years ago this month. It was a different world. A world of Windows, shareware and CDs. A lot has changed since then, but PerfectTablePlan is now at version 7 and still going strong.

PerfectTablePlan v1

PerfectTablePlan v7

I have released several other products since then, and done some training and consulting, but PerfectTablePlan remains my most successful product. It’s success is due to a lot of hard work, and a certain amount of dumb luck.

I was getting married and I volunteered to do the seating plan for our wedding reception. It sounded like a relatively straightforward optimization problem, as we only had 60 guests and no family feuds to worry about. But it was surprisingly difficult to get right. I looked around for some software to help me. There were a couple of software packages, but I wasn’t impressed. I could do better myself! So I wrote a (very rough) first version, which I used for our wedding.

Things weren’t going great at my day job, at a small software startup. Maybe I could commercialize my table planner? I was a bit wary, as my potential competitors all seemed rather moribund and I didn’t think I would be able to make a living off it. But I thought I could do everything worth doing in 6-12 months and then start on the next product. Wrong on both counts!

Web-based software was still in its infancy in 2005. So I decided to write it as desktop software using C++ and cross-platform framework Qt, which I had plenty of experience in. Initially, I just released a Windows version. But I later added a Mac version as well. Qt has had its commercial ups and downs in the last 20 years, but it has grown with me and is now very robust, comprehensive and well documented. I think I made a good choice.

I financed PerfectTablePlan out of my own savings and it has been profitable every year since version 1 was launched. I could have taken on employees and grown the business, but I preferred to keep it as a lifestyle business. My wife does the accounts and proof reading and I do nearly everything else, with a bit of help from my accountant, web designers and a few other contractors. I don’t regret that decision. 20 years without meetings, ties or alarm clocks. My son was born 18 months after PerfectTablePlan was launched and it has been great to have the flexibility to be fully present as a Dad.

CDs, remember them? I sent out around 5,000 CDs (with some help from my father), before I stopped shipping CDs in 2016.

During the lifetime of PerfectTablePlan it became clear that things were increasingly moving to the web. But I couldn’t face rewriting PerfectTablePlan from scratch for the web. Javascript. Ugh. Also PerfectTablePlan is quite compute intensive, using a genetic algorithm to generate an automated seating plan and I felt it was better running this on the customer’s local computers than my server. And some of my customers consider their seating plans to be confidential and don’t want to store them on third party servers. So I decided to stick with desktop. But, if I was starting PerfectTablePlan from scratch now, I might make a different decision.

Plenty of strange and wonderful things have happened over the last 20 years, including:

  • PerfectTablePlan has been used by some very famous organizations for some very famous events (which we mostly don’t have permission to mention). It has seated royalty, celebrities and heads of state.
  • PerfectTablePlan was used as part of a demonstration of the (controversial) first commercial quantum computer by D-Wave.
  • A mock-up of PerfectTablePlan, including icons I did myself, was used without our permission by Sony in their ‘Big day’ TV comedy series. I threated them with legal action. Years later, I am still awaiting a reply.
  • I got to grapple with some interesting problems, including the mathematics of large combinatorial problems and elliptical tables. Some customers have seated 4,000 guests and 4000! (4000x3999x3998 .. x 1) is a mind-bogglingly huge number.
  • A well known wedding magazine ran a promotion with a valid licence key clearly visible in a photograph of a PerfectTablePlan CD. I worked through the night to release a new version of PerfectTablePlan that didn’t work with this key.
  • I found out that CDs are edible.
  • I sponsored the building of a kindergarten in Nepal.
  • I once had to stay up late, in a state of some inebriation, to fix an issue so that a world famous event wasn’t a disaster (no I can’t tell you the event).

The lowest point was the pandemic, when sales pretty much dropped to zero.

Competitors and operating systems have come and gone and the ecosystem for software has changed a lot, but PerfectTablePlan is still here and still paying the bills. It is about 145,000 lines of C++. Some of the code is a bit ugly and not how I would write it now. But the product is very solid, with very few bugs. The website and user documentation are also substantial pieces of work. The PDF version of the documentation is nearly 500 pages.

I now divide my time between PerfectTablePlan and my 2 other products: data wrangling software Easy Data Transform and visual planner Hyper Plan. Having multiple products keeps things varied and avoids having all my eggs in one basket. In May 2024 I released PerfectTablePlan v7 with a load of improvements and new features. And I have plenty of ideas for future improvements. I fully expect to keep working on PerfectTablePlan until I retire (I’m 59 now).