12 rules for software business happiness

Here are a few rules for happiness that I have learned (often the hard way) running a solo software business since 2005

Make sure your important stuff is backed-up automatically

Any sort of manual back-up is going to get forgotten. Back-up to more than one place, at least one of which is offsite.

Stay away from the bleeding edge

Stick with tried and trusty tools and technologies, where you can. JQuery will probably be here in another 10 years, but the latest and greatest Javascript framework might not.

Use good suppliers

You need your hosting company, payment processor and other critical suppliers to be rock solid. Think twice about going with a supplier just because they are cheap. Changing suppliers can be a pain, so ask around before trying a supplier.

Use version control for everything important

It matters less which version control system it is. Periodically making a copy of your source folder is not a version control system!

Don’t promise ship dates

Developers are notoriously bad at predicting dates. If you promise a date and get it wrong (and you will) then you either have to miss the date or cut corners. Neither is good.

Never send an email you might later regret

If you are starting to feel angry writing an email, then stop writing. Come back to it later. Or maybe write it, feel a bit better, then delete it without sending.

Write documentation as you go

Few people enjoy writing documentation. But if you leave all the documentation until you have finished programming, then you are likely to rush it and forget stuff.

Have a checklist

Automate where you can. Have checklists for everything else. Keep updating your checklists.

Get someone else to proof read everything

Typos are embarrassing, but it is impossible to proof read your own stuff. So get someone else to proof read any stuff that customers see: web pages, newsletters, documentation etc.

Never release changes just before going on holiday

You don’t want to have to be fire-fighting a new bug when you should be on the beach with your family/friends.

Don’t try to do everything yourself

You could spend weeks learning about taxes, web hosting, CSS or any number of other topics that aren’t central to your business. But why bother? Pay someone who already know this stuff.

Embrace imperfection

If you wait for perfection, then you are never going to ship anything. Just make sure each release is better than the last. Good enough is good enough.

Easy Data Transform progress

I have been gradually improving my data wrangling tool, Easy Data Transform, putting out 70 public releases since 2019. While the product’s emphasis is on ease of use, rather than pure performance, I have been trying to make it fast as well, so it can cope with the multi-million row datasets customers like to throw at it. To see how I was doing, I did a simple benchmark of the most recent version of Easy Data Transform (v1.37.0) against several other desktop data wrangling tools. The benchmark did a read, sort, join and write of a 1 million row CSV file. I did the benchmarking on my Windows development PC and my Mac M1 laptop.

Easy Data Transform screenshot

Here is an overview of the results:

Time by task (seconds), on Windows without Power Query (smaller is better):

data wrangling/ETL benchmark Windows

I have left Excel Power Query off this graph, as it is so slow you can hardly see the other bars when it is included!

Time by task (seconds) on Mac (smaller is better):

data wrangling/ETL benchmark M1 Mac

Memory usage (MB), Windows vs Mac (smaller is better):

data wrangling/ETL benchmark memory Windows vs Mac

So Easy Data Transform is nearly as fast as it’s nearest competitor, Knime, on Windows and a fair bit faster on an M1 Mac. It is also uses a lot less memory than Knime. However we have got some way to go to catch up with the Pandas library for Python and the data.table package for R, when it comes to raw performance. Hopefully I can get nearer to their performance in time. I was forbidden from including benchmarks for Tableau Prep and Alteryx by their licensing terms, which seems unnecessarily restrictive.

Looking at just the Easy Data Transform results, it is interesting to notice that a newish Macbook Air M1 laptop is significantly faster than a desktop AMD Ryzen 7 desktop PC from a few years ago.

Windows vs Mac M1 benchmark

See the full comparison:

Comparison of data wrangling/ETL tools : R, Pandas, Knime, Power Query, Tableau Prep, Alteryx and Easy Data Transform, with benchmarks

Got some data to clean, merge, reshape or analyze? Why not download a free trial of Easy Data Transform ? No sign up required.

Discussing lifestyle businesses on Bootstrapped.fm podcast

I had a discussion with fellow British bootstrapper Robin Warren about what running a lifestyle business means to us. Click the link to listen:

Bootstrapped.fm podcast #233: Running a lifestyle business and proud of it

Creating a Mac Universal binary for Intel and ARM M1/M2 with Qt

Apple has transitioned Macs from Intel to ARM (M1/M2) chips. In the process it has provided an emulation layer (Rosetta2) to ensure that the new ARM Macs can still run applications created for Intel Macs. The emulation works very well, but is quoted to be some 20% slower than running native ARM binaries. That may not seem like a lot, but it is significant on processor intensive applications such as my own data wrangling software, which often processes datasets with millions of rows through complex sequences of merging, splitting, reformatting, filtering and reshaping. Also people who have just spent a small fortune on a shiny new ARM Mac can get grumpy about not having a native ARM binary to run on it. So I have been investigating moving Easy Data Transform from an Intel binary to a Universal (‘fat'[1]) binary containing both Intel and ARM binaries. This is a process familiar from moving my seating planner software for Mac from PowerPC to Intel chips some years ago. Hopefully I will have retired before the next chip change on the Mac.

My software is built on-top of the excellent Qt cross-platfom framework. Qt announced support for Mac Universal binaries in Qt 6.2 and Qt 5.15.9. I am sticking with Qt 5 for now, because it better supports multiple text encodings and because I don’t see any particular advantage to switching to Qt 6 yet. But, there is a wrinkle. Qt 5.15.3 and later are only available to Qt customers with commercial licenses. I want to use the QtCharts component in Easy Data Transform v2, and QtCharts requires a commercial license (or GPL, which is a no-go for me). I also want access to all the latest bug fixes for Qt 5. So I decided to switch from the free LGPL license and buy a commercial Qt license. Thankfully I was eligible for the Qt small business license which is currently $499 per year. The push towards commercial licensing is controversial with Qt developers, but I really appreciate Qt and all the work that goes into it, so I am happy to support the business (not enough to pay the eye-watering fee for a full enterprise license though!).

Moving from producing an Intel binary using LGPL Qt to producing a Universal binary using commercial Qt involved several major stumbling points that took me hours and a lot of googling to sort out. I’m going to spell them out here to save you that pain. You’re welcome.

  • The latest Qt 5 LTS releases are not available via the Qt maintenance tool if you have open source Qt installed. After you buy your commercial licence you need to delete your open source installation and all the associated license files. Here is the information I got from Qt support:
I assume that you were previously using open source version, is that correct?

Qt 5.15.10 should be available through the maintenance tool but it is required to remove the old open source installation completely and also remove the open source license files from your system.

So, first step is to remove the old Qt installation completely. Then remove the old open source licenses which might exist. Instructions for removing the license files:

****************************
Unified installer/maintenancetool/qtcreator will save all licenses (downloaded from the used Qt Account) inside the new qtlicenses.ini file. You need to remove the following files to fully reset the license information.

Windows
"C:/Users/%USERNAME%/AppData/Roaming/Qt/qtlicenses.ini"
"C:/Users/%USERNAME%/AppData/Roaming/Qt/qtaccount.ini"

Linux
"/home/$USERNAME/.local/share/Qt/qtlicenses.ini"
"/home/$USERNAME/.local/share/Qt/qtaccount.ini"

OS X
"/Users/$USERNAME/Library/Application Support/Qt/qtlicenses.ini"
"/Users/$USERNAME/Library/Application Support/Qt/qtaccount.ini"

As a side note: If the files above cannot be found $HOME/.qt-license(Linux/macOS) or %USERPROFILE%\.qt-license(Windows) file is used as a fallback. .qt-license file can be downloaded from Qt Account. https://account.qt.io/licenses
Be sure to name the Qt license file as ".qt-license" and not for example ".qt-license.txt".

***********************************************************************

After removing the old installation and the license files, please download the new online installer via your commercial Qt Account.
You can login there at:
https://login.qt.io/login

After installing Qt with commercial license, it should be able to find the Qt 5.15.10 also through the maintenance tool in addition to online installer.
  • Then you need to download the commercial installer from your online Qt account and reinstall all the Qt versions you need. Gigabytes of it. Time to drink some coffee. A lot of coffee.
  • In your .pro file you need to add:
macx {
QMAKE_APPLE_DEVICE_ARCHS = x86_64 arm64
}
  • Note that the above doubles the build time of your application, so you probably don’t want it set for day to day development.
  • You can use macdeployqt to create your deployable Universal .app but, and this is the critical step that took me hours to work out, you need to use <QtDir>/macos/bin/macdeployqt not <QtDir>/clang_64/bin/macdeployqt . Doh!
  • You can check the .app is Universal using the lipo command, e.g.:
lipo -detailed_info EasyDataTransform.app/Contents/MacOS/EasyDataTransform
  • I was able to use my existing practise of copying extra files (third party libraries, help etc) into the .app file and then digitally signing everything using codesign –deep [2]. Thankfully the only third party library I use apart from Qt (the excellent libXL library for Excel) is available as a Universal framework.
  • I notarize the application, as before.

I did all the above on an Intel iMac using the latest Qt 5 LTS release (Qt 5.15.10) and XCode 13.4 on macOS 12. I then tested it on an ARM MacBook Air. No doubt you can also build Universal binaries on an ARM Mac.

Unsurprisingly the Universal app is substantially larger than the Intel-only version. My Easy Data Transform .dmg file (which also includes a lot of help documentation) went from ~56 MB to ~69 MB. However that is still positively anorexic compared to many bloated modern apps (looking at you Electron).

A couple of tests I did on an ARM MacBook Air showed ~16% improvement in performance. For example joining two 500,000 row x 10 column tables went from 4.5 seconds to 3.8 seconds. Obviously the performance improvement depends on the task and the system. One customer reported batch processing 3,541 JSON Files and writing the results to CSV went from 12.8 to 8.1 seconds, a 37% improvement.

[1] I’m not judging.

[2] Apparently the use of –deep is frowned on by Apple. But it works (for now anyway). Bite me, Apple.

Fogbugz goes dark

Today I got an email titled ‘Your Manuscript Account Balance’ which said:

Your Manuscript account is now in arrears in the amount of $31.25.

Use this link to specify a credit card for your account and we will use it to settle your balance:

https://****.fogbugz.com/default.asp?pg=pgEditCC

You can prepay for a year in advance using a check, corporate purchase order, or credit card:

https://shop.fogbugz.com/FogBugz/?sCategory=prepayfb&nUsers=0&ixOrderCoveredBySupport=*****

Thank you,

FogBugz Customer Success
success@fogbugz.com

Fogbugz was an online bugtracker developed by the much respected Fog Creek Software. They sold it some years ago when their company headed off into the stratosphere (Stackoverflow, Trello etc). I think I might have had a free Fogbugz account 10+ years ago. I asked around on Twitter and quite a few people seem to have got the exact same email. It appears that:

  • The venerable FogBugz application was recently sold on to https://ignitetech.com and they appear to be trying to demand money from anyone who ever used FogBugz. Which is likely to be thousands of people.
  • IgniteTech describe themselves as “Where software goes to live”, but “Where software goes to die” might be a better description.
  • FogBugz was renamed to ‘Manuscript’ at some point.

I tried replying to the email, telling them in no uncertain terms that they weren’t going to get a penny from me. This is the reply I got:

Hi, 

In an effort to provide a better experience to our existing customers, we are consolidating our email support channel into our Help-Center. 

In case you need technical support for FogBugz you will be able to submit a case from our 'FogBugz Support Page'. 

Replies made to this mailbox are routed to an unmonitored mailbox. 

Please do not reply to this message. 

Thanks, FogBugz Support Team

It is sad to see someone using dark business practices to try to wring a few pennies from the corpse of a once much-respected product. Does Joel Spolsky know? I don’t think he’d be impressed.

** Update 24-Sep-2022 **

I got this email late on the 22nd:

Dear FogBugz User,

Mea culpa — sincerely.

We’ve become well aware of the maelstrom of concern and comments caused by a series of emails that some of you received, but to explain, we were just as surprised as you when multiple emails were sent. Our original and only intended email, which was controlled by humans, was to inform you that the “free,” non-expiring version of FogBugz is being phased out on October 17, 2022. Additionally, we wanted to offer you the option to continue using FogBugz by updating your account to a paid subscription. This was the entire, planned effort.

However, once we updated the accounts that had been identified as free and non-expiring in the internal FogBugz accounting system, the software automatically generated a form email, notifying you that we had summarily converted you to a paid account, and worse, actually triggering collection/dunning notices to some. This was unintended and is not accurate. Yes, we’ve owned the software for some time and should know all of the nooks and crannies by now. We don’t know if this was a nook or a cranny, but it decided to act on its own. Truth. It’s embarrassing, and we’d react the same way as many of you have. So this email is to set the record straight.

We do not automatically charge any customer for usage of the software unless a subscription has been expressly elected, despite what the erroneous auto-email stated.

Here’s what you need to know:

    All free subscriptions for FogBugz will expire on Oct. 17, 2022. If your FogBugz subscription has already expired, and you do not want to continue using FogBugz, you don’t need to take any further action. Your data will be deleted from the system; you will not be billed and you will receive no further emails from either the nook or the cranny.

    If you were erroneously charged because your free account still had your credit card on file (e.g. some users were on paid accounts and had downgraded), we will ensure that you are promptly refunded. Please use the email fogbugzbilling@ignitetech.com if you have not seen the refund already posted.

    Despite this mishap, we hope the benefits of the software will encourage you to continue. If your FogBugz free-account subscription is currently active, and you would like to continue as a subscriber, please authorize your paid subscription by following these instructions.

    Please note: When you upgrade, your account will qualify for 50% off any subscription tier.

    If you need any assistance while upgrading, please contact us at  support.fogbugz.com.

We sincerely apologize for the confusion that was caused and are working hard behind the scenes to resolve all issues related to this action.

Sincerely,

The (humbled) FogBugz Team

So Fogbugz became sentient and started demanding money? And this first happened at least a week ago, but the new owners didn’t notice and a whole load of emails went out days later (including the one I got several days after the first post on HN)? Hmm.

Software Startup Founders Academy

My friend Stuart Prestedge is launching his Software Startup Founders Academy next week. He is giving away a limited number of free 1:1 advisory sessions for software startup founders (and prospective founders) .

Stuart has been creating software startups for over 30 years, including 2 successful exits. He is now making his knowledge and experience available to any founders or prospective founders who join him in the academy. This means you can get help and guidance with whatever your issues, hurdles, blockers, worries and concerns are, right now. You’ll be part of a community that helps each other to succeed, answering questions, solving problems and making decisions big and small. You’ll also have access to resources (videos and documents) to accelerate your progress. And you can join the regular group coaching calls to discuss the topics that members are asking about. The combination of community, content & coaching will help you to grow and succeed.

To request your free session go to:

https://softwarestartupfounders.academy/academy-launch-request-1-1

Summerfest 2022

Easy Data Transform and Hyper Plan Professional edition are both on sale for 25% off at Summerfest 2022. So now might be a good time to give them a try (both have free trials). There is also some other great products from other small vendors on sale, including Tinderbox, Scrivener and Devonthink. Some of the software is Mac only, but Easy Data Transform and Hyper Plan are available for both Mac and Windows (one license covers both). Sale ends 12th July.

No-one knows what they are doing

When I was a child I assumed that all the adults running the world knew what they were doing. Now that I am an adult, I am under no such illusions. Just look at the current British government. They clearly don’t have a clue. A more mediocre bunch of individuals would be hard to find.

I’m going to let you in on a little secret. Most of us who are running businesses had no real idea what they were doing when they started, and still struggle with decisions now. I’ve been making a full-time living selling my own software since 2005. But when I launched my seating planner software, I really had no idea if I would sell a single licence. After 17 years I know a lot more about my market and running a software business. But things are constantly changing and I still don’t know day-to-day if I should be spending more time on SEO, partnerships, Youtube videos, new features, a better website, or thousand other things I could be doing. A lot of guessing and gut feel is still involved.

It is easy to read 20/20 hindsight accounts of successful businesses and assume they they knew exactly what they needed to do at each stage. They didn’t. Running a business involves making a lot of decisions under great uncertainty in a constantly changing environment. So if you want to start a business, don’t be put off by not knowing what you are doing. No-one does.

Why isn’t there a decent file format for tabular data?

Tabular data is everywhere. I support reading and writing tabular data in various formats in all 3 of my software application. It is an important part of my data transformation software. But all the tabular data formats suck. There doesn’t seem to be anything that is reasonably space efficient, simple and quick to parse and text based (not binary) so you can view and edit it with a standard editor.

Most tabular data currently gets exchanged as: CSV, Tab separated, XML, JSON or Excel. And they are all highly sub-optimal for the job.

CSV is a mess. One quote in the wrong place and the file is invalid. It is difficult to parse efficiently using multiple cores, due to the quoting (you can’t start parsing from part way through a file). Different quoting schemes are in use. You don’t know what encoding it is in. Use of separators and line endings are inconsistent (sometimes comma, sometimes semicolon). Writing a parser to handle all the different dialects is not at all trivial. Microsoft Excel and Apple Numbers don’t even agree on how to interpret some edge cases for CSV.

Tab separated is a bit better than CSV. But can’t store tabs and still has issues with line endings, encodings etc.

XML and JSON are tree structures and not suitable for efficiently storing tabular data (plus other issues).

There is Parquet. It is very efficient with it’s columnar storage and compression. But it is binary, so can’t be viewed or edited with standard tools, which is a pain.

Don’t even get me started on Excel’s proprietary, ghastly binary format.

Why can’t we have a format where:

  • Encoding is always UTF-8
  • Values stored in row major order (row 1, row2 etc)
  • Columns are separated by \u001F (ASCII unit separator)
  • Rows are separated by \u001E (ASCII record separator)
  • Er, that’s the entire specification.

No escaping. If you want to put \u001F or \u001E in your data – tough you can’t. Use a different format.

It would be reasonably compact, efficient to parse and easy to manually edit (Notepad++ shows the unit separator as a ‘US’ symbol). You could write a fast parser for it in minutes. Typing \u001F or \u001E in some editors might be a faff, but it is hardly a showstopper.

It could be called something like “unicode separated value” (hat tip to @fakeunicode on Twitter for the name) or “unit separated value” with file extension .usv. Maybe a different extension could used when values are stored in column major order (column1, column 2 etc).

Is there nothing like this already? Maybe there is and I just haven’t heard of it. If not, shouldn’t there be?

And yes I am aware of the relevant XKCD cartoon ( https://xkcd.com/927/ ).

** Edit 4-May-2022 **

“Javascript” -> “JSON” in para 5.

It has been pointed at the above will give you a single line of text in an editor, which is not great for human readability. A quick fix for this would be to make the record delimiter a \u001E character followed by an LF character. Any LF that comes immediately after an \u001E would be ignored when parsing. Any LF not immediately after an \u001E is part of the data. I don’t know about other editors, but it is easy to view and edit in Notepad++.

Verifone seems to be having issues processing UK payments

An ‘unfinished’ transaction is when an order ‘has incomplete/delayed/invalid payment details’. For example a payment not completed after being flagged (correctly or incorrectly) as fradulent will be marked ‘unfinished’. Recently I have been getting a lot more ‘unfinished’ transactions than usual through my main payment processor, Verifone[1]. This seemed to be particularly for people ordering from the UK.

I dug down into my table planning software sales data using my own data munging software. Here is what I found looking through data since 2017:

(April results are for half the month. There were fewer transaction during 2020 and 2021 as not much table planning was happening during the pandemic!)

The brown bars show the ‘unfinished’ transaction rate for all countries. The blue bars show the ‘unfinished’ transaction rate for the UK. So my suspicions were correct – there has been a huge jump in ‘unfinished’ UK transactions. In March and April the number of unfinished transaction is about 10x what I would expect historically.

Some of these lost sales I am able to recover by emailing them and sending a Stripe payment link. But it isn’t ideal, as it is a hassle for me, and the customer and Stripe doesn’t handle the tax. But many of these sales are just lost for good.

I emailed some of the prospective customers with unfinished transactions. Here are a couple of responses I got:

“Hi my bank tell me that you are not set up with the new security banking system. That is why my payment is not going through.”

“I was told to ring my bank to ask why the payment was denied. I spent ages waiting for [my bank] to answer the phone and had to answer goodness knows how many security questions before they were able to tell me that the payment company had not updated their software to be on Visa’s list of acceptable people to pay. Something to do with preventing fraud.”

What is going on Verifone? Why has my ‘unfinished’ rate for UK customers sky rocketed? Is it going to be fixed soon? This is costing me time and money. Quite a lot of money. Is anyone else seeing the same thing?

I emailed Verifone on the 11th of March to ask what is going on. I am still awaiting a substantive reponse from Verifone, over a month later. It isn’t the first time I’ve had to send several follow-up emails and wait weeks for a response. Verifone support response times are glacial. Unfortunately great little payment processing companies frequently get bought and become not-so great large payment processing companies. Back when they were Avangate, you would often get a reponse on the same day. I miss those days.

** Update 10-Jun-2022 **

Things got even worse in May, with some 30% of UK customer transactions failings. I kept on at Verifone and I finally got an email on 23-May-2022:

“Thank you for the patience you have shown us during the investigation. Our development team has resolved the described issue and released the fix into the production environment. We have tested it and confirm that the fix is working as intended.”

The rejection rate then went very quickly back to normal. When I asked what the problem had been I was told:

“There was a setup issue with the GBP payment terminal. Our engineers identified and fixed it.”

So I am glad it is fixed. But it took from 11-Mar, when I first reported it, to 23-May, when it was reported fixed. It cost me a lot of time and and money. It must have cost Verifone a *lot* more, Possibly millions in commission for an operation of their scale. You would think they would have spotted and fixed something like this very quickly. But apparently not.

[1] I was originally with Avangate, who merged with 2Checkout and then were bought by Verifone.