Author Archives: Andy Brice

Stalking website visitors with Microsoft Clarity

Microsoft Clarity is a new service that allows you to see, in detail, how visitors are interacting with your website. It includes:

  • heat maps, showing where visitors are clicking or touching or how far they are scrolling
  • recordings of visitor sessions, including mouse movement, clicks, touches and scrolling

You just need to get a Javascript snippet from clarity.microsoft.com (for which you need a Microsoft login) and paste it into the header of each page. You can then login to clarity.microsoft.com at a later time to see your results. The service is free.

I tried it on my www.easydatatransform.com website. Here you can see a click heatmap for the buy page:

People are clicking all over non-hyperlinked text. Hmm. Perhaps they somehow couldn’t the see the effing enormous blue button next to it? Notice that numbers are starred out to avoid personal information, such as credit card numbers.

You can also see how far visitors scroll down the page with scroll heatmaps:

So I can see that the buy button is appearing well above the fold.

You can also watch recodings of people interacting with the website, showing their mouse movements, clicks, touches and scrolling. This is where things start to feel a bit stalkerish. You don’t get any identifiable information on the visitor beyond their country, browser and their operating system and I’m ok with people watching me interact with their websites like this. But it still feels a bit voyeuristic. The results are also a bit strange. Some people just click all over the place and highlight random text (touches are tracked separately from clicks). There is a distinct danger that you could watch hours of sessions and come away without much actionable information.

You can filter the information in various ways, including by country or referring website. You can even filter to see sessions with ‘Rage clicks’ (where the user has clicked or tapped repeatedly in the same area).

Watching a few sessions with ‘rage clicks’ I could see that some people indeed seem completely unable to see the effing enormous blue ‘buy now’ button on the buy page. So I have also added a text hyperlink where most people are clicking in the text and will probably try changing the button colour. Perhaps to shocking pink!

Running Pingdom Website Speed Test on the Easy Data Transform home page, both with and without Clarity a few minutes apart, I can see that it had some effect on speed, but not too much.

Without Clarity script: 175.2kb of scripts, 7 script requests.

With Clarity script: 194.3kb of scripts, 9 script requests.

The load time was actually 0.1s faster with Clarity. That is probably just an anomaly.

I have disabled Clarity for now. But may reenable it after I have made some changes to the website, to see the effect of the changes. Overall I was quite impressed with the service and it was surprisingly easy to set-up. But the cynic in me does wonder what exactly Microsoft is getting out of it.

Running Qt apps on M1 ARM Macs

Apple is switching the processor architecture of it’s Macs. Again (I transitioned PerfectTablePlan from PowerPC to Intel some hears ago). This time to their own M1 ARM chips. Reports so far have been very positive about speed and battery life of the new processors. Obviously most current Mac software has been written for Intel Macs, so they are using the Rosetta2 emulation layer to run apps compiled for Intel Macs on the ARM chips. I’m not sure how much of a performance hit this causes, but clearly it would be better to run native ARM binaries on an ARM machine. Also Apple, being Apple, want to move everyone to ARM as quickly as possible. Tough luck if you just spent big bucks on a shiny new Intel Mac.

One of my customers emailed me that the latest version of my Hyper Plan visual planner, built with Qt 5.13.1, didn’t run on an new M1 Mac. I don’t currently have an M1 Mac to test it on. But my Easy Data Transform software , built with Qt 5.15.2, apparently works fine on an M1 Mac. So I recompiled Hyper Plan using Qt 5.15.2, and was told it now works. I have found a couple of minor differences in behaviour between Qt 5.13.1 and 5.15.2, but they are too obscure to go into here. Some Qt apps may still have issues on ARM.

Currently Qt is only available as Intel binaries. Efforts are in progress to be able to build Qt as M1 (ARM) binaries. When that is complete it should be possible to ship Qt applications as a ‘fat binary’ with both Intel and ARM executables, as I did with the PowerPC to Intel transition. I’m not sure if this is going to be supported on Qt 5 and 6 or just Qt 6.

Issues with Qt applications on macOS 11.0 (Big Sur)

In my previous post I wrote about the trials and tribulations of upgrading my iMac to macOS 11.0. Here I am going to list some of the issues I know about deploying Qt applications on macOS 11.0. More issues may subsequently come to light.

The QFileDialog::DontConfirmOverwrite flag is ignored when passed to QFileDialog::getSaveFileName(). Which means that you can’t use this flag and handle the message yourself, or you will end up with a double warning. This seems to have been an issue since macOS 10.15. It still isn’t fixed in Qt 5.15.2. It is annoying, but relatively easy to work around. The Qt bug report is QTBUG-39791.

QMessageBox::information() shows a placeholder icon instead of the information icon:

It is only cosmetic. But it looks shonky and Mac users tend to care a lot about this sort of thing. I can reproduce it in Qt 5.15.2. I don’t know of a workaround. The Qt bug report is QTBUG-88928.

QMessageBox::warning(), QMessageBox::information() etc show the default focus button incorrectly. For example:

QMessageBox::warning( this, "App", "text", QMessageBox::Ok|QMessageBox::Cancel, QMessageBox::Ok );

Gives:

Again it is only cosmetic, but it looks jarring. I can reproduce it in Qt 5.15.2. I don’t know of a workaround. The Qt bug report is QTBUG-89133.

There are also some other styling issues. The Qt bug report is QTBUG-86513.

Dark Mode still doesn’t work properly for Qt apps.

There was an issue on macOS 10.15 where using QFileDialog::getSaveFileName() to save over an existing file could cause a crash. Thankfully that doesn’t seem to be an issue in macOS 10.11. The Qt bug report is QTBUG-83342.

Unfortunately issues with Qt on Mac are nothing new. I realize it is a big challenge for the Qt developers to keep such a large codebase up-to-date with so many continually evolving platforms. But the Mac version always feels rather neglected compared to the Windows version. I wish they would prioritise basic issues such as the above over adding whizzy new features, 80% of which most Qt developers probably never use. macOS 11.0 was released a couple of weeks ago and betas have been available for a while.

I would be interested to hear of the experience of other developers with macOS 11.0. Any other Qt macOS 11.0 issues I should know about? Please let me know in the comments.

Upgrading to MacOS 11.0 (Big Sur)

It is always a bit of fraught process upgrading a computer OS, especially for a development machine with loads of tools and libraries installed. So I try to do it as infrequently as I can get away with. On Windows I generally buy a new PC rather than upgrade OS. However glitches had been reported in Easy Data Transform on macOS 11.0 (Big Sur) and I wasn’t ready to abandon my 2017 iMac, so I decided to bite the bullet and upgrade it from macOS 10.13 to 11.0.

The initial upgrade of OS was straightforward enough. But when I tried to run Qt Creator the CPU shot to 99% and stayed there, making the machine unusable. A glance at Activity Monitor showed that several XCode related processes were going crazy. After a bit a Googling I managed to find this magic incantation to type into the terminal on a forum post:

defaults write com.apple.dt.Xcode DVTDisableMainThreadChecker 1

I was then able to rebuild my Qt-based products: Easy Data Transform, PerfectTablePlan and Hyper Plan using the existing installs of Qt 5.13.1 and Qt Creator 4.8.0.

I had to update some of the software I use:

  • DropDMG
  • Beyond Compare
  • SnagIt

Annoyingly, I had to buy an upgrade of SnagIt as the 2018 version doesn’t work on Big Sur. Even more annoyingly the upgrade costs nearly as much as a new licence, which feels predatory.

The Subversion command line no longer worked from the terminal, but that was easily fixed by adding /Applications/XCode.app/Developer/usr/bin to PATH in my .profile.

So far I haven’t been able to get the following to work:

  • XCode
  • Hammer4Mac

XCode 10.1 falls over if I try to start it. It says that it requires additional components and then fails to install them. I may upgrade XCode at some point. But I only use the compiler from the command line via QtCreator, so it doesn’t really matter at present.

Hammer4Mac is a static website builder I use to build the PerfectTablePlan website and a couple of other mini sites. I upgraded to the latest version. It starts, but returns ‘Build failed’ for all 3 websites. No clue as to why. I Tweeted the creator, but got no reply. It appears to be abandonware. If so they should really take down the Hammer4Mac website. I guess I will use it from my macOS 10.14 laptop and then eventually do the tedious job of porting those websites to Jekyll.

Hopefully I won’t have to do another major upgrade of macOS any time soon (I may buy a new Mac next time).

Microsoft PPC broad match goes nuts

With Microsoft Advertising (formerly Bing Ads) you can bid on 3 different match types: exact, phrase and broad. These match types take a successively broader interpretation of your search term. Broad match has to be used with care, but can be useful for casting a wider net. However broad match seems to have gone a bit nuts for Microsoft pay per click. Witness these recent results:

broad match keywordsearch term matched
perfecttablecore muscle machine
perfecttablemy plate
perfecttableto be able to
perfecttablenumista threepence
perfecttablesplitwise
plans tableofloxacin tablets
table plan softwaresmall business software
table plan softwarehr software for small business

What!? How is “numista threepence” in any way a match for “perfecttable”? That is stretching the concept of ‘broad match’ to weird levels. I have deleted most of my broad match keywords. Is anyone else seeing this?

How I finally beat my son at a computer game

TL;DR: I cheated, using programming.

I play computer games with my son. But he is 14 and I am 54, so I just can’t compete on reflexes. Just yesterday he thrashed me 10-3 at the silly and fun Spelunky Deathmatch. Then he gloated about my pitiful score.

spelunky

We’ve also been writing our own games together in Python, for fun and so that I can teach him some programming. We’ve written a little jet dogfight game together. You each get a little plane that can turn left, right, accelerate or shoot. You score 2 points for shooting down your opponent and 1 point for flying over a powerup. First to 20 points wins. We are both Python novices (my day job is writing software in C++), so the program is quite hacky. Lots of globals and cut and paste. The planes are triangles and the clouds are square. But it is a fast and fun game to play.

plane-game

Predictably my son was winning most the games. Then gloating about it. However I had recently seen an article about an AI winning dogfights against a human fighter pilot. This gave me an idea. While he was asleep I modified the program so that you can press a key to toggle a cheat mode on either plane. In the cheat mode pressing the left key aims automatically at the powerup and pressing the right key aims automatically at the opponent’s plane. Suddenly I started thrashing him. He got suspicious and insisted we swap planes. Which is fine, I just toggled cheat mode on the other plane. He got even more  suspicious. I told him I had been practising. He went off to practise with an old version of the code I gave him. I then thrashed him several more times and told him I have being doing a lot of practise. ;0)

Sooner or later he will figure out what is going on. I’m not sure what the take away lesson will be. Coding is a powerful skill? Don’t trust your Father?

I offer up the code for any competitive Dad’s (or Mum’s) who feel they need a little help against their cocky offspring. See how long you get away with cheat mode. You can always toggle it off for a while when they get suspicious.

Notes about the game

You can download the game’s Python code.

The game runs from inside the Python variant of the free processing.org environment, which you can download here for Windows, Mac or Linux. You need to select Sketch>Import Library to get the Sound library. File>Open the .pyde file in processing.org and press the run button. You can adjust the szx and szy variables according to your screen size. There seems to be a bug where the sound only works for the first game after you start the IDE.

The keyboard controls are:

Player 1:

a – turn left (aim for powerup in cheat mode)

d – turn right (aim for opponent in cheat mode)

w – accelerate

s – fire

z – toggle cheat mode (off at start)

Player 2:

left cursor – turn left (aim for powerup in cheat mode)

right cursor – turn right (aim for opponent in cheat mode)

up cursor – accelerate

down cursor – fire

end – toggle cheat mode (off at start)

Currently the aim cheat aims at where the opponent’s plane is. To be a bit more sophisticated, it could aim at where it thinks the opponent’s plane will be. But the current approach turns out to be good enough, and less likely to make an opponent suspicious.

It would be interesting to write a little AI that completely controls the plane and then put it up against other people’s AIs. A future project perhaps. But processing.org isn’t an ideal environment for that.

Google Ads can charge you anything they like for a click on their partner network

I have been using Google Ads (previously Adwords) since 2005. A lot has changed in that in time. But it still remains basically an auction. You bid how much you are prepared to pay for a click when someone types a particular phrase into Google. Google then decides which ads to show based on bids and a variety of factors. It has always been a basic article of faith that (if you choose manual bidding) you are never charged more than the maximum cost per click you set. But this is no longer true. Check out this recent Google Ads report for my data transformation software, Easy Data Transform.

adwords-cpc

How can the average cost per click be 3x more than the maximum cost per click? I am using manual bidding, am not using ‘enhanced CPC’, have no positive bid adjustments and I haven’t recently changed these bids. I emailed Google technical support. This is the reply I got:

Post reviewing the search campaign : Easy Data Transform, I can see that  you have applied Bidding strategy as Manual CPC, also you have included Google search partners due to which system has charged extra amount. As in Google search partners maximum capping would not work.

For example the keywords: [keyword redacted] in this the maximum cpc is set as while as you have included Google search partners due to which system has avg cpc charged £0.45.

‘Google search partners’ are “hundreds of non-Google websites, as well as YouTube and other Google sites”. New Adwords campaigns are opted in to search partners by default.

So I emailed back:

So when did this policy change?

Is there a limit what you can change for Search partners clicks? If I bid a Max CPC of £0.10 CPC can Adwords charge me £100.00, if it feels like it?

And the reply was:

Andy, I would like to inform you that there is no recent changes made in this policy. However, I would not be able to provide the exact date of change made in this policy. If you have applied the  Max CPC of £0.10, system may charge you £100 if the search partner is enabled at your campaign level.

WHAT? They can ignore my maximum bid and charge me £100 per click (if I am using manual bidding, tracking conversions and opted in to search partners)! I don’t trust Google’s algorithms to bid for me based on prior experience. Presumably they can’t charge more than my daily budget, but they can use this whole budget in a single click, if they choose.

Running a report you can see that the average cost per click of the partner network (red) has shot up recently for my Easy Data Transform campaign:

adwords-network

Digging a bit more I found this.

smart-bidding

So it looks like they quietly introduced this new policy in October 2018. I don’t remember being told about this. It doesn’t mention this policy if you hover over max CPC:

adwords-tip

It is only if you click ‘Learn more’ and read to the bottom of that page that they tell you about the exception for network partners.

I checked with my goto Google Ads expert, Aaron of softwarepromotions.com. He knows Google Ads inside out, but he hadn’t heard of this policy either.

Google Ads have a long history of quietly introducing major changes and not telling their customers, let alone asking them to opt-in. For example with in-app ads.

This change is particularly galling given that clicks from the partner network are generally lower quality and convert significantly worse than clicks from Google itsself.

So far the costs to me of this change don’t appear to be significant. But they could be very significant to people with larger accounts. I did a quick surf and I found this on webmasterworld.com, dated March 2019:

Don’t have all the numbers yet but it looks like so far this month about 60% of our spend this month has been on these inflated clicks. And when I say inflated, on the extreme end we have a keyword set to $0.27 CPC and the average CPC has been $4.67, and almost all clicks for the keyword have been from the search partner network.

I have turned off the Google partner network on all my campaigns. You might want to do the same.

disable

Easy Data Transform v1.6.0

I have been working hard on Easy Data Transform. The last few releases have included:

  • a 64 bit version for Windows
  • JSON, XML and vCard format input
  • output of nested JSON and XML
  • a batch processing mode
  • command line arguments
  • keyboard shortcuts
  • various improvements to transforms

Plus lots of other improvements.

The installer now includes 32 and 64 bit version of Easy Data Transform for Windows and installs the one appropriate to your operating sytem. This is pretty easy to with the Inno Setup installer. You just need to use Check: Is64BitInstallMode, for example:

[Files]
Source: "..\binaries\windows\program\{#MyAppExeName}"; DestDir: "{app}"; Flags: ignoreversion; Check: not Is64BitInstallMode
Source: "..\binaries\windows\program64\{#MyAppExeName}"; DestDir: "{app}"; Flags: ignoreversion; Check: Is64BitInstallMode

But it does pretty much double the size of the installer (from 25MB to 47MB in my case).

The 32 bit version is restricted to addressing 4GB of memory. In practise, this means you may run out of memory if you get much above a million data values. The 64 bit version is able to address as much memory as your system can provide. So datasets with tens or hundreds of millions of values are now within reach.

I have kept the 32 bit version for compatibility reasons. But data on the percentage of customers still using 32 bit Windows is surprisingly hard to come by. Some figures I have seen suggest <5%. So I will probably drop 32 bit Windows support at some point. Apple, being Apple, made the transition to a 64 bit OS much more aggressively and so the Mac version of Easy Data Transform has always been 64 bit only.

I have also been doing some benchmarking and Easy Data Transform is fast. On my development PC it can perform an inner join of 2 datasets of 900 thousand rows x 14 columns in 5 seconds. The same operation on the same machine in Excel Power Query took 4 minutes 20 seconds. So Easy Data Transform is some 52 times faster than Excel Power Query. Easy Data Transform is written in C++ with reference counting and hashing for performance, but I am surprised by just how much faster it is.

The Excel Power Query user interface also seems very clunky by comparison. The process to join two CSV files is:

  • Start Excel.
  • Start Power Query.
  • Load the two data files into Excel power query.
  • Choose the key columns.
  • Choose merge.
  • Choose inner join. Wait 4 minutes 20 seconds.
  • Load the results back into Excel. Wait several more minutes.
  • Save to a .csv file.
  • Total time: ~600 seconds

Whereas in Easy Data Transform you just:

  • Start Easy Data Transform.
  • Drag the 2 files onto the center pane.
  • Click ‘join’.
  • Select the key columns. Wait 5 seconds.
  • Click ‘To file’ and save to a .csv file.
  • Total time: ~30 seconds

join-op

If you have some data to transform, clean or analyze please give Easy Data Transform a try. There is a fully functional free trial. Email me if you have any questions.

 

Analyzing COVID19 data with Easy Data Transform

I have continued to make lots of improvements to Easy Data Transform, including:

Here is a video of me using Easy Data Transform to analyze the data.europa.eu COVID19 dataset. Hopefully it gives an idea of what the software is capable of.

covid19-data-wrangling

 

 

Easy Data Transform v1.1.0

I released v1.1.0 of Easy Data Transform this week. It is a big upgrade, with some major new features.

easy-data-transform-v110

There is a new Javascript transform. This allows you to create custom transforms for anything that is too specialist to do with the other 37 built-in transforms. I’m not a fan of Javascript, with its horrible scoping and typing, and I would have preferred Python or Lua. But there is a Javascript engine built into Qt, so this was the easiest way to add scripting. Now if you want to multiply two columns of your data together in Easy Data Transform, you can just do this:

javascript-data-transform-v110

You can also access Javascript maths, date and string functions. So you can do some pretty complicated stuff. Hopefully the built-in transforms are enough to cover 95% of data transformations. But the new Javascript transform adds some serious flexibility for the remainder. The Qt Javascript engine is also pretty fast. In testing I was able to multiply values from 2 columns together across 10,000 rows in less than 0.03 seconds.

There is a new Lookup transform. This allows you to lookup values for one dataset in another dataset. For example, if you have a dataset with a column for country code and another dataset with columns for the country code and tax rate, you can look up the tax rate by country code.

Previously you could only output your data in Excel and delimited text formats (including CSV and TSV). The new release also adds output to JSON, HTML, Markdown, vCard, YAML and XML formats.

I have improved the speed of the Join transform significantly using hashing. This makes a big difference with large datasets.

To save time, Easy Data Transform guesses the likely columns you want to use as keys when you Join, Intersect, Lookup or Subtract two datasets. For example if 2 datasets both have colummns called ‘ID’ with lots of unique values that are common to both columns, it will choose those two columns as the default key columns. I have improved the heuristic used to set the default columns.

You can now add comments to input, transform and output nodes as a note to a colleague or your future self.

You can now snap your input, transform and output nodes to a grid, so you can keep your layout all lovely and neat.

I have also made some bug fixes and minor improvements.

Haven’t tried Easy Data Transform yet? Got some table or list data that you need to wrangle into a more useful form? Take the free trial for a spin.