Category Archives: tools

Visual vs text based programming, which is better?

Visual programming tools (also called ‘no-code’ or ‘low-code’) have been getting a lot of press recently. This, in turn, has generated a lot of discussion about whether visual or text based programming (coding) is ‘best’. As someone who uses text programming (C++) to create a visual programming data wrangling tool (Easy Data Transform) I have some skin in this game and have thought about it quite a bit.

At some level, everything is visual. Text is still visual (glyphs). By visual programming here I specifically mean software that allows you to program using nodes (boxes) and vertexes (arrows), laid out on a virtual canvas using drag and drop. 

A famous example of this sort of drag and drop visual programming is Yahoo Pipes:

Yahoo Pipes
Credit: Tony Hirst

But there are many others, including my own Easy Data Transform:

Note that I’m not talking about Excel, Scratch or drag and drop GUI designers. Although some of the discussion might apply to them.

By text programming, I mean mainstream programming languages such as Python, Javascript or C++, and associated tools. Here is the QtCreator Interactive Development Environment (IDE) that I use to write C++ in, to create Easy Data Transform:

The advantages of visual programming are:

  • Intuitive. Humans are very visual creatures. A lot of our brain is given over to visual processing and our visual processing bandwidth is high. Look at pretty much any whiteboard, at any company, and there is a good chance you will see boxes and arrows. Even in non-techie companies.
  • Quicker to get started. Drag and drop tools can allow you to start solving problems in minutes.
  • Higher level abstractions. Which means you can work faster (assuming they are the right abstractions).
  • Less hidden state. The connections between nodes are shown on screen, rather than you having to build an internal model in your own memory.
  • Less configuration. The system components work together without modification.
  • No syntax to remember. Which means it is less arcane for people who aren’t experienced programmers.
  • Less run-time errors, because the system generally won’t let you do anything invalid. You don’t have to worry about getting function names or parameter ordering and types right.
  • Immediate feedback on every action. No need to compile and run.

The advantages of text programming are:

  • Denser representation of information.
  • Greater flexibility. Easier to do things like looping and recursion.
  • Better tooling. There is a vast ecosystem of tools for manipulating text, such as editors and version control systems.
  • Less lock-in. You can generally move your C++ or Python code from one IDE to another without much problem.
  • More opportunities for optimization. Because you have lower-level access there is more scope to optimize speed and/or memory as required.

The advantages and disadvantages of each are two sides of the same coin. A higher level of abstraction makes things simpler, but also reduces the expressiveness and flexibility. The explicit showing of connections can make things clearer, but can also increase on-screen clutter.

The typical complaints you hear online about visual programming systems are:

It makes 95% of things easy and 5% of things impossible

Visual programming systems are not as flexible. However many visual programming systems will let you drop down into text programming, when required, to implement that additional 5%.

Jokes aside, I think this hybrid approach does a lot to combine the strengths of both approaches.

It doesn’t scale to complex systems

Managing complex systems has been much improved over the years in text programming, using techniques such as hierarchy and encapsulation. But there is no reason these same techniques can’t also be applied to visual programming.

It isn’t high enough performance

The creators of a visual programming system are making a lot of design decisions for you. If you need to tune a system for high performance on a particular problem, then you probably need the low level control that text based programming allows. But with most problems you probably don’t care if it takes a few extra seconds to run, if you can do the programming in a fraction of the time. Also, a lot of visual programming systems are pretty fast. Easy Data Transform can join 2 one million row datasets on a laptop in ~5 seconds, which is faster than base R.

It ends up as spaghetti

Labview spaghetti from DailyWTF
Unreal Blueprint spaghetti from reddit.com/r/ProgrammerHumor/

I’m sure we’ve all seen examples of spaghetti diagrams. But you can also create horrible spaghetti code with text programming. Also, being able to immediately see that a visual program has been sloppily constructed might serve as a useful cue.

If you are careful to layout your nodes, you can keep things manageable (ravioli, rather than spaghetti). But it starts to become tricky when you have 50+ nodes with a moderate to high degree of connectivity, especially if there is no support for hierarchy (nodes within nodes).

Automatic layout of graphs for easier comprehension (e.g. to minimize line crossings) is hard (NP-complete, in the same class of problems as the ‘travelling salesman’).

No support for versioning

It is possible to version visual programming tools if they store the information in a text based file (e.g XML). Trying to diff raw XML isn’t ideal, but some visual based programming tools do have built-in diff and merge tools.

It isn’t searchable

There is no reason why visual programming tools should not be searchable.

Too much mousing

Professional programmers love their keyboard shortcuts. But there is no reason why visual programming tools can’t also make good use of keyboard shortcuts.

Vendor lock-in

Many visual programming tools are proprietary, which means the cost can be high for switching from one to another. So, if you are going to invest time and/or money heavily in a visual programming tool, take time to make a good choice and consider how you could move away from it if you need to. If you are doing quick and dirty one-offs to solve a particular problem that you don’t need to solve again, then this doesn’t really matter.

‘No code’ just means ‘someone else’s code’

If you are using Python+Pandas or R instead of Easy Data Transform, then you are also balancing on top of an enormous pile of someone else’s code.

We are experts, we don’t need no stinkin drag and drop

If you are an experienced text programmer, then you aren’t really the target market for these tools. Easy Data Transform is aimed at the analyst or business guy trying to wrangle a motley collection of Excel and CSV files, not the professional data scientist who dreams in R or Pandas. However even a professional code jockey might find visual tools faster for some jobs.

Both visual and text programming have their places. Visual programming is excellent for exploratory work and prototyping. Text based programming is almost always a better choice for experts creating production systems where performance is important. When I want to analyse some sales data, I use Easy Data Transform. But when I work on Easy Data Transform itself, I use C++.

Text programming is more mature than visual programming. FORTRAN appeared in the 1950s. Applications with graphical user interfaces only started becoming mainstream in the 1980s. Some of the shortcomings with visual programming reflect it’s relative lack of maturity and I think we can expect to see continued improvements in the tooling associated with visual programming.

Visual programming works best in specific domains, such as:

  • 3d graphics and animations
  • image processing
  • audio processing
  • game design
  • data wrangling

These domains tend to have:

  • A single, well defined data type. Such as a table of data (dataframe) for data wrangling.
  • Well defined abstractions. Such as join, to merge 2 tables of data using a common key column.
  • A relatively straightforward control flow. Typically a step-by-step pipeline, without loops, recursion or complex control flow.

My teenage son has been able to do some (I think) pretty impressive 3D modelling and animations just with Blender’s visual tools.

Visual programming has been much less successful when applied to generic programming, where you need lots of different data types, a wide range of abstractions and potentially complex control flow.

I’ve been a professional software developer since 1987. People (mostly in marketing) have talked about replacing code and programmers with point and click tools for much of that time. That is clearly not going to happen. Text programming is the best approach for some kinds of problems and will remain so for the foreseeable future. But domain-specific visual programming can be very powerful and has a much lower barrier to entry. Visual programming empowers people to do things that might be out of their reach with text programming and might never get done if they have to wait for the IT department to do it.

So, unsurprisingly, the answer to ‘which is better?’ is very much ‘it depends’. Both have their place and neither is going away.

Further reading:

Hacker News folk wisdom on visual programming

Visual Programming Codex

The life and times of Yahoo Pipes

The ‘No Code’ Delusion and HN discussion

‘Visual programming doesnt suck’ HN discussion (original article seems to have disappeared)

Visual Programming Languages – Snapshots

A Personal History of Visual Programming Environments

Is the future of data science drag and drop?

Rethinking Visual Programming with Go

Responses to this post on Reddit:

reddit.com/r/Programminglanguages

reddit.com/r/nocode

reddit.com/r/datascience

Summerfest 2023

Summerfest 2023 is on. Loads of quality software for Mac and Windows from independent vendors, at a discount. This includes my own Easy Data Transform and Hyper Plan, which are on sale with a 25% discount.

Find out more at artisanalsoftwarefestival.com .

Moving from altool to notarytool for Mac notarization

This is an update to my 2018 article How to notarize your software on macOS.

I have been using altool to notarize my Mac apps for some years. However Apple, being Apple, have deprecated altool in favour of the new notarytool. altool will stop working at some point in 2023. And Apple, being Apple, have made little attempt to keep consistency between the two.

I didn’t find anything online to tell me how arguments between the two tools related. Consequently I spent a while trying to guess which arguments mapped to which. I got locked out for a while for trying to wrong combination too many times. In the end I went from this:

xcrun altool -t osx -f <mydmg>.dmg --primary-bundle-id <com.company.product> --notarize-app --username <apple-account-email> --password <password>

... wait for approval email ...

xcrun altool --username <apple-account-email> --password <password> --notarization-info <RequestUUID>

To this:

xcrun notarytool submit <mydmg>.dmg --apple-id <apple-account-email> --team-id <teamid> --password <password> --verbose --wait

On the plus side the --wait option doesn’t exit until the notarization is complete, which means you can easily do you whole build, sign and notarize process in a single script. Hoorah.

Note that you still need to run the ‘stapling’ step after notarization:

xcrun stapler staple -v <mydmg>.dmg

More details on notarytool arguments at:

https://keith.github.io/xcode-man-pages/notarytool.1.html

Winterfest 2022

Easy Data Transform and Hyper Plan Professional edition are both on sale for 25% off at Winterfest 2022. So now might be a good time to give them a try (both have free trials). There is also some other great products from other small vendors on sale, including Tinderbox, Scrivener and Devonthink. Some of the software is Mac only, but Easy Data Transform and Hyper Plan are available for both Mac and Windows (one license covers both OSs).

Easy Data Transform progress

I have been gradually improving my data wrangling tool, Easy Data Transform, putting out 70 public releases since 2019. While the product’s emphasis is on ease of use, rather than pure performance, I have been trying to make it fast as well, so it can cope with the multi-million row datasets customers like to throw at it. To see how I was doing, I did a simple benchmark of the most recent version of Easy Data Transform (v1.37.0) against several other desktop data wrangling tools. The benchmark did a read, sort, join and write of a 1 million row CSV file. I did the benchmarking on my Windows development PC and my Mac M1 laptop.

Easy Data Transform screenshot

Here is an overview of the results:

Time by task (seconds), on Windows without Power Query (smaller is better):

data wrangling/ETL benchmark Windows

I have left Excel Power Query off this graph, as it is so slow you can hardly see the other bars when it is included!

Time by task (seconds) on Mac (smaller is better):

data wrangling/ETL benchmark M1 Mac

Memory usage (MB), Windows vs Mac (smaller is better):

data wrangling/ETL benchmark memory Windows vs Mac

So Easy Data Transform is nearly as fast as it’s nearest competitor, Knime, on Windows and a fair bit faster on an M1 Mac. It is also uses a lot less memory than Knime. However we have got some way to go to catch up with the Pandas library for Python and the data.table package for R, when it comes to raw performance. Hopefully I can get nearer to their performance in time. I was forbidden from including benchmarks for Tableau Prep and Alteryx by their licensing terms, which seems unnecessarily restrictive.

Looking at just the Easy Data Transform results, it is interesting to notice that a newish Macbook Air M1 laptop is significantly faster than a desktop AMD Ryzen 7 desktop PC from a few years ago.

Windows vs Mac M1 benchmark

See the full comparison:

Comparison of data wrangling/ETL tools : R, Pandas, Knime, Power Query, Tableau Prep, Alteryx and Easy Data Transform, with benchmarks

Got some data to clean, merge, reshape or analyze? Why not download a free trial of Easy Data Transform ? No sign up required.

Making explainer videos for your software

If you want to find out how to do something, such as do a mail merge in Word or fix a leaky valve on a radiator, where do you look first? Probably Youtube. Videos are an excellent way to explain something. More bandwidth than text and more scaleable than a 1-to-1 demo.

I’ve done explainer videos for all 3 of my products. But I found it a real struggle. I would write a script and then try to read the script and do the screencast at the same time and do it all in one take. I would stutter and stumble and it would take multiple attempts. It took ages and results were passable at best. I got some better software to edit the stumbles out, so I didn’t have to do it in one go. But it still took me a fair few attempts and quite a bit of editing. It became one of my least favourite things to do and so I did less and less of it.

Recently, I came across these slides on video by Christian Genco. These and subsequent Twitter exchanges with Christian convinced me that I should stop being a perfectionist about video and just start cranking them out on the grounds that a ‘good enough’ video is better than no video at all (‘the perfect is the enemy of the good’) and I would get better at it over time. As Stalin supposedly said “Quantity has a quality all of it’s own”.

So I have ditched the scripts and the perfectionism and I’ve managed to create 13 short Easy Data Transform explainer videos in the last week or so. And I am getting faster at it and (hopefully) a bit more polished. I’m definitely not an expert on this (and probably never will be) but here are some tips I have picked up along the way:

  • Get some decent software. I use Camtasia on Windows and it seems pretty good.
Camtasia
  • Try to talk slower.
  • Try to sound upbeat (not easy if you are British and could voice double for Eeyore).
  • Try not to move the mouse and talk at the same time. This makes editing a lot easier. Some people like to do the audio and the visual separately, but that seems like too much hassle.
  • If you stumble, just take a deep breath, say it again and then edit the stumble out later.
  • Get a reasonable mic. I have a snowball mic on a cantilevered stand. I covered it with a thin cloth to try to reduce pops.
  • The occasional ‘um’ is fine.
  • Have a checklist of things to do for each video, so you don’t forget anything (such as disabling your phrase expander software or muting the phone).
My setup. Note the high tech use of rubber bands.

I’m lucky to have a very quiet office, so I don’t have much background noise to contend with.

Using Camtasia I can easily add intos and outros, edit out stumbles and add various effects, such a mouse position highlighting and movement smoothing. I just File>Save as the previous project so that I don’t have to re-add the intro and outro. Unsurprisingly, Camtasia have lots of explainer videos. I wish there was a way to automatically ‘ripple delete’ any sections where there is no audio and no mouse movement (if there is, I haven’t found it). Some people recommend descript.com. It looks interesting, but I haven’t tried it.

I did an A/B test of recordings with my Senheiser headset mic against my Snowball mic and the consensus was that the headset was ok but the the Snowball mic sound quality was better.

Some people prefer to use synthetic voices, instead of their own voice. While these synthetic voices have improved a lot, they never sound quite right to me. Also it must be time consuming to type out all the text. Or you can pay to have a professional voiceover done, but this is surprisingly expensive (around $100 per minute, last time I checked) and almost certainly more time consuming than doing it yourself.

Some people aren’t confident about speaking on videos because they are not native speakers of that language and have an accent. Personally accents don’t bother me at all. In fact I like hearing English spoken with a foreign accent, as long as I can understand it. Also I think there is an authenticity to hearing a creator talk about their product in their own voice.

I’m not a big fan of music on explainer videos, so I don’t add any.

I let Youtube generate automatic captions for people that want them (which could be people in busy offices and on trains and planes, as well as the hearing impaired). They aren’t perfect, but they are good enough.

My videos are aimed at least as much at finding new users as helping existing users. So I make sure I research keyword terms (mostly in Google Adwords) before I decide which videos to make and what to title them. Currently I am targetting very specific keyword searches, such as How to convert CSV to Markdown. Easy Data Transform can do a lot more than just format conversion, but from an SEO point of view it is better to target the phrases that people are actually searching for.

I upload the videos as 1080P (1920 x 1080 pixels) on to the Easy Data Transform Youtube channel and onto my screencast.com account (which I pay a yearly fee for). I then embed the screencast.com videos on relevant easydatatransform.com pages using IFRAME embed codes created by screencast.com. I don’t use the Youtube videos on my website, because I don’t want people to be distracted by Youtube ads and ‘you may also like’ recommendations. They might be showing a competitor! I don’t host the videos on the website itsself as I worry that might slow down the website. I also link to the videos in screencast.com from my help documentation, as appropriate.

Some people like to embed video of themselves in screencasts, in the hope of making it more engaging. But personally I want people to concentrate on my software, rather than being distracted by the horror of my face. And not having to comb my hair or look smart was part of what got me into running my own software business.

In the next few months I will be checking my analytics to see how many views these videos get and whether they increase the time on page and reduce the bounce rate.

If you can spare a few seconds to go to my Youtube page and ‘like’ a video ot two or subscribe, that would be a big help!

Note that some of the above doesn’t apply when you are creating a demo video for your home page, rather than an explainer video. Your main demo video should be slick and polished.

WinterFest 2021

Winterfest 2021

Easy Data Transform and Hyper Plan Professional edition are both on sale for 25% off at Winterfest 2021. There is also some other great products from other small vendors on sale, including Tinderbox, Scrivener and Devonthink. Some of the software is Mac only, but Easy Data Transform and Hyper Plan are available for both Mac and Windows (one license covers both). Sale ends 11th January.

A Windows Developer in Mac Land

This is a guest post from fellow software developer, Simon Kravis.

Few developers would choose their development platform on the merits of their respective Integrated Development Environments (IDEs)  but it happens that applications developed in Windows need to be made available on the Mac platform.

There are many environments offering cross-platform (Mac, Windows and sometimes Android) functionality, but close inspection shows that they all have limitations.  Visual Studio (the native Windows IDE) can produce apps which will run on a Mac using .Net Core – but only if they are command line apps on Windows. Other environments (like Xamarin) do support interfaces, but only involving simple controls like text boxes or drop-downs. There are other cross-platform IDEs (such as Qt)  which offer better graphics support, but they are not cheap and the extent of their support is not evident. If you need functionality such as computer vision, there seems to be no alternative to creating a separate code base for the Mac. Once you start on this path it becomes obvious that Macs handle graphics (and interfaces) very differently from Windows.

Macs have evolved rather more than PCs over the decades: they abandoned their proprietary Mac operating system in favour of UNIX in 1999, adopting the NeXTSTEP platform created by NeXT. Apple originally used PowerPC chips, replacing them with Intel Core processors in 2006, and they are currently transitioning to RISC chips. The Mac NeXTSTEP programming language was Objective C, developed in the 1980s and this is still supported, although the modern Swift language was introduced in 2014, and the Xcode IDE appeared in 2003. Xcode is free, even for teams. It uses the Cocoa API, which is accessible from other environments. The current release (MacOS  13.0) supports both Objective-C and Swift and is also used for developing iPhone and iPad apps. Mac operating systems since Catalina (released in 2019) are 64-bit only.  Xcode can only develop apps for Apple operating systems, notably iOS, which powers the iPhone. Most of the web questions and examples relate to iOS rather than MacOS. MacOS uses different frameworks from iOS, so some functions used in iOS are not available in MacOS, or have different parameters.

The Windows IDE (Visual Studio) dates from 1997, when it bundled together Visual Basic, Visual Fox Pro and Visual Source Safe and Visual C++.  It has an open architecture based on plug-ins and supports 36 different programming languages, but the major ones are C#, VB.Net and C++. Visual Studio can develop apps for any platform via the .NetCore framework, but capability for non-Windows platforms is limited. The Community edition is free, and has almost all the functionality of paid versions.

Both Visual Studio and Xcode are highly complex applications. They both have graphical interface builders where controls are dragged from a library onto a form.  Each application has a vocal supporters and detractors. My experience comes from about 5 years with Visual Studio developing C# applications. Before this I worked with Visual Basic for Applications in Microsoft Access, so I am well-versed in the Microsoft way of doing things.

Like most complex applications, Visual Studio and Xcode each have plenty of bugs, often producing completely unhelpful error messages. Reporting an Xcode bug through standard channels resulted in … nothing. Not even an automated message saying “Thank you for feedback. It will be used to improve future versions”. I haven’t even tried to report a Visual Studio bug, but I suspect that the much larger user base for Visual Studio will mean that workarounds are more readily available, even if the giant ship of Microsoft takes years to respond.

Moving to the Mac and Xcode for development was a shock as I found I didn’t know how to do the most basic things. String manipulation (used in most applications) in Objective C is highly verbose compared to C#. Google was invaluable for finding answers – mostly they were from Stack Overflow, but often from 10 or more years ago, sometimes from Apple Developer Forums. As Xcode has changed considerably since then, answers often had to be adjusted before they could be used.  Another problem is that functionality once provided externally has since been incorporated into Cocoa, so attempts to find a current version of a component (or framework as they called in Cocoa) are often unsuccessful.

MacOS provides more native functionality than Windows. Features such as computer vision and PDF generation are included in MacOS, rather than requiring the use of 3rd party components, which may not as robust as desired, and may require a license for commercial use. However, documentation of MacOS functionality, if present at all, was rarely useful. A few times I asked questions on Stack Overflow which attracted the ire of the Mac gurus for either through having obvious (to them) answers or through not conforming to the forum guidelines (in their opinion). However, the integration of NuGet with Visual Studio provides easy access to the massive number of 3rd party  libraries available for .Net on Windows.

The model-view-controller paradigm used on the Mac took some getting used to, as did the design of the main Xcode screen. Sometimes a useful display would disappear and I had difficulty in finding it how to bring it back. I often had to resort to retrieving earlier versions from the excellent Time Machine backup.  Form design is similar on both platforms – dragging and dropping components from a library. Both Xcode and Visual Studio have bugs, as would be expected for such complex apps. Events from components are generated automatically in Windows, but have to be defined on the Mac (as Actions). References to the component you’ve added also need to be defined on the Mac (as Outlets) and are not a property of the component, whereas on Windows they are.

The Xcode environment provides only basic facilities from scratch: if you need to do something more sophisticated you’ll have to Google around to find out how. Once you know – it’s easy, but the learning curve for Xcode is much higher than for Visual Studio.   

Rather than starting from scratch with the Mac version of my Caption Pro  app, which uses local computer vision functionality to detect multiple photos, changes image dimensions and adds text to images,  I found an existing open-source project on GitHub with similar basic functionality. This dated from 7 years ago and used Objective-C, so that was the language I opted for. An immediate handicap was that many of the answers I found to my questions used Swift in their example code, which is not interconvertible with Objective-C  in the way that C# and VB.Net are. iOS applications for the iPhone (which are most common) use different frameworks from Mac apps, and routines in them sometimes have completely different syntax.

The user interfaces for the Mac and Windows versions look quite different, as shown below. There are some basic differences – menus appear separately to the application window on the Mac and are locked to the top of the screen, whereas Windows menus are part of the application screen. Toolbars offer access to common functionality on the Mac. Differences also arise from the fact the Mac application was adapted from existing code rather than created from scratch.

Figure 1 Windows App main screen
Figure 2 Mac App main screen

Open-source examples (often from GitHub) are useful, but rarely work out-of-the- box. Sometimes the modifications need are minor – like defining the development team-  but sometimes it’s not possible to get them to build in a current version of Xcode.

Debugging on Xcode is frustrating – the call stack frequently contains assembler (which is perhaps why app performance tends to be better on Macs), and the debug variables window does not list all relevant variable values. Variable types may not be correct – Boolean values may appear as dates, and sometimes variables cannot even be evaluated by po (print out) statements. Printing out structure variables may show nothing.  Despite the generally superior performance of Mac apps, building apps in Xcode appears to be much slower than in Visual Studio on similar vintage machines, and after code stops at a breakpoint, it may take a long time before the variables window is filled. Deployment of Mac apps can still be done on an ad-hoc basis, but you have to register as an Apple Developer to avoid blockages in installation arising from being an ‘untrusted source’. Bypassing these blockages is more than a matter of clicking “Install anyway” so it’s hard to avoid forking out US$100 per year for registration. Windows has similar blockages, which can be bypassed with a code-signing certificate. These certificates are available from many vendors, and are slightly cheaper than Apple developer registration, but the process of obtaining one may be very involved.

Ad-hoc deployment is somewhat easier on the Mac than on Windows, but the method of doing it via Archive generation is anything but obvious. Mac applications are actually disk images and applications keep all of relevant files in a folder. This makes uninstallation a matter of dragging the application icon into the recycle bin, a far simpler process than on Windows. dmg files are not recognized by IIS web servers (and may not be by Apache either), so unless the file type is registered, download from a web site will not be possible.

Apple pioneered the App Store for iPhones (it is the only way in which iPhone apps can be installed) and Mac apps can also be put there. Apple takes a commission of 30% (or 15% if you are a small company) and they review all apps before adding them. Passing the review process may be a lngthy process, as not all problems are detected in a review cycle. Fixing these issues and resubmitting may result in further problems coming to light.  The review process may also be somewhat arbitrary. One App Store app presented an interface in German by default. English was available as Preferences option, but only after guessing where the Preferences option was located. App Store apps operate within a sandbox, which places restrictions on filesystem operations. Whether App Store deployment makes economic sense depends on the nature of the app, its market and price structure. Its advantages are that it targets the 16% of desktop users who use Macs, and streamlines installation (and payment, if applicable). The App Store supports ‘freemium’ pricing, where additional features are made available to paying users, but apps with free trial periods are shown as being free but with ‘in-app purchases’, which annoys some users.

Windows deployment can use .msi files, which have been around for decades, but are not easily installed by non-admin users. Self-extracting executables are more tractable, but 3rd party tools have to be used to create them. Windows 10 introduced Universal Windows Programs, which are easier to install and can be placed in the Microsoft Store, which operates in a similar way to the Apple App store, but for Windows desktops and tablets.

A key question which is very difficult to answer is “How long will it take me to convert my Windows app to run on a Mac?” Factors affecting this are app complexity, functionality and programmer skill.  The time between starting work on the Mac app and first deploying it on the company web site was about 3 months, but the amount of time spent on the project each day varied between zero and 3 or 4 hours. If you are a paid resource, then the cost of a cross-platform IDE may be justified, but the requirement for local computer vision functionality added a great deal of complexity to my requirements, which is one reason why I opted for a separate code base. Substantial evaluation would be required before deciding if a cross-platform environment could support any required  functionality.

Simon Kravis runs Aleka Consulting, a small software and consultancy company in Canberra, Australia specializing in information management and offering a number of software products. He has mainly developed scientific and engineering programs, starting in the era of paper tape.

Creating a forum for your product

I started selling software online 16 years ago. Until this year I never had a forum for any of my products. I handled customer support for PerfectTablePlan and Hyper Plan by email and kept customers up-to-date with an opt-in email newsletter. But I rethought this position with my latest product, Easy Data Transform and started a forum at forum.easydatatransform.com in December 2020.

My ISP offered various forum software packages, but I really wanted Discourse, as I consider it head and shoulders above all the other forum software I have interacted with as a user (even if I find the badge system a bit patronising). I didn’t want the hassle of setting up and patching a Discourse server, so created the forum through www.communiteq.com (previously discoursehosting.com). It was suprisingly easy to set-up. And it gives the option to export everything, in case I want to part ways with them. The sheer number of options in Discourse are quite daunting, but I stuck with the defaults for the most part.

Some people use Facebook Groups for their product forums. Ugh. You have almost no control of such a forum. Facebook could even be showing ads for your competitors on your forums. Or they could just decide to shut you down and delete all the content. That is before we get on to the fact that Facebook make their money monetising hatred and abusing our privacy at an industrial scale. No thanks.

The advantages of a forum are:

  • Letting customers talk to each other, and post content helps to create a community around the product. Which, in turn, can add a lot of value to your product.
  • Customers can help each other with support questions. Sometimes they will answer before you are able to or will give a different perspective. Or even give a better answer.
  • If a customer asks a question that has already been asked, you can send them a link to the appropriate forum page.
  • It is a quick and easy channel to communicate with customers. I can post a link to a new snapshot release in a few minutes. This is much quicker than sending out an email newsletter. It is also more interactive as customers can respond on the forum and see each other’s responses.
  • A lively forum is ‘social proof’ that your product is worth buying.
  • A forum with lots of content should have a large SEO footprint.

The disadvantages of a forum are:

  • The time to maintain it. A forum that is broken or full of spam and unanswered questions is worse than no forum.
  • Disgruntled customers potentially airing their grievances in public.
  • The cost of the forum.
  • An empty forum looks bad.
  • Bad actors can be a pain. For example, people posting links to spam or competing products.

It probably only takes me 1-2 hours per week to post on the forum at present. Some of that is time I would have spent answering support emails. If that rises substantially then I may have to delegate it.

I try very hard to provide a good product, with good support and haven’t had any issues with negativity, so far. But I know from my experiences moderating Joel Spolsky’s Business of Software forum that moderating a busy forum can be tricky, time-consuming and emotionally draining.

The cost of the forum is currently around $20 per month, so pretty low. That may climb, but hopefully sales will be climbing as well.

I was a bit worried about whether the forum was going to look empty. I warned customers that the forum was an experiment and would be closed if there wasn’t enough activity, to manage their expectations. I also created a ‘sock puppet’ account and ‘seeded’ the forum with a few support questions that I had been previously asked by email (with the permission of those that asked) and then posted answers. But I only did this a handful of times and then the forum started to take off.

I have heard stories of people getting 1000+ spam posts a day on their forum. But I haven’t had any issues with bad actors, so far. I’m not sure how much of that is down to Discourse and how much of it is down to luck. But, no doubt issues will occur at some point.

I still have my product newsletter, which I send out every few weeks when there is a new production release.

Overall I am pretty happy with how the forum is going. Should you have a forum for your product? As always, it depends. I think you should consider it if:

  • Your customer base isn’t tiny.
  • You want to interact with your customers and get feedback. This might be less the case with mature products.
  • You have the time and energy to police and maintain it.
  • Your product is relatively open ended or complex. For example, if your product just checks whether website are up or down, there is probably a very limited amount you can discuss.

Stalking website visitors with Microsoft Clarity

Microsoft Clarity is a new service that allows you to see, in detail, how visitors are interacting with your website. It includes:

  • heat maps, showing where visitors are clicking or touching or how far they are scrolling
  • recordings of visitor sessions, including mouse movement, clicks, touches and scrolling

You just need to get a Javascript snippet from clarity.microsoft.com (for which you need a Microsoft login) and paste it into the header of each page. You can then login to clarity.microsoft.com at a later time to see your results. The service is free.

I tried it on my www.easydatatransform.com website. Here you can see a click heatmap for the buy page:

People are clicking all over non-hyperlinked text. Hmm. Perhaps they somehow couldn’t the see the effing enormous blue button next to it? Notice that numbers are starred out to avoid personal information, such as credit card numbers.

You can also see how far visitors scroll down the page with scroll heatmaps:

So I can see that the buy button is appearing well above the fold.

You can also watch recodings of people interacting with the website, showing their mouse movements, clicks, touches and scrolling. This is where things start to feel a bit stalkerish. You don’t get any identifiable information on the visitor beyond their country, browser and their operating system and I’m ok with people watching me interact with their websites like this. But it still feels a bit voyeuristic. The results are also a bit strange. Some people just click all over the place and highlight random text (touches are tracked separately from clicks). There is a distinct danger that you could watch hours of sessions and come away without much actionable information.

You can filter the information in various ways, including by country or referring website. You can even filter to see sessions with ‘Rage clicks’ (where the user has clicked or tapped repeatedly in the same area).

Watching a few sessions with ‘rage clicks’ I could see that some people indeed seem completely unable to see the effing enormous blue ‘buy now’ button on the buy page. So I have also added a text hyperlink where most people are clicking in the text and will probably try changing the button colour. Perhaps to shocking pink!

Running Pingdom Website Speed Test on the Easy Data Transform home page, both with and without Clarity a few minutes apart, I can see that it had some effect on speed, but not too much.

Without Clarity script: 175.2kb of scripts, 7 script requests.

With Clarity script: 194.3kb of scripts, 9 script requests.

The load time was actually 0.1s faster with Clarity. That is probably just an anomaly.

I have disabled Clarity for now. But may reenable it after I have made some changes to the website, to see the effect of the changes. Overall I was quite impressed with the service and it was surprisingly easy to set-up. But the cynic in me does wonder what exactly Microsoft is getting out of it.