Category Archives: Windows

WinterFest 2021

Winterfest 2021

Easy Data Transform and Hyper Plan Professional edition are both on sale for 25% off at Winterfest 2021. There is also some other great products from other small vendors on sale, including Tinderbox, Scrivener and Devonthink. Some of the software is Mac only, but Easy Data Transform and Hyper Plan are available for both Mac and Windows (one license covers both). Sale ends 11th January.

A Windows Developer in Mac Land

This is a guest post from fellow software developer, Simon Kravis.

Few developers would choose their development platform on the merits of their respective Integrated Development Environments (IDEs)  but it happens that applications developed in Windows need to be made available on the Mac platform.

There are many environments offering cross-platform (Mac, Windows and sometimes Android) functionality, but close inspection shows that they all have limitations.  Visual Studio (the native Windows IDE) can produce apps which will run on a Mac using .Net Core – but only if they are command line apps on Windows. Other environments (like Xamarin) do support interfaces, but only involving simple controls like text boxes or drop-downs. There are other cross-platform IDEs (such as Qt)  which offer better graphics support, but they are not cheap and the extent of their support is not evident. If you need functionality such as computer vision, there seems to be no alternative to creating a separate code base for the Mac. Once you start on this path it becomes obvious that Macs handle graphics (and interfaces) very differently from Windows.

Macs have evolved rather more than PCs over the decades: they abandoned their proprietary Mac operating system in favour of UNIX in 1999, adopting the NeXTSTEP platform created by NeXT. Apple originally used PowerPC chips, replacing them with Intel Core processors in 2006, and they are currently transitioning to RISC chips. The Mac NeXTSTEP programming language was Objective C, developed in the 1980s and this is still supported, although the modern Swift language was introduced in 2014, and the Xcode IDE appeared in 2003. Xcode is free, even for teams. It uses the Cocoa API, which is accessible from other environments. The current release (MacOS  13.0) supports both Objective-C and Swift and is also used for developing iPhone and iPad apps. Mac operating systems since Catalina (released in 2019) are 64-bit only.  Xcode can only develop apps for Apple operating systems, notably iOS, which powers the iPhone. Most of the web questions and examples relate to iOS rather than MacOS. MacOS uses different frameworks from iOS, so some functions used in iOS are not available in MacOS, or have different parameters.

The Windows IDE (Visual Studio) dates from 1997, when it bundled together Visual Basic, Visual Fox Pro and Visual Source Safe and Visual C++.  It has an open architecture based on plug-ins and supports 36 different programming languages, but the major ones are C#, VB.Net and C++. Visual Studio can develop apps for any platform via the .NetCore framework, but capability for non-Windows platforms is limited. The Community edition is free, and has almost all the functionality of paid versions.

Both Visual Studio and Xcode are highly complex applications. They both have graphical interface builders where controls are dragged from a library onto a form.  Each application has a vocal supporters and detractors. My experience comes from about 5 years with Visual Studio developing C# applications. Before this I worked with Visual Basic for Applications in Microsoft Access, so I am well-versed in the Microsoft way of doing things.

Like most complex applications, Visual Studio and Xcode each have plenty of bugs, often producing completely unhelpful error messages. Reporting an Xcode bug through standard channels resulted in … nothing. Not even an automated message saying “Thank you for feedback. It will be used to improve future versions”. I haven’t even tried to report a Visual Studio bug, but I suspect that the much larger user base for Visual Studio will mean that workarounds are more readily available, even if the giant ship of Microsoft takes years to respond.

Moving to the Mac and Xcode for development was a shock as I found I didn’t know how to do the most basic things. String manipulation (used in most applications) in Objective C is highly verbose compared to C#. Google was invaluable for finding answers – mostly they were from Stack Overflow, but often from 10 or more years ago, sometimes from Apple Developer Forums. As Xcode has changed considerably since then, answers often had to be adjusted before they could be used.  Another problem is that functionality once provided externally has since been incorporated into Cocoa, so attempts to find a current version of a component (or framework as they called in Cocoa) are often unsuccessful.

MacOS provides more native functionality than Windows. Features such as computer vision and PDF generation are included in MacOS, rather than requiring the use of 3rd party components, which may not as robust as desired, and may require a license for commercial use. However, documentation of MacOS functionality, if present at all, was rarely useful. A few times I asked questions on Stack Overflow which attracted the ire of the Mac gurus for either through having obvious (to them) answers or through not conforming to the forum guidelines (in their opinion). However, the integration of NuGet with Visual Studio provides easy access to the massive number of 3rd party  libraries available for .Net on Windows.

The model-view-controller paradigm used on the Mac took some getting used to, as did the design of the main Xcode screen. Sometimes a useful display would disappear and I had difficulty in finding it how to bring it back. I often had to resort to retrieving earlier versions from the excellent Time Machine backup.  Form design is similar on both platforms – dragging and dropping components from a library. Both Xcode and Visual Studio have bugs, as would be expected for such complex apps. Events from components are generated automatically in Windows, but have to be defined on the Mac (as Actions). References to the component you’ve added also need to be defined on the Mac (as Outlets) and are not a property of the component, whereas on Windows they are.

The Xcode environment provides only basic facilities from scratch: if you need to do something more sophisticated you’ll have to Google around to find out how. Once you know – it’s easy, but the learning curve for Xcode is much higher than for Visual Studio.   

Rather than starting from scratch with the Mac version of my Caption Pro  app, which uses local computer vision functionality to detect multiple photos, changes image dimensions and adds text to images,  I found an existing open-source project on GitHub with similar basic functionality. This dated from 7 years ago and used Objective-C, so that was the language I opted for. An immediate handicap was that many of the answers I found to my questions used Swift in their example code, which is not interconvertible with Objective-C  in the way that C# and VB.Net are. iOS applications for the iPhone (which are most common) use different frameworks from Mac apps, and routines in them sometimes have completely different syntax.

The user interfaces for the Mac and Windows versions look quite different, as shown below. There are some basic differences – menus appear separately to the application window on the Mac and are locked to the top of the screen, whereas Windows menus are part of the application screen. Toolbars offer access to common functionality on the Mac. Differences also arise from the fact the Mac application was adapted from existing code rather than created from scratch.

Figure 1 Windows App main screen
Figure 2 Mac App main screen

Open-source examples (often from GitHub) are useful, but rarely work out-of-the- box. Sometimes the modifications need are minor – like defining the development team-  but sometimes it’s not possible to get them to build in a current version of Xcode.

Debugging on Xcode is frustrating – the call stack frequently contains assembler (which is perhaps why app performance tends to be better on Macs), and the debug variables window does not list all relevant variable values. Variable types may not be correct – Boolean values may appear as dates, and sometimes variables cannot even be evaluated by po (print out) statements. Printing out structure variables may show nothing.  Despite the generally superior performance of Mac apps, building apps in Xcode appears to be much slower than in Visual Studio on similar vintage machines, and after code stops at a breakpoint, it may take a long time before the variables window is filled. Deployment of Mac apps can still be done on an ad-hoc basis, but you have to register as an Apple Developer to avoid blockages in installation arising from being an ‘untrusted source’. Bypassing these blockages is more than a matter of clicking “Install anyway” so it’s hard to avoid forking out US$100 per year for registration. Windows has similar blockages, which can be bypassed with a code-signing certificate. These certificates are available from many vendors, and are slightly cheaper than Apple developer registration, but the process of obtaining one may be very involved.

Ad-hoc deployment is somewhat easier on the Mac than on Windows, but the method of doing it via Archive generation is anything but obvious. Mac applications are actually disk images and applications keep all of relevant files in a folder. This makes uninstallation a matter of dragging the application icon into the recycle bin, a far simpler process than on Windows. dmg files are not recognized by IIS web servers (and may not be by Apache either), so unless the file type is registered, download from a web site will not be possible.

Apple pioneered the App Store for iPhones (it is the only way in which iPhone apps can be installed) and Mac apps can also be put there. Apple takes a commission of 30% (or 15% if you are a small company) and they review all apps before adding them. Passing the review process may be a lngthy process, as not all problems are detected in a review cycle. Fixing these issues and resubmitting may result in further problems coming to light.  The review process may also be somewhat arbitrary. One App Store app presented an interface in German by default. English was available as Preferences option, but only after guessing where the Preferences option was located. App Store apps operate within a sandbox, which places restrictions on filesystem operations. Whether App Store deployment makes economic sense depends on the nature of the app, its market and price structure. Its advantages are that it targets the 16% of desktop users who use Macs, and streamlines installation (and payment, if applicable). The App Store supports ‘freemium’ pricing, where additional features are made available to paying users, but apps with free trial periods are shown as being free but with ‘in-app purchases’, which annoys some users.

Windows deployment can use .msi files, which have been around for decades, but are not easily installed by non-admin users. Self-extracting executables are more tractable, but 3rd party tools have to be used to create them. Windows 10 introduced Universal Windows Programs, which are easier to install and can be placed in the Microsoft Store, which operates in a similar way to the Apple App store, but for Windows desktops and tablets.

A key question which is very difficult to answer is “How long will it take me to convert my Windows app to run on a Mac?” Factors affecting this are app complexity, functionality and programmer skill.  The time between starting work on the Mac app and first deploying it on the company web site was about 3 months, but the amount of time spent on the project each day varied between zero and 3 or 4 hours. If you are a paid resource, then the cost of a cross-platform IDE may be justified, but the requirement for local computer vision functionality added a great deal of complexity to my requirements, which is one reason why I opted for a separate code base. Substantial evaluation would be required before deciding if a cross-platform environment could support any required  functionality.

Simon Kravis runs Aleka Consulting, a small software and consultancy company in Canberra, Australia specializing in information management and offering a number of software products. He has mainly developed scientific and engineering programs, starting in the era of paper tape.

How to add a dark theme to your Qt application

Dark themes are now available for Windows 10 and Mac and it is increasingly expected that desktop applications will offer a dark theme. Previously Qt support for dark themes was patchy. But I am happy to say that it now seems to work fine with Qt 5.12.2, and I have added dark themes to both Windows and Mac versions of my Easy Data Transform and Hyper Plan applications.

Easy Data Transform for Mac with a dark theme:

Easy Data Transform for Windows with a dark theme:

Hyper Plan for Mac with a dark theme:

Hyper Plan for Windows with a dark theme:

I haven’t decided yet whether to add a dark theme to PerfectTablePlan.

Adding dark themes was a fair amount of work. But a lot of that was scouring forums to work out how to integrate with macOS and Windows. Hopefully this article will mean you don’t have to duplicate that work.

Dark themes work a bit differently on Windows and Mac. On Windows changing the UI theme to dark won’t directly affect your Qt application. But you can use an application stylesheet to set the appearance. On Mac changing the UI theme to dark will automatically change your application palette, unless you explicitly block this in your Info.plist file (see below). On both platforms you will need to change any icons you have set to the appropriate light/dark version when the theme changes. Some of this may change in future as dark themes are more closely integrated into Qt on Windows and Mac.

macOS

You can add the following helper functions to a .mm (Objective-C) file:

#include "Mac.h"
#import <Cocoa/Cocoa.h>

bool macDarkThemeAvailable()
{
    if (__builtin_available(macOS 10.14, *))
    {
        return true;
    }
    else
    {
        return false;
    }
}

bool macIsInDarkTheme()
{
    if (__builtin_available(macOS 10.14, *))
    {
        auto appearance = [NSApp.effectiveAppearance bestMatchFromAppearancesWithNames:
                @[ NSAppearanceNameAqua, NSAppearanceNameDarkAqua ]];
        return [appearance isEqualToString:NSAppearanceNameDarkAqua];
    }
    return false;
}

void macSetToDarkTheme()
{
   // https://stackoverflow.com/questions/55925862/how-can-i-set-my-os-x-application-theme-in-code
   if (__builtin_available(macOS 10.14, *))
   {
        [NSApp setAppearance:[NSAppearance appearanceNamed:NSAppearanceNameDarkAqua]];
   }
}

void macSetToLightTheme()
{
    // https://stackoverflow.com/questions/55925862/how-can-i-set-my-os-x-application-theme-in-code
    if (__builtin_available(macOS 10.14, *))
    {
        [NSApp setAppearance:[NSAppearance appearanceNamed:NSAppearanceNameAqua]];
    }
}

void macSetToAutoTheme()
{
    if (__builtin_available(macOS 10.14, *))
    {
        [NSApp setAppearance:nil];
    }
}

The macSetToLightTheme() and macSetToDarkTheme() are useful if you want to give the user the option to ignore the OS theme. Call macSetToAutoTheme() to set it back to the default.

Corresponding header file:

#ifndef MAC_H
#define MAC_H

bool macDarkThemeAvailable();
bool macIsInDarkTheme();
void macSetToDarkTheme();
void macSetToLightTheme();
void macSetToAutoTheme();

#endif // MAC_H

You then need to add these files into your .pro file:

macx {
   ...
   HEADERS += Mac.h
   OBJECTIVE_SOURCES += Mac.mm
}

You can detect a change of theme by overriding changeEvent():

void MainWindow::changeEvent( QEvent* e )
{
#ifdef Q_OS_MACX
    if ( e->type() == QEvent::PaletteChange )
    {
        // update icons to appropriate theme
        ...
    }
#endif
    QMainWindow::changeEvent( e );
}

If you decide you *don’t* want to add a dark theme to your Mac app, the you should add the bold entry below to your Info.plist file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    ...
    <key>NSRequiresAquaSystemAppearance</key>
    <true/>
</dict>
</plist>

This will then force it to be shown in a light theme, regardless of the theme the operating system is in.

Windows

To set a dark theme palette you can use a stylesheet:

QFile f( ":qdarkstyle/style.qss" );
if ( !f.exists() )
{
   qWarning() << "Unable to set dark stylesheet, file not found";
}
else
{
   f.open( QFile::ReadOnly | QFile::Text );
   QTextStream ts( &f );
   getApp()->setStyleSheet( ts.readAll() );
}

The stylesheet I used was a modified version of qdarkstyle from a few years ago.

To unset the stylesheet and return to a light theme just call:

getApp()->setStyleSheet( "" );

Alternatively you can do it by changing the application palette.

Windows helper functions:

bool windowsDarkThemeAvailable()
{
    // dark mode supported Windows 10 1809 10.0.17763 onward
    // https://stackoverflow.com/questions/53501268/win10-dark-theme-how-to-use-in-winapi
    if ( QOperatingSystemVersion::current().majorVersion() == 10 )
    {
        return QOperatingSystemVersion::current().microVersion() >= 17763;
    }
    else if ( QOperatingSystemVersion::current().majorVersion() > 10 )
    {
        return true;
    }
    else
    {
        return false;
    }
}

bool windowsIsInDarkTheme()
{
    QSettings settings( "HKEY_CURRENT_USER\\Software\\Microsoft\\Windows\\CurrentVersion\\Themes\\Personalize", QSettings::NativeFormat );
    return settings.value( "AppsUseLightTheme", 1 ).toInt() == 0;
}

Currently there seems to be no way to conect to a signal or event that shows the theme has changed in Windows. So I connected to a signal from a QTimer that fires every 5 seconds to check windowsIsInDarkTheme().

Icons

When the theme changes you potentially need to update any icons you have set, e.g. for the toolbar.

In a light theme you can usually set the active icons and let Qt calculate the corresponding disabled icons. This doesn’t work for a dark theme as you want the disabled icons to be darker than the enabled icons, rather than lighter. So you can either calculate the disabled icons programmatically or you can provide a set of disabled icons as well. I opted for the former.

Assuming your icons are set up as resources under :/icons/dark and :/icons/light you can do something like this:

QString getResourceName( const QString& iconName, bool dark )
{
    return QString( ":/icons/%1/%2" ).arg( dark ? "dark" : "light" ).arg( iconName );
}

QPixmap getPixmapResource( const QString& iconName, bool dark )
{
    QString resourceName = getResourceName( iconName, dark );
    QPixmap pixmap = QPixmap( resourceName );
    Q_ASSERT( !pixmap.isNull() );
    return pixmap;
}

QIcon getIconResource( const QString& iconName, bool dark )
{
    QIcon icon;
    QPixmap pixmap = getPixmapResource( iconName, dark );
    icon.addPixmap( pixmap );
    if ( dark )
    {
        // automatic disabled icon is no good for dark
        // paint transparent black to get disabled look
        QPainter p( &pixmap );
        p.fillRect( pixmap.rect(), QColor( 48, 47, 47, 128 ) );
        icon.addPixmap( pixmap, QIcon::Disabled );
    }
    return icon;
}

Then you can reset the icon for the appropriate theme with:

bool isDark()
{
#ifdef Q_OS_MACX
   return macIsInDarkTheme();
#else
   return windowsIsInDarkTheme();
#endif
}
...
myButton->setIcon( getIconResource( "my_icon.png", isDark() ) );
...

You may also be able to update icons through QIcon::setThemeName(). But I didn’t explore this in any detail.

Note that you probably don’t want the enabled icons to be pure white, as it is a bit too visually jarring against a dark theme.

Easy Data Transform v1.6.0

I have been working hard on Easy Data Transform. The last few releases have included:

  • a 64 bit version for Windows
  • JSON, XML and vCard format input
  • output of nested JSON and XML
  • a batch processing mode
  • command line arguments
  • keyboard shortcuts
  • various improvements to transforms

Plus lots of other improvements.

The installer now includes 32 and 64 bit version of Easy Data Transform for Windows and installs the one appropriate to your operating sytem. This is pretty easy to with the Inno Setup installer. You just need to use Check: Is64BitInstallMode, for example:

[Files]
Source: "..\binaries\windows\program\{#MyAppExeName}"; DestDir: "{app}"; Flags: ignoreversion; Check: not Is64BitInstallMode
Source: "..\binaries\windows\program64\{#MyAppExeName}"; DestDir: "{app}"; Flags: ignoreversion; Check: Is64BitInstallMode

But it does pretty much double the size of the installer (from 25MB to 47MB in my case).

The 32 bit version is restricted to addressing 4GB of memory. In practise, this means you may run out of memory if you get much above a million data values. The 64 bit version is able to address as much memory as your system can provide. So datasets with tens or hundreds of millions of values are now within reach.

I have kept the 32 bit version for compatibility reasons. But data on the percentage of customers still using 32 bit Windows is surprisingly hard to come by. Some figures I have seen suggest <5%. So I will probably drop 32 bit Windows support at some point. Apple, being Apple, made the transition to a 64 bit OS much more aggressively and so the Mac version of Easy Data Transform has always been 64 bit only.

I have also been doing some benchmarking and Easy Data Transform is fast. On my development PC it can perform an inner join of 2 datasets of 900 thousand rows x 14 columns in 5 seconds. The same operation on the same machine in Excel Power Query took 4 minutes 20 seconds. So Easy Data Transform is some 52 times faster than Excel Power Query. Easy Data Transform is written in C++ with reference counting and hashing for performance, but I am surprised by just how much faster it is.

The Excel Power Query user interface also seems very clunky by comparison. The process to join two CSV files is:

  • Start Excel.
  • Start Power Query.
  • Load the two data files into Excel power query.
  • Choose the key columns.
  • Choose merge.
  • Choose inner join. Wait 4 minutes 20 seconds.
  • Load the results back into Excel. Wait several more minutes.
  • Save to a .csv file.
  • Total time: ~600 seconds

Whereas in Easy Data Transform you just:

  • Start Easy Data Transform.
  • Drag the 2 files onto the center pane.
  • Click ‘join’.
  • Select the key columns. Wait 5 seconds.
  • Click ‘To file’ and save to a .csv file.
  • Total time: ~30 seconds

join-op

If you have some data to transform, clean or analyze please give Easy Data Transform a try. There is a fully functional free trial. Email me if you have any questions.

 

Easy Data Transform v1.0.0 released

v100-screen-cap

I finally released a paid version of Easy Data Transform today, for both Windows and Mac. I am very pleased with how it has turned out. Obviously it is only v1.0.0, so there is plenty of additional features I could add, including:

  • Batch processing
  • Support for JSON, XML, SQLite input/output
  • More transforms
  • A 64 bit version for Windows
  • A Linux version

But I need to listen carefully to prospective customers to decide which additional features to prioritize in future releases. It might be something I haven’t even thought of.

But v1.0.0 already has a really useful core of features. And, if you aren’t embarrassed by v1.0, you didn’t release it early enough. That said, I haven’t cut corners on quality. It has proper documentation and has been through extended beta testing, dogfooding and several rounds of usability and third party testing.

The product has a fully-functional 7 (non-consecutive) day free trial. I think that is enough for prospective customers to decide if it does what they need. I also have a 60 day money-back guarantee.

I have decided to go with a subscription model: $99 / €90 / £75 + tax per person per year. Which covers up to 3 computers. At this price point I can afford some paid promotion and to provide a decent level of support. I am not offering a monthly subscription, as I don’t really want people who are going to pay for 1 month (to do their annual TPS reports) and then cancel.

Have you got some data you need to merge, clean, reformat or de-dupe? Give it a try. You can get a 25% discount if you buy a subscription by the 27th December 2019 using this link.

 

Easy Data Transform

I have been furiously coding a new product. Easy Data Transform. It is a Windows and Mac tool for transforming table and list data from one form to another. Joining, splitting, reformatting, filtering, sorting etc.

easydatatransform

I have been thinking about this product idea for years. In fact I threw together a janky prototype back in 2008. It allows you to perform various operations on a pair of lists.

list-weaver

I used this prototype for jobs such as creating a list of emails of people who had bought Perfect Table Plan v5, but hadn’t upgraded to v6 yet. It worked. But it wasn’t very good. The biggest annoyance was that each operation obliterated everything that came before. Which made it very easy to lose track of where you had got to. And there was no repeatability. It was also limited to lists and it became clear that I really needed something that could also handle tabular data. I never released it.

But the idea has been running as a background process in my brain for 11 years since. And I think I have come up with a much better design in that time. Finally I had mature, stable versions of my Perfect Table Plan and Hyper Plan products out, so I decided to go for it. I am really pleased with how it has turned out so far.

If you aren’t embarrassed by v1.0 you didn’t release it early enough. And so I have cut lots of corners to get this first public version out. The documentation is only part written. I created the application icon myself  in 10 minutes. There is no licensing. The GUI is lacking polish. The website would make a designer cry. But the software seems fairly robust. My 13 year old son wasn’t able to crash it after 10 minutes of trying, despite financial incentives to do so.

I did some market research and spoke to some people who knew a bit about this market. But I deliberately didn’t look closely at any competing products, as I didn’t want to be mentally restricted by what others have done. For better or worse, I want to blaze my own trail. Copying other people’s stuff is a zero-sum game with no net benefit to society.

Most of the things that Easy Data Transform you can do, you can also do in Excel or SQL. My claim is that it is much quicker, easier and less error prone to do in Easy Data Transform. No programming or scripting required. I am hoping that people will be able to start using it within a couple of minutes of downloading it (I plan to do lots of usability testing). Will people pay for that? I hope so. I’m not aiming it at programmers. Perish the thought.

Naming is hard. I came up with some 70 names. Things like ‘Data Hero’, ‘Transform Flow’, ‘Transmogrify’ and ‘Data Rapture’. But the domains were taken, people I asked hated them or there was an existing service or product with that name. So I ended up with Easy Data Transform. It does what it says on the tin.

Why desktop? Surely no-one is writing new desktop apps in 2019? I believe a desktop solution has some real advantages in this market. The biggest ones are:

  • You don’t need to load your (potentially highly sensitive) data on to a third party server.
  • Not having to upload and download (potentially very large) data sets makes it much more responsive.

Easy Data Transform is currently free for anyone to use. You can get it from the super-minimalist easydatatransform.com website. The current 0.9.0 version expires on the 4th August 2019. You will then be able to get another free version. Once the product is mature enough, and if I am convinced there is enough demand, I will release a paid version. The free beta will probably last several months. Please try it and let me know how you get on. I am particularly interested to get feedback from anyone using it for real day-to-day tasks.

Of course the real challenge is always marketing. How to get noticed amongst many competing products. As well as helping to improve the product I am hoping that this extended beta will also help me to get some traction and better understand the market. For example, what price to charge and what trial model to use. Watch this space.

Bloviate

I wondered what it would look like if you took a body of text and then used it to generate new text, using Markov chains of different lengths. So I knocked up  quick program to try it.  ‘Bloviate’.

bloviate

Bloviate analyses your source text to find every sequence of N characters and then works out the frequency of characters that come next.

For example, if you set N=3 and your source text contains the following character sequences staring with ‘the’:

‘the ‘, ‘then’, ‘they’, ‘the ‘

Then ‘the’ should be followed 50% of the time by a space, 25% of the time by an ‘n’ and 25% of the time by a ‘y’.

Bloviate then creates output text, starting with the first N characters of the source text and filling in the rest randomly using the same sequence frequencies as the source text.

Note that a character is a character to Bloviate. It treats upper and lower case as different characters, makes no attempt to differentiate between letters, punctuation and white space and does not attempt to clean up the source text. Which also means it works on any language.

Bloviate also tells you the average number of different characters following each unique sequence of N, which I will call F here. As F approaches 1.0 the output text becomes closer and closer to the input text.

Using ‘Goldilocks and the 3 bears’ as input:

If N=1 (F=7.05) the output is garbage. Albeit garbage with the same character pair frequency as the original.

On cre She sl s ramy raked cheais Bus ore than s sherd up m. ged. bend staireomest p!”Sof ckstirigrorr a ry ps.

” f waine tind s aso Sowa t antthee aime bupis stht stooomed pie k is beche p!

At N=3 (F=1.44) it looks close to English, but jibberish:

Once up and been sight,” she this timed. Pretty so soon, she second soft. She screame up and she screame hot!” cried the Mama bed the Papa been sleeping in the Papa bear

“Someone’s bear growl.

At N=5 (F=1.14) it starts to look like proper English, but semantically weird:

Once upon a time, so she went for a walked right,” she lay down into the kitchen, Goldilocks sat in the porridge from the three chair,” growled, “Someone’s been sitting my porridge and she tasted the door, and ran down the bedroom. Goldilocks woke up and she second bowl.

And it comes out with occasional gems such as:

“Someone’s been sitting my porridge,” said the bedroom.

At N=10 (F=1.03) it starts to become reasonably coherent:

Once upon a time, there was a little tired. So, she walked into the forest. Pretty soon, she came upon a house. She knocked and, when no one answered, she walked right in.

At the table in the kitchen, there were three bowls of porridge.

At N=15 (F=1.01) it starts to get pretty close to the original text, but doesn’t follow quite the same order:

Once upon a time, there was a little girl named Goldilocks. She went for a walk in the forest. Pretty soon, she came upon a house. She knocked and, when no one answered, she walked right in.

At the table in the kitchen, there were three bowls of porridge. Goldilocks was very tired by this time, so she went upstairs to the bedroom. She lay down in the first bed, but it was too hard. Then she lay down in the third bed and it was just right. Goldilocks fell asleep.

At N=12 (F=1.07) the whole 680k characters of ‘Pride and prejudice’ produces:

It is a truth universally contradict it. Besides, there was a motive within her of goodwill which could not help saying:

“Oh, that my dear mother had more command over herself! She can have her own way.”

As she spoke she observed him looking at her earnest desire for their folly or their vice. He was fond of them.”

Obviously the source text is important. The Bohemian Rhapsody lyrics make nearly as much (or as little sense) at N=5 (F=1.08) as the original:

Is this to me, for me, to me

Mama, just a poor boy from this to me

Any way the truth

Mama, life? Is this time tomorrow

Carry on as if nothing all behind and face the truth

Mama, ooh, didn’t mean to me, baby!

Just gotta leave me and lightning, very fright out, just killed a man

Put a gun against his head

Pulled my time to die?

At N=12 (F=1.05) 160k characters of Trump election speeches produces:

Hillary brought death and disaster to Iraq, Syria and Libya, she empowered Iran, and she unleashed ISIS. Now she wants to raise your taxes very substantially. Highest taxed nation in the world is a tenant of mine in Manhattan, so many great people. These are people that have been stolen, stolen by either very stupid politicians ask me the question, how are you going to get rid of all the emails?” “Yes, ma’am, they’re gonna stay in this country blind. My contract with the American voter begins with a plan to end government that will not protect its people is a government corruption at the State Department of Justice is trying as hard as they can to protect religious liberty;

Supply your own joke.

I knocked together Bloviate in C++/Qt in a couple of hours, so it is far from commercial quality. But it is fairly robust, runs on Windows and Mac and can rewrite the whole of ‘Pride and prejudice’ in a few seconds. The core of Bloviate is just a map of the frequency of characters mapped to the character sequence they follow:

QMap< QString, QMap< QChar, int > >

You can get the Windows binaries here (~8MB, should work from Windows 7 onwards).

You can get the Mac binaries here (~11MB, should work from macOS 10.12 onwards).

Note that the Bloviate executable is tiny compared to the Qt library files. I could have tried to reduce the size of the downloads, but I didn’t.

To use Bloviate just:

  1. paste your source text in the left pane
  2. set the sequence length
  3. press the ‘Go >’ button

I included some source text files in the downloads.

You can get the source for Bloviate here (~1MB).

It should build on Qt 4 or 5 and is licensed as creative commons. If you modify it, just give me an attribution and send me a link to anything interesting you come up with.

Getting Qt 5.9 working on Windows (eventually)

I have had Qt 5.5 and 5.6 installed on my development machines for some time. Now that I have purchased a new Mac development box (an iMac with a lickably beautiful 27″ screen) I thought it was a good time to update to a more recent version of Qt. I went for Qt 5.9, rather than Qt 5.10, as 5.9 has been designated as an LTS (long term support) release. Upgrading turned into a real chore. I am quickly writing it up here in the hope that it helps someone else, and as a reminder to myself a few years down the line.

I like to build Qt from source. Because then I know it was built using the same compiler, headers, SDK etc as I am using to build my product. And I have more control over how Qt is configured. Also I can patch the source and rebuild it, if I need to. But I have had problems building Qt on Mac before. So I decided to install the pre-built binaries on my new Mac. I installed the latest version of XCode and then the Q5.9.4 binaries. This was a couple of big downloads, but it all went pretty smoothly.

I successfully built Qt 5.5 from source on my Windows machine previously, so I decided to try that for Qt 5.9. I have Visual Studio 2010 installed. This isn’t supported for Qt 5.9.4, so I downloaded Visual Studio 2017. I unzipped the Qt source into C:\Qt\5.9.4, ran ‘x86 native tools command prompt for VS 2017’, made sure Python and Perl were in the path and then:

cd C:\Qt\5.9.4

set QTDIR=C:\Qt\5.9.4\qtbase

set PATH=%QTDIR%\bin;%PATH%

configure -opensource -confirm-license -opengl desktop -nomake tests -nomake examples -no-plugin-manifests -debug-and-release -platform win32-msvc -verbose

nmake

Note that you are told by the nmake script to do nmake install at the end of this. But it tells you somewhere in the Qt Windows documentation not to do this, unless you have set the prefix argument (confusing, I know)

The build failed part way through making qtwebengine. Something to do with a path being too long for Perl or Python (I forget). It seems to be a known problem. Odd as the root path was just C:\Qt\5.9.4. I don’t need qtwebengine at present, so I deleted everything and tried again with -skip qtwebengine:

configure -opensource -confirm-license -opengl desktop -skip qtwebengine -nomake tests -nomake examples -no-plugin-manifests -debug-and-release -platform win32-msvc -verbose

nmake

It seemed to complete ok this time. But using this version of Qt to build Hyper Plan I got an error:

Unknown module(s) in QT:svg

On further examination the SVG DLL  had been built, but hadn’t been copied to the C:\Qt\5.9.4\qtbase\bin folder. Similarly for a lot of the other Qt DLLs. I couldn’t find any obvious reason for this looking through logs, Stackoverflow and Googling. I could possibly do without the SVG functionality, but I wasn’t sure what else was broken. So I decided to give up on bulding from source on Windows as well.

I download the Qt 5.9.4 binaries for Visual Studio 2017. This seemed to go ok, but then I discovered that I could only build a 64-bit application from these. No 32-bit version was available for Visual Studio 2017. Many of my customers are still on 32 bit versions of Windows. So I need to be able to ship my product as a 32 bit executable + DLLs[1].

So I uninstalled Visual Studio 2017 and installed Visual Studio 2015. I then got an error message about Visual Studio 2017 redistributables that I hadn’t uninstalled. So I had to uninstall those and run a repair install on Visual Studio 2015. That seemed to work ok. So then I download the 32-bit Qt 5.9.4 binaries for Visual Studio 2015. I had to download these into a different top level folder (C:\Qtb), so as not to risk wiping existing Qt installs that I had previously managed to build from source.

Eventually I managed to build Hyper Plan and PerfectTablePlan on Mac and Windows. What a palaver though! Qt is an amazing framework and I am very grateful for everyone who works on it. But I wish they would make it a bit easier to install and upgrade! Has anyone actually managed to get Qt 5.9 built from source on Windows?

[1] I don’t bother shipping a 64-bit executable on Windows as the 32-bit executable works fine on 64-bit versions of Windows (my software doesn’t require excessive amounts of memory). I only ship a 64-bit executable on macOS as almost no-one uses 32-bit versions of macOS now.

Promoting your software through 1-day sales and bundles

Hyper Plan, my visual planning software for Windows and Mac, has now been for sale for a bit less than 2 years. Given that I am (by choice) doing all the development, marketing and support for both Hyper Plan and my other product, PerfectTablePlan, I have had a limited amount of time to promote Hyper Plan. But Hyper Plan is in a  competitive market, where it is hard to get noticed using traditional promotional techniques such as SEO and PPC. So I have been experimenting with promotion via 1-day sales sites and bundles.

I did several promotions through both bitsdujour.com and macupdate.com promo. These were 50%-off sales for 1 day (sometimes extended for another day). The site takes 50% commission on the sale, so I only got $10 of my normal $40 ticket price. But I also got exposure to a whole new audience I wouldn’t normally reach.

I also included Hyper Plan in bundlehunt.com and macupdate.com software bundles. In these bundles customers purchased some 10 items of software at a big discount. The promotions lasted for a few weeks each. I am not at liberty to divulge how much I got for each licence, but a quick calculation based on the price of the bundles and the number of items in the bundle tells you that it was a lot less than $10!

My hopes related to sales sites and bundles were:

  1. A worthwhile amount additional sales revenue.
  2. Increased feedback, giving me more insight for improving the product.
  3. Making money further down the line from major upgrades (e.g. v1 to v2).
  4. That I wouldn’t be swamped in support emails from people who were paying me a lot less than the standard price.
  5. More word-of-mouth sales after the discount has finished.

On analysing the results, the first 4 turned out to be true.

I had previously tried promoting my PerfectTablePlan table planning software on bitsdujour.com, but the results were disappointing. It just wasn’t a good match for their audience. However Hyper Plan is a more general tool and it did a lot better. The bundles also sold in impressive volumes. The source of Hyper Plan sales revenues to date after commission (but not including upgrades) is show below.

sales-revenue-source

So the extra sales were certainly significant from a revenue point of view, bearing in mind that Hyper Plan is a relatively young and unknown product.

I also got some very useful feedback from the bitsdujour comments section.

I released v2 of Hyper Plan in March 2016. I have crunched the numbers to see how many v1 customers to date have paid for upgrades to v2.

percentage upgrades

I expected that the 1-day sale customers who had paid $20 for the initial licence would be less likely to pay $16 to upgrade to v2 than those who had hadn’t purchased at a heavy discount. I was surprised that the opposite turned out to be true. I don’t have a good theory why.

I don’t have any figures for bundle customer upgrades, as the bundles happened after v2 was released. Given that bundle purchasers probably only wanted a subset of the software in the bundle, I expect the upgrade percentages to be a lot lower than above.

I wasn’t swamped in support emails. In fact things were surprisingly quiet during the bundles, which makes me wonder how many people who purchased the bundle were interested in Hyper Plan.

There were no sustained jumps in traffic or sales after the 1-day sales or bundles ended.

Best of all, the 1-day sales and bundles don’t cost anything, apart from a modest amount of time to set-up.

I know some vendors promote these 1-day sales and bundles to existing customers. But I don’t understand why you would do that. The whole point of these channels is to reach new audiences. Also you risk annoying customers who have paid list price. If you already have an audience you can promote a sale to, then you don’t need 1-day sales sites or bundles. Just email them a discount voucher.

I had one complaint from an existing customer on a forum who had paid full price and then saw Hyper Plan in a 1-day sale. I offered to refund the difference back to them, but they didn’t take me up on it.

In conclusion, the sales and bundle sites brought in useful spikes of additional sales (especially when you include upgrades later on) and feedback, without a big jump in support burden. But they didn’t lead to a noticeable long-term increase in traffic or sales. Obviously every product is different. But if you have a product that needs exposure, isn’t too niche and doesn’t require a lot of support, it may be worth giving 1-day sales and bundles a try.

What every software vendor needs to know about SHA1/SHA2 and digital certificates

TL;DR : If you digitally sign your software you need to make sure you have an SHA2 certificate and use it to dual sign your software with both SHA1 and SHA2 digests.

Digital certificates are used to prove who authored a piece of software and that it hasn’t subsequently been tampered with. Starting with Windows XP SP2 you get a warning message if you download software that that isn’t signed with an appropriate digital certificate. So most commercial software vendors digitally sign their software. We grumble about price gouging by the certificate vendors and the hoops we have to jump through to get a certificate. But, apart from that, the system seems to work tolerably well. However Microsoft have thrown a spanner into the works by deprecating digital certificates using the SHA1 algorithm. I only found out about this a few weeks ago from a fellow vendor’s blog. Thanks for nothing Microsoft. If you are using a digital certificate you purchased more than a year ago, it is probably SHA1. This post explains what this means for software vendors, based on my research so far. I am not an expert on this topic and things seem to be changing fast, so please let me know if there are any mistakes or omissions.

I don’t digitally sign Windows software, does this affect me?

No. But perhaps treat Windows unsigned software warning with some skepticism until Windows software vendors sort this mess out. If you only develop for Mac OS X you can feel a bit smug (at least until the next time Apple nukes your development ecosystem from orbit).

What is SHA1?

SHA1 (Secure Hash Algorithm 1) is a cryptographic hash function that was used in digital certificates issued until recently. SHA1 was known to have weaknesses as far back as 2005. Microsoft (and Google) have finally decided that SHA1 is too vulnerable and SHA2 digital certificates should be used instead.

What happens if my certificate is SHA1?

If you signed your software with a timestamp before 01-Jan-2016:

  • It will be treated by Windows XP SP2/XP SP3/Vista as signed.
  • It will be treated by Windows 7/8/10 as signed only until 01-Jan-2017.

If you signed your software with a timestamp on or after 01-Jan-2016:

  • It will be treated by Windows XP SP2/XP SP3/Vista as signed.
  • On Windows 7/8/10 and you will get an ugly “The signature of <file> is corrupt or invalid” or “The signature of this program is corrupt or invalid” error when downloading. If you don’t see this, it might be because you haven’t done a Windows Update recently (shame on you).

Windows seems to treat software that has been downloaded from the web (with ‘mark of the web’) differently. So make sure you test a version of your software you have downloaded from the web. I carried out some tests on 01-Mar-2016 using an SHA1 certificate to sign an executable and then dowload it. It worked ok when downloaded using Firefox or Chrome, but was shown as corrupt when downloaded using IE.

How do I know if my current certificate is SHA1?

  1. Right click on your most recently signed installer and select Properties.
  2. Click on the Digital Signatures tab.
  3. Select the signature and click on the Details button.
  4. Click the View Certificate button.
  5. Click the Details tab.
  6. Look at the Signature hash algorithm.sha1 digital certificate

What should I do if my certificate is SHA1?

If you certificate hasn’t expired you should ask the company you purchased it from to issue you a new SHA2 certificate. They should do this free of charge. In the process they will revoke your SHA1 certificate, so you can no longer use it for signing. You should then use your new SHA2 certificate to double sign new releases (see below).

I have an SHA2 certificate, now what?

If you want a new release to be treated as signed on both Windows XP SP3/Vista and Windows 7/8/10 then you need to double sign the file for SHA1 and SHA2:

signtool.exe sign /f <pfx file> /p <pfx password> /t <sha1 timestamp server> /v <installer>

signtool.exe sign /f <pfx file> /p <pfx password> /tr <sha2 timestamp server> /fd sha256 /td sha256 /as /v <installer>

Note the the order of the above is important (SHA1 first).

The Comodo SHA1 and SHA2 timestamp server is:
http://timestamp.comodoca.com

You can add a /debug flag for verbose output.

If you only want to support Windows 7/8/10, then you can omit the first line (but why would you?).

You can use chktrust.exe to check the signature:

chktrust.exe <installer>

Note that only version 6.3 and later of signtool.exe (which comes with Windows 8.1 SDK and is also available here) supports the /as flag.

I always sign the program, as well as the installer.

Can I double sign .msi files?

I have seen reports that .msi installers don’t support double signing. But I don’t use .msi installers, so I haven’t investigated further.

What happens to software I signed with my SHA1 certificate after the certificate is revoked?

Software you signed previously will not be affected, e.g. it will be treated as signed by Windows 7/8/10 until 01-Jan-2017

How do I sign Windows XP SP1/XP SP2 software?

Windows XP SP1 doesn’t warn you if there is no signature, so you can ignore XP SP1. SHA2 signatures are not supported in Windows XP SP2. So you will need to have both valid SHA1 and SHA2 certificates to support XP SP2 and all the later versions of Windows. Its not clear that certificate vendors will allow this. Also, how many people with Windows XP SP2 (an unsupported OS) are out there buying software? I won’t be bothering to support signing for XP SP2.

Does this affect SSL certificates as well as code signing (Authenticode) certificates?

I believe so. But I don’t have any SSL certificates, so I haven’t investigated further.

How does this affect signing of device drivers?

I understand there are some differences for device drivers. But I don’t create device drivers, so I haven’t investigated further.

What is the difference between SHA2 and SHA256?

SHA2 is a family of two similar hash functions known as SHA256 and SHA512. SHA256 uses 32-bit words where SHA512 uses 64-bit words.

How secure is SHA2?

Er, it was designed by the NSA. Supply your own joke.

I don’t have a digital certificate, where can I get one?

I got my Comodo code signing certificate from reseller codesigning.ksoftware.net. They have a good reputation, and are significantly cheaper than Comodo. I don’t have any business relationship with them beyond being a happy customer.

Anything else I should know?

Microsoft has reserved the right to move the SHA1 deprecation date forward from 01-Jan-2017.

Acknowledgements

Thanks to Nikos Bozinis for first alerting me to this issue and to Mitchell Vincent of ksoftware.net for fact checking this article.

Further reading

http://zabkat.com/blog/code-signing-sha1-armageddon.htm

http://support.ksoftware.net/support/solutions/articles/215805-the-truth-about-sha1-sha256-and-code-signing-certificates-

http://social.technet.microsoft.com/wiki/contents/articles/32288.windows-enforcement-of-authenticode-code-signing-and-timestamping.aspx

Updates

02-Mar-2016: Added missing link and minor update.

03-Mar-2016: Minor update.