Tag Archives: qt

Bloviate

I wondered what it would look like if you took a body of text and then used it to generate new text, using Markov chains of different lengths. So I knocked up  quick program to try it.  ‘Bloviate’.

bloviate

Bloviate analyses your source text to find every sequence of N characters and then works out the frequency of characters that come next.

For example, if you set N=3 and your source text contains the following character sequences staring with ‘the’:

‘the ‘, ‘then’, ‘they’, ‘the ‘

Then ‘the’ should be followed 50% of the time by a space, 25% of the time by an ‘n’ and 25% of the time by a ‘y’.

Bloviate then creates output text, starting with the first N characters of the source text and filling in the rest randomly using the same sequence frequencies as the source text.

Note that a character is a character to Bloviate. It treats upper and lower case as different characters, makes no attempt to differentiate between letters, punctuation and white space and does not attempt to clean up the source text. Which also means it works on any language.

Bloviate also tells you the average number of different characters following each unique sequence of N, which I will call F here. As F approaches 1.0 the output text becomes closer and closer to the input text.

Using ‘Goldilocks and the 3 bears’ as input:

If N=1 (F=7.05) the output is garbage. Albeit garbage with the same character pair frequency as the original.

On cre She sl s ramy raked cheais Bus ore than s sherd up m. ged. bend staireomest p!”Sof ckstirigrorr a ry ps.

” f waine tind s aso Sowa t antthee aime bupis stht stooomed pie k is beche p!

At N=3 (F=1.44) it looks close to English, but jibberish:

Once up and been sight,” she this timed. Pretty so soon, she second soft. She screame up and she screame hot!” cried the Mama bed the Papa been sleeping in the Papa bear

“Someone’s bear growl.

At N=5 (F=1.14) it starts to look like proper English, but semantically weird:

Once upon a time, so she went for a walked right,” she lay down into the kitchen, Goldilocks sat in the porridge from the three chair,” growled, “Someone’s been sitting my porridge and she tasted the door, and ran down the bedroom. Goldilocks woke up and she second bowl.

And it comes out with occasional gems such as:

“Someone’s been sitting my porridge,” said the bedroom.

At N=10 (F=1.03) it starts to become reasonably coherent:

Once upon a time, there was a little tired. So, she walked into the forest. Pretty soon, she came upon a house. She knocked and, when no one answered, she walked right in.

At the table in the kitchen, there were three bowls of porridge.

At N=15 (F=1.01) it starts to get pretty close to the original text, but doesn’t follow quite the same order:

Once upon a time, there was a little girl named Goldilocks. She went for a walk in the forest. Pretty soon, she came upon a house. She knocked and, when no one answered, she walked right in.

At the table in the kitchen, there were three bowls of porridge. Goldilocks was very tired by this time, so she went upstairs to the bedroom. She lay down in the first bed, but it was too hard. Then she lay down in the third bed and it was just right. Goldilocks fell asleep.

At N=12 (F=1.07) the whole 680k characters of ‘Pride and prejudice’ produces:

It is a truth universally contradict it. Besides, there was a motive within her of goodwill which could not help saying:

“Oh, that my dear mother had more command over herself! She can have her own way.”

As she spoke she observed him looking at her earnest desire for their folly or their vice. He was fond of them.”

Obviously the source text is important. The Bohemian Rhapsody lyrics make nearly as much (or as little sense) at N=5 (F=1.08) as the original:

Is this to me, for me, to me

Mama, just a poor boy from this to me

Any way the truth

Mama, life? Is this time tomorrow

Carry on as if nothing all behind and face the truth

Mama, ooh, didn’t mean to me, baby!

Just gotta leave me and lightning, very fright out, just killed a man

Put a gun against his head

Pulled my time to die?

At N=12 (F=1.05) 160k characters of Trump election speeches produces:

Hillary brought death and disaster to Iraq, Syria and Libya, she empowered Iran, and she unleashed ISIS. Now she wants to raise your taxes very substantially. Highest taxed nation in the world is a tenant of mine in Manhattan, so many great people. These are people that have been stolen, stolen by either very stupid politicians ask me the question, how are you going to get rid of all the emails?” “Yes, ma’am, they’re gonna stay in this country blind. My contract with the American voter begins with a plan to end government that will not protect its people is a government corruption at the State Department of Justice is trying as hard as they can to protect religious liberty;

Supply your own joke.

I knocked together Bloviate in C++/Qt in a couple of hours, so it is far from commercial quality. But it is fairly robust, runs on Windows and Mac and can rewrite the whole of ‘Pride and prejudice’ in a few seconds. The core of Bloviate is just a map of the frequency of characters mapped to the character sequence they follow:

QMap< QString, QMap< QChar, int > >

You can get the Windows binaries here (~8MB, should work from Windows 7 onwards).

You can get the Mac binaries here (~11MB, should work from macOS 10.12 onwards).

Note that the Bloviate executable is tiny compared to the Qt library files. I could have tried to reduce the size of the downloads, but I didn’t.

To use Bloviate just:

  1. paste your source text in the left pane
  2. set the sequence length
  3. press the ‘Go >’ button

I included some source text files in the downloads.

You can get the source for Bloviate here (~1MB).

It should build on Qt 4 or 5 and is licensed as creative commons. If you modify it, just give me an attribution and send me a link to anything interesting you come up with.

Getting Qt 5.9 working on Windows (eventually)

I have had Qt 5.5 and 5.6 installed on my development machines for some time. Now that I have purchased a new Mac development box (an iMac with a lickably beautiful 27″ screen) I thought it was a good time to update to a more recent version of Qt. I went for Qt 5.9, rather than Qt 5.10, as 5.9 has been designated as an LTS (long term support) release. Upgrading turned into a real chore. I am quickly writing it up here in the hope that it helps someone else, and as a reminder to myself a few years down the line.

I like to build Qt from source. Because then I know it was built using the same compiler, headers, SDK etc as I am using to build my product. And I have more control over how Qt is configured. Also I can patch the source and rebuild it, if I need to. But I have had problems building Qt on Mac before. So I decided to install the pre-built binaries on my new Mac. I installed the latest version of XCode and then the Q5.9.4 binaries. This was a couple of big downloads, but it all went pretty smoothly.

I successfully built Qt 5.5 from source on my Windows machine previously, so I decided to try that for Qt 5.9. I have Visual Studio 2010 installed. This isn’t supported for Qt 5.9.4, so I downloaded Visual Studio 2017. I unzipped the Qt source into C:\Qt\5.9.4, ran ‘x86 native tools command prompt for VS 2017’, made sure Python and Perl were in the path and then:

cd C:\Qt\5.9.4

set QTDIR=C:\Qt\5.9.4\qtbase

set PATH=%QTDIR%\bin;%PATH%

configure -opensource -confirm-license -opengl desktop -nomake tests -nomake examples -no-plugin-manifests -debug-and-release -platform win32-msvc -verbose

nmake

Note that you are told by the nmake script to do nmake install at the end of this. But it tells you somewhere in the Qt Windows documentation not to do this, unless you have set the prefix argument (confusing, I know)

The build failed part way through making qtwebengine. Something to do with a path being too long for Perl or Python (I forget). It seems to be a known problem. Odd as the root path was just C:\Qt\5.9.4. I don’t need qtwebengine at present, so I deleted everything and tried again with -skip qtwebengine:

configure -opensource -confirm-license -opengl desktop -skip qtwebengine -nomake tests -nomake examples -no-plugin-manifests -debug-and-release -platform win32-msvc -verbose

nmake

It seemed to complete ok this time. But using this version of Qt to build Hyper Plan I got an error:

Unknown module(s) in QT:svg

On further examination the SVG DLL  had been built, but hadn’t been copied to the C:\Qt\5.9.4\qtbase\bin folder. Similarly for a lot of the other Qt DLLs. I couldn’t find any obvious reason for this looking through logs, Stackoverflow and Googling. I could possibly do without the SVG functionality, but I wasn’t sure what else was broken. So I decided to give up on bulding from source on Windows as well.

I download the Qt 5.9.4 binaries for Visual Studio 2017. This seemed to go ok, but then I discovered that I could only build a 64-bit application from these. No 32-bit version was available for Visual Studio 2017. Many of my customers are still on 32 bit versions of Windows. So I need to be able to ship my product as a 32 bit executable + DLLs[1].

So I uninstalled Visual Studio 2017 and installed Visual Studio 2015. I then got an error message about Visual Studio 2017 redistributables that I hadn’t uninstalled. So I had to uninstall those and run a repair install on Visual Studio 2015. That seemed to work ok. So then I download the 32-bit Qt 5.9.4 binaries for Visual Studio 2015. I had to download these into a different top level folder (C:\Qtb), so as not to risk wiping existing Qt installs that I had previously managed to build from source.

Eventually I managed to build Hyper Plan and PerfectTablePlan on Mac and Windows. What a palaver though! Qt is an amazing framework and I am very grateful for everyone who works on it. But I wish they would make it a bit easier to install and upgrade! Has anyone actually managed to get Qt 5.9 built from source on Windows?

[1] I don’t bother shipping a 64-bit executable on Windows as the 32-bit executable works fine on 64-bit versions of Windows (my software doesn’t require excessive amounts of memory). I only ship a 64-bit executable on macOS as almost no-one uses 32-bit versions of macOS now.

Signing Qt applications for Mac OS X 10.9.5 and 10.10

I have written previously about signing Qt applications for Mac OS X. It all worked fine until I upgraded to Mac OS X 10.9.5, which broke my signing script. Those Apple chaps do love to break stuff. Grrr.

The problem appears to be that the directory structure of the app bundle has changed and the Qt4 macdeployqt command does not conform to the new layout (I believe this is also the case for Qt5). Oh joy. I managed to work out how to get it working again after a bit of digging around. The good news is that Apple have also made the codesign command easier with a --deep option to traverse and sign the whole bundle in a single command. About time.

So here is the basic process to build and sign your Qt .app on the latest versions of Mac OS X:

# deploy Qt frameworks into .app bundle
$QTDIR/bin/macdeployqt <your_app>.app -verbose=1
# optionally delete unwanted framework and plugin folders, e.g.:
# rm -f -r <your_app>.app/Contents/Frameworks/QtDeclarative.framework
# rm -f -r <your_app>.app/Contents/PlugIns/sqldrivers
# correct .app bundle structure
python rebundle.py $QTDIR <your_app>.app
# sign .app bundle (including frameworks and plugins)
codesign --deep --force --verify --verbose --sign "Developer ID Application: <your developer id>" <your_app>.app
# the 2 lines below are just for verification/diagnostics
otool -L <your_app>.app/Contents/MacOS/<your_app>
codesign --verify --verbose=4 <your_app>.app

(Sorry about the small font, but I wanted to avoid confusing line wraps).

I then invoke DropDmg to create a .dmg image file complete with licence and background image. This is all stuck it all in a bash script, which I can pretty much forget about it (until Apple break something else).

In the above rebundle.py is a Python script  written by some public spirited individual that can be downloaded from github (thank you, ‘kingcheez’). Note that you can just find and replace all the ‘5’ characters in the script by ‘4’ if you are still using Qt4.

The first time I ran my script I ended up with a whopping 50MB .app file. It turns out that the cp -r commands in my script don’t preserve symbolic links. So you end up with 3 copies of each framework library. You can avoid this by using cp -R instead.

On the subject of signing for Mac, Apple recently sent out an email stating:

Signatures created with OS X Mountain Lion 10.8.5 or earlier (v1 signatures) will be obsoleted and Gatekeeper will no longer recognize them. Users may receive a Gatekeeper warning and will need to exempt your app to continue using it. To ensure your apps will run without warning on updated versions of OS X, they must be signed on OS X Mavericks 10.9 or later (v2 signatures). … Apps signed with v2 signatures will work on older versions of OS X.

So you are going to have to start signing using 10.9, whether you like it or not.

Fixing Qt 4 for Mac OS X 10.9

Mac OS X 10.9 (Mavericks) was released yesterday. And those nice people at Apple made it free, so you can be sure lots of people are downloading it. However Qt 4 apps look at bit strange on the new OS. Look at the text alignment in these buttons:

buttons1The text isn’t centre aligned. It doesn’t look like much of an issue out of context. But it looks wrong when you look at a whole UI. The good news is that there is a simple fix:

#ifdef Q_OS_MACX
    if ( QSysInfo::MacintoshVersion > QSysInfo::MV_10_8 )
    {
        // fix Mac OS X 10.9 (mavericks) font issue
        // https://bugreports.qt-project.org/browse/QTBUG-32789
        QFont::insertSubstitution(".Lucida Grande UI", "Lucida Grande");
    }
#endif

You need to place this code in your main() before creating your QApplication. For more details see this bug report.

With the fix the buttons look like this:

buttons2Much better! There are some console warnings:

CoreText performance note: Client called CTFontCreateWithName() using
name "Lucida Grande" and got font with PostScript name "LucidaGrande". 
For best performance, only use PostScript names when calling this API.

I am not sure how how significant these are.

I have also found that updating to Qt 4.8.5 fixes a printing crash bug in my table plan software. This crash happened when rotated pixmaps were printed from Mac OS X 10.8.

I have seen on forums that Qt 5 is completely broken on 10.9. But I don’t know if that is true.

Boostrapped.fm podcast

I was a guest on episode 21 of Bootstrapped.fm, the podcast of Andrey Butov and Ian Landsman. The discussion was very wide-ranging, touching on SAAS vs web, the Qt development environment, the royal wedding, A/B testing, capoeira, Adwords, the history of shareware, my new training course and lots more besides. I really enjoyed it. Boostrapped.fm also has a thriving discussion forum at discuss.bootstrapped.fm.

Programming skills wanted

I am looking to outsource some self-contained programming tasks in areas that I don’t have expertise in. I am hoping that someone reading this blog might be able to help (or know someone that can) so I don’t have to go through outsourcing sites. These are the two skills sets I am currently looking for:

  1. Javascript/CSS/HTML – To write a single page web app. This will have a relatively simple UI displaying data read from XML. The app will need to work on a wide range of browsers and devices. Ideally you should also have some web design skills, but this isn’t essential.
  2. C++/Qt 4/OpenGL – To write a relatively simple 3D visualization model that runs on Windows and Mac. This will involve populating a 3D space with specified shapes and allowing simple movement around it.

Details:

  • I am expecting that I will need 2 different people, but it is possible there might be someone out there with experience in both.
  • These are small projects (probably less than 2 weeks for task 1 and less than 1 week for task 2). But they might lead on to more work in future.
  • Time scales are reasonably relaxed. Ideally I would like the work to be finished by the end of September.
  • You can be based anywhere in the world, but must be able to communicate in English (written and spoken).
  • Full copyright to the work will pass to my company on full payment.
  • Obviously cost is an issue. If I have 2 promising candidates, I am likely to pick the cheaper one.

If you are interested in doing either of these tasks please email me ( andy at oryxdigital.com ) before the end of Friday 26th August with subject “programming work” and a brief outline of:

  • Which of the 2 tasks you are interested in.
  • Your relevant experience. Ideally including details of related projects completed.
  • Your daily rate in Pounds Sterlings or US dollars.

I will send detailed specs to a shortlist of the best candidates. The work will be awarded on the basis of fixed price bids against the spec. Please don’t apply unless you have relevant experience – if I wanted a programmer without experience in these areas I could do it myself. ;0)

Qt visual artefacts on Mac OS X 10.6

deploy_nearly_everywhereThe release of Mac OS X 10.6 (Snow Leopard) snuck up on me while I have been working hard on a major new release of PerfectTablePlan plan. I didn’t really want to risk messing up my stable Mac development machine by installing it, so I asked testlab2.com (who have been doing some third party testing for me) to test my latest beta on it.

Bad news. All the combo boxes in my application (which work fine in Mac OS X 10.3, 10.4 and 10.5) have visual artefacts on Mac OS X 10.6. They work fine, but they don’t look right. I wondered if the artefact could be unique to some weird configuration on testlab2’s test machine, so I asked the fine folk on macsb mailing list to see if they could replicate it. 4 of them tried (thanks Cesar, Jon, Aaron and Pierre) and they all saw the visual artefacts. Damn.

I posted a question on the qt-interest forum. The single response (thanks Francesco) mentioned that this issue had been reported in the Qt development blog. I managed to find the blog post, applied the recommended single line code patch to Qt 4.4.2 and rebuilt everything. Cesar from macsb kindly re-tried the new binaries and reports that this improved things, but some artefacts are still visible the first time a combo box is shown. Perhaps the patch works better for Qt 4.5.

Note the artefacts at the top and bottom of the combo box drop-down list

Note the artefacts at the top and bottom of the combo box drop-down list

It looks like I am going to have to release it like this, unless someone knows of a better patch. This is very annoying after the months of hard work I have put into the new release. Aesthetics are important for commercial software, especially on the Mac. This could cost me quite a few sales.

I am not on the very latest release of Qt, but apparently this issue would still occur even if I was. Qt/Nokia have announced that they don’t expect to  support Mac OS X 10.6 until they release Qt 4.6, whenever that might be. What use is a cross-platform toolkit that doesn’t support the latest major OSs?

Developer releases of 10.6 have been available for a while – I believe the Qt team should have burnt the midnight oil to make sure they had a release that properly supported 10.6 as soon as it became generally available.  I know that isn’t easy given the size of the Qt framework and Apple’s penchant for secretiveness. But that is the game they are in and that is what I expect. I think they have seriously dropped the ball here, and this is coming from a longtime Qt fan-boy. Perhaps they have spread themselves a bit too thin by moving to LGPL licensing.

I have written this post as a quick heads up to other Qt developers. Thanks to everyone that helped me get this far.

Using a Mac mini for development

mac miniI have been using a Mac mini to port my C++/Qt based code to Mac OS X for the last 3.5 years. This is one of the early PowerPC based Mac minis, upgraded to 1GB of RAM. Being Apple hardware, it is expensive for what you get. But it has served me well. The small form factor (approx 17 x 17 x 5 cm) has also been useful in my cramped office, where I have it attached to the same monitor, mouse and keyboard as my Windows box through a KVM switch. But it is struggling to keep up with PerfectTablePlan’s ever increasing code base. A clean build of the PerfectTablePlan source into a Universal (fat) binary now takes an eye-watering 36 minutes to compile and link on the Mac mini. Building a PowerPC-only debug version still takes nearly half that time. That is painful, even just for occasional porting work.

As my main development environment is Windows, I can’t really justify the cost (or office space requirements) of a Mac Pro. So I decided to buy a new Mac mini, with an Intel Core 2 Duo processor. I did look around to see if I could find one at a discount. However, this being Apple hardware, no-one dares sell  at anything significantly less than Apple’s RRP. I bought the smaller (120GB) disk variant and had the dealer upgrade it to 2GB RAM, which tests on my old Mac mini indicated should be plenty for compiling and linking. I didn’t want to do the memory upgrade myself as I know, from experience with my first Mac mini, that removing the case involves putty knives and some very worrying cracking noises.

I had all sorts of problems trying to get the right cables. Firstly I wanted a Firewire cable so I could copy the set-up across from the old machine to the new machine using Apple’s Migration Assistant software. But it turns out that the old Mac Mini has a Firewire 400 6-pin socket, whereas the new Mac Mini has a Firewire 800 9-pin socket. I ordered a 6-pin to 9-pin Firewire cable cable. Then I discovered that there is more than one type of DVI cable. The old Mac mini was attached to my KVM switch with a DVI-I cable. The new Mac mini only accepts mini-DVI or (via a supplied adaptor) DVI-D. So I ordered a dual link DVI-D to DVI-D cable as well.

Once I had the right cables things went relatively smoothly. The Migration Assistant software copied almost all the apps and data across from the old machine to the new one. It even preserved settings for the apps, e.g. the email accounts in my Thunderbird email client. I just had to re-install XCode (which wasn’t copied across) and rebuild my Qt libraries (to avoid copious warnings due to the fact they had been built with an earlier version of XCode/gcc).

To use the migration assistant you simply:

  1. connect the 2 machines with a Firewire cable
  2. start-up the old machine with the ‘T’ key depresses to put it in ‘Target’ mode
  3. start-up the new machine
  4. follow the on-screen instructions

Nice. If only it was was that easy to set-up a new Windows machine.

A quick test shows that the new Mac mini is nearly 6 times faster at compiling and linking a Universal binary of PerfectTablePlan from scratch[1]:

mac mini performance

The time the new Mac mini takes to compile and link an Intel-only debug release of PerfectTablePlan also compares favourably with a similar build on my Windows 2.13 GHz Intel Core 2 Duo box with 4GB of RAM[2].

mac mini performance 2

This isn’t a fair hardware comparison, as the two machines are using completely different compilers and linkers and the Windows box was running various background services. But it certainly shows that Intel-based Mac minis are worth considering for use as development machines.

[1] The newer machine is using a newer version of XCode/gcc.

[2] The Windows box is using Visual Studio 2005.

Qt to be available for free under LGPL

qtToday Nokia announced that the cross-platform Qt framework is to be released under the LGPL, with no developer licensing fees or royalties. As someone who has been using Qt continuously for the last 9 years, this is of particular interest to me. Especially since the hefty annual renewal fee for my commercial Qt licence is due in a few months.

Here is the email I received from Nokia:

Dear Qt User:

Nokia is pleased to announce that with the release of Qt 4.5 you will be able to use Qt under the Lesser General Public License (LGPL) version 2.1 terms. When released in March 2009, Qt will be made available under three licensing options: Commercial, LGPL and GPL. Prior versions of Qt are not impacted by this announcement.

Nokia is committed to Qt and its continued development. By offering Qt under LGPL version 2.1 license terms alongside today’s licensing options Nokia hopes to:

– facilitate wider adoption of Qt across industries, desktop, web and embedded platforms.

– establish Qt as a de facto standard for application development.

– receive more valuable feedback and increased user contributions to ensure that Qt remains the best-in-class, cross-platform framework.

– extend Nokia’s existing platform commitment to the open source community.

By offering a cost-free LGPL license as well as commercial and GPL licenses to Qt, you can choose the license model that best fits your development requirements.

Irrespective of which license model you choose:

– Qt Software is committed to continuing to provide our customers with the same level of professional support, services and regular releases you have come to expect of Qt Software.

– We will continue to actively develop Qt, and with a greater degree of cooperation with the community through a new contribution model, we hope to make Qt even more valuable to our users.

For more information on the introduction of the LGPL license and what this means for you, please consult the Frequently Asked Questions section on http://www.qtsoftware.com.

Best regards

Tom Miller

Director of Sales

Nokia, Qt Software

I am a big fan of QT. Over the years it has evolved into an extremely polished and comprehensive framework, with: impressive cross-platform capabilities across a wide range of desktop OSs and embedded devices; C++ and Java APIs; excellent documentation and  a wide range of supporting tools (there is now even a cross-platform Qt IDE). The introduction of WebKit also takes Qt some way towards bridging the desktop/web divide. Widely admired by developers, the main stumbling block to Qt’s wider adoption has been the relatively high cost of commercial development licences.

Qt has been available for a while with both commercial and GPL licensing. The commercial version is expensive and the GPL version is free. However, using the GPL means you have to release the source of your own application, which is enough to make it unattractive to the vast majority of commercial software vendors. With the LGPL you can use the Qt libraries for free while keeping your own code proprietary.

So why would Nokia licence Qt under the LGPL? They even have a page on their site saying why they don’t think the LGPL is a good fit for Qt. A commercial licence for Qt is expensive, both in initial purchase costs and annual maintenance. Why is Nokia giving up a fat revenue stream? I am too cynical to believe that it is pure altruism. I guess the Qt licence fees are fairly insignificant to their new owner, Nokia, and they see it as an important strategic step to allow their mobile devices to compete against the free iPhone and Android APIs. Feel free to speculate on alternative motivations in the comments below.

As a commercial Qt licencee I am still working out the full implications of switching to LGPL Qt.

  • Including a copyright notice, the licence agreement and a link to the downloadable Qt source shouldn’t be a problem.
  • Shipping a linker and object files isn’t realistic, so I would probably have to dynamically link Qt. I much prefer static linking to avoid ‘DLL hell’ issues.
  • It isn’t clear whether all the Qt classes and widgets available in the commercial version are available in the LGPL version. Does it include the Qt 3 -> Qt 4 backward compatibility layer?
  • Will I still be able to get decent technical support, or will I need to buy a support contract?

I haven’t had time to assimilate it all yet. But I think a number of trends could make the free LGPL Qt into a big player in the future:

  • The increasing interest in cross-platform tools due to the growing Mac market share and an increasing numbers of mobile devices.
  • The neverending uncertainties about the future of Delphi.
  • The shortcomings of .Net for some types of development, e.g. ‘shrinkwrap’ software.
  • Microsoft’s apparent lack of enthusiasm for MFC and Windows Forms.
  • The many technical strengths of Qt.

It certainly looks like very bad news for directly competing cross-platform technologies such RealBasic and WxWidgets.

Further reading:

PS/ Thanks Nokia (I think)!