Category Archives: miscellaneous

Bloviate

I wondered what it would look like if you took a body of text and then used it to generate new text, using Markov chains of different lengths. So I knocked up  quick program to try it.  ‘Bloviate’.

bloviate

Bloviate analyses your source text to find every sequence of N characters and then works out the frequency of characters that come next.

For example, if you set N=3 and your source text contains the following character sequences staring with ‘the’:

‘the ‘, ‘then’, ‘they’, ‘the ‘

Then ‘the’ should be followed 50% of the time by a space, 25% of the time by an ‘n’ and 25% of the time by a ‘y’.

Bloviate then creates output text, starting with the first N characters of the source text and filling in the rest randomly using the same sequence frequencies as the source text.

Note that a character is a character to Bloviate. It treats upper and lower case as different characters, makes no attempt to differentiate between letters, punctuation and white space and does not attempt to clean up the source text. Which also means it works on any language.

Bloviate also tells you the average number of different characters following each unique sequence of N, which I will call F here. As F approaches 1.0 the output text becomes closer and closer to the input text.

Using ‘Goldilocks and the 3 bears’ as input:

If N=1 (F=7.05) the output is garbage. Albeit garbage with the same character pair frequency as the original.

On cre She sl s ramy raked cheais Bus ore than s sherd up m. ged. bend staireomest p!”Sof ckstirigrorr a ry ps.

” f waine tind s aso Sowa t antthee aime bupis stht stooomed pie k is beche p!

At N=3 (F=1.44) it looks close to English, but jibberish:

Once up and been sight,” she this timed. Pretty so soon, she second soft. She screame up and she screame hot!” cried the Mama bed the Papa been sleeping in the Papa bear

“Someone’s bear growl.

At N=5 (F=1.14) it starts to look like proper English, but semantically weird:

Once upon a time, so she went for a walked right,” she lay down into the kitchen, Goldilocks sat in the porridge from the three chair,” growled, “Someone’s been sitting my porridge and she tasted the door, and ran down the bedroom. Goldilocks woke up and she second bowl.

And it comes out with occasional gems such as:

“Someone’s been sitting my porridge,” said the bedroom.

At N=10 (F=1.03) it starts to become reasonably coherent:

Once upon a time, there was a little tired. So, she walked into the forest. Pretty soon, she came upon a house. She knocked and, when no one answered, she walked right in.

At the table in the kitchen, there were three bowls of porridge.

At N=15 (F=1.01) it starts to get pretty close to the original text, but doesn’t follow quite the same order:

Once upon a time, there was a little girl named Goldilocks. She went for a walk in the forest. Pretty soon, she came upon a house. She knocked and, when no one answered, she walked right in.

At the table in the kitchen, there were three bowls of porridge. Goldilocks was very tired by this time, so she went upstairs to the bedroom. She lay down in the first bed, but it was too hard. Then she lay down in the third bed and it was just right. Goldilocks fell asleep.

At N=12 (F=1.07) the whole 680k characters of ‘Pride and prejudice’ produces:

It is a truth universally contradict it. Besides, there was a motive within her of goodwill which could not help saying:

“Oh, that my dear mother had more command over herself! She can have her own way.”

As she spoke she observed him looking at her earnest desire for their folly or their vice. He was fond of them.”

Obviously the source text is important. The Bohemian Rhapsody lyrics make nearly as much (or as little sense) at N=5 (F=1.08) as the original:

Is this to me, for me, to me

Mama, just a poor boy from this to me

Any way the truth

Mama, life? Is this time tomorrow

Carry on as if nothing all behind and face the truth

Mama, ooh, didn’t mean to me, baby!

Just gotta leave me and lightning, very fright out, just killed a man

Put a gun against his head

Pulled my time to die?

At N=12 (F=1.05) 160k characters of Trump election speeches produces:

Hillary brought death and disaster to Iraq, Syria and Libya, she empowered Iran, and she unleashed ISIS. Now she wants to raise your taxes very substantially. Highest taxed nation in the world is a tenant of mine in Manhattan, so many great people. These are people that have been stolen, stolen by either very stupid politicians ask me the question, how are you going to get rid of all the emails?” “Yes, ma’am, they’re gonna stay in this country blind. My contract with the American voter begins with a plan to end government that will not protect its people is a government corruption at the State Department of Justice is trying as hard as they can to protect religious liberty;

Supply your own joke.

I knocked together Bloviate in C++/Qt in a couple of hours, so it is far from commercial quality. But it is fairly robust, runs on Windows and Mac and can rewrite the whole of ‘Pride and prejudice’ in a few seconds. The core of Bloviate is just a map of the frequency of characters mapped to the character sequence they follow:

QMap< QString, QMap< QChar, int > >

You can get the Windows binaries here (~8MB, should work from Windows 7 onwards).

You can get the Mac binaries here (~11MB, should work from macOS 10.12 onwards).

Note that the Bloviate executable is tiny compared to the Qt library files. I could have tried to reduce the size of the downloads, but I didn’t.

To use Bloviate just:

  1. paste your source text in the left pane
  2. set the sequence length
  3. press the ‘Go >’ button

I included some source text files in the downloads.

You can get the source for Bloviate here (~1MB).

It should build on Qt 4 or 5 and is licensed as creative commons. If you modify it, just give me an attribution and send me a link to anything interesting you come up with.

Volunteering Your IT Skills

There is a lot to be said for running a small software business (just me, with my wife doing some of the admin). For a start it gives me a great deal of flexibility, which I used to spend 2 months travelling abroad with my family last year. It is also low in stress, as I don’t have any employees to manage (my wife manages herself!). But even with some consulting work, going to the occasional conference and running some face-to-face training courses, I was starting to feel a little bit isolated after 13 years working mostly on my own. At that time the news was full of heart-rending stories of the suffering of refugees trying to flee war and repression. I don’t like the way the world is heading at present and wanted to do something to help these people in whatever small way I could. So I started volunteering at a charity for refugees and asylum seekers in my home town.

I initially tried to avoid doing computer things for the charity. But it quickly became clear that my IT skills were much more useful to them than anything else I could offer. Consequently I have been sorting out various IT issues, teaching people basic IT skills and have built them a simple CRM and reporting system, based on Airtable. Replacing the previous paper system with an electronic one is saving the staff and volunteers a lot of busy-work, which frees up their time to do more useful things. It is also giving the charity a lot more insight into how they are doing. And it has got me out from office and meeting some really great people who I wouldn’t have met otherwise. Win-win. Sometimes I feel spread a bit thin with my various work, charity and personal commitments (which is partly why the blog has been a bit quiet recently) but overall I am very glad I started volunteering.

So, if you are feeling a bit isolated, consider volunteering for a local charity. I suspect many small charities are desperate for volunteers with IT skills. Even if you are a programmer with quite specialist skills (like me), it is easy to forget that you are still an IT systems god compared with 99% of the population.

Giving a shit

Sturgeon’s law states that “90% of everything is crap”. He was probably an optimist. Here are some recent examples of the sort of crap I come across day to day:

The school selection website

My wife and I had to select which secondary school we want out son to go to, an important decision for our family. We had to do this via a website created on behalf of Swindon council. I won’t bore you with all the painful details, but only an impressive combination of incompetence and apathy could have produced something so egregiously awful. At the end of the process we got an error message and the promised confirmation email never arrived. We were left feeling confused and angry. Every other parent we spoke to had a similar experience.

The ATM

Feast your eyes on my local ATM:

ctnybk8wiaaqkza-jpg-large

Yes, that’s right, the buttons aren’t correctly aligned with the screen, so they have added some shonky visual cues in a feeble attempt to compensate for it. They failed – I have pressed the wrong button more than once. If they couldn’t move the buttons, why didn’t they just change the text positions in the software? I would like to know what sort of horrific set of bad decisions and sloppy planning led to this laughably bad design.

The in-flight meal

Check out this British Airways in-flight meal I was served:

IMG_1083.jpg

Behold, the cutlery is in a sealed plastic bag in the pasta. To get at your cutlery you have to open a slippery plastic bag covered in sauce with your fingers, which are now also covered in sauce. Who could have possibly have thought this was a good experience? You might as well just eat the pasta with your fingers. Or stick your face in the plate. Maybe the subliminal message is: if you won’t pay for business class we are going to make you eat like an animal.


You don’t have to look very hard to find crappy design. Badly designed parking buildings, confusing ticket machines, painful to use Sat Navs, packaging that is almost impossible to open, web forms that won’t let you use a space or a dash in a telephone number. I could go on, but I’m sure you could come up with plenty of examples from your own life. The most frustrating thing is that these issues could have been avoided with a little bit of thought and care. I doubt it would have added more than an extra 1% more effort or cost to get them right.

Crappy products and services make everyone’s life worse. Hold yourself to a higher standard. Take pride in your work. Do usability tests. Get feedback from your users. Fix things that are broken. Keep improving. Above all, give a shit.

How to build a gym in your garden

Human physiology has evolved for a challenging existence on the African savannah. It doesn’t cope well with sitting in front of a computer all day, with high energy foods constantly within easy reach. But going to the gym is a hassle: get your gear together, drive to the gym, get changed, do your workout, have a shower, get changed back, drive home. Even just going for a run means 2 changes of clothes and a shower. I wanted something high intensity that I could do in a few minutes every day. I work from home, so I built a gym in my garden, right outside my office. I posted some pictures of it on social media and a few people asked for details of how I made it. So I thought I would write it up here, in case anyone else was interested.

Construction materials:

  • 3.0m x 0.1m x 0.1m fence posts (2 of)
  • 1.8m x 0.1m x 0.1m fence posts (3 of)
  • 1.2m outdoor pull-up bars with fixings (coach bolts and washers) (3 of)
  • 20kg bags of Postcrete (19 of)
  • 20kg bags of gravel (2 of)

The total cost of all the above was about £240, including delivery of the fence posts and pull-up bars.

You should be able to get the fence posts from any fencing supplier. Make sure they are pressure treated, so they don’t rot away in a few years. Anything narrower than 0.1m x 0.1m might not be strong enough. Anything bigger is going to be pretty unwieldy to work with.

You can buy outdoor pull-up bars from various sources. I got mine here. Make sure the bars and their fixings are either galvanized or powder coated, so they don’t rust. I choose bars long enough that I have the option to do wide-grip pull-ups.

Postcrete is a special form of concrete for fence posts (I think it might be called Quickcrete is some countries). You just add water and it sets solid in minutes. Leave it to ‘cure’ for 24 hours before putting any weight on it. I used 5 bags of Postcrete for each of the 3.0m pull-up posts and 3 bags of Postcrete for each of the 1.8m dips posts. You could probably get away with less, but I preferred to ‘over-engineer’ it. I also threw some old bricks and hardcore into the holes for extra bulk. You can  use standard cement, which is cheaper, but not as convenient.

scan013.jpg

You need to dig your post holes according to the height and spacing you want for the bars, which will depend on your height. The pull-up bar should be roughly the same height as your knuckles with your arms full outstretched above your head. The dips bars should be slightly more than shoulder width apart and level with your lower ribs. If you are very tall, you might need longer posts than I did. The holes should be approximately 3 times the width of the fence posts. Put approximately 0.1m of gravel in the bottom of each hole for drainage. The gravel also helps with getting the posts at the same level.

Digging a 1.0m deep by 0.1m x 0.1m across hole is difficult using a spade. I recommend you use a post hole digging tool. I bought one from building supplier Wickes for £25. The bolts were a bit loose, but once I had tightened them up it was fine. You can also rent them, but 3 days rental was as expensive as buying one new.post hole digging toolDigging the holes is hard work! I did 30 minutes of digging every now and then. Usually when I got fed up with whatever I was working on. Tip: Cover the loose dirt from the hole with something waterproof as it is much harder to move later if it gets wet.

Attaching the bars before you set the posts isn’t practical. Setting all the posts before attaching any bars is asking for trouble. So we alternated setting the posts and attaching the bars.

Setting the posts and attaching the bars is definitely not a one-person job, so I conscripted the family to help. We used rubber bands to hold 2 spirit levels onto 2 adjacent sides of a post, to make sure it was completely vertical (you can also buy specialist post levellers). One person then held the post while the other one added the Postcrete and water. To attach the bars just drill 4 pilot holes into a post and then use a socket and ratchet to tighten the coach bolts onto the washers.

iPhone 040.jpg

I also bought a heavy duty rubber mat and post caps to finish things off.

Normally I only create digital things (software, websites, documentation, blog posts etc) so it was really nice to make something physical for a change. Given my modest DIY skills, I am very pleased with how it turned out. It feels very solid and everything is pretty straight and level. Not bad for a software engineer!

Pull-ups, dips and leg raises cover a lot of the major muscle groups between them. Currently I am trying to do pull-ups and dips on alternate days. I usually do 3 sets of as many as I can, with at least a few minutes rest in between. I also do some negative reps. A negative pull-up is where you jump up and then lower yourself as slooooooowly as you can. This sort of eccentric training is very good for building strength (and also useful if you aren’t yet strong enough to do a pull-up). Just hanging from the bar is good for stretching your back muscles.

Because my gym is right outside my office and only takes a minute or so per set, there is no excuse. I also have a reminder set up in the Balanced app on my iPhone. In a few weeks I have gone from 3 pull-ups to 8 pull-ups (with good form). Once I have improved my strength futher and reached a plateau on those exercises, I may try some more exotic exercises. I hope eventually to be able to do a ‘muscle up’!

Muscle-up

Rocket Science

My son, my wife and I have been messing around with model rockets. They seem to be a big thing in the USA, but are a lot less common here in the UK. They are a lot of fun.

I bought the above rocket + launch pad + launch controller kit from a local model shop, with some recovery wadding and 3 class C rocket motors with igniters:

rocket kit amazon.co.uk link

rocket kit amazon.com link

The total cost was £30.

Making the rocket involved a bit of glueing and assembly, but was fairly straightforward. Then we inserted some wadding (to protect the internals from the hot gas of the rocket motor), the recovery parachute and the nose cone with rotors. When it was finished we took it to a big open space, inserted a rocket motor and igniter, put it on the launch pad and used the 9v battery operated remote control to launch it.

We had a few non-launches because the crocodile clips (connecting the launch control to the igniter) touched, causing a short-circuit, or fell off. Not a great design. Once we had sorted that out we successfully launched and the rocket went well over 100 metres in the air. Cool!

In theory the motor should burn for a couple of seconds and then a little explosive charge fires to separate the nose cone from the main body. The main body then floats down on the parachute while the nose cone deploys spring-loaded rotors and auto-rotates down. In theory.  However, in our inexperience, we put in too much wadding and packed it too tightly. Consequently the rocket blew itself apart in mid-air and the parachute and rotors didn’t deploy. We managed to recover all the bits. The parachute was ok, but the rotor blades were too damaged to use again.

A video of our first launch

So we cut off the damaged section and added the nose cone back on to make a new, shorter rocket and did 2 more launches. Being lighter with the same motor it went a lot higher. Possibly over 200 metres!

We made a new rocket from the nose cone and tail of the kit, plus a long cardboard tube and lots of duct tape. We did another 3 launches using C class rocket motors. Even managing to get one successful parachute deployment. However as the new rocket was  heavier it got noticeablely less height, probably less than 100 metres.

A few things we learnt along the way:

  • Don’t force the parachute and nose cone in too hard or use too much wadding.
  • If the parachute doesn’t deploy the rocket can survive hitting the ground at speed surprisingly well. But they make quite a hole in the ground, so you REALLY don’t want to get in the way.
  • Even in light wind the rockets can land a fair distance away. Especially if the parachute deploys successfully. So pick a still day for the launch. You can also cut some extra vents in the parachute to make it fall faster.
  • You need a BIG open space, free from other people, animals and trees. Preferably at least 200 metres across, if you want to stand a good chance of recovering your rocket for another launch.
  • The maximum height of your rocket depends critically on the thrust to weight ratio.

Hopefully it goes without saying that pyrotechnics and objects travelling at high speed are potentially dangerous and require common sense and adult supervision.

Being a software geek with a physics background I couldn’t resist doing a few calculations. Here is a little Python script I wrote to calculate the maximum height and flight time based on the mass of the rocket and the thrust and duration of the motor. It applies a simple time-step approach to F=ma. Just modify the mass, thrust and duration variables.

rocket science codeIt assumes the rocket goes straight up and doesn’t allow for air resistance. But the values it calculates seem fairly plausible based on my observations. You can get the code via this link:

Python rocket calculation code

For example with a thrust of 6N for 1.6s I calculate a maximum height of:

Mass (Kg) Max height (Metres)
0.1 388
0.15 156
0.2 78
0.25 43

So you can see how critically important thrust to weight ratio is to maximum height.

Presumably it is possible to derive an analytic solution as well. I leave that as an exercise for the interested reader. ;0)

I think we will try a D-class motor next time (each step up the alphabet doubles the impulse). This seems to be the biggest that you can get hold of in the UK without a license. Watch out passing aircraft.

To infinity and beyond!

It’s great to be in the software products business

hard at work on my software businessThose of us who own software product businesses sometimes grumble about what a difficult business it is. Although its indoor work with no heavy lifting, it has it’s frustrations: software piracy, customers who moan about paying a whole $0.99 for thousands of hours of work, buggy third party software, RSI, chargebacks and the catastrophic consequence of accidentally offending the great god Google, to name but a few.

But reading Kitchen Confidential brought home to me just what a hard business it is to run a restaurant. You have to make a major financial outlay to fit out the restaurant and kitchen. You have rent and staff salaries to pay every month, regardless of whether customers come or not. Staff turnover is generally very high in the catering business, so you are continually having to hire new staff. You have to deal with drunken, unreasonable and dishonest customers. Possibly also drunken, unreasonable and dishonest staff, who have ready access to sharp knives and boiling liquids. Theft by staff can be a real problem. You have highly perishable stock. If you don’t order enough, you have to turn people away. If you order too much, you have to throw away the excess or risk poisoning your customers. You have endless deliveries from suppliers, which you have to check to ensure they are the correct amount and quality. You have to keep the restaurant clean. Extremely long hours are standard. Even if you are doing well, you can’t seat more people than the restaurant can physically hold. A restaurant that has to turn people away Fridays and Saturdays might be empty on Monday. And success brings its own problems as you can only increase the scale of the operation by expensive and disruptive  measures such as opening a new restaurant or moving venue. The relentless overheads of staff, rent and stock mean that cash flow is a huge issue. It’s no wonder that restaurants fail so frequently.

Running a software product business is pretty cushy by comparison. You can start your own software product business with just a PC and a generous dollop of time. Nearly all the issues related to manufacturing, suppliers, stock and shipping go away when you are dealing with electrons rather than atoms. If you do make a mistake, you can usually put it right just by making another release. The worst a disgruntled customer is likely to do is post a snarky comment on a forum or send you a nasty email. High margins and low overheads means that cash flow is much less of an issue than for most other businesses. Software businesses also scale much more easily than other businesses. You aren’t tied to a particular location and don’t even need to rent an office building (billion dollar company Automattic has a fully distributed workforce and no company office).

The software business is a great business to be in!

 

The mystery of the Chinese downloads

A spider (probably not Chinese)It’s a good to idea to regularly look through the logs of your website. You’ll often find something interesting. In March 2013 I was looking through the web logs for my seating planner software and I noticed the number of downloads of the Windows version of my software had gone up by a factor of 5, compared to the previous month. Everything else stayed pretty much the same:

  • The number of visits to the download page hardly changed.
  • The number of completed Windows installs hardly changed.
  • The number of downloads of my Mac installer hardly changed.

Odd. On further investigation it turned out that a number of Chinese IP addresses were downloading my Windows installer again and again. My software is not localised into Chinese and I get very few sales from China. Also there were no installs from these IP addresses (my software puts up a ‘thank you for trying’ page when it is first run). It was a substantial increase in bandwidth, but not enough to be a serious denial of service attack. Very odd.

I am on an unlimited bandwidth hosting contract so I wasn’t paying for the extra bandwidth. But I was worried that the volume of requests would slow down my web site. So I put a .htaccess file in the downloads directory to block the worst offenders.

After a few months I got the bandwidth from China down from ~30GB per day to ~100MB per day. I have been playing this game of ‘whack a mole’ every since. Currently I have some 1700 Chinese IP addresses blocked.

downloads per month

PerfectTablePlan for Windows downloads per month 2013/2014

As an example I recently blocked IP 211.136.10.56, which was downloading PerfectTablePlan around 20 times per day, but never visiting a page on my website.

Here are the logs from one day (via Web Log Storming), picked at random before I blocked their IP:

logsAnd here is one of those records in more detail:

logWeb Log Storming classifies it as a ‘spider’.  whois.domaintools.com says the IP belongs to ‘China Mobile Communications Corporation’. The IP is not listed on projecthoneypot.org and I wasn’t able to find out any more from casual Googling.

To block the this IP I just added this line to my .htaccess file:

Deny from 211.136.10.56

But it is a bit of a nuisance to keep having to do this.

Other software companies are having similar issues. But I haven’t come across any compelling answers about why this is happening. Perhaps it is a way of masking some other nefarious activity? Does anyone have any idea what is going on?