Tag Archives: python

Easy Data Transform progress

I have been gradually improving my data wrangling tool, Easy Data Transform, putting out 70 public releases since 2019. While the product’s emphasis is on ease of use, rather than pure performance, I have been trying to make it fast as well, so it can cope with the multi-million row datasets customers like to throw at it. To see how I was doing, I did a simple benchmark of the most recent version of Easy Data Transform (v1.37.0) against several other desktop data wrangling tools. The benchmark did a read, sort, join and write of a 1 million row CSV file. I did the benchmarking on my Windows development PC and my Mac M1 laptop.

Easy Data Transform screenshot

Here is an overview of the results:

Time by task (seconds), on Windows without Power Query (smaller is better):

data wrangling/ETL benchmark Windows

I have left Excel Power Query off this graph, as it is so slow you can hardly see the other bars when it is included!

Time by task (seconds) on Mac (smaller is better):

data wrangling/ETL benchmark M1 Mac

Memory usage (MB), Windows vs Mac (smaller is better):

data wrangling/ETL benchmark memory Windows vs Mac

So Easy Data Transform is nearly as fast as it’s nearest competitor, Knime, on Windows and a fair bit faster on an M1 Mac. It is also uses a lot less memory than Knime. However we have got some way to go to catch up with the Pandas library for Python and the data.table package for R, when it comes to raw performance. Hopefully I can get nearer to their performance in time. I was forbidden from including benchmarks for Tableau Prep and Alteryx by their licensing terms, which seems unnecessarily restrictive.

Looking at just the Easy Data Transform results, it is interesting to notice that a newish Macbook Air M1 laptop is significantly faster than a desktop AMD Ryzen 7 desktop PC from a few years ago.

Windows vs Mac M1 benchmark

See the full comparison:

Comparison of data wrangling/ETL tools : R, Pandas, Knime, Power Query, Tableau Prep, Alteryx and Easy Data Transform, with benchmarks

Got some data to clean, merge, reshape or analyze? Why not download a free trial of Easy Data Transform ? No sign up required.

Battlesnake

I have been doing some recreational programming at play.battlesnake.com. It is a series of online leagues where you enter a program to play competitive ‘snake’.

The rules are pretty simple:

  • eat food to grow
  • die of starvation if you go too long without eating
  • die if you collide with a wall or the body of another snake
  • die if you collide head-on with another snake that if of equal size or bigger
  • last snake standing wins the match

There are also some variants, such as ‘royale’ where hazards move in from the walls.

You can program your snake in pretty much any language and host it where you like. When a match starts your program recieves JSON data with the board state and has 500 milliseconds to return either “left”, “right”, “up” or “down” for each move. You can write something super-simple (move to the nearest food, avoid other snakes) or you can get as complex as you like (machine learning or full game tree with alpha-pruning).

I have written my snake in Python (which seems appropriate) and host it on a free replit.com account. It uses a series of heuristics to decide it’s next action. It uses flood fill to assess how much space is available and A* for path finding.

You can see my snake (‘RhinoCrocoPede’) in action below, it is the purple one:

You can also see it more clearly here.

At the time of writing RhinoCrocoPede is 132nd in the global league (out of 450) and steadily rising.

The Battlesnake documentation is good and I was able to get the starter Python/Replit snake up and running in 15 minutes or so. I then just built on that. Replit is a nice online IDE. I did have issues with the free Replit account timing out. But I fixed this with a 90 day free upgrade code I found on the Battlesnake discord. This allowed me to set my REPL as ‘always’ on and ‘boosted’. I still have a fairly long ping time to the server (which is in California). This eats into my 500 milliseconds. But the time remaining is plenty for my current heuristic approach, even in Python.

If I wanted to get really serious I would rewrite my snake in C++, use a full game tree or Monte Carlo approach and host it on a fast server near the battlesnake server (to reduce ping time). But it is just a bit of fun and I don’t think I’ll get that serious.

My son has also written his own snake, which has been useful programming experience for him.

Battlesnake is really slick and well done. If you feel like doing some recreational programming, I recommend you give it a try.

How I finally beat my son at a computer game

TL;DR: I cheated, using programming.

I play computer games with my son. But he is 14 and I am 54, so I just can’t compete on reflexes. Just yesterday he thrashed me 10-3 at the silly and fun Spelunky Deathmatch. Then he gloated about my pitiful score.

spelunky

We’ve also been writing our own games together in Python, for fun and so that I can teach him some programming. We’ve written a little jet dogfight game together. You each get a little plane that can turn left, right, accelerate or shoot. You score 2 points for shooting down your opponent and 1 point for flying over a powerup. First to 20 points wins. We are both Python novices (my day job is writing software in C++), so the program is quite hacky. Lots of globals and cut and paste. The planes are triangles and the clouds are square. But it is a fast and fun game to play.

plane-game

Predictably my son was winning most the games. Then gloating about it. However I had recently seen an article about an AI winning dogfights against a human fighter pilot. This gave me an idea. While he was asleep I modified the program so that you can press a key to toggle a cheat mode on either plane. In the cheat mode pressing the left key aims automatically at the powerup and pressing the right key aims automatically at the opponent’s plane. Suddenly I started thrashing him. He got suspicious and insisted we swap planes. Which is fine, I just toggled cheat mode on the other plane. He got even more  suspicious. I told him I had been practising. He went off to practise with an old version of the code I gave him. I then thrashed him several more times and told him I have being doing a lot of practise. ;0)

Sooner or later he will figure out what is going on. I’m not sure what the take away lesson will be. Coding is a powerful skill? Don’t trust your Father?

I offer up the code for any competitive Dad’s (or Mum’s) who feel they need a little help against their cocky offspring. See how long you get away with cheat mode. You can always toggle it off for a while when they get suspicious.

Notes about the game

You can download the game’s Python code.

The game runs from inside the Python variant of the free processing.org environment, which you can download here for Windows, Mac or Linux. You need to select Sketch>Import Library to get the Sound library. File>Open the .pyde file in processing.org and press the run button. You can adjust the szx and szy variables according to your screen size. There seems to be a bug where the sound only works for the first game after you start the IDE.

The keyboard controls are:

Player 1:

a – turn left (aim for powerup in cheat mode)

d – turn right (aim for opponent in cheat mode)

w – accelerate

s – fire

z – toggle cheat mode (off at start)

Player 2:

left cursor – turn left (aim for powerup in cheat mode)

right cursor – turn right (aim for opponent in cheat mode)

up cursor – accelerate

down cursor – fire

end – toggle cheat mode (off at start)

Currently the aim cheat aims at where the opponent’s plane is. To be a bit more sophisticated, it could aim at where it thinks the opponent’s plane will be. But the current approach turns out to be good enough, and less likely to make an opponent suspicious.

It would be interesting to write a little AI that completely controls the plane and then put it up against other people’s AIs. A future project perhaps. But processing.org isn’t an ideal environment for that.

Rocket Science

My son, my wife and I have been messing around with model rockets. They seem to be a big thing in the USA, but are a lot less common here in the UK. They are a lot of fun.

I bought the above rocket + launch pad + launch controller kit from a local model shop, with some recovery wadding and 3 class C rocket motors with igniters:

rocket kit amazon.co.uk link

rocket kit amazon.com link

The total cost was £30.

Making the rocket involved a bit of glueing and assembly, but was fairly straightforward. Then we inserted some wadding (to protect the internals from the hot gas of the rocket motor), the recovery parachute and the nose cone with rotors. When it was finished we took it to a big open space, inserted a rocket motor and igniter, put it on the launch pad and used the 9v battery operated remote control to launch it.

We had a few non-launches because the crocodile clips (connecting the launch control to the igniter) touched, causing a short-circuit, or fell off. Not a great design. Once we had sorted that out we successfully launched and the rocket went well over 100 metres in the air. Cool!

In theory the motor should burn for a couple of seconds and then a little explosive charge fires to separate the nose cone from the main body. The main body then floats down on the parachute while the nose cone deploys spring-loaded rotors and auto-rotates down. In theory.  However, in our inexperience, we put in too much wadding and packed it too tightly. Consequently the rocket blew itself apart in mid-air and the parachute and rotors didn’t deploy. We managed to recover all the bits. The parachute was ok, but the rotor blades were too damaged to use again.

A video of our first launch

So we cut off the damaged section and added the nose cone back on to make a new, shorter rocket and did 2 more launches. Being lighter with the same motor it went a lot higher. Possibly over 200 metres!

We made a new rocket from the nose cone and tail of the kit, plus a long cardboard tube and lots of duct tape. We did another 3 launches using C class rocket motors. Even managing to get one successful parachute deployment. However as the new rocket was  heavier it got noticeablely less height, probably less than 100 metres.

A few things we learnt along the way:

  • Don’t force the parachute and nose cone in too hard or use too much wadding.
  • If the parachute doesn’t deploy the rocket can survive hitting the ground at speed surprisingly well. But they make quite a hole in the ground, so you REALLY don’t want to get in the way.
  • Even in light wind the rockets can land a fair distance away. Especially if the parachute deploys successfully. So pick a still day for the launch. You can also cut some extra vents in the parachute to make it fall faster.
  • You need a BIG open space, free from other people, animals and trees. Preferably at least 200 metres across, if you want to stand a good chance of recovering your rocket for another launch.
  • The maximum height of your rocket depends critically on the thrust to weight ratio.

Hopefully it goes without saying that pyrotechnics and objects travelling at high speed are potentially dangerous and require common sense and adult supervision.

Being a software geek with a physics background I couldn’t resist doing a few calculations. Here is a little Python script I wrote to calculate the maximum height and flight time based on the mass of the rocket and the thrust and duration of the motor. It applies a simple time-step approach to F=ma. Just modify the mass, thrust and duration variables.

rocket science codeIt assumes the rocket goes straight up and doesn’t allow for air resistance. But the values it calculates seem fairly plausible based on my observations. You can get the code via this link:

Python rocket calculation code

For example with a thrust of 6N for 1.6s I calculate a maximum height of:

Mass (Kg) Max height (Metres)
0.1 388
0.15 156
0.2 78
0.25 43

So you can see how critically important thrust to weight ratio is to maximum height.

Presumably it is possible to derive an analytic solution as well. I leave that as an exercise for the interested reader. ;0)

I think we will try a D-class motor next time (each step up the alphabet doubles the impulse). This seems to be the biggest that you can get hold of in the UK without a license. Watch out passing aircraft.

To infinity and beyond!

** Update 23-Apr-2023 **

One thing led to another and we are now launching rockets with F motors to over 1000 feet and taking part in UK national rocketry competition (placed 9th last year). My son is considering doing a degree in aerospace and astronautic engineering.

Correction: The biggest motor you can buy in the UK without certification is a G.