Easy Data Transform and Hyper Plan Professional edition are both on sale for 25% off at Summerfest 2022. So now might be a good time to give them a try (both have free trials). There is also some other great products from other small vendors on sale, including Tinderbox, Scrivener and Devonthink. Some of the software is Mac only, but Easy Data Transform and Hyper Plan are available for both Mac and Windows (one license covers both). Sale ends 12th July.
Tabular data is everywhere. I support reading and writing tabular data in various formats in all 3 of my software application. It is an important part of my data transformation software. But all the tabular data formats suck. There doesn’t seem to be anything that is reasonably space efficient, simple and quick to parse and text based (not binary) so you can view and edit it with a standard editor.
Most tabular data currently gets exchanged as: CSV, Tab separated, XML, JSON or Excel. And they are all highly sub-optimal for the job.
CSV is a mess. One quote in the wrong place and the file is invalid. It is difficult to parse efficiently using multiple cores, due to the quoting (you can’t start parsing from part way through a file). Different quoting schemes are in use. You don’t know what encoding it is in. Use of separators and line endings are inconsistent (sometimes comma, sometimes semicolon). Writing a parser to handle all the different dialects is not at all trivial. Microsoft Excel and Apple Numbers don’t even agree on how to interpret some edge cases for CSV.
Tab separated is a bit better than CSV. But can’t store tabs and still has issues with line endings, encodings etc.
XML and JSON are tree structures and not suitable for efficiently storing tabular data (plus other issues).
There is Parquet. It is very efficient with it’s columnar storage and compression. But it is binary, so can’t be viewed or edited with standard tools, which is a pain.
Don’t even get me started on Excel’s proprietary, ghastly binary format.
Why can’t we have a format where:
- Encoding is always UTF-8
- Values stored in row major order (row 1, row2 etc)
- Columns are separated by \u001F (ASCII unit separator)
- Rows are separated by \u001E (ASCII record separator)
- Er, that’s the entire specification.
No escaping. If you want to put \u001F or \u001E in your data – tough you can’t. Use a different format.
It would be reasonably compact, efficient to parse and easy to manually edit (Notepad++ shows the unit separator as a ‘US’ symbol). You could write a fast parser for it in minutes. Typing \u001F or \u001E in some editors might be a faff, but it is hardly a showstopper.
It could be called something like “unicode separated value” (hat tip to @fakeunicode on Twitter for the name) or “unit separated value” with file extension .usv. Maybe a different extension could used when values are stored in column major order (column1, column 2 etc).
Is there nothing like this already? Maybe there is and I just haven’t heard of it. If not, shouldn’t there be?
And yes I am aware of the relevant XKCD cartoon ( https://xkcd.com/927/ ).
** Edit 4-May-2022 **
It has been pointed at the above will give you a single line of text in an editor, which is not great for human readability. A quick fix for this would be to make the record delimiter a \u001E character followed by an LF character. Any LF that comes immediately after an \u001E would be ignored when parsing. Any LF not immediately after an \u001E is part of the data. I don’t know about other editors, but it is easy to view and edit in Notepad++.
This is a guest post from serial software entrepreneur Dennis Gurock.
Thinking about product positioning (and matching branding) is especially important if you build a product for a crowded market with many established competitors (and there are many reasons why this can be a good idea). We were in exactly this situation when we initially thought about building and marketing our new test management tool.
Positioning will allow you to better focus on a specific market segment to target, it makes it easier to build a clearer and stronger message to reach customers, and it helps develop the initial product vision and feature set.
What does successful positioning mean for software products? It can mean identifying a unique angle to focus on so you can stand out with your product among other products and competitors. Especially if you are entering a crowded market, this allows you to better communicate the key benefits and features you have to offer. It will help you reach the right customers and ensures that customers remember you when they look for a new product to try.
To come up with positioning for your new product, you can focus on a specific customer segment or niche that you think will be easier to market to or that you think is underserved by existing offerings. It can also help you limit the initial product scope, so you can go to market faster. Then rigorously optimizing for this initial customer segment allows you to establish a market presence and expand to other segments more easily later.
Why is positioning useful?
There are many benefits of coming up with and deciding on positioning for your new software product early on. Once you decide on the positioning, many marketing, product management and sales decisions become more straightforward.
- Clear message & benefits: it is not easy to stand out in a crowded market. Positioning allows you to come up with clear messaging so you can explain and highlight unique selling points in few words.
- Target and identify niche/marketing opportunities: it can be difficult to decide which marketing options to try, which campaigns to book and which niches to target. Focusing on a specific market segment based on the product positioning can be a great way to identify matching niches and opportunities.
- Identify customer fit during sales: one of the most important aspects of the sales process is identifying and ensuring prospects are actually a great fit for your product. It’s wasted time for both you and for your prospects to invest a lot of effort evaluating and piloting a product if they will not benefit from it. Positioning can help you quickly filter and identify which customers to focus on.
- Better focus on initial product vision: there are a lot of directions to choose when building a new product. If you don’t have a clear vision to guide you, it is easy to be distracted by different directions and work on too many things at the same time. Clear positioning makes it easy to focus product management on specific goals and use cases.
- Easier to choose features: when you start working with customers, you will (hopefully) receive a lot of feedback on features you should add. Positioning helps you decide which of these features you should actually implement. Often times the most successful products are developed by following strong opinions and saying ‘No’ to many requests.
Examples of software product positioning
Let’s look at a few examples of companies that use positioning to market and build their products. All these examples are from industries and product categories with many existing competitors and products.
- Testmo: we entered a crowded market with many established testing tools when we developed our new product. Most existing offerings either focus on manual testing, or they offer a complete ALM toolset to handle the entire development lifecycle. With Testmo we had other ideas and wanted to position it differently, focusing on unified testing. This means we combine test cases, automation and exploratory testing in a single platform. At the same time it allows us to limit the scope of the product. We won’t add our own issue tracking, or CI pipelines, or existing DevOps features. Instead of we focus on integrating with other tools customers already use.
- Another example is the documentation and wiki product GitBook. They heavily focus on software developers and position themselves as the primary tool for developers to publish user docs and to document internal knowledge. With this positioning in mind, they can focus on features that primarily make sense for developers, such as Git synchronization, Markdown support and code snippets. It also allows them to more easily market directly to software developers with a clear message.
- Then there’s the application monitoring service Checkly. There are many services and products that enable you to monitor apps and sites for downtime and notify you about issues. Checkly positions itself as a tool that enables end-to-end monitoring with flexible scripting. So it doesn’t just make simple web requests to see if a site is still live. It allows customers to write custom scripts to implement complex user flows and thereby not just check if a site is reachable, but also test the entire stack with the front-end, database, authentication and much more. This focus allows them to build more targeted features for advanced use cases and thereby provides more value to customers compared to simpler competitors.
- The popular email marketing service Campaign Monitor also started with very focused positioning. In the first few years they concentrated on providing the best possible campaign tool for web designers and design agencies. This focus allowed them to invest more in features designers needed, such as white labeling, reusable themes and live email previews. Once they established their market presence, they started to expand their customer base to capture a larger part of the overall market for newsletter tools.
These are just some examples of companies and products that have benefited from clear positioning. Of course there are also countless of examples of companies choosing not to have such clear positioning. There is nothing wrong with this and you can certainly be very successful even if you ignore these points. But more often than not positioning is a useful tool to improve focus on specific goals and customer needs, which increases your chance to build a successful software business.
Dennis Gurock is one of the founders of Testmo, a QA testing tool that unifies test case management with exploratory testing and test automation in one platform. He has been working on products that help teams improve software quality for more than 15 years.
If you want to find out how to do something, such as do a mail merge in Word or fix a leaky valve on a radiator, where do you look first? Probably Youtube. Videos are an excellent way to explain something. More bandwidth than text and more scaleable than a 1-to-1 demo.
I’ve done explainer videos for all 3 of my products. But I found it a real struggle. I would write a script and then try to read the script and do the screencast at the same time and do it all in one take. I would stutter and stumble and it would take multiple attempts. It took ages and results were passable at best. I got some better software to edit the stumbles out, so I didn’t have to do it in one go. But it still took me a fair few attempts and quite a bit of editing. It became one of my least favourite things to do and so I did less and less of it.
Recently, I came across these slides on video by Christian Genco. These and subsequent Twitter exchanges with Christian convinced me that I should stop being a perfectionist about video and just start cranking them out on the grounds that a ‘good enough’ video is better than no video at all (‘the perfect is the enemy of the good’) and I would get better at it over time. As Stalin supposedly said “Quantity has a quality all of it’s own”.
So I have ditched the scripts and the perfectionism and I’ve managed to create 13 short Easy Data Transform explainer videos in the last week or so. And I am getting faster at it and (hopefully) a bit more polished. I’m definitely not an expert on this (and probably never will be) but here are some tips I have picked up along the way:
- Get some decent software. I use Camtasia on Windows and it seems pretty good.
- Try to talk slower.
- Try to sound upbeat (not easy if you are British and could voice double for Eeyore).
- Try not to move the mouse and talk at the same time. This makes editing a lot easier. Some people like to do the audio and the visual separately, but that seems like too much hassle.
- If you stumble, just take a deep breath, say it again and then edit the stumble out later.
- Get a reasonable mic. I have a snowball mic on a cantilevered stand. I covered it with a thin cloth to try to reduce pops.
- The occasional ‘um’ is fine.
- Have a checklist of things to do for each video, so you don’t forget anything (such as disabling your phrase expander software or muting the phone).
I’m lucky to have a very quiet office, so I don’t have much background noise to contend with.
Using Camtasia I can easily add intos and outros, edit out stumbles and add various effects, such a mouse position highlighting and movement smoothing. I just File>Save as the previous project so that I don’t have to re-add the intro and outro. Unsurprisingly, Camtasia have lots of explainer videos. I wish there was a way to automatically ‘ripple delete’ any sections where there is no audio and no mouse movement (if there is, I haven’t found it). Some people recommend descript.com. It looks interesting, but I haven’t tried it.
I did an A/B test of recordings with my Senheiser headset mic against my Snowball mic and the consensus was that the headset was ok but the the Snowball mic sound quality was better.
Some people prefer to use synthetic voices, instead of their own voice. While these synthetic voices have improved a lot, they never sound quite right to me. Also it must be time consuming to type out all the text. Or you can pay to have a professional voiceover done, but this is surprisingly expensive (around $100 per minute, last time I checked) and almost certainly more time consuming than doing it yourself.
Some people aren’t confident about speaking on videos because they are not native speakers of that language and have an accent. Personally accents don’t bother me at all. In fact I like hearing English spoken with a foreign accent, as long as I can understand it. Also I think there is an authenticity to hearing a creator talk about their product in their own voice.
I’m not a big fan of music on explainer videos, so I don’t add any.
I let Youtube generate automatic captions for people that want them (which could be people in busy offices and on trains and planes, as well as the hearing impaired). They aren’t perfect, but they are good enough.
My videos are aimed at least as much at finding new users as helping existing users. So I make sure I research keyword terms (mostly in Google Adwords) before I decide which videos to make and what to title them. Currently I am targetting very specific keyword searches, such as How to convert CSV to Markdown. Easy Data Transform can do a lot more than just format conversion, but from an SEO point of view it is better to target the phrases that people are actually searching for.
I upload the videos as 1080P (1920 x 1080 pixels) on to the Easy Data Transform Youtube channel and onto my screencast.com account (which I pay a yearly fee for). I then embed the screencast.com videos on relevant easydatatransform.com pages using IFRAME embed codes created by screencast.com. I don’t use the Youtube videos on my website, because I don’t want people to be distracted by Youtube ads and ‘you may also like’ recommendations. They might be showing a competitor! I don’t host the videos on the website itsself as I worry that might slow down the website. I also link to the videos in screencast.com from my help documentation, as appropriate.
Some people like to embed video of themselves in screencasts, in the hope of making it more engaging. But personally I want people to concentrate on my software, rather than being distracted by the horror of my face. And not having to comb my hair or look smart was part of what got me into running my own software business.
In the next few months I will be checking my analytics to see how many views these videos get and whether they increase the time on page and reduce the bounce rate.
If you can spare a few seconds to go to my Youtube page and ‘like’ a video ot two or subscribe, that would be a big help!
Note that some of the above doesn’t apply when you are creating a demo video for your home page, rather than an explainer video. Your main demo video should be slick and polished.
This is a guest post from fellow software developer, Simon Kravis.
Few developers would choose their development platform on the merits of their respective Integrated Development Environments (IDEs) but it happens that applications developed in Windows need to be made available on the Mac platform.
There are many environments offering cross-platform (Mac, Windows and sometimes Android) functionality, but close inspection shows that they all have limitations. Visual Studio (the native Windows IDE) can produce apps which will run on a Mac using .Net Core – but only if they are command line apps on Windows. Other environments (like Xamarin) do support interfaces, but only involving simple controls like text boxes or drop-downs. There are other cross-platform IDEs (such as Qt) which offer better graphics support, but they are not cheap and the extent of their support is not evident. If you need functionality such as computer vision, there seems to be no alternative to creating a separate code base for the Mac. Once you start on this path it becomes obvious that Macs handle graphics (and interfaces) very differently from Windows.
Macs have evolved rather more than PCs over the decades: they abandoned their proprietary Mac operating system in favour of UNIX in 1999, adopting the NeXTSTEP platform created by NeXT. Apple originally used PowerPC chips, replacing them with Intel Core processors in 2006, and they are currently transitioning to RISC chips. The Mac NeXTSTEP programming language was Objective C, developed in the 1980s and this is still supported, although the modern Swift language was introduced in 2014, and the Xcode IDE appeared in 2003. Xcode is free, even for teams. It uses the Cocoa API, which is accessible from other environments. The current release (MacOS 13.0) supports both Objective-C and Swift and is also used for developing iPhone and iPad apps. Mac operating systems since Catalina (released in 2019) are 64-bit only. Xcode can only develop apps for Apple operating systems, notably iOS, which powers the iPhone. Most of the web questions and examples relate to iOS rather than MacOS. MacOS uses different frameworks from iOS, so some functions used in iOS are not available in MacOS, or have different parameters.
The Windows IDE (Visual Studio) dates from 1997, when it bundled together Visual Basic, Visual Fox Pro and Visual Source Safe and Visual C++. It has an open architecture based on plug-ins and supports 36 different programming languages, but the major ones are C#, VB.Net and C++. Visual Studio can develop apps for any platform via the .NetCore framework, but capability for non-Windows platforms is limited. The Community edition is free, and has almost all the functionality of paid versions.
Both Visual Studio and Xcode are highly complex applications. They both have graphical interface builders where controls are dragged from a library onto a form. Each application has a vocal supporters and detractors. My experience comes from about 5 years with Visual Studio developing C# applications. Before this I worked with Visual Basic for Applications in Microsoft Access, so I am well-versed in the Microsoft way of doing things.
Like most complex applications, Visual Studio and Xcode each have plenty of bugs, often producing completely unhelpful error messages. Reporting an Xcode bug through standard channels resulted in … nothing. Not even an automated message saying “Thank you for feedback. It will be used to improve future versions”. I haven’t even tried to report a Visual Studio bug, but I suspect that the much larger user base for Visual Studio will mean that workarounds are more readily available, even if the giant ship of Microsoft takes years to respond.
Moving to the Mac and Xcode for development was a shock as I found I didn’t know how to do the most basic things. String manipulation (used in most applications) in Objective C is highly verbose compared to C#. Google was invaluable for finding answers – mostly they were from Stack Overflow, but often from 10 or more years ago, sometimes from Apple Developer Forums. As Xcode has changed considerably since then, answers often had to be adjusted before they could be used. Another problem is that functionality once provided externally has since been incorporated into Cocoa, so attempts to find a current version of a component (or framework as they called in Cocoa) are often unsuccessful.
MacOS provides more native functionality than Windows. Features such as computer vision and PDF generation are included in MacOS, rather than requiring the use of 3rd party components, which may not as robust as desired, and may require a license for commercial use. However, documentation of MacOS functionality, if present at all, was rarely useful. A few times I asked questions on Stack Overflow which attracted the ire of the Mac gurus for either through having obvious (to them) answers or through not conforming to the forum guidelines (in their opinion). However, the integration of NuGet with Visual Studio provides easy access to the massive number of 3rd party libraries available for .Net on Windows.
The model-view-controller paradigm used on the Mac took some getting used to, as did the design of the main Xcode screen. Sometimes a useful display would disappear and I had difficulty in finding it how to bring it back. I often had to resort to retrieving earlier versions from the excellent Time Machine backup. Form design is similar on both platforms – dragging and dropping components from a library. Both Xcode and Visual Studio have bugs, as would be expected for such complex apps. Events from components are generated automatically in Windows, but have to be defined on the Mac (as Actions). References to the component you’ve added also need to be defined on the Mac (as Outlets) and are not a property of the component, whereas on Windows they are.
The Xcode environment provides only basic facilities from scratch: if you need to do something more sophisticated you’ll have to Google around to find out how. Once you know – it’s easy, but the learning curve for Xcode is much higher than for Visual Studio.
Rather than starting from scratch with the Mac version of my Caption Pro app, which uses local computer vision functionality to detect multiple photos, changes image dimensions and adds text to images, I found an existing open-source project on GitHub with similar basic functionality. This dated from 7 years ago and used Objective-C, so that was the language I opted for. An immediate handicap was that many of the answers I found to my questions used Swift in their example code, which is not interconvertible with Objective-C in the way that C# and VB.Net are. iOS applications for the iPhone (which are most common) use different frameworks from Mac apps, and routines in them sometimes have completely different syntax.
The user interfaces for the Mac and Windows versions look quite different, as shown below. There are some basic differences – menus appear separately to the application window on the Mac and are locked to the top of the screen, whereas Windows menus are part of the application screen. Toolbars offer access to common functionality on the Mac. Differences also arise from the fact the Mac application was adapted from existing code rather than created from scratch.
Open-source examples (often from GitHub) are useful, but rarely work out-of-the- box. Sometimes the modifications need are minor – like defining the development team- but sometimes it’s not possible to get them to build in a current version of Xcode.
Debugging on Xcode is frustrating – the call stack frequently contains assembler (which is perhaps why app performance tends to be better on Macs), and the debug variables window does not list all relevant variable values. Variable types may not be correct – Boolean values may appear as dates, and sometimes variables cannot even be evaluated by po (print out) statements. Printing out structure variables may show nothing. Despite the generally superior performance of Mac apps, building apps in Xcode appears to be much slower than in Visual Studio on similar vintage machines, and after code stops at a breakpoint, it may take a long time before the variables window is filled. Deployment of Mac apps can still be done on an ad-hoc basis, but you have to register as an Apple Developer to avoid blockages in installation arising from being an ‘untrusted source’. Bypassing these blockages is more than a matter of clicking “Install anyway” so it’s hard to avoid forking out US$100 per year for registration. Windows has similar blockages, which can be bypassed with a code-signing certificate. These certificates are available from many vendors, and are slightly cheaper than Apple developer registration, but the process of obtaining one may be very involved.
Ad-hoc deployment is somewhat easier on the Mac than on Windows, but the method of doing it via Archive generation is anything but obvious. Mac applications are actually disk images and applications keep all of relevant files in a folder. This makes uninstallation a matter of dragging the application icon into the recycle bin, a far simpler process than on Windows. dmg files are not recognized by IIS web servers (and may not be by Apache either), so unless the file type is registered, download from a web site will not be possible.
Apple pioneered the App Store for iPhones (it is the only way in which iPhone apps can be installed) and Mac apps can also be put there. Apple takes a commission of 30% (or 15% if you are a small company) and they review all apps before adding them. Passing the review process may be a lngthy process, as not all problems are detected in a review cycle. Fixing these issues and resubmitting may result in further problems coming to light. The review process may also be somewhat arbitrary. One App Store app presented an interface in German by default. English was available as Preferences option, but only after guessing where the Preferences option was located. App Store apps operate within a sandbox, which places restrictions on filesystem operations. Whether App Store deployment makes economic sense depends on the nature of the app, its market and price structure. Its advantages are that it targets the 16% of desktop users who use Macs, and streamlines installation (and payment, if applicable). The App Store supports ‘freemium’ pricing, where additional features are made available to paying users, but apps with free trial periods are shown as being free but with ‘in-app purchases’, which annoys some users.
Windows deployment can use .msi files, which have been around for decades, but are not easily installed by non-admin users. Self-extracting executables are more tractable, but 3rd party tools have to be used to create them. Windows 10 introduced Universal Windows Programs, which are easier to install and can be placed in the Microsoft Store, which operates in a similar way to the Apple App store, but for Windows desktops and tablets.
A key question which is very difficult to answer is “How long will it take me to convert my Windows app to run on a Mac?” Factors affecting this are app complexity, functionality and programmer skill. The time between starting work on the Mac app and first deploying it on the company web site was about 3 months, but the amount of time spent on the project each day varied between zero and 3 or 4 hours. If you are a paid resource, then the cost of a cross-platform IDE may be justified, but the requirement for local computer vision functionality added a great deal of complexity to my requirements, which is one reason why I opted for a separate code base. Substantial evaluation would be required before deciding if a cross-platform environment could support any required functionality.
Simon Kravis runs Aleka Consulting, a small software and consultancy company in Canberra, Australia specializing in information management and offering a number of software products. He has mainly developed scientific and engineering programs, starting in the era of paper tape.
I started selling software online 16 years ago. Until this year I never had a forum for any of my products. I handled customer support for PerfectTablePlan and Hyper Plan by email and kept customers up-to-date with an opt-in email newsletter. But I rethought this position with my latest product, Easy Data Transform and started a forum at forum.easydatatransform.com in December 2020.
My ISP offered various forum software packages, but I really wanted Discourse, as I consider it head and shoulders above all the other forum software I have interacted with as a user (even if I find the badge system a bit patronising). I didn’t want the hassle of setting up and patching a Discourse server, so created the forum through www.communiteq.com (previously discoursehosting.com). It was suprisingly easy to set-up. And it gives the option to export everything, in case I want to part ways with them. The sheer number of options in Discourse are quite daunting, but I stuck with the defaults for the most part.
Some people use Facebook Groups for their product forums. Ugh. You have almost no control of such a forum. Facebook could even be showing ads for your competitors on your forums. Or they could just decide to shut you down and delete all the content. That is before we get on to the fact that Facebook make their money monetising hatred and abusing our privacy at an industrial scale. No thanks.
The advantages of a forum are:
- Letting customers talk to each other, and post content helps to create a community around the product. Which, in turn, can add a lot of value to your product.
- Customers can help each other with support questions. Sometimes they will answer before you are able to or will give a different perspective. Or even give a better answer.
- If a customer asks a question that has already been asked, you can send them a link to the appropriate forum page.
- It is a quick and easy channel to communicate with customers. I can post a link to a new snapshot release in a few minutes. This is much quicker than sending out an email newsletter. It is also more interactive as customers can respond on the forum and see each other’s responses.
- A lively forum is ‘social proof’ that your product is worth buying.
- A forum with lots of content should have a large SEO footprint.
The disadvantages of a forum are:
- The time to maintain it. A forum that is broken or full of spam and unanswered questions is worse than no forum.
- Disgruntled customers potentially airing their grievances in public.
- The cost of the forum.
- An empty forum looks bad.
- Bad actors can be a pain. For example, people posting links to spam or competing products.
It probably only takes me 1-2 hours per week to post on the forum at present. Some of that is time I would have spent answering support emails. If that rises substantially then I may have to delegate it.
I try very hard to provide a good product, with good support and haven’t had any issues with negativity, so far. But I know from my experiences moderating Joel Spolsky’s Business of Software forum that moderating a busy forum can be tricky, time-consuming and emotionally draining.
The cost of the forum is currently around $20 per month, so pretty low. That may climb, but hopefully sales will be climbing as well.
I was a bit worried about whether the forum was going to look empty. I warned customers that the forum was an experiment and would be closed if there wasn’t enough activity, to manage their expectations. I also created a ‘sock puppet’ account and ‘seeded’ the forum with a few support questions that I had been previously asked by email (with the permission of those that asked) and then posted answers. But I only did this a handful of times and then the forum started to take off.
I have heard stories of people getting 1000+ spam posts a day on their forum. But I haven’t had any issues with bad actors, so far. I’m not sure how much of that is down to Discourse and how much of it is down to luck. But, no doubt issues will occur at some point.
I still have my product newsletter, which I send out every few weeks when there is a new production release.
Overall I am pretty happy with how the forum is going. Should you have a forum for your product? As always, it depends. I think you should consider it if:
- Your customer base isn’t tiny.
- You want to interact with your customers and get feedback. This might be less the case with mature products.
- You have the time and energy to police and maintain it.
- Your product is relatively open ended or complex. For example, if your product just checks whether website are up or down, there is probably a very limited amount you can discuss.
I wondered what it would look like if you took a body of text and then used it to generate new text, using Markov chains of different lengths. So I knocked up quick program to try it. ‘Bloviate’.
Bloviate analyses your source text to find every sequence of N characters and then works out the frequency of characters that come next.
For example, if you set N=3 and your source text contains the following character sequences staring with ‘the’:
‘the ‘, ‘then’, ‘they’, ‘the ‘
Then ‘the’ should be followed 50% of the time by a space, 25% of the time by an ‘n’ and 25% of the time by a ‘y’.
Bloviate then creates output text, starting with the first N characters of the source text and filling in the rest randomly using the same sequence frequencies as the source text.
Note that a character is a character to Bloviate. It treats upper and lower case as different characters, makes no attempt to differentiate between letters, punctuation and white space and does not attempt to clean up the source text. Which also means it works on any language.
Bloviate also tells you the average number of different characters following each unique sequence of N, which I will call F here. As F approaches 1.0 the output text becomes closer and closer to the input text.
Using ‘Goldilocks and the 3 bears’ as input:
If N=1 (F=7.05) the output is garbage. Albeit garbage with the same character pair frequency as the original.
On cre She sl s ramy raked cheais Bus ore than s sherd up m. ged. bend staireomest p!”Sof ckstirigrorr a ry ps.
” f waine tind s aso Sowa t antthee aime bupis stht stooomed pie k is beche p!
At N=3 (F=1.44) it looks close to English, but jibberish:
Once up and been sight,” she this timed. Pretty so soon, she second soft. She screame up and she screame hot!” cried the Mama bed the Papa been sleeping in the Papa bear
“Someone’s bear growl.
At N=5 (F=1.14) it starts to look like proper English, but semantically weird:
Once upon a time, so she went for a walked right,” she lay down into the kitchen, Goldilocks sat in the porridge from the three chair,” growled, “Someone’s been sitting my porridge and she tasted the door, and ran down the bedroom. Goldilocks woke up and she second bowl.
And it comes out with occasional gems such as:
“Someone’s been sitting my porridge,” said the bedroom.
At N=10 (F=1.03) it starts to become reasonably coherent:
Once upon a time, there was a little tired. So, she walked into the forest. Pretty soon, she came upon a house. She knocked and, when no one answered, she walked right in.
At the table in the kitchen, there were three bowls of porridge.
At N=15 (F=1.01) it starts to get pretty close to the original text, but doesn’t follow quite the same order:
Once upon a time, there was a little girl named Goldilocks. She went for a walk in the forest. Pretty soon, she came upon a house. She knocked and, when no one answered, she walked right in.
At the table in the kitchen, there were three bowls of porridge. Goldilocks was very tired by this time, so she went upstairs to the bedroom. She lay down in the first bed, but it was too hard. Then she lay down in the third bed and it was just right. Goldilocks fell asleep.
At N=12 (F=1.07) the whole 680k characters of ‘Pride and prejudice’ produces:
It is a truth universally contradict it. Besides, there was a motive within her of goodwill which could not help saying:
“Oh, that my dear mother had more command over herself! She can have her own way.”
As she spoke she observed him looking at her earnest desire for their folly or their vice. He was fond of them.”
Obviously the source text is important. The Bohemian Rhapsody lyrics make nearly as much (or as little sense) at N=5 (F=1.08) as the original:
Is this to me, for me, to me
Mama, just a poor boy from this to me
Any way the truth
Mama, life? Is this time tomorrow
Carry on as if nothing all behind and face the truth
Mama, ooh, didn’t mean to me, baby!
Just gotta leave me and lightning, very fright out, just killed a man
Put a gun against his head
Pulled my time to die?
At N=12 (F=1.05) 160k characters of Trump election speeches produces:
Hillary brought death and disaster to Iraq, Syria and Libya, she empowered Iran, and she unleashed ISIS. Now she wants to raise your taxes very substantially. Highest taxed nation in the world is a tenant of mine in Manhattan, so many great people. These are people that have been stolen, stolen by either very stupid politicians ask me the question, how are you going to get rid of all the emails?” “Yes, ma’am, they’re gonna stay in this country blind. My contract with the American voter begins with a plan to end government that will not protect its people is a government corruption at the State Department of Justice is trying as hard as they can to protect religious liberty;
Supply your own joke.
I knocked together Bloviate in C++/Qt in a couple of hours, so it is far from commercial quality. But it is fairly robust, runs on Windows and Mac and can rewrite the whole of ‘Pride and prejudice’ in a few seconds. The core of Bloviate is just a map of the frequency of characters mapped to the character sequence they follow:
QMap< QString, QMap< QChar, int > >
You can get the Windows binaries here (~8MB, should work from Windows 7 onwards).
You can get the Mac binaries here (~11MB, should work from macOS 10.12 onwards).
Note that the Bloviate executable is tiny compared to the Qt library files. I could have tried to reduce the size of the downloads, but I didn’t.
To use Bloviate just:
- paste your source text in the left pane
- set the sequence length
- press the ‘Go >’ button
I included some source text files in the downloads.
You can get the source for Bloviate here (~1MB).
If a film director is only as good as their last film, then I guess a software developer is only as good as their last software release. In more than 30 years of writing software professionally I have shipped my fair share of releases. For the last 13 years I have been shipping software as a solo developer. Here are a few things I have learned along the way. Some of them are specific to downloadable software, but some of them apply equally to SaaS products.
Use a version control system
I occasionally hear about software developers who don’t use a version control system. Instead they usually create some sort of janky system using dated copies of source folders or zip files. This send shivers down my spine. Don’t be that guy. A version control system should be an essential part of every professional software developer’s tool kit. It matters less which version control system you use. All the cool kids now use distributed version control systems, such as git. But I find that Subversion is fine for my requirements.
Tag each release in version control
This makes it easy to go back and compare any two releases. A bug appeared in the printing between v1.1.1 and v1.1.2? Go back and diff the source files related to printing and review all the changes.
Store your release binaries in version control
I store every binary I ship to customers in my version control system. Many people will tell you that you should only store the source in version control. Then you can use this to regenerate the binaries if you need to. This was sound advice back in the day when harddisks were small, networks were slow, version control systems were clunky (SourceSafe!) and developer environments didn’t change very frequently. But I don’t think it is valid advice now. Harddisks are as cheap as chips, networks are much faster and online updates mean your SDK, compiler or some other element of your toolchain is likely to be updated between your releases, making it impossible for you to recreate an identical release binary later on.
‘Test what you ship, ship what you test’
In an analog system (such as a bridge) a tiny change in the system will usually only cause a small change in the behaviour. In a discrete system (such as software) a change to a single bit can make the difference between a solid release and a showstopper bug. The Mariner I rocket was destroyed by a single missing hyphen in the code. So test the binaries that you plan to ship to the customer. And if you change a single bit in the release, re-test it. You probably don’t need to run all the tests again. But you certainly can’t assume that a small change won’t cause a big problem.
This issue often manifests itself when the developers test the debug version of their executable and then ship the release version. They then find that the two have different behaviour, e.g. due to a compiler optimization, different memory layout or code inside an ASSERT.
Make each executable individually identifiable
As a corollary of the above, you need to be able to uniquely identify each executable. I do this by having a timestamp visible in the ‘About’ box (you can use __DATE__ and __TIME__ macros in C++ for this) and ensure that I rebuild this source file for every release.
Diff your release with the previous one
Do a quick diff of your new release files versus the previous ones. Have any of the files changed unexpectedly? Are any files missing?
Be more cautious as you get nearer the release
Try not to make major changes to your code or toolchain near a release. It is too risky and it means lots of extra testing. Sometimes it is better to ship a release with a minor bug than fix it near the release and risk causing a much worse problem that might not get detected in testing.
Test your release on a clean machine
Most of us have probably sent out a release that didn’t work on a customer’s machine due to a missing dynamic library. Oops. Make sure you test your release on a non-development machine. VMs are useful for this. Don’t expect customers to be very impressed when you tell them ‘It works on my machine‘.
Test on a representative range of platforms
At least run a smoke test on the oldest and most recent version of each operating system you support.
Automate the testing where you can
Use unit tests and test harnesses to automate testing where practical. For example I can build a command line version of the seating optimization engine for my table plan software and run it on hundreds of sample seating plans overnight, to test changes haven’t broken anything.
If you set up a continuous integration server you can build a release and test it daily or even every commit. You can then quickly spot issues as soon as they appear. This makes bug fixing a lot easier than trying to work out what went wrong weeks down the line.
But still do manual testing
Automated test won’t pick up everything, especially with graphical user interface issues. So you still need to do manual testing. I find it is very useful to see real-time path coverage data during testing, for which I use Coverage Validator.
Use third parties
You can’t properly test your own software or proof read your own documentation any more than you can tickle yourself. So try to get other people involved. I have found that it is sometimes useful to pay testing companies to do additional testing. But I always do this in addition to (not instead of) my own testing.
Your documentation is an important part of the release. So make sure you get it proof read by someone different to the person that wrote it.
Use your customers
Even two computers with the same hardware specifications and operating system can be set up with an almost infinite range of user options (e.g. screen resolutions, mouse and language settings) and third party software (e.g. anti-virus). Getting customers involved in beta testing means you can cover a much wider range of setups.
When I am putting out a major new release I invite customers to join a beta mailing list and email them each time there is a new version they can test. In the past I have offered free upgrades to the customers who found the most bugs.
Don’t rely only on testing
I believe in a defence in depth approach to QA. Testing is just one element.
Automate the release process as much as you can
Typically a release process involves quite a few steps: building the executable, copying files, building the installer, adding a digital signature etc. Write a script to automate as much of this as possible. This saves time and reduces the likelihood of errors.
Use a checklist for everything else
There are typically lots of tasks that can’t be automated, such as writing release notes, updating the online FAQ, writing a newsletter etc. Create a comprehensive checklist that covers all these tasks and go through it every release. Whenever you make a mistake, add an item to the checklist to catch it next time. Here is a delightfully meta checklist for checklists.
Write release notes
Customers are entitled to know what changes are in a release before they decide whether to install it. So write some release notes describing the changes. Use screen captures and/or videos, where appropriate, to break up the text. Release notes can also be very useful for yourself later on.
Email customers whose issues you have fixed
Whenever I record a customer bug report or a feature request, I also record the email of the customer. I then email them when there is a release with a fix. It seems only polite when they have taken the effort to contact me. But it also encourages them to report bugs and suggest features in future. I will also let them access the release before I make it public, so they can let me know if there are any problems with the fix that I might not have spotted.
Don’t force people to upgrade
Don’t force customers to upgrade if I don’t want to. And don’t nag them every day if they don’t. A case in point is Skype. It has (predictably) turned from a great piece of software into a piece of crap now that Microsoft have purchased it. Every release is worst than the last. And, to add insult to injury, it just keeps bleating at me to upgrade and there doesn’t seem to be any way to turn off the notifications.
Don’t promise ship dates
If you promise a ship date and you get your estimate wrong (which you will) then either:
- You have ship software that isn’t finished; or
- You miss your ship date
Neither are good. So don’t promise ship dates. I never do and it makes my life a lot less stressful. It’s ready when it’s ready. I realize that some companies with investors, business partners and large marketing departments don’t have that luxury. I’m just glad that I am not them.
Inform existing customers of the release
There isn’t much point in putting out releases if no-one knows about them. By default my software checks an XML file on my server weekly and informs the customer if a new update is available. I also send out a newsletter with each software release. I generally get a spike in upgrades after each newsletter.
Don’t release too often
Creating a stable release is a lot of work, even if you manage to automate some of it. The more releases you do, the higher percentage of your time you will spend testing, proof reading and updating your website.
Adobe Acrobat seems to go through phases of nagging at me almost daily for updates. Do I think “Wow, I am so happy that those Adobe engineers keep putting out releases of their useful free software”? No. I hate them for it. If you have an early stage product with early-adopters, they may be ok with an update every few days. But most mainstream customers won’t thank you for it.
Don’t release too infrequently
Fixing a bug or usability issue doesn’t help the customer until you ship it. Also a product with very infrequent updates looks dead. The appropriate release frequency will vary with the type of product and how complex and mature it is.
Digitally sign your releases
Digital certificates are a rip-off. But unsigned software makes you look like an an amateur. I am wary of downloading any software that isn’t digitally signed. Apple now prevents you downloading unsigned software by default. Signing is just an extra line in your build script. It is a bit tedious getting a digital certificate though, so get one that lasts several years.
Check your binaries against major anti-virus software
Over zealous anti-virus software can be a real headache for developers of downloadable software. So it is worth checking if your release is likely to get flagged. You can do this using free online resource virustotal.com. If you are flagged, contact the vendor and ask them to whitelist you.
‘The perfect is the enemy of the good’
Beware second system effect. If you wait for perfection, then you will never ship anything. As long as this release is a significant improvement on the last release, then it is good enough to ship.
Creating a release is exhausting. Even maths, physics and software prodigy Stephen Wolfram of Mathematica says so:
I’ve led a few dozen major software releases in my life. And one might think that by now I’d have got to the point where doing a software release would just be a calm and straightforward process. But it never is. Perhaps it’s because we’re always trying to do majorly new and innovative things. Or perhaps it’s just the nature of such projects. But I’ve found that to get the project done to the quality level I want always requires a remarkable degree of personal intensity. Yes, at least in the case of our company, there are always extremely talented people working on the project. But somehow there are always things to do that nobody expected, and it takes a lot of energy, focus and pushing to get them all together.
So look after yourself. Make sure you get enough sleep, exercise and eat healthily. Also things may be at their most intense straight after the release with promotion, support, bug fixing etc. So it may be a good idea to take a day or two off before you send the release out.
Don’t release anything just before you go away
There is always a chance a new release is going to mess things up. If you are a one-man band like me, you really don’t want to make a software release just before you go away on holiday or to a conference. Wait until you get back!
Fix screwups ASAP
We all make mistakes from time to time. I recently put out a release of my card planning software, Hyper Plan, that crashed on start-up on some older versions of macOS. Oops. But I got out a release with a fix as soon as I could.
Treat yourself after a release
Releases are hard work. A successful release deserves a treat!
Anything I missed?
This is a guest post from roving software entrepreneur, Steve McLeod.
Each feature you add to your software product takes time to implement, adds ongoing complexity, and is hard to get rid of later. So you need to choose wisely when adding new features.
Here are some tips on choosing which features to implement.
Does your product really need new features?
This question might be surprising, but it is an important one to ask yourself. A software product is never completely finished. There is always scope for improvement. This makes it hard to know when to stop working on it.
If your product is mature and has stable revenue, there might not be much opportunity to increase sales. Adding new features would please some customers, but would not be a good use of your time.
Products that should be considered “done” can still have a large backlog. Don’t be fooled into thinking that your product will only be done once all existing feature requests are done.
Your product might have great potential to grow, but not by adding new features. Perhaps you need to improve your sales and marketing efforts. Don’t use your feature request backlog as an excuse to neglect the other important parts of running your product.
It’s okay to ignore your backlog
Feature request backlogs seem to never shrink. Implementing improvements leads to even more feature requests.
Looking at your backlog can be overwhelming. You’ll see feature requests that are years old. There are others that you intended to implement, but never did. It is okay to ignore these.
Don’t feel that the entire backlog consists of promises that must be kept. Some requests are no longer relevant. Some are from customers who no longer need your product. Some are simply not worth the substantial effort.
Keep your grand vision in mind
Prefer to implement highly requested features. But before you do, make sure each feature fits your long-term plans. Ask yourself:
- who is your ideal customer? Is the feature relevant to them? For example, your enterprise customers might be asking for “single sign-on”, whereas you are more interested in working with small businesses.
- will this be an ongoing cost- and time-sink? For example, a particular feature could require HIPAA (a US health industry regulation) compliance, while you have no interest in the extra workload and costs of HIPAA compliance.
- does this proposed feature agree with your marketing angle?
Each new feature has ongoing costs
Remember that each additional feature comes with ongoing costs in support and complexity. Each new feature is an additional place for bugs to hide, adding to your future workload. Take these future costs into account when deciding if a feature should be added.
Features interact with each other, sometimes in unexpected ways. Even one additional checkbox in a settings panel can lead to ongoing confusion.
Be wary in particular of features that require integration with third party products. For example, many a product owner was burned when Twitter turned off some parts of their public API. Another example is my own experience adding support for connecting to customers’ email inboxes. I then found myself supporting not just my own product’s problems, but those caused by customers’ email service provider.
Sales-blockers are high priority
Not all feature requests are equal. Requests for feature A might come only from long time customers of your product, while feature B is requested only by your trial customers. It sounds ruthless, but you should prioritise the feature requested by trial customers.
When running a business, you do need to care for financial concerns. Take into consideration that:
- A trial customer is likely to report a showstopper (for them). When you implement features requested by several trial customers, you are likely to increase revenue.
- An existing customer is likely to report an inconvenience. You probably won’t lose the customer by delaying the improvement they asked for.
- You need new customers to keep your business healthy.
It is important, of course, to care for existing customers. Don’t take this as an argument to ignore existing customers altogether in favour of potential customers.
Identifying and fixing sales blockers as a matter of priority is especially important for new products. It takes time to find the right balance of features a new product needs to meet the demands of the market.
Prefer the easy things, but not always
Let’s say you are deciding which of two features to implement next. Both feature A and feature B have been requested dozens of times. Feature A will take a week’s worth of work, and feature B will take a month’s work. Which one do you do?
Feature A, right? Maybe not.
Feature A is usually the correct choice. But you’ll be repeating this scenario over and over. Your product will gradually gain many relatively trivial features, while never getting the important features you need to create a compelling product.
Sometimes you need to do the hard work to add the features that are difficult to implement.
Listen to your customers…
Your customers are a great source of ideas for improvements to your product. As they use your product they discover what’s missing and what’s poorly implemented.
Therefore make it exceedingly easy for your customers to get suggestions to you. Consider adding a “Suggest an Improvement” link to your support page and to your help menu, if applicable. This makes it clear to your customers that you are actively seeking suggestions. My experience has been that customers notice these links and use them.
A “Suggest an Improvement” link can be as simple as a “mailto” link. A basic dialog or web form also works well. If you are seeking inspiration on how to do this, look at the “Report an Issue” form linked from the Google Chrome’s Help menu.
Another way to get customer suggestions is by asking them via a survey. This is okay, but it requires people to imagine back to when they last used your product. Us humans tend to be pretty poor at recalling things. It is much better to encourage customers to send suggestions as they encounter difficulties, while it is foremost in their mind.
I found collecting and organizing feedback for my own product, Poker Copilot, to be a pain. So I launched a new product to help manage feature requests. It allows your customers to suggest and upvote improvements for your product.
…and not your competitors
You’ve probably already heard that you should “listen to your customers, not your competitors.” I’m repeating it, because forgetting this can be deadly to your business.
Feature-envy leads us to believe we have to add every feature our competitors offer. Like regular envy, it is best to ignore it.
When your competitor announces a new feature, it can be demoralising. You feel you are falling behind and unable to compete. If the feature looks impressive, you’ll be tempted to add it to your product as soon as you can.
But wait. Have any of your customers actually requested this feature that your competitor added? If not, it is probably because it is not all that useful to your customers.
Your competitor might eventually regret adding that feature, as it turns out that only a small but demanding percentage of their customers use it. By not adding it, you could be giving yourself an advantage.
I’ve discovered that customers often learn about the good features my competitors have. They then tell me that they want these features. Only then do I start considering adding them.
Small tweaks versus major features
I just recommended listening to customers. But be careful of only using customer requests as a source for improvements.
Your customers tend to ask for incremental improvements. Customers are more likely to ask for an additional option in a drop-down menu than they are for a innovative new way to view their data.
If you rely on customer suggestions alone, you’ll never get the innovative new features that can make your product something much grander.
Make sure you balance your customer-contributed requests with your own innovative improvements.
Beware of preferring “pet” features
I just argued in favour of choosing innovative improvements. However this comes with a warning. Innovation can be used to justify poor choices. When you are in charge of decisions for your product, you are in danger of choosing improvements that you find interesting instead of those that have a strong business case.
Be careful. You could spend months working on a feature that is not very helpful to your business.
This is the type of feature that only one customer requested, but because it sounds more interesting to you than anything else in your massive backlog of feature requests, you immediately start working on it.
My own experience with this: I added scripting to a product even though no-one had asked for it and my target customer was not the type of person who would be interested in custom scripting. When I announced the scripting feature, not a single person seemed to use it. I eventually got rid of it.
How do you decide?
The tips I’ve offered you in this article come from my own experience. Do you have tips of your own for deciding which features to implement? If so, please add them in the comments below. I’d love to read them!
Steve McLeod runs a small software company in Barcelona, Spain. His products are Poker Copilot, a desktop analytics app for online poker players, and Feature Upvote, a web app that allows your customers to openly suggest and upvote features they want to see in your product.
There is a lot to be said for running a small software business (just me, with my wife doing some of the admin). For a start it gives me a great deal of flexibility, which I used to spend 2 months travelling abroad with my family last year. It is also low in stress, as I don’t have any employees to manage (my wife manages herself!). But even with some consulting work, going to the occasional conference and running some face-to-face training courses, I was starting to feel a little bit isolated after 13 years working mostly on my own. At that time the news was full of heart-rending stories of the suffering of refugees trying to flee war and repression. I don’t like the way the world is heading at present and wanted to do something to help these people in whatever small way I could. So I started volunteering at a charity for refugees and asylum seekers in my home town.
I initially tried to avoid doing computer things for the charity. But it quickly became clear that my IT skills were much more useful to them than anything else I could offer. Consequently I have been sorting out various IT issues, teaching people basic IT skills and have built them a simple CRM and reporting system, based on Airtable. Replacing the previous paper system with an electronic one is saving the staff and volunteers a lot of busy-work, which frees up their time to do more useful things. It is also giving the charity a lot more insight into how they are doing. And it has got me out from office and meeting some really great people who I wouldn’t have met otherwise. Win-win. Sometimes I feel spread a bit thin with my various work, charity and personal commitments (which is partly why the blog has been a bit quiet recently) but overall I am very glad I started volunteering.
So, if you are feeling a bit isolated, consider volunteering for a local charity. I suspect many small charities are desperate for volunteers with IT skills. Even if you are a programmer with quite specialist skills (like me), it is easy to forget that you are still an IT systems god compared with 99% of the population.