Not TDD or BDD but DDD: Documentation-driven development

I used to advocate test-driven development (TDD), but now I advocate DDD – documentation-driven development. I don’t even know if it’s a “thing”, I just decided that’s how we should work.

Writing tests first ensures that your program jumps through some arbitrary set of developer-designed hoops. Worthwhile for sure, but may or may not represent things that are important to the user, and doesn’t help with the design process.

Writing documentation first ensures that your program *makes sense* from the point of view of the user of that program (or API or whatever). It’s too easy otherwise to design from our own point of view, and do something a particular way because it’s easier to code that way, or that’s the way the code happens to fall out that day, without adequately considering the user perspective.

Tests, being still written in code, keep you in the programmer’s mindset. Only by switching to English (or other human language) and writing the actual docs for the poor sod who’s going to use this, can you flip into the user’s mindset and see how actually it would make more sense if you do this, don’t do that, simplify this, make that optional, etc.

A handy side-effect is that you get good, accurate and up-to-date documentation – in fact, this is the *only* method I’ve ever found that results in that. If you do documentation last, it doesn’t get done, or gets out of sync with the code. You could use a tool to auto-generate docs from code but those are hardly docs in any menaingful sense at all.

DDD is also implementation-agnostic. Tests tend to be implementation-specific, which means they need to be rewritten if you change the backend language.

So, write documentation first, then write tests for the key parts of the documentation. Then, importantly, still write your code based on the documentation, not the tests. The tests simply confirm that you’ve done that right. If you code to the tests, you might code to *only* the tests, meaning that if there’s something in the docs that you don’t have a test for, you might miss it or get it wrong in your code. Just like the current problem in the education system – a heavy reliance on testing, and the results of those tests having implications for the school’s funding, means that teachers “teach to the test” rather than teaching what kids actually need to learn. Don’t do that with your code – code to the docs, not the tests.

Since I came up with DDD, BDD (behaviour-driven development) has become popular… this seems to me like it’s just a variant of TDD – still quite “code-ish” even if written in pseudo-English, but possibly useful for requirements capture, or to help with translating docs into tests, and to ensure that your tests have good coverage of what’s in the documentation. But it’s not something I’ve used thus far.

Bypass cut / copy / paste blocking on websites

An increasing number of websites seem to be blocking the use of copy and paste.

They have their reasons for doing so, but they’re rarely good reasons. For example, they frequently justify banning c&p during signup, to ensure that when they ask you to enter a password twice, the purpose of entering it twice (to ensure you didn’t make a typo) isn’t subverted by the user blindly c&p’ing whatever they entered first time, typos and all. However, doing this also prevents pasting a strong, generated password from a password vault program like Keepass. It harms security to force everyone to use a password which is simple enough to type in.

Youtube probably bans pasting into comments in an effort to cut spam and blind trolling; but doing this also denies you the ability to edit and re-arrange your comment, or paste in a link to another site or even another Youtube video (which is stupid, because they do allow links, but force you to type them in, awkward for URLs like Youtube’s which contain a random string of upper and lowercase letters and numbers…)

Fortunately, bypassing these restrictions is easy, at least in Firefox:
In about:config, set dom.event.clipboardevents.enabled to false. That’s it.

Screen Shot 2016-01-26 at 13.00.15

Of course this will probably also break things that automatically copy to your clipboard when you press a button, but that’s no great hardship.

For Chrome and Chromium-based browsers there’s an add-on called “Don’t Fuck With Paste” – haven’t tried it but I agree with the sentiment 🙂

How to make a video clip with avconv / ffmpeg

The following commandline will make a 30 second clip in output.avi, from the file input.avi starting at 2 mins 25.5 seconds:

avconv -i input.avi -ss 00:02:25.500 -t 30 -codec copy output.avi

ffmpeg takes the same arguments if you have that (avconv is just the newer name for ffmpeg)
The -codec copy is important, and should be used as long as you are using the same output and input file type. Without it, the video will be decoded and re-encoded, which may cause a dramatic loss in quality.

I got a lot of warnings about renumbering frames or somesuch, but the output file worked fine.

A better, faster way to write a Raspberry Pi SD card image

The standard way of writing an SD card image for a Raspberry Pi (or any other purpose that requires writing a whole disk image to the card) from any unix-like system (eg Linux or Mac OSX) is to use the venerable dd(1) utility.

dd has been around, basically unchanged, since the dawn of time. It has an arcane syntax that’s completely different to every other unix command, and its screen output is spectacularly unhelpful. While it’s working, it remains silent. You have no idea how fast it’s going or how long it’s going to take, until it eventually finishes and tells you how many blocks came in and went out – and even that minimal information is presented in a rather obscure format.

On my Mac I found using dd to write a Raspi boot image to an SD card to be very slow and unreliable. For some reason it refused to write to the raw disk device (/dev/rdiskn), even though there were no active mounts. Using the buffered device file instead (/dev/diskn) it took a full 30 minutes to write a 650MB image… and even then, when I pulled the card and put it in my Raspi, somehow it still booted the old OS that was on there before.

Then I discovered a lovely utility called pv, which stands for Pipe Viewer. On a Mac it’s available in Homebrew (brew install pv) and Macports (port install pv).

Using this, instead of dd if=osimage.img of=/dev/diskn (30 minutes, remember), I did pv osimage.img > /dev/rdiskn and it took just over 2 minutes, and the card worked perfectly. (These commands need to run as root, of course).

The other big advantage of pv over dd, is that it shows you what’s going on – it provides a progress bar, data transfer rate, ETA and so on (this is actually its raison d’etre, it just seems to be much more efficient than dd at piping data too) – whereas dd just sits there saying nothing until it’s finished.

pv has a ton of potential uses, and is a shining example of the strength of the UNIX philosophy: a small program that does one thing and does it well, interconnecting with other programs in a standard way to make them better. I don’t know how long it’s been around, but I love the fact that even after 20 years of using Linux I can still stumble upon something new and neat that I know will immediately become a well-used part of my toolkit and make life easier. And I have to laugh, because I’ve just found out it was written by Andrew Wood, an old friend from years ago. Looks like it’s been around for quite a while, and is still being actively maintained. Kudos.


* Wherever you see n in this article, you’ll need to replace it with the appropriate number for the SD card on your system. On my Macbook Pro it’s 2, but your system may be different, and if you get it wrong, you could overwrite a critical system disk. My first ever big data loss event back in 1994 was caused by an error like this, which is why I’ve not put the number in above, I wouldn’t want anyone to blindly copy&paste – find out the correct number for your system, double, triple and quadruple check the commandline before you hit enter, take backups and take care!

Verifying the Tube station “mackerel” factoid

I’ve heard the following little nugget of information before:

St John’s Wood is the only station on the London Underground network which does not contain any of the letters in the word “mackerel”.

I have never had any reason to doubt this, but neither had I checked it. It popped up again recently on a Facebook thread. Someone suggested that it was not true (or perhaps no longer true) because of Hoxton. However, strictly speaking Hoxton is on the London Overground network, not London Underground.

Anyway, I decided that I should go ahead and check the veracity of the statement. The geeky way.

  1. After a while searching for an existing plaintext list of Tube station names, I couldn’t find one, so instead I downloaded the Wikipedia page List of London Underground stations, had a quick look at the source HTML to see if it would be easy to pull the station names out of the page.

  2. It was; each line of the table starts with <th scope=”row”>, and follows a set pattern after that, as you might expect:

    <th scope="row"><a href="/wiki/Acton_Town_tube_station"
    title="Acton Town tube station">Acton Town</a></th>

    This is all one line in the original HTML, I’ve just broken it to two for display.

    So I can pull out just the lines containing station names using a simple grep.

  3. All I’m interested in is the bit between the opening <a …> and closing </a> tag. At this point I tend to resort to Perl to do anything remotely complex with regex replacement.

    $ grep '<th scope="row">' \
    list_of_london_underground_stations_from_wikipedia.html \
    | perl -pe 's/.*>([^<]+)<\/a>.*/$1/' \
    > list_of_london_underground_stations.txt

    Since I’ve broken out the Perl for this job, I could have thrown away the initial grep and incorporated it into the Perl instead, but this is just a quick hack and it already works, so why bother? I’m not aiming for elegance here.

  4. Verify the list:

    $ cat list_of_london_underground_stations.txt
    Acton Town
    Aldgate East
    Watford High Street
    Watford Junction
    Watford Vicarage Road
    $ wc -l  list_of_london_underground_stations.txt
         275 list_of_london_underground_stations.txt

    Looks good. We have just the station names, one per line, and there are 275 lines, which sounds about right. [The list includes a few planned stations at the end. I decided to keep these in.]

  5. Now I can grep for those station names for those not containing any of the letters in mackerel:
    $ egrep -vi '[mackerl]' list_of_london_underground_stations.txt
    St. John's Wood

And there you have it. Assuming of course that the Wikipedia list is correct and complete, the factoid is confirmed. And it took just a couple of minutes. Plus about 20 minutes writing it up afterwards…

Of course, now I can substitute other words for ‘mackerel’ too. For example, there are three stations that do not contain any of the letters in ‘herring’.

For your convenience and further exploration, you may download my plain text list of London Underground stations. Remember, it includes every station on the Wikipedia page as of 4th March 2015, including several stations that are at the planning stage so do not actually exist yet (or not on the current Tube network anyway); these are at the end of the list and are: Battersea, Cassiobridge, Nine Elms, Watford High Street, Watford Junction, Watford Vicarage Road.

Bring back the Firefox (wait / busy) mouse (cursor / pointer) (hourglass / clock)

Old versions of Firefox used to indicate when you were waiting for the next page to load, by changing the cursor to one with an hourglass or clock. This provided immediate visual feedback at the screen position where you were already looking that the link click had succeeded and something was happening. This is simply good UI practice.

Some time ago, in version 3.5, this feature got removed, without fanfare. This meant that now, to see if the browser is doing something, the user must look away from where they were looking, either at the tab bar or the status bar. Neither of these have the immediacy of the pointer change, and not everyone has the luxury of having them.

Many people felt this was a bad idea, as did I, but at the time no option was provided to restore it.

However I recently revisited the page, and found that, soon after I reluctantly gave up on finding a solution, they did indeed add an option to restore it. Go to about:config, and set the value ui.use_activity_cursor to true

This is far from the only bad decision that Mozilla developers have made in recent years. Removing the status bar is probably the most ludicrous one (and the one that would have sent me running to another browser immediately had there not been an add-on to bring it back), amd I’m sure I’m not alone in feeling that their current release schedule and version number policy is barmy. Overall, it’s still great, and for modern web development nothing else comes close to Firefox armed with a stack of extensions like Firebug, Web Developer, Tilt etc. But it seems to me that Mozilla have repeatedly forgotten or ignored the diverse needs of their users in a bid to follow idealistic policies like having a “zen” interface, or upping their version number every 2 seconds (slight exaggeration) whether anything significant has changed or not.

Edit: 12 Nov 2014:
Another about:config option I recently discovered is the ability to stop the backspace key from going back a page. I never, ever, ever want it to do that, and whoever decided that was a good keybinding better not meet me in a dark alley lest they have to pay for all the times it’s bitten me. Anyway, fortunately in Firefox you can switch it off easily. The setting is browser.backspace_action and the magic number to change it to is 2. (1 makes it behave like Page Up – less annoying than Back, but still, just why?)