ContentedLife

I used to advocate test-driven development (TDD), but now I advocate DDD – documentation-driven development. I don’t even know if it’s a “thing”, I just decided that’s how we should work.

Writing tests first ensures that your program jumps through some arbitrary set of developer-designed hoops. Worthwhile for sure, but may or may not represent things that are important to the user, and doesn’t help with the design process.

Writing documentation first ensures that your program *makes sense* from the point of view of the user of that program (or API or whatever). It’s too easy otherwise to design from our own point of view, and do something a particular way because it’s easier to code that way, or that’s the way the code happens to fall out that day, without adequately considering the user perspective.

Tests, being still written in code, keep you in the programmer’s mindset. Only by switching to English (or other human language) and writing the actual docs for the poor sod who’s going to use this, can you flip into the user’s mindset and see how actually it would make more sense if you do this, don’t do that, simplify this, make that optional, etc.

A handy side-effect is that you get good, accurate and up-to-date documentation – in fact, this is the *only* method I’ve ever found that results in that. If you do documentation last, it doesn’t get done, or gets out of sync with the code. You could use a tool to auto-generate docs from code but those are hardly docs in any menaingful sense at all.

DDD is also implementation-agnostic. Tests tend to be implementation-specific, which means they need to be rewritten if you change the backend language.

So, write documentation first, then write tests for the key parts of the documentation. Then, importantly, still write your code based on the documentation, not the tests. The tests simply confirm that you’ve done that right. If you code to the tests, you might code to *only* the tests, meaning that if there’s something in the docs that you don’t have a test for, you might miss it or get it wrong in your code. Just like the current problem in the education system – a heavy reliance on testing, and the results of those tests having implications for the school’s funding, means that teachers “teach to the test” rather than teaching what kids actually need to learn. Don’t do that with your code – code to the docs, not the tests.

Since I came up with DDD, BDD (behaviour-driven development) has become popular… this seems to me like it’s just a variant of TDD – still quite “code-ish” even if written in pseudo-English, but possibly useful for requirements capture, or to help with translating docs into tests, and to ensure that your tests have good coverage of what’s in the documentation. But it’s not something I’ve used thus far.

I’ve heard the following little nugget of information before:

St John’s Wood is the only station on the London Underground network which does not contain any of the letters in the word “mackerel”.

I have never had any reason to doubt this, but neither had I checked it. It popped up again recently on a Facebook thread. Someone suggested that it was not true (or perhaps no longer true) because of Hoxton. However, strictly speaking Hoxton is on the London Overground network, not London Underground.

Anyway, I decided that I should go ahead and check the veracity of the statement. The geeky way.

1. After a while searching for an existing plaintext list of Tube station names, I couldn’t find one, so instead I downloaded the Wikipedia page List of London Underground stations, had a quick look at the source HTML to see if it would be easy to pull the station names out of the page.

2. It was; each line of the table starts with <th scope=”row”>, and follows a set pattern after that, as you might expect:

   <th scope="row"><a href="/wiki/Acton_Town_tube_station"
   title="Acton Town tube station">Acton Town</a></th>

   This is all one line in the original HTML, I’ve just broken it to two for display.

   So I can pull out just the lines containing station names using a simple grep.

3. All I’m interested in is the bit between the opening <a …> and closing </a> tag. At this point I tend to resort to Perl to do anything remotely complex with regex replacement.

   $ grep '<th scope="row">' \
   list_of_london_underground_stations_from_wikipedia.html \
   | perl -pe 's/.*>([^<]+)<\/a>.*/$1/' \
   > list_of_london_underground_stations.txt

   Since I’ve broken out the Perl for this job, I could have thrown away the initial grep and incorporated it into the Perl instead, but this is just a quick hack and it already works, so why bother? I’m not aiming for elegance here.

4. Verify the list:

   $ cat list_of_london_underground_stations.txt
   Acton Town
   Aldgate
   Aldgate East
   Alperton
   [...]
   Watford High Street
   Watford Junction
   Watford Vicarage Road
   $ wc -l  list_of_london_underground_stations.txt
        275 list_of_london_underground_stations.txt
   

   Looks good. We have just the station names, one per line, and there are 275 lines, which sounds about right. [The list includes a few planned stations at the end. I decided to keep these in.]

5. Now I can grep for those station names for those not containing any of the letters in mackerel:

   $ egrep -vi '[mackerl]' list_of_london_underground_stations.txt
   St. John's Wood

And there you have it. Assuming of course that the Wikipedia list is correct and complete, the factoid is confirmed. And it took just a couple of minutes. Plus about 20 minutes writing it up afterwards…

Of course, now I can substitute other words for ‘mackerel’ too. For example, there are three stations that do not contain any of the letters in ‘herring’.

For your convenience and further exploration, you may download my plain text list of London Underground stations. Remember, it includes every station on the Wikipedia page as of 4th March 2015, including several stations that are at the planning stage so do not actually exist yet (or not on the current Tube network anyway); these are at the end of the list and are: Battersea, Cassiobridge, Nine Elms, Watford High Street, Watford Junction, Watford Vicarage Road.

Can’t find the customer’s home country in the database? That’s ok; just pick any country with a vaguely similar-sounding name, that’s good enough.

A bizarre bit of code in the e-commerce software¹ I’m currently fixing up does exactly that; it uses the Soundex algorithm to look for an approximate match to where the customer lives, according to the similarity of how the countries’ names are pronounced, rather than the more conventional considerations like geography.

There are 27 countries in the database that share a Soundex value with at least one other (mostly just pairs, but the largest matching group is 4: Ghana, Guam, Guinea and Guyana). In each group, all the countries would be rewritten to whichever was first alphabetically. Addresses in Greece would appear to be in Georgia; Norway became Nauru.

This sort of thing is nice and easy to fix (finding it is the hard part), but leaves a strange aftertaste… the insoluble mystery of just what was going on in the mind of whoever decided to write that code that made them think it would be a good idea…

No, it’s not really my DJing that’s script-powered, but over the last couple of years that I’ve been doing regular net radio shows, I have written a number of Perl scripts to help with some of the more tedious aspects of the job, particularly related to the posting of the MP3 archives and tracklists of those shows (and my occasional promo mixes) on quextal.com, but also for the broadcasting process itself.

In fact one of the first scripts I wrote was to assist with the fact that I broadcast (using darkice on my Linux box) on different stations, necessitating having multiple different configurations for darkice. What began as a one-liner to do the equivalent of darkice -c /path/to/darkice/configs/$1.cfg then expanded to do things like shut down certain daemons before broadcasting, and start them again afterwards, as my elderly PC would occasionally struggle to cope with the demands of running two MP3 encoders if it was also dealing with a large incoming mail or a disk-heavy cronjob.

I then tired of hitting reload on the server stats page to keep an eye on my listener count, so now I have a script which fetches that page every couple of minutes, parses the relevant number out of it, and shows it with a timestamp, so I have a full record of how many were tuned in at each point of the show, what the peak was etc.

Scripts followed to automate filling in the ID3 tag, and renaming darkice’s output spool name into a standard format prior to uploading it to the site.

quextal.com is a WordPress-based site with a heavily customised skin and a couple of extra plugins, nothing too fancy. After writing the first few posts by hand, I came up with a simple template-driven script which would simply wrap my plain-text tracklist of the show in some HTML to make it look a bit prettier for the site. This evolved so that it would read the metadata from the MP3 (eg filesize, bitrate, length in minutes and seconds) and put that info in there as well.

After a while I decided to have my online tracklists in table format rather than just reproducing what I write in plain text. So this meant adapting the script to split up each entry in the tracklist for the separate columns. I had the prescience to choose a roughly standardised format for my plain text tracklists anyway — at its simplest, it’s just “Artist – Title” or “Artist – Title – Label” — but over time it’s evolved a number of variations to deal with, for example, marking out who played which track when I have a guest in. I sensed it was time to create a separate library (Perl module) to parse tracklists into separate information, and a number of my scripts now use this.

Just this year I expanded the templating script into a more complex system which interfaces directly with the WordPress API. It determines which radio station the broadcast was on (which is in the filename), searches for some of my past mixes for that station on the site, and offers a selection of their post titles so I can choose one (eg with, or without, a guest DJ, as applicable) on which to base the default title for the new one, helping to keep the title format consistent. Both my current regular shows feature the number of the show in the title – the script will automatically increment this, be it in ordinary numerals or Roman numerals. Appropriate tags are chosen automatically, and any additional words for the article can be added before the script posts it directly to the site via the API.

Why stop there? Since my Tracklist library conveniently gives me information about the artists and labels played in each show, the script now also creates a Custom Field entry for each. I don’t really know why I’ve done that… just a vague sense that it might be useful at some point in the future. For now, a slight tweak at the WordPress end provides A-Z lists of artists and labels for each mix at the end of the article. At some point, if so desired, it should make it easier to search for all the mixes containing a specific artist or label…

Most recently, the thing I was finding particularly time-consuming was to fill in the label for each tune, which information I often don’t have handy during the show when I’m writing down the track. So now I have a couple of scripts to help with that. The first just looks for the “artist – title” string in all my previous tracklists and copies the label info from there if it finds it. The second, which is a work in progress, attempts to automate looking up the track details on the sites where I do most of my tune shopping, and screen-scraping the label from there.

Curiously, the net effect of all this automation has not really made it significantly quicker or easier to post a mix, compared to when I first started out and was doing it all by hand. What it has done is escalated the amount and quality of information I’m putting up, its consistency and reliability, while taking about the same amount of time and effort. Obviously that doesn’t include the effort required to write the scripts… but that’s not effort. That’s fun. It’s been a whole series of interesting little coding tasks… which of course is the main reason I did it.

Re Bugtraq post PHP import_request_variables() arbitrary variable overwrite.

This sort of thing really brings it home how the PHP core team still
don’t seem to really understand security… or would rather sacrifice it
in the name of backwards, very backwards, compatibility.

If you’re going to provide a function like import_request_variables()
to replace the blatantly-unsafe register_globals, how on earth can you
get it so badly wrong that it’s even more unsafe?? I mean, security 101
for a function like this would be:

1. Don’t overwrite any existing global variables (come on!)
2. Failing to specify a prefix should be at least an E_WARNING.
3. Stop pandering to people who wrote or still use bad code originating back in PHP3 days, and aim towards getting rid of the whole concept of registering global variable symbols from user-supplied data. It was always a bad idea, it’s never going to stop being a bad idea, just drop it.

I also get terribly bored of seeing bugtraq reports of a “full path disclosure” bug in some app (like it’s a big deal too), only to find that it is, once again, PHP itself that’s at fault. Sure, the sysadmin on a production machine should set display_errors = Off, but if it’s left on why does PHP show the full path? If the purpose is to help the programmer developing code (who somehow doesn’t have access to the server errorlog), then the programmer doesn’t need the full path, they already know where the docroot is — so when showing errors from scripts in the docroot, why not only show the path relative to the docroot? And if from an included or required file outside the docroot, just the last directory component ought to suffice. The full path can still go in the error log, and then maybe developers would learn to use it instead of relying on errors going to screen…

PHP can be secure, but it really needs to stop offering features that are insecure by definition.

I’ve tentatively placed a toe aboard the AJAX bandwagon, thanks largely to Rails which makes it possible, even painless, to achieve AJAX functionality without having to learn Javascript.¹

While AJAX, done properly, can be a big win for Usability for those whose browsers support it, I am also concerned with issues of Accessibility, and indeed keeping things usable for those with non-AJAX browsers.

Standards-Schmandards have come up with some useful hints about how to make AJAX pages more accessible to screenreaders, such as providing an option to pop up an alert box instead of, or as well as, simply updating the page. This is good advice, but it’s only half the story. These methods don’t help those without Javascript at all, or with non-AJAX-aware browsers.

In this article I will detail one method to help ensure accessibility for such people, without compromising on AJAX spiffiness, and most importantly without causing much extra work for us poor developers. The example is in Rails, but may be of interest to those using other frameworks. (more…)