Archive for the 'Linux' Category

How to make a video clip with avconv / ffmpeg

Thursday 7th January 2016

The following commandline will make a 30 second clip in output.avi, from the file input.avi starting at 2 mins 25.5 seconds:

avconv -i input.avi -ss 00:02:25.500 -t 30 -codec copy output.avi

ffmpeg takes the same arguments if you have that (avconv is just the newer name for ffmpeg)
The -codec copy is important, and should be used as long as you are using the same output and input file type. Without it, the video will be decoded and re-encoded, which may cause a dramatic loss in quality.

I got a lot of warnings about renumbering frames or somesuch, but the output file worked fine.

A better, faster way to write a Raspberry Pi SD card image

Sunday 31st May 2015

The standard way of writing an SD card image for a Raspberry Pi (or any other purpose that requires writing a whole disk image to the card) from any unix-like system (eg Linux or Mac OSX) is to use the venerable dd(1) utility.

dd has been around, basically unchanged, since the dawn of time. It has an arcane syntax that’s completely different to every other unix command, and its screen output is spectacularly unhelpful. While it’s working, it remains silent. You have no idea how fast it’s going or how long it’s going to take, until it eventually finishes and tells you how many blocks came in and went out – and even that minimal information is presented in a rather obscure format.

On my Mac I found using dd to write a Raspi boot image to an SD card to be very slow and unreliable. For some reason it refused to write to the raw disk device (/dev/rdiskn), even though there were no active mounts. Using the buffered device file instead (/dev/diskn) it took a full 30 minutes to write a 650MB image… and even then, when I pulled the card and put it in my Raspi, somehow it still booted the old OS that was on there before.

Then I discovered a lovely utility called pv, which stands for Pipe Viewer. On a Mac it’s available in Homebrew (brew install pv) and Macports (port install pv).

Using this, instead of dd if=osimage.img of=/dev/diskn (30 minutes, remember), I did pv osimage.img > /dev/rdiskn and it took just over 2 minutes, and the card worked perfectly. (These commands need to run as root, of course).

The other big advantage of pv over dd, is that it shows you what’s going on – it provides a progress bar, data transfer rate, ETA and so on (this is actually its raison d’etre, it just seems to be much more efficient than dd at piping data too) – whereas dd just sits there saying nothing until it’s finished.

pv has a ton of potential uses, and is a shining example of the strength of the UNIX philosophy: a small program that does one thing and does it well, interconnecting with other programs in a standard way to make them better. I don’t know how long it’s been around, but I love the fact that even after 20 years of using Linux I can still stumble upon something new and neat that I know will immediately become a well-used part of my toolkit and make life easier. And I have to laugh, because I’ve just found out it was written by Andrew Wood, an old friend from years ago. Looks like it’s been around for quite a while, and is still being actively maintained. Kudos.

Links

* Wherever you see n in this article, you’ll need to replace it with the appropriate number for the SD card on your system. On my Macbook Pro it’s 2, but your system may be different, and if you get it wrong, you could overwrite a critical system disk. My first ever big data loss event back in 1994 was caused by an error like this, which is why I’ve not put the number in above, I wouldn’t want anyone to blindly copy&paste – find out the correct number for your system, double, triple and quadruple check the commandline before you hit enter, take backups and take care!

Verifying the Tube station “mackerel” factoid

Wednesday 4th March 2015

I’ve heard the following little nugget of information before:

St John’s Wood is the only station on the London Underground network which does not contain any of the letters in the word “mackerel”.

I have never had any reason to doubt this, but neither had I checked it. It popped up again recently on a Facebook thread. Someone suggested that it was not true (or perhaps no longer true) because of Hoxton. However, strictly speaking Hoxton is on the London Overground network, not London Underground.

Anyway, I decided that I should go ahead and check the veracity of the statement. The geeky way.

  1. After a while searching for an existing plaintext list of Tube station names, I couldn’t find one, so instead I downloaded the Wikipedia page List of London Underground stations, had a quick look at the source HTML to see if it would be easy to pull the station names out of the page.

  2. It was; each line of the table starts with <th scope=”row”>, and follows a set pattern after that, as you might expect:

    <th scope="row"><a href="/wiki/Acton_Town_tube_station"
    title="Acton Town tube station">Acton Town</a></th>

    This is all one line in the original HTML, I’ve just broken it to two for display.

    So I can pull out just the lines containing station names using a simple grep.

  3. All I’m interested in is the bit between the opening <a …> and closing </a> tag. At this point I tend to resort to Perl to do anything remotely complex with regex replacement.

    $ grep '<th scope="row">' \
    list_of_london_underground_stations_from_wikipedia.html \
    | perl -pe 's/.*>([^<]+)<\/a>.*/$1/' \
    > list_of_london_underground_stations.txt

    Since I’ve broken out the Perl for this job, I could have thrown away the initial grep and incorporated it into the Perl instead, but this is just a quick hack and it already works, so why bother? I’m not aiming for elegance here.

  4. Verify the list:

    $ cat list_of_london_underground_stations.txt
    Acton Town
    Aldgate
    Aldgate East
    Alperton
    [...]
    Watford High Street
    Watford Junction
    Watford Vicarage Road
    $ wc -l  list_of_london_underground_stations.txt
         275 list_of_london_underground_stations.txt
    

    Looks good. We have just the station names, one per line, and there are 275 lines, which sounds about right. [The list includes a few planned stations at the end. I decided to keep these in.]

  5. Now I can grep for those station names for those not containing any of the letters in mackerel:
    $ egrep -vi '[mackerl]' list_of_london_underground_stations.txt
    St. John's Wood

And there you have it. Assuming of course that the Wikipedia list is correct and complete, the factoid is confirmed. And it took just a couple of minutes. Plus about 20 minutes writing it up afterwards…

Of course, now I can substitute other words for ‘mackerel’ too. For example, there are three stations that do not contain any of the letters in ‘herring’.

For your convenience and further exploration, you may download my plain text list of London Underground stations. Remember, it includes every station on the Wikipedia page as of 4th March 2015, including several stations that are at the planning stage so do not actually exist yet (or not on the current Tube network anyway); these are at the end of the list and are: Battersea, Cassiobridge, Nine Elms, Watford High Street, Watford Junction, Watford Vicarage Road.

Bring back the Firefox (wait / busy) mouse (cursor / pointer) (hourglass / clock)

Wednesday 30th January 2013

Old versions of Firefox used to indicate when you were waiting for the next page to load, by changing the cursor to one with an hourglass or clock. This provided immediate visual feedback at the screen position where you were already looking that the link click had succeeded and something was happening. This is simply good UI practice.

Some time ago, in version 3.5, this feature got removed, without fanfare. This meant that now, to see if the browser is doing something, the user must look away from where they were looking, either at the tab bar or the status bar. Neither of these have the immediacy of the pointer change, and not everyone has the luxury of having them.

Many people felt this was a bad idea, as did I, but at the time no option was provided to restore it.

However I recently revisited the page, and found that, soon after I reluctantly gave up on finding a solution, they did indeed add an option to restore it. Go to about:config, and set the value ui.use_activity_cursor to true

This is far from the only bad decision that Mozilla developers have made in recent years. Removing the status bar is probably the most ludicrous one (and the one that would have sent me running to another browser immediately had there not been an add-on to bring it back), amd I’m sure I’m not alone in feeling that their current release schedule and version number policy is barmy. Overall, it’s still great, and for modern web development nothing else comes close to Firefox armed with a stack of extensions like Firebug, Web Developer, Tilt etc. But it seems to me that Mozilla have repeatedly forgotten or ignored the diverse needs of their users in a bid to follow idealistic policies like having a “zen” interface, or upping their version number every 2 seconds (slight exaggeration) whether anything significant has changed or not.

Edit: 12 Nov 2014:
Another about:config option I recently discovered is the ability to stop the backspace key from going back a page. I never, ever, ever want it to do that, and whoever decided that was a good keybinding better not meet me in a dark alley lest they have to pay for all the times it’s bitten me. Anyway, fortunately in Firefox you can switch it off easily. The setting is browser.backspace_action and the magic number to change it to is 2. (1 makes it behave like Page Up – less annoying than Back, but still, just why?)

Perl-Powered DJ

Friday 20th August 2010

No, it’s not really my DJing that’s script-powered, but over the last couple of years that I’ve been doing regular net radio shows, I have written a number of Perl scripts to help with some of the more tedious aspects of the job, particularly related to the posting of the MP3 archives and tracklists of those shows (and my occasional promo mixes) on quextal.com, but also for the broadcasting process itself.

In fact one of the first scripts I wrote was to assist with the fact that I broadcast (using darkice on my Linux box) on different stations, necessitating having multiple different configurations for darkice. What began as a one-liner to do the equivalent of darkice -c /path/to/darkice/configs/$1.cfg then expanded to do things like shut down certain daemons before broadcasting, and start them again afterwards, as my elderly PC would occasionally struggle to cope with the demands of running two MP3 encoders if it was also dealing with a large incoming mail or a disk-heavy cronjob.

I then tired of hitting reload on the server stats page to keep an eye on my listener count, so now I have a script which fetches that page every couple of minutes, parses the relevant number out of it, and shows it with a timestamp, so I have a full record of how many were tuned in at each point of the show, what the peak was etc.

Scripts followed to automate filling in the ID3 tag, and renaming darkice’s output spool name into a standard format prior to uploading it to the site.

quextal.com is a WordPress-based site with a heavily customised skin and a couple of extra plugins, nothing too fancy. After writing the first few posts by hand, I came up with a simple template-driven script which would simply wrap my plain-text tracklist of the show in some HTML to make it look a bit prettier for the site. This evolved so that it would read the metadata from the MP3 (eg filesize, bitrate, length in minutes and seconds) and put that info in there as well.

After a while I decided to have my online tracklists in table format rather than just reproducing what I write in plain text. So this meant adapting the script to split up each entry in the tracklist for the separate columns. I had the prescience to choose a roughly standardised format for my plain text tracklists anyway — at its simplest, it’s just “Artist – Title” or “Artist – Title – Label” — but over time it’s evolved a number of variations to deal with, for example, marking out who played which track when I have a guest in. I sensed it was time to create a separate library (Perl module) to parse tracklists into separate information, and a number of my scripts now use this.

Just this year I expanded the templating script into a more complex system which interfaces directly with the WordPress API. It determines which radio station the broadcast was on (which is in the filename), searches for some of my past mixes for that station on the site, and offers a selection of their post titles so I can choose one (eg with, or without, a guest DJ, as applicable) on which to base the default title for the new one, helping to keep the title format consistent. Both my current regular shows feature the number of the show in the title – the script will automatically increment this, be it in ordinary numerals or Roman numerals. Appropriate tags are chosen automatically, and any additional words for the article can be added before the script posts it directly to the site via the API.

Why stop there? Since my Tracklist library conveniently gives me information about the artists and labels played in each show, the script now also creates a Custom Field entry for each. I don’t really know why I’ve done that… just a vague sense that it might be useful at some point in the future. For now, a slight tweak at the WordPress end provides A-Z lists of artists and labels for each mix at the end of the article. At some point, if so desired, it should make it easier to search for all the mixes containing a specific artist or label…

Most recently, the thing I was finding particularly time-consuming was to fill in the label for each tune, which information I often don’t have handy during the show when I’m writing down the track. So now I have a couple of scripts to help with that. The first just looks for the “artist – title” string in all my previous tracklists and copies the label info from there if it finds it. The second, which is a work in progress, attempts to automate looking up the track details on the sites where I do most of my tune shopping, and screen-scraping the label from there.

Curiously, the net effect of all this automation has not really made it significantly quicker or easier to post a mix, compared to when I first started out and was doing it all by hand. What it has done is escalated the amount and quality of information I’m putting up, its consistency and reliability, while taking about the same amount of time and effort. Obviously that doesn’t include the effort required to write the scripts… but that’s not effort. That’s fun. It’s been a whole series of interesting little coding tasks… which of course is the main reason I did it.

Review: Edirol UA-25 24-bit 96kHz 2in 2out USB soundcard

Friday 15th January 2010

I’ve become quite a fan of this sound device since I got it about a year ago.

For its price, the sound quality is excellent. It’s fairly packed with features, has a good range of options for input and output connectivity, plus MIDI. And it works flawlessly, out of the box, with Linux — no special setup or drivers required, ALSA knows what it is and how to deal with it in any mode.

The same is true of Mac OS, but only in the basic mode which restricts you to 16-bit 44.1kHz I/O – a driver is required for Advance mode to get up to 24-bit 96kHz support (either in, or out – we’ll come to this under Limitations). This driver can be downloaded free from the Edirol website, and seems to work fine on my new unibody Macbook Pro with OSX 10.6 Snow Leopard, though I haven’t used it extensively on there yet.

I guess it probably works in Windows too, but I wouldn’t know anything about that 🙂

The sound quality (for what I’ve used it for anyway) is very good. It’s stacked with features, and quite versatile… within certain limits.

First we’ll take a quick look at the features packed into this gadget, which is information you could probably find elsewhere but I include for ease of reference, and after that we’ll get to discussing those limitations in more detail.

(more…)