The Plural of Anecdote

Optimizing TensorFlow for your laptop

2017-05-28T00:00:00+01:00

I’ve recently been teaching myself TensorFlow, and haven’t spent the time and money to set up a cloud server (or physical machine!) with a GPU. I was originally running it from a pre-built Docker image, inside a Jupyter notebook, and saw a bunch of warnings like this in the console output:

W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't
compiled to use SSE3 instructions, but these are available on your machine and
could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't
compiled to use SSE4.1 instructions, but these are available on your machine and
could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't
compiled to use SSE4.2 instructions, but these are available on your machine and
could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't
compiled to use AVX instructions, but these are available on your machine and
could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't
compiled to use AVX2 instructions, but these are available on your machine and
could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't
compiled to use FMA instructions, but these are available on your machine and
could speed up CPU computations.

The pre-built versions available through Docker, pip etc. tend to go for wide compatibility, which means disabling a bunch of optional speedups that aren’t supported on all hardware. Thankfully, it turns out not to be too hard to build it from scratch yourself, which lets you switch all this good stuff on, and makes the process of experimentation and learning a bit less tedious.

This could also be useful if you’re training a model that’s just too damn big to fit on GPU hardware that you can actually afford, or if you want to train on GPUs but squeeze extra performance out of cheap CPU-based inference servers at query time.

Here’s a walkthrough of how I did it, on a 2016 Macbook Pro running Sierra (10.12.5). I expect the steps for Windows and Linux are similar. There are quite a few prerequisites, but it’s fairly likely you have some of these already.

Xcode

First off you’ll need Xcode if you don’t already have it, you can get this from the App Store.

Python

TensorFlow works with various different versions of Python, I believe, but I was using the Anaconda distribution of Python 2.7.

Homebrew

You’ll need this if you don’t have it already, in order to install Bazel (see below). Download it from here.

Java

Bazel also requires JDK8 which you can get from Oracle. I used 1.8.0_131 which was the latest at time of writing, but I don’t think it matters too much.

Bazel

Bazel is Google’s build tool which is used to compile and package TF. Once Homebrew and Java are installed — you may want to log out and log back in again to make sure environment variables etc. are set up right — you can install Bazel by typing:

brew install bazel

TensorFlow

Finally we can actually install TensorFlow itself.

I created a Conda environment to work in, first:

conda create -n tensorflow
source activate tensorflow

Although I think this is a bit moot, as the pip install step (see below) seems to install it into your Conda site-packages and make it available in all environments automatically.

Then checkout TensorFlow from GitHub and cd into your local copy, and

./configure

to configure the build. You can switch on various optional features here, it’s probably fine to leave everything as defaults though. The actual options we’re interested in aren’t controlled here, but via command line params when you actually build it.

That’s done by typing the following:

bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-msse4.2 -k \
  //tensorflow/tools/pip_package:build_pip_package

This will compile TF itself, and also output a script to generate a Python package. Be warned, this will take a while! Well over an hour, probably more like two. I didn’t time it exactly.

On some platforms you need to add --copt=-mfpmath=both to the set of flags above, but recent versions of clang provided with macOS don’t need this, and will barf if you do.

Once it’s finished, you need to bundle it into a pip wheel — a binary distribution with a Python wrapper:

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

Then you can install this into your Conda environment. First, double-check your paths are set up right by typing which pip — it should be pointing to a version of pip in your anaconda install directory. Then type:

pip install /tmp/tensorflow_pkg/tensorflow-<blah>.whl

where <blah> is some long version string. Just tab-complete it — this should be the only .whl file in that directory anyway.

Now we can test it works! But before that, and this is important, cd out of the tensorflow directory to somewhere completely different, e.g. your home directory. If you don’t do this, when you try to import the package, Python will try to import it from the local directory and not the installed library, and fail with a cryptic error. This confused me the first time.

Anyway, to test it’s installed OK, run python or ipython, and:

import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))

If this prints a message with no errors, you’re good to go.

And we’re done! I didn’t do any specific speed comparisons but training and evaluating toy models was noticeably faster with the custom build than it had been before. Of course, some of this may be down to the Docker virtualization overhead, not just the CPU flags, but a win’s a win — and you save a load of memory by not using Docker too.

Interview with a Data Scientist

2015-06-06T00:00:00+01:00

I was recently interviewed by Peadar Coyle for his data science blog, Models are Illuminating and Wrong, as the latest in a series of features on people in the field. It’s archived here for posterity, but I would recommend taking a look at the whole collection.

1. What project have you worked on do you wish you could go back to, and do better?

The one that most springs to mind was an analytics and visualization platform called Palomino that my team at Pearson built: a custom JS/HTML5 app on top of Elasticsearch, Hadoop and HBase, plus a bunch of other pipeline components, some open source and some in-house. It kind of worked, and we learnt a lot, but it was buggy, flaky at the scale we tried to push it to, and reliant on constant supervision. And it’s no longer in use, mostly for those reasons.

It was pretty ambitious to begin with, but I got dazzled by shiny new toys and the lure of realtime intelligence, and brought in too many new bits of tech that there was no organisational support for. We discovered that distributed data stores and message queues are never as robust as they claim (c.f. Jepsen); that most people don’t really need realtime interactive analytics; and that supporting complex clustered applications (even internal ones) is really hard, especially in an organisation that doesn’t really have a devops culture.

These days, I’d try very hard to find a solution using existing tools — Kibana for example looks much more mature and powerful than it did when we started out, and has a whole community and coherent ecosystem around it. And I’d definitely shoot for a much simpler architecture with fewer moving parts and unfamiliar components. Dan McKinley’s article Choose Boring Technology is very relevant here.

2. What advice do you have to younger analytics professionals and in particular PhD students in the Sciences?

I was asked this the other day by a recent PhD grad who was interested in a data science career, so I’ll pass on what I told him.

I think there are broadly three kinds of work that take place under the general heading of “data scientist”, although, there are also plenty of exceptions to this.

The first is about turning data into business insight, via statistical modelling, forecasting, predictive analytics, customer segmentation and clustering, survival analysis, churn prediction, visualization, online experiment design, and selection or design of meaningful metrics and KPIs.

The second is about developing data-driven products and features for the web, e.g. recommendation engines, trend detectors, anomaly detectors, search and ranking engines, ad placement algorithms, spam and abuse classifiers, content fingerprinting and similarity scoring, etc.

The third is really a more modern take on what used to be called operational research, i.e. optimizing business processes algorithmically to reduce time or cost, or increase coverage or reported satisfaction.

In many companies these will be separate roles, and not all companies do all three. But you’ll also see roles that involve two or occasionally all three of these, in varying proportions. I guess a good start is to think about which appeals to you the most, and that will help guide you.

Don’t get confused by the nomenclature: “data scientist” could mean any of those things, or something else entirely that’s been rebranded to look cool. And you could be doing any of those things and not be called a data scientist. Read the job specs closely and ask lots of questions.

3. What do you wish you knew earlier about being a data scientist?

Well, I wish I’d taken double maths for A level, all those years ago! As it was, I took the single option, and chose the mechanics module over statistics, something that held me back ever since despite various post-graduate courses. There are certain things that are just harder to crowbar into an adult brain, if you don’t internalize the concepts early enough. I think languages and music are in that category too.

(For our global readers: A-levels are the qualifications from the last two years of high school. You usually do three or four subjects, or at least you did in my day. You could do standard maths with mechanics or stats, or standard + further with both, which counted as two qualifications.)

I had a similar experience with biology — I dropped it when I was 16 but ended up working in bioinformatics for several years. Statistics and biology are both subjects that are much more interesting than school makes them seem, and I wish I’d known that at the time.

4. How do you respond when you hear the phrase ‘big data’?

Well, I used to react with anger and contempt, and have given some pretty opinionated talks on that subject before. It’s one of those things you can’t get away from in the enterprise IT world, but ironically, since I joined Etsy I’ve been numbed to the phrase by over-exposure… Just because the Github repo for our Scalding and Cascading code is called “BigData”.

It’s a marketing term with very little information content — rather like “cloud”. But unlike “cloud” I actually think it’s actively misleading — it focuses attention on the size aspect, when most organisations have interesting and potentially valuable datasets that can fit on a laptop, or at least a medium-sized server. For that matter, a server with a terabyte of RAM isn’t much over $20K these days. “Big data” makes IT departments go all weak-kneed with delight or terror at the prospect of getting a Hadoop (or Spark) cluster, even though that’s often not the right fit at all.

And as a noun phrase, it sucks, as it really doesn’t refer to anything. You can’t say “we solved this problem with big data” as big data isn’t really a thing with any consistent definition.

5. What is the most exciting thing about your field?

That’s an interesting one. Deep learning is huge right now, but part of me still suspects it’s a passing fad, partly because I’m old enough to remember when plain-old neural networks were at the same stage of the hype cycle. Then they fell by the wayside for years. That said, the concrete improvements shown by convolutional nets on image recognition tasks are pretty impressive.

Time will tell whether that feat can be replicated in other domains. Recent work on recurrent nets for modelling sequences (text, music, etc.) is interesting, and there’s been some fascinating work from Google (and their acquihires DeepMind) on learning to play video games or parse and execute code. These last two examples both combine deep learning with non-standard training methods (reinforcement learning and curriculum learning respectively), and my money’s on this being the direction that will really shake things up. But I’m a layman as far as this stuff goes.

One problem with neural architectures is that they’re often black boxes, or at least pretty dark grey — hard to interpret or gain much insight from. There are still a lot of huge domains where this is a hard sell, education and healthcare being good examples. Maybe someone will invent a learning method with the transparency of decision trees but the power of deep nets, and win over those people in jobs where “just trust the machine” doesn’t work.

6. How do you go about framing a data problem - in particular, how do you avoid spending too long, how do you manage expectations etc. How do you know what is good enough?

It took me a long time to realise this, but short release cycles with small iterative improvements are the way to go. Any result that shows an improvement over your current baseline is a result — so even if you think there are much bigger wins to be had, get it into production, and test it on real data, while you work on its replacement. (Or if you’re in academia, get a quick workshop paper out while you work on its replacement!)

This is also a great way to avoid overfitting, especially if you are in industry, or a service-driven academic field like bioinformatics. Instead of constantly bashing away at the error rate on a well-worn standard test set, get some new data from actual users (or cultures or sensors or whatever) and see if your model holds up in real life. And make sure you’re optimizing for the right thing — i.e. that your evaluation metrics really reflect the true cost of a misprediction.

I worked in natural language processing for quite a while, and I’m sure that field was held back for a while by collective, cultural overfitting to the same-old datasets, like Penn Treebank section 23. There’s an old John Langford article about this and other non-obvious ways to overfit, which is always worth a re-read.

Original version at Peadar’s blog

Kale: timeseries data mining at Etsy

2015-02-21T00:00:00+00:00

I’m very pleased to announce that my talk proposal on Kale v2 has been accepted by both Monitorama 2015 and Berlin Buzzwords 2015. Here’s the abstract.

Signatures, patterns and trends: Timeseries data mining at Etsy

Etsy loves metrics. Everything that happens in our data centres gets recorded, graphed and stored. But with over a million metrics flowing in constantly, it’s hard for any team to keep on top of all that information. Graphing everything doesn’t scale, and traditional alerting methods based on thresholds become very prone to false positives.

That’s why we started Kale, an open-source software suite for pattern mining and anomaly detection in operational data streams. These are big topics with decades of research, but many of the methods in the literature are ineffective on terabytes of noisy data with unusual statistical characteristics, and techniques that require extensive manual analysis are unsuitable when your ops teams have service levels to maintain.

In this talk I’ll briefly cover the main challenges that traditional statistical methods face in this environment, and introduce some pragmatic alternatives that scale well and are easy to implement (and automate) on Elasticsearch and similar platforms. I’ll talk about the stumbling blocks we encountered with the first release of Kale, and the resulting architectural changes coming in version 2.0. And I’ll go into a little technical detail on the fingerprinting algorithms we use for fast approximate querying of metrics and their associated statistical metadata. These techniques have applications in clustering, outlier detection, similarity search and supervised learning, and they are not limited to the data centre but can be applied to any high-volume timeseries data.

If you can’t make it to either of these, videos will hopefully be available online soon afterwards.

You sure don’t look like a scientist

2014-10-12T00:00:00+01:00

You get a little tired of people asking what you do, when the answer’s “computational linguistics research in a bioinformatics group at the Institute for Structural Molecular Biology.” There are enough unfamiliar terms in there to derail a light social conversation several times over, especially when you don’t usually enjoy talking about work at parties. So I got into the habit of saying “I’m a scientist” and glossing over the details unless people were interested.

The reactions from laypeople tend to run the whole gamut from intrigue to wariness and even, back in the days, occasionally outright hostility. Ten years ago, the 90s “Frankenscience” narrative was still lingering around, and the popular images associated with bioscience included GM crop protests, ears grown on mice, Dolly the sheep, and the horrific undercover pics from Huntingdon Life Sciences that the ALF put on every high street. Sometimes even fellow academics I met from arts and humanities departments would totally change their demeanour when they heard a trigger word like “lab.” I always felt particularly disappointed when that happened.

Another pretty common response was “Really? You don’t look like a scientist.” Now this usually just amused and exasperated me in roughly equal measures — and still does occasionally. But one time it nearly got me deported.

Boston manners

It’s 2004, and I’m arriving in Boston for HLT-NAACL‘04. I’m in the first year of my PhD, going to my first international conference — and my first trip outside Europe for any reason. It’s the height of summer, and I’m in combat shorts and a sleeveless t-shirt as usual. I get to the front of the immigration queue and hand over my documents. “Business or vacation?” asks the border guard.

Neither really, I think to myself. “I’m going to an academic conference,” I say instead, and give its full name. “I’m a PhD student.”

This wasn’t in the script. She stops pecking at her keyboard, squints at me over her glasses and chews this over for a moment. “You’re a scientist?” Nod. “You coulda fooled me. You sure don’t look like a scientist.”

I’m a bit floored by this — particularly as there’s an unsettling mix of contempt and distrust creeping into her voice. I can’t think of anything to say that isn’t hazardously sarcastic, so I wave the conference details printout at her. She glances at it, not really interested, and calls over a colleague. “This guy says he’s a scientist, huh. Going to a conference.”

“Oh really?” chuckles the guy. “Whereabouts?” The Park Plaza. He laughs a bit more. “They won’t even let you in the Park Plaza looking like that! Now please come with me.”

So he leads me to an inspection booth, and somewhere along the way, acquires another goon with a very large sniffer dog.

Waive goodbye to your rights

By now I’m starting to get seriously worried. Why the disbelief? I’m in my work clothes. I’ve got a slightly punky haircut and a couple of tattoos visible, but nothing you wouldn’t see in London every day. And this is Massachusetts, not Arkansas, right?

But more importantly — I’m travelling on a visa waiver form. This gives you exactly no rights whatsoever. You can be summarily deported even if your papers are entirely in order, with no means of appeal. And I have absolutely no idea what happens then. Do they pay for your flight home? Or are you detained until you can arrange a ticket? As a student I have no spare funds to draw on, my parents are on holiday, and my girlfriend (now wife) is delerious with the ‘flu at home.

And a deportation makes it very hard to ever get into the USA again. With the American authorities still in over-compensation mode post-11/9, there’s a very real chance my academic career could get hobbled here, before it’s even started.

I almost hit panic level when Laughing Guy snaps on the rubber gloves, but it’s just to rifle through my bag. Meanwhile Dog Dude is going “Find it! Find it! Find it!” and his hound is all over me. British sniffer dogs tend to be small friendly beagles or similar — cute and floppy-eared — not this Rottweiler-sized beast. (Bit vague on breeds I’m afraid — not a dog person.)

And then just like that, it’s over. They don’t phone the conference or my college for confirmation, as I suggest. But suddenly my suspicious credentials are just fine. They don’t even search me all that thoroughly, which is obviously a blessing, but makes me wonder if they actually expected to find anything, or just wanted to put me in my place. Welcome to America.

Bio diversity

From talking to friends and colleagues at the time, and discussions online more recently, “you don’t look like a scientist” is a pretty common reaction. I’d hope it’s much less common today, with the likes of Ben Goldacre, Simon Singh, Alice Roberts, Jim Al-Khalili and of course Brian Cox putting a more realistic face on science in the media, and I’d really hope the consequences of that reaction are typically much less extreme than my story.

But there’s still a long way to go. Women scientists are still under-represented in the media, and our most prominent and influential is frankly turning out to be a bit of a crank. Dawkins has retired from his Public Understanding of Science post in order to pursue his hobby of Public Misrepresentation of Science full-time, which isn’t doing anyone any favours. Of course, this is very much a UK-centric view, and without the BBC and the Guardian things would look a lot less encouraging.

And let’s not forget that presence of visible role models and a more welcoming atmosphere for younger scientists won’t necessarily equate to better career prospects. Recent UK figures show that only 22% of professors are women and less than half a percent are black. These numbers are across all subjects but I doubt sciences fair any better than others.

I have no idea what the picture’s like in the States, but I haven’t been detained there recently, so maybe there’s hope yet.

Updated 2014-10-13 to add new figures from THE article

The unwisdom of the crowd: marketing, metrics & machine learning

2014-06-16T00:00:00+01:00

Or, why Google Analytics is not a good analytics tool.

This is a slightly modified repost of an article I wrote for an internal Pearson blog recently. My colleague Tendayi encouraged me to post it here, so it could reach a wider audience.

“Vanity metrics” is a phrase I’ve heard cropping up a few times recently, in the context of growth engineering, the lean startup movement, and discussions around product lifecycles. It’s hard to find any agreement on what metrics are vanity metrics, but in general it seems to refer to metrics that product owners measure and report in order to boast about growth, but which don’t themselves prove anything about value or guide decision-making. Numbers of logins, of resources viewed, of content items downloaded, or of messages sent — these might be considered vanity metrics, depending on the product. This seemed a bit strange to me, because in my world, all these metrics are extremely valuable.

Suddenly it occurred to me this morning: when product and marketing experts talk about these metrics, they’re almost always referring to aggregates — counts or averages across all users, or specific predefined segments. But when data scientists talk about them, we’re often talking about individual users.

So why would I want to record and measure login counts or download counts for individual users? Or in the education domain, where I work: why count learning resources viewed, time taken on tasks, etc.? Well, all of these things are essential components of a user profile, and without a good user profile, you can’t do things like:

Predict which users will suffer poor educational outcomes
- Contact those with poor predicted outcomes, to offer them more support
Recommend content to users
- Educational tasks that are likely to improve their assessment scores
- Purchases they’re likely to make
Score users for loyalty
- Target low-scoring users in order to reduce churn
Estimate the probability of a given user responding to an offer
- Tailor offers to specific users in order to lift acceptance rate or overall revenue
Explain negative trends in the behaviour of certain users
- Substandard educational outcomes
- Failure to renew subscriptions
- Low click-through rate on recommendations

… and so on.

Some of these examples are specific to our area of business, and I’m sure you have similar domain-specific challenges, but others apply across the commercial web. And all of them need activity data down to the level of individual users. Something that may seem like a “vanity metric” when reported across all users — number of logins per week, for example — can be a very important indicator of an individual user’s engagement, when it’s fed into a machine learning model for risk scoring.

At the top of this article I mentioned Google Analytics. I don’t want to pick on GA specifically, as this is common across web analytics tools in general, but I sometimes hear product owners saying “we don’t need a way to record user activity, we have Google Analytics already.” But this misses the point that GA is designed to report aggregates across all your users, or across segments that you’ve defined in advance. Unless it’s changed considerably since I last used it, GA makes it very hard or impossible to report these metrics for individual users, or for retrospectively-defined groups of users that are interesting because of some shared (usually negative) behaviour. And it also makes it hard or impossible to connect the activity of individual users with other elements of a user profile, such as demographic information, educational records, purchase history and so on.

So please take the term “vanity metrics” with a pinch of salt. A particular KPI may be worth very little when reported at the level of your entire user base, but it might be the differentiator between a loyal customer or satisfied learner, a borderline case who needs help and support right now, or a lost cause.

All your Bayes are belong to us! PyMC, PyStan and emcee in Snake Charmer

2014-06-04T00:00:00+01:00

PyMC is the most widely-used Python package for Bayesian modelling, learning and inference, but it isn’t the only choice, by far. Jake Vanderplas recently posted a pretty detailed comparison of PyMC, PyStan and emcee on his blog, and even if you’re not an expert in Bayesian methods (I’m certainly not) I’d recommend you give it a read. It walks you through the same task in all three packages in order to get a feel for how they work, which is a great way to do it.

As well as functionality, speed and accuracy, Jake also covered installation, and mentioned that PyStan can be painful to install, especially if you don’t already have a full C/C++ toolset. While many Linux users do, these days a lot of Mac users don’t, and I’d hazard a guess that most Windows users don’t, even if they program in other languages frequently. Many Python packages need to compile native code during installation, but some of them get round this by offering binaries for various platforms — although this often not a foolproof solution either. But PyStan also uses the compiler at runtime, in the process of translating its own model definition language into machine code, so I guess there’s no easy way for them to build a one-click installer.

This is exactly the kind of reason why I started Snake Charmer in the first place. Installing it and getting the tests passing on Snake Charmer’s Ubuntu VM was — well not entirely trivial, but an hour or two at the very most. And once it works for me, it works for everyone. You’ll now get PyStan and the slightly less fiddly emcee as standard with every Snake Charmer VM — so, err, forget your unpleasant prior experiences and raise your expectations. (Sorry…)

I think this makes Snake Charmer the first Python distribution to include all three of these tools.

One last note: if you try to reproduce Jake’s tutorial in Snake Charmer, you may notice some differences. He uses PyMC 2 but Snake Charmer’s on a pre-release of version 3 already. (Given limited time to work on this, I’d rather stay ahead of the game…) However, here’s a note from Thomas Wiecki showing the equivalent steps in PyMC 3.

Lyric clouds, genre maps and distinctive words

2014-06-04T00:00:00+01:00

This is a repost of an article I wrote for the Last.fm blog while working there in 2011. The images disappeared from Last.fm so I’m serving them out of the wonderful Internet Archive.

One of the interesting things that sets even superficially similiar genres of music apart is their lyrical content. Last.fm tags can overlap to a great degree, but we were interested to see what the words can tell you about the subtler shades of meaning that go along with those tags. As usual around here, the best way to answer questions like these is by asking the data.

So I downloaded the musiXmatch dataset, a collection of lyric tables for nearly 240,000 songs from all around the world (and the musical universe). They are tables in the sense that they don’t contain the intact lyrics of each song, but rather a list of words present in each song, along with the number of times that word occurs. No use for karaoke, but perfect for investigating the overall properties of a genre. I then matched up the songs in the dataset with tracks in our own catalogue, and correlated this with Last.fm tag data, in order to count the number of times a given word appeared in each of several prominent genres.

Lyric clouds

Of course, lists of words and frequencies are a little dry, but thankfully IBM have released a Word-Cloud Generator which can take a weighted list of words and display it graphically, as seen on the Wordle website. The more often a word appears, the bigger it will be rendered. Here’s what it came up with for the genres I tried — the software did the layout, but you can blame me for the font selection.

Click to open full images in a new window. Warning: they contain lyrics you may find offensive. Not safe for work.	Blues	Country

Electronic	Folk	Goth

Hip-hop	Indie	Metal

Rap	Rock	Soul

I did a bit of pre-processing to remove common ‘stopwords’ that don’t really hold any information about the topics of the lyrics (and, for, I, you, the, plus many more), but this only took into account English words — and if you look closely, you’ll see a few common words from German, French and Spanish (and probably others) that are from foreign-language songs in the dataset. But what’s most striking for me about these is not how much they differ, but in fact how often some of the words appear prominently across genres. Almost everyone sings about love, for example, with the exception of Rap and Hip-Hop, and time comes up… time and time again.

Genre maps

A limitation of word clouds is that while they’re great for showing the comparative popularity of words within a genre, they’re not so good for looking at the overall similarities or differences of several genres at once. To do that, you need some measure of similarity which can be rendered graphically as a kind of ‘genre neighbourhood map’. So I measured the similarities between the word lists for each genre, ranked by popularity, using a method which was developed to compare the result rankings from different search engines. This gives a single value for how similar the lyric choices are between each pair of genres, where differences towards the top of the lists (the most popular words) are considered more important than differences further down. A bit of extra number crunching in R can convert these similarity scores into a 2D map, which I imported into OpenOffice to render:

Click image to open larger version in a new window.

This map is really interesting for its combination of expected and unexpected neighbours, and also for the way it clearly shows Rap and Hip-Hop as outliers from the main axis on the left. Goth and Metal, which may appear similar to the un-trained ear (and eye!), are considerably separated, while Metal and Folk are — surprisingly — much closer. Electronic (a very broad tag) is clustered together with Soul and Blues, presumably because of the soulful origins of house music, which is one of the more lyrical electronic sub-genres. And Rap and Hip-Hop, which might be considered synonymous by the layman, are about as different as Indie and Country in terms of lyric ranking.

Distinctive words

The word clouds as shown draw the viewer’s attention to the very frequent words, but these also tend to be the ones like love and time which are popular across genres. This is a problem if you want to find out which words are most distinctive or characteristic of a given genre — the words which, if used as search terms for example, would be best at selecting songs from that genre correctly (true positives), while minimizing the number of songs retrieved from other genres (false positives). Once again, information retrieval (the science behind search engines) can help us — the F measure or F score is specifically designed for measuring the tradeoff between true positives and false positives in a set of results. It’s a score between 0 and 1, where 0 means “no relevant documents retrieved”, but 1 means “all relevant documents retrieved” and “no additional spurious documents retrieved”.

So I calculated the F score that each word would have as a search term for each genre in some notional lyric-based search engine: “how relevant would the results be if I searched for Indie tracks with the search term friend” for example. This doesn’t take into account the number of times each word occurs within a song, just the fact that it occurs at all, but it does let us redraw the lyric clouds with each word’s size determined by its F score for that genre. As you can see, this brings out the words that are characteristic of each genre, rather than emphasizing those that are globally popular:

Click to open full images in a new window. Warning: they contain lyrics you may find offensive. Not safe for work.	Blues	Country

Electronic	Folk	Goth

Hip-hop	Indie	Metal

Rap	Rock	Soul

I think they bring out the unique character of each genre much more effectively, and the variation in size between the words is much less, so the less prominent words are easier to see. There are some interesting quirks visible too. For example, many German words are much more clearly visible in the Goth cloud than they were before, reflecting both the comparatively large number of songs in German in that genre, and the lack of German lyrics in most other genres. Country for example is entirely English.

Finally, a little extra present from the data. The word with the highest F score in the whole dataset is Christmas, with an F score of 0.3892 for the tag… Christmas. So, unseasonal greetings from the data crunchers here at Last.HQ!

Thanks to musiXmatch for making the lyric database available, and Thierry Bertin-Mahieux for helping me to reconstruct the full words from the stems in the database.

Snake Charmer: the all-in-one data science toolbox for Python 3

2014-06-04T00:00:00+01:00

Python’s an amazing language, but it’s constantly let down by its packaging and distribution infrastructure. Apologies to those people who’ve worked hard to fix this, but it sucks. I’ve wasted days in the past getting a particular set of packages to cooperate on a particular machine, only to find that the resulting combination of library files, symlinks, chicken blood and chanting doesn’t work on any other machines.

The virtualenv tool helps a bit, by letting you have entirely separate Pythons, different versions if necessary, with their own sandboxed libraries and tools. But not all packages behave well in virtualenvs. Some are just sloppily written, but others have very specific dependencies on libraries supplied by your operating system or compiler toolchain, which may be written in C or Fortran or Ancient Enochian, and can’t easily be localized to a virtualenv. Some even have specific hardware requirements, or platform-specific behavioural quirks.

This is particularly true in scientific and statistical computing, where some of the most popular Python tools make use of low-level libraries for linear algebra or parallel computing, and others rely on OS-dependent features for image rendering or interprocess communication. And behavioural differences and incompatibilities can be particularly pernicious — maybe your code will run without any errors, but quietly produce slightly inaccurate predictions.

Meanwhile, in Shoreditch…

Thankfully, there’s a solution to many of these problems, if we borrow an idea from our much hipper web developer friends. These days it’s quite common to build standardized, pre-packaged virtual machines (VMs) for developers to work in, meaning your code has the same runtime environment as all your teammates’ code does, and this closely resembles the servers you will eventually deploy to. By running a little server inside each developer’s laptop or desktop, with its own OS and emulated ‘hardware’, you mask the differences between all those machines and the software installed on them.

Who cares if Alice has Mountain Lion on a MacBook Air, and Bob has Lucid Lynx on a water-cooled tower PC with a Cylon-themed case mod? When they’re building code and running tests, they’re both using the exact same version of Debian that their server farm uses. So the scope for incompatibilities is hugely reduced. But Alice and Bob still get to use the same text editors, IDEs and other dev tools they rely on — their VMs augment the capabilities of their original OSes, but don’t replace them. This is not really a new idea, but end-user virtualization tools like Vagrant and automation frameworks such as Salt have made it much easier.

The science bit

My goal with Snake Charmer is to provide the same hassle-free experience to scientists, engineers, statisticians and data miners. Any two VMs built from the same Snake Charmer config should behave exactly the same, even if they’re on a completely different hardware or OS platform. And installation needs to be as smooth as possible. When you run Snake Charmer, it does the following, entirely unsupervised, through the mechanisms provided by Vagrant and Salt:

Downloads a VM image for a specific version of Ubuntu
Boots this up in VirtualBox
Installs a precise versioned set of Ubuntu packages, including Python 3.4
Installs an equally exact list of Python packages, patching them where necessary so they play well together
Starts up an IPython Notebook server on the VM, and gives you its URL

The packages installed include NumPy, SciPy, Pandas, IPython, Matplotlib, Seaborn, scikit-learn, PyMC, statsmodels, PyTables, SymPy, Numexpr, Theano, DEAP, gensim, NLTK, Beautiful Soup, Cython, Numba and more. And it even comes bundled with R and Octave, and connectors to plug these into IPython — on the slim chance you find something that you really can’t do in Python.

Of course, since it’s just a standard Ubuntu box running standard Python, you can install whatever other packages you need, from the Python package index, the Ubuntu repositories or elsewhere. And if there’s a package that you need all the time, you can just add it to Snake Charmer’s config files so that it gets installed automatically any time you need to build a fresh VM. But that’s just the tip of the iceberg. The fact that these are complete virtual machines, not just software packages, brings all kinds of added benefits.

Big fish, little fish, virtual box

Here are some of the useful things you can do with VMs, either via Vagrant commands, or through VirtualBox’s admin app.

Disposability. Normally, you wouldn’t store data or code within the VM, you would use it to run notebooks and read data files from your physical computer. This means if you mess up your VM somehow, you can just delete it and create a new, identical one. Think of them as disposable commodities that aren’t even worth fixing.

Snapshots and rollbacks. On the other hand, you might want to make some persistent changes to a VM, like trying out a new package, or a new version of a library. You can take a snapshot of a whole VM before making a change or taking an action, and if necessary, undo it by rolling back the whole VM to the snapshot. You could also use this feature to save the state of the VM at intermediate points along an analysis pipeline. It’s also a good idea to make a snapshot after you first create a fresh VM, as rolling back is much quicker than installing a new one from scratch.

This process can be managed via a Vagrant plugin or through VirtualBox itself.

Shareability. You can bundle up your VM into a redistributable package, including any changes you’ve made to it, and then send it to your colleagues, or make it available for download. Plus you can include data, notebooks and scripts within the package, meaning your entire workflow can be replicated. I believe this has lots of potential in education and in scientific publishing, beyond the more obvious uses in academic and commercial R&D environments.

Collaboration. Using Vagrant Cloud, you can easily let remote users connect to your VMs over the internet — either securely authenticated individuals, or anyone with the URL. This is a natural extension of IPython’s broadcasting features, and could be used for collaborative working, running demos, presenting results, or teaching. In general, it won’t be affected by firewalls, broadband NAT etc.

Resource management. You can put hard limits on the amount of memory, CPU and storage space a VM can consume, so a runaway process can’t crash your whole computer. (We’ve all been there…) And you can suspend a running VM, freeing up all the resources it’s using, even if it’s in the middle of a long-running task — then resume where you left off later.

Elasticity. If your resource limits are too tight, for example there’s not enough RAM to build a model in, you can extend them any time. If there’s not enough physical RAM, CPU or disk space in your computer, you can move the entire VM to a server — or simply create a new, identical one there. With a bit of extra work you can even host a Snake Charmer VM on cloud services like Amazon EC2 and professional virtualization platforms like VMWare. And you could even create an IPython cluster of identical VMs for parallel processing.

What next?

It’s still early days for Snake Charmer, which currently provides one version of Python (3.4) on one version of Ubuntu (12.04), and installs over 25 data science packages. I want to keep adding new packages on a regular basis, and stay reasonably up-to-date with new versions of Python and Ubuntu. I plan to create a Python 2.7 build too. At some point in the future it might be possible for me to host pre-built VM images for download, but that would need a sponsor, as I want to keep it entirely free.

To get started, clone the git repository, and follow the steps in the README. All you really need to do is install VirtualBox and Vagrant, and run one command. Please let me know how you get on, either here or on Twitter.

Note: A previous version of this post talked about ‘official releases’ but I’ve since decided not to do numbered releases. The reason being, it gives a false sense of permanence. A third-party package listed in a numbered release can still disappear from a repo which will break a numbered release. It’s probably better just to ensure that master at any point in time is a valid build. Happy to hear any alternative ideas you might have…

Lies, Damned Lies and Dataviz

2014-06-03T00:00:00+01:00

I recently gave a talk at the Big Data Innovation Summit called Lies, Damned Lies & Dataviz: Bad visualization, and how to avoid it. The aim: educate and entertain. I’m not above resorting to mockery when a graphic is particularly incompetent or manipulative, and I’ve never had that many laughs from a conference crowd before.

Pearson Labs asked to write up a short blog post version which is here — enjoy. And if you liked that, I’ll be revisiting the same subject at a Skills Matter event in November.

Introduction

2014-06-02T00:00:00+01:00

Releasing Snake Charmer has inspired me to start blogging again, so welcome to the new blog.

I’ve had a couple of Wordpress blogs in the past, but this time I’ve decided to give Pelican a go, for a bit more control. And because Markdown and Python.

Pelican’s basically a toolkit for rendering source documents (Markdown, reStructuredText, AsciiDoc) as static HTML pages, and generating all the usual blogging extras like index pages, category pages and so on. It’s very flexible, hackable, scriptable and command-line-driven, which is ideal for me. So let’s see how it goes.

A note on the blog title, in case you’re interested. There’s an often-quoted quotation that goes “the plural of anecdote is not data,” referring to the comparative credibility of different kinds of evidence. Or at least, many people seem to think that’s how it goes — but without any particular reason to believe so. Ironically, the original quote was “the plural of anecdote is data” (my emphasis).