Colin Green

Is the Netflix Prize winnable? (2)

cgreen@dsl.pipex.com — Tue, 29 May 07 21:52:00 UT

Some more thoughts on the subject I wanted to note down before I forgot them, as I'm keen in giving this experiment a go soonish.

The SVD should probably be run with the regularisation fact (k) set to 0. Actually it migth not matter but you get the maximum amount of overfitting with k=0 and the RMSE becomes less overfit as we supply more data to the algorithm.

We also need to keep the size of the feature arrays (vectors if you prefer) the same size for each data subset, so this means we need to keep the number of movies and customers constant. This presents a problem for the smaller sample sets were we have, say 0.78m ratings spread out over 0.48m customers, that's 1.625 ratings/customer on average, not nearly enough to get a good idea of a customer's preferences. This probably means I'm going to end up with a few less sample points on my RMSE convergence plot, say 6 points now instead of 8 (loose the two smallest data sets).

It's also worth noting at this point that Netflix's 1 billion results are of course going to be made up largely of ratings on customers not in the netflix data set. This means they have an even larger matrix than the prize data set, but not necessarily a matrix that is significantly less sparse.

Is the Netflix Prize winnable?

cgreen@dsl.pipex.com — Mon, 28 May 07 18:12:00 UT

Over the last 3 months I've been working on (or competing in) the Netflix Prize (www.netflixprize.com). Actually I came to the party late as the competition started in October '06 and (alias)Simon Funk posted a blog entry describing his SVD (Singular Value Decomposition) technique in December '06. Simon's entry was something of a bombshell as most competitors, if not all, are now using some variation on it, at least in part.

I'm fairly proud of my own SVD algorithm written in C#/.Net 2.0 as it crams the Netflix data consisting of 100 million ratings into just 275 MB, this packing of the data actually improves performance because each rating is used on each iteration(epoch) of the SVD alogorithm, thus it is effectively a memory bandwidth bound algorithm. As evidence of this I find that running two SVD algorithms together on a dual core machine results in both running at half speed (or thereabouts), basically because each core spends most of it's time twiddling it's thumbs waiting for data to arrive on the memory bus.

This along with some knowledge of optimisation and cutting edge PC hardware (Intel Core 2 Duo) means I can perform one SVD iteration in under 2 seconds. Pretty darn fast.

Even so my current setup requires at least 120k epochs to get a good result, and in fact some recent results are suggesting that more like like 200-300k+ epochs are in order to get the best possible result.

So while waiting for one of these runs to complete I got thinking about whether this competition is actually winnable. It's a subject that has come up a few times in the Netflix Prize forums.

Basically the goal is to get the root mean squared error (RMSE) of your predictions <=0.8563. Netflix's own Cinematch algorithm scores 0.9514 on the competition data, my own SVD algorithm scores about 0.9060 right now and the top entry on the leaderboard at time of writing is 0.8808, pretty good. As a reference point just using the average rating of each movie for all predictiosn gives an RMSE of abut 1.06 and some basic refinements on that approach (as outlined by Simon Funk) can get you down to 0.9840.

So the whole competition is basically compressed into that tiny range between 0.9840 (the easy to obtain baseline RMSE if you like) and 0.8563 (the million dollar prize!). That's a range of 0.1277. In those terms the prize seems teasingly close, but of course the RMSE score is subject to the law of diminishing returns - expend twice as much effort to achieve half as much gain. So as the leading score inches forward it's easy to come to the conclusion that they are simply approaching some lower bound by smaller and smaller increments, and although near to 0.8563, never actually attaining it.

OK but a couple of times over the (relatively short) life of the competition there have been significant leaps in best RMSE as new entrants have come in with new algorithms. So who is to say the lower bound is actually some way off where we are right now, and below 0.8563? So are there any ways of getting an estimate for where this lower bound is? Maybe.

SVD old timers will be well aware of the problem of over fitting. Getting a much better RMSE on the training data set than on the probe set. Simon Funk introduced a 'regularisation' aspect to his algorithm to help alleviate this problem, but it is still very much present. Bascially Simon decays the SVD feature values by some small amount (generally referred to as K, a proportion) in the feature value update expression. Without this decay the probe RMSE actually starts to rise after a few features as the algorithm overfits to the training data. With a relatively small K of 0.015 each feature seems to last for more epochs before completing (no longer falling on the probe RMSE) and also gives a better probe RMSE at the end of each feature. Also the training set RMSE is actually higher as it is not overfitting quite so much.

Discussions on the Netflix Prize forums mostly seem to be referring to K=0.015 as a good near optimal value, which I had taken at face value. Some recent experiments though suggest that a higher K gives better results, albeit at the expense of longer clock time to achieve those results (more epochs required per feature). K=0.025 seems like a better K to me right now, 0.05 at first sight seems to be one step too far as the resulting probe RMSE per feature is now higher than that obtained for K=0.025.

But is 0.05 worse over the long term? My hunch is that higher values of K will result in runs taking longer and longer but ultimately reaching lower and lower scores. I already have some fairly minimal evidence to support this so lets run with it for now. Lets say that we can just keep on increasing K, eventually the plot of probe and traing RMSE might actually be very close to each other, the two plots will have (almost) converged. If we experimentally find where this convergence point is we might have found the lower bound for the competion RMSE, at least for SVD, and just maybe that lower bound will be <=0.8563, maybe.

Hmm, maybe not. Consider this, the baseline predictions (as set out by Simon) give a probe RMSE of 0.9840. We know that K=1 will prevent any learning and thus the probe and training RMSE converge at 0.9840. Reducing K now results in the probe and trainign RMSE falling but separating. Thus our convergence point isn't between the probe and training RMSE we observe in 'normal' runs, it is actually above those values, at the baseline RMSE!

Ok so lets take another line. SVD can achieve an RMSE of <0.8 on the training set with just a dozen features. We have approximated 100 million predictions with an RMSE of just 0.8, that's pretty good for 100m ratings. Presumably if we increase the number of ratings we train against that figure would go up, as overfitting would be reduced. If we double, quadrupled the # of ratings that RMSE would rise by less and less as it converged on (but never reached) the magic lower bound, but this time from below.

We don't have more ratings (Netflix do BTW) but we should be able to get a good idea of the relationship between # of ratings and achievable RMSE by running SVD with fewer ratings and extrapolating. Expect the curve of RMSE achieved by each subsequent increase in # of ratings to plot an exponential curve, doubling the # of ratings each time results in an increase of RMSE that halves each time, something like that.

So for number of ratings we could use...

100m
50m
25m
12.5m
6.25m
3.125m
1.5625m
0.78125m

I expect that the smaller sample sizes will be noisier, so going much below 1m probably isn't worth it, thus we are left with just 8 points on our curve, which should give some idea of where the convergence point is. In fact for the smaller sets we can sample multiple sets of that size from within the total 100m to help squash the noise. So maybe we could get a couple more points, or a less noisy curve for the above 8 points.

I haven't done this yet but it would be fairly easy to do, the one area I'm fuzzy on is ensuring the sub-sets are statistically similar, but I don't think it matters that much, so long as the sub-sets aren't wildly dissimilar.

Whether it's best to use base-2 or not I'm not sure, we could take a base 10 logarithmic scale, but then we're left with sample points at 100m, 10m, 1m, 100k. Which is not really enough points to extratpolate a trend from, especially if there is significant noise in there. Base-2 seems like a good place to start.

They think it's all over... well it is now

cgreen@dsl.pipex.com — Thu, 4 Nov 04 01:29:00 UT

Wow, where to start. The US election has been quite a show as viewed from across the Atlantic, it looked like it might descend into farce again but ultimately Bush won by a clear majority, which at least puts all of the allegations of dirty tricks to bed. I found Kerry's concession speach quite moving, I think there was almost a tear in his eye at one point when he talked about taking everyone in his arms, and how he had taken on board the struggles of all of the people he had met on his journey.

Bush's speech was IMO less spectacular, essentially repeating the call for unity from four years ago. The trouble with this is that Bush's general approach to political issues is to take a position, probably based on his religious faith and his advisers, and to deeply entrench himself in that position without really debating the issues or considering the alternative viewpoints. In that respect he leads very much like a medieval monarch, in that deep down I suspect that he believes he has divine right to rule. The logic goes something like this; God works in mysterious ways, I have become leader of a powerful country by whatever means, therefore God must have meant this to be. And then the scary bit is; if god wanted me to be leader then my viewpoints must be the correct ones, and thus everyone elses viewpoint is wrong. Henry VIII took this doctrine to an extreme by splitting with the Catholic church and adopting protestantism and thus changing history - mainly because he just wanted to get a divorce :)

Far fetched? Whether you agree with the analysis or not, it is a point of record that Bush selects a position and sticks to it, and if no-one else shares that position, which is often the case, then that is their problem. Kyoto, Iraq and Israel/Palestine being notable examples. The same pattern is also repeated in his governance of Texas - the policies he adopted out of his black and white viewpoint on teenage pregnancy resulted in Texas having the fifth highest teenage STD and pregnancy rates in the US in 2000.

One thing I will concede is that I have arrived at the opinion that the idiot image of Bush is not accurate. There are times when his performances are dire, for confirmation look no further than the first presidential debate or his personal message to the Iraqi survey group. But on other days he performs well in front of a crowd, he tells them what they want to hear and makes the right signals. His policies *are* backed with an intellectual rigour, the problem being IMO, that the viepoints and logic that underpin the intellectual analysis are wrong, and quite fundamentally so in some cases.

As powerful as the US is it cannot control everyone, and therefore in the long term his entrenchment tactic will probably cause more divisions in the world as a whole which in term will generate bad will and possibly more terrorists. More significantly he risks creating a bipolar political and economic world in which the US inhabits a lonely political space distinct from and at odds with China, Europe and Asia. In an odd twist of fate it's possible that Russia could become the USA's closest friend - the emerging run-away capitalism coupled with the a non-invasive legal framework is not disimilar to the political and economic environment that gave birth to the idea of the 'American Dream'.

My final and perhaps most important point is that of economic management. The economy was of course an issue in the recent debates but it was never center stage, which arguably it should have been. The main points made were that the US economy is strong and that employment is steady (actually slightly down, but still high). The problem is that this success level has been achieved on the back of massive borrowing. Under Bush the US has been spending more money than it has, right now both the US government and consumers are overspending by about 3% per anum. In short the US consumes more than it produces. The gap is filled by borrowing from foreign nations. At some point in the future this debt will need to be repaid, which means cutting spending and/or increasing taxes, which in turn will likely put the breaks on economic development. If the debt continues to increase at the rate it has done over the last four years then economic realities will set in - the value of the dollar will fall further, thus making foreign produce (that the US imports a lot of) relatively more expensive, which again has a detrimental effect on the US economy and the US people's overall quality of life. It is my opinion that if this sole issue is not addressed soon that it will result in the Republicans loosing the next election. Clinton you will recall balanced the books well thus affording Bush a considerable economic leeway, but that leeway surely cannot last another four years without serious economic implications.

Bush conspiracy theory details filled out.

cgreen@dsl.pipex.com — Fri, 29 Oct 04 11:02:00 UT

The US election is receiving a seemingly unprecedented amount of coverage in the worlds press, which I would have to say is largely brought on by the policies of this chap, his administration and their foreign policies.

His verbal blunders during his first term in office have been the focus of much mocking and satire around the world, according to a discussion on Radio 4 recently this has been particularly noted in the Italian, French and some South American press. China has been neutral, as decreed from above. Russia I don't think was mentioned, oddly.

Much of this mocking entered a new level following the presidential debates. Having watched parts of the debates I have to admit that George Bush's demeanor is quite perplexing. Charlie Brooker, the Guardian's acerbic TV critic (of which I'm a big fan BTW) put aside TV banter for one week to comment on the debates, I quote:

"So I sit here there and I start scratching my head, because I'm trying to work out why Bush is afforded any kind of credence or respect whatsoever in his native country. His performance is so transparently bizarre, so feeble and stumbling, it's a miracle he wasn't laughed off the stage."

A sentiment I have to agree with, where do I start. He does this thing were he emits a burst of facts and then smiles in a way reminiscent of a nervous child at a bring and tell, looking for approval from the teacher. His verbal blunders are astounding, in his ending speech in the 3rd debate there is one word that he just gives up on altogether half way through saying it.

At the end of Brookers article he crosses the line bigtime by jokingly hinting that the world might be better off if Bush were assasinated. drudgereport.com picked up on this and it wasn't long (a few hours) before complaints were made, an apology was published and the article was withdrawn (from the net anyhow). Some of the more radical inhabitants on the net were asking for his extradiction, amongst other less tasteful things which I shall not relay here.

So what can I summise from this performance? Well I've been looking at footage of his past speeches were he seems perfectly eloquent while debating as governor in 1994 and during his inauguration speech in 2000. But of course these are largely pre-arranged, scripted speeches that he is reciting. One exception to this rule is his personal message to the Iraq Survey Group where he appears extremely anxious and just not, well, with it:

http://www.dailykos.com/story/2004/10/15/20363/328

Watching the presidential debates I put his demeanor down to being anxious, perhaps the combination of speaking to such a large audience, so much being at stake *and* having to debate with someone who might catch him out. Essentially this would boil down to a lack of confidence, in his ability to debate and perhaps in his level of knowledge?

Another explanation for the wide variation in his performances would be medication - many types of medication have a sedating effect and this can definitely be a cause of anxiety if you know you have to speak in public and might not have the mental accuity to cope with everything that is 'thrown' at you.

Alternative explanations discussed on conspiracy type web sites are that he has suffered a minor stroke at some point - citing the apparent drooping of the left side of his mouth (It looks to me though that the drooping was already there in 2000, bu tnot in '94) - or that he is suffering from pre-senile dementia . Certainly if you look at his performance in 1994 he appears far more confident than in any recent footage and speaks far more eloquently.

So who know's what's going on. What I do know is that if I were judging Bush by those performances alone, I would have serious doubts about his mental state and thus ability to govern.

This then leads to the whole 'Mystery Bulge' thing. The press have brushed it off but the speculation on the net has been rife, some of it is just junk conpiracy theories, but there is some intelligent discussion going on. The bulge could be there for legitimate reasons, a radio receiver might be in place so that security can quickly notify him of a threat, some kind of bullet proof vest, or perhaps a medical device. So to me this would have been a non-issue, except for the fact that Bush denies that any such device was in place and that the bulge was simply a crease in his shirt - which it clearly is not upon close examination of the footage. There is also a wire clearly visible behind his tie for several frames of the 2nd (I think it was) debate. It could well be legitimate, but why the denial which then allows conspiracies to fester on the net?

I then can't help thinking that the bulge story ties in with his possible deminished mental state - is someone assisting him through an earpiece? Some have discounted the whole theory stating that radio receivers are far smaller than the bulge, which is true, except that if he really were receiving an audio feed it would almost certainly be an encrypted digital signal via a spread spectrum rf carrier - why? because otherwise anyone could listen in and prove the feed's existence, also a spread spectrum carrier is hard or impossible to pick up on with a standard radio scanner or frequency analyzer, and they are far harder signals to 'jam'. Receiving such a signal and converting it to audio cannot be done within a miniturised earpiece, a second device would be required to receive and decode the signal and then relay it to the earpiece, probably via a very low power inductance loop similar to those used in cinemas, a low power loop would prevent pickup by nearby microphones. To power all of that for 90 minutes would require a reasonable quantity of batteries. So all of this speculation ties in with the bulge and the wire that is visible running up his right shoulder (to an inductance loop as close to his ear as possible):

http://www.salon.com/news/feature/2004/10/29/bulge/index_np.html

If we follow this theory through the bulge would be the receiver/relayer, the wire up the right shoulder is to an inductance loop, perhaps in his collar, and it also looks to me that there is a wire travelling down his back, which would correspond to the wire seen behind his tie. This wire that we can see is actually quite prominent by way of its thickness, say 5mm. Such thickness would only normally be used to convey power - from a battery pack somewhere on his person, to the receiver on his back. Also note that the tie wire disappears into the left hand side of Bush's jacket - the same side as the wire seen down his lower back.

It's all pure speculation on my(and web conspirators) part of course. If I had to put money on it then I think that I would say that he is indeed recieving an audio feed. It may well be part of his security, but since he's not telling there is always the possibility that it's not.

John Peel R.I.P.

cgreen@dsl.pipex.com — Wed, 27 Oct 04 18:18:00 UT

Was out walking and listening to the World Service when I heard the news. JP died of a massive heart attack while on a working holiday in Peru.

For the last 24hrs practically every live TV and radio show has been running the story, interviewing friends, colleagues and fans, many shows receiving an unprecedented number of calls and emails. His picture was prominently placed on the front page of all of the 'proper' newspapers, with the exception of The Times.

At 65 I guess he had a good innings, but I think everyone would agree that he was one of the good guys and died too young. The amount of blog and forum posting on this subject is quite astounding, everyone seems to have a JP story from some point over the 40 years he was hosting his radio 1 show. Some stories pre-date even that era, Paul Gambaccini recounted a story from his early DJ'ing days in Dallas, Texas - he apparently encountered JFK while taking photographs, JFK on hearing his liverpool accent invited him to take some pictures, at one point John was taking snapshots standing on the car bonnet!(not sure of this story is true, but if it is then JFK would have been president at the time peelie was in Dallas!) Later on John was present at the press conference just prior to Lee Harvey Oswald being shot.

JP was truly a one off, the product of his upbringing, his self-deprecating, modest and caring personality, his presence on the music scene at the birth of rock'n'roll, and his subsequent involvement in the fledgling careers of more bands probably than anyone other single person in the history of music. He had accumulated followers from all over the world and I think many have been genuinely shocked to learn of his death, I know I was.

The friendly tones of John's voice coming over the airwaves late at night on Radio 1 and on Saturday morning on Radio4 will be missed by many. His absence from the music scene will leave behind a black hole. Yesterday PM Tony Blair paid tribute and today I hear that Micheal Eavis of Glastonbury festival fame will be permanently naming one of the Galstonbury stages after John, and no doubt this will be the first of many great tributes to come.

Sorry to see you go old friend.