tag:blogger.com,1999:blog-19303585Sat, 20 Mar 2010 09:10:10 +0000http://www.jgc.org/blog/noreply@blogger.com (John Graham-Cumming)Blogger382125tag:blogger.com,1999:blog-19303585.post-8391498145672798504Fri, 19 Mar 2010 10:15:00 +00002010-03-19T10:42:25.585Zpseudo-randomnessmachine learningThe other day on Hacker News a user posted an anonymous comment. Regular Hacker News participant jacquesm wanted to unmask the writer and posted a challenge to unmask the user.He also emailed me because he thought I might be the anonymous user. I agreed to help him with a little bit of text mining. Jacques had a nice database of all Hacker News submissions and comments and gave me a 250Mb SQL http://www.jgc.org/blog/2010/03/responding-to-jacquesms-challenge.htmlnoreply@blogger.com (John Graham-Cumming)1tag:blogger.com,1999:blog-19303585.post-1014381764096527035Thu, 18 Mar 2010 19:52:00 +00002010-03-19T09:08:49.505Zthe geek atlasBack in 2004 I was living in New York and commuting between New York and Washington, DC on the Acela. I was working in a fairly rural part of Virginia and was lucky enough to accidentally experience a once in 17 years event: the emergence of Magicicada Brood X.(Picture from Wikipedia)Now I realize that most people probably don't think that being in a place where millions of large winged insects http://www.jgc.org/blog/2010/03/fascinating-little-beastie.htmlnoreply@blogger.com (John Graham-Cumming)4tag:blogger.com,1999:blog-19303585.post-5714125616449877250Tue, 16 Mar 2010 12:48:00 +00002010-03-16T14:18:15.179Zthe geek atlasThis past weekend the London Transport Museum held an open weekend at its Acton Depot where they keep a collection of trams, trolley cards, buses and underground trains, plus all the associated equipment. They only open the depot twice a year so this was a chance to see some things that are rarely open to the public.I didn't include this museum in The Geek Atlas but after a visit it's likely a http://www.jgc.org/blog/2010/03/london-transport-museum-acton-depot.htmlnoreply@blogger.com (John Graham-Cumming)0tag:blogger.com,1999:blog-19303585.post-2897426477460604160Thu, 11 Mar 2010 15:40:00 +00002010-03-11T15:58:43.312Zday jobpseudo-randomnessOccasionally I get asked for some sort of official bio. Here's one people can use:John Graham-Cumming is computer programmer and author. He studied mathematics and computation at Oxford and stayed for a doctorate in computer security. As a programmer he has worked in Silicon Valley and New York, and the UK and France. His open source POPFile program won a Jolt Productivity Award in 2004.He ishttp://www.jgc.org/blog/2010/03/my-bio.htmlnoreply@blogger.com (John Graham-Cumming)1tag:blogger.com,1999:blog-19303585.post-3051580967854407851Wed, 10 Mar 2010 12:39:00 +00002010-03-10T12:40:56.427Zalan turingOver at The Guardian I write:Last year I led a campaign to obtain an apology for the mistreatment of the British mathematician Alan Turing. Turing's prosecution for homosexuality led to the death of a true genius at the age of only 41 in 1954. On 10 September last year, Gordon Brown issued an apology that recognised Turing's stature as one of the greatest Britons. But Britain has a final http://www.jgc.org/blog/2010/03/olympic-honour-for-alan-turing.htmlnoreply@blogger.com (John Graham-Cumming)0tag:blogger.com,1999:blog-19303585.post-2200348446890589156Tue, 09 Mar 2010 14:19:00 +00002010-03-09T14:35:25.242Zclimate changeIn The Guardian's Comment is Free section there's an article by George Monbiot called The trouble with trusting complex science which argues that:The detail of modern science is incomprehensible to almost everyone, which means that we have to take what scientists say on trust.He does this in the context of climate change science. I wonder if he actually tried to read the key paper that describeshttp://www.jgc.org/blog/2010/03/did-monbiot-try-to-understand-climate.htmlnoreply@blogger.com (John Graham-Cumming)5tag:blogger.com,1999:blog-19303585.post-918555874631291972Wed, 03 Mar 2010 12:50:00 +00002010-03-03T12:51:52.423Zpseudo-randomnessHere's me writing in The Guardian's Comment is Free section:We're all the children of amateurs: amateur parents. There's no government department that will certify you as a parent (thankfully), nor a university department where you get your PhD in being a daddy, nor a professional body ready to strike you off for not following mothering standards. But any parent who's held a newborn child in http://www.jgc.org/blog/2010/03/welcome-bunch-of-amateurs.htmlnoreply@blogger.com (John Graham-Cumming)0tag:blogger.com,1999:blog-19303585.post-2150221012031444681Mon, 01 Mar 2010 11:41:00 +00002010-03-01T14:41:10.694Zclimate changeIt turns out the reason is rather simple. In the evidence being given today to the Parliamentary Science and Technology Committee the Met Office says of their quality control procedures:Manual inspection, including real-time quality control using GIS software; quality control described in literature for the various regional studies.Contrast that with what they say about NOAA's procedures for thehttp://www.jgc.org/blog/2010/03/reason-i-managed-to-find-errors-in-met.htmlnoreply@blogger.com (John Graham-Cumming)1tag:blogger.com,1999:blog-19303585.post-6758623446879471383Thu, 25 Feb 2010 16:46:00 +00002010-03-02T17:41:54.931Zclimate changeUEA and CRU have issued a document that they have submitted to the Parliamentary Select Committee on Science and Technology who are looking into the taking of email and documents from CRU. The document can be found here.In it there are two interesting paragraphs concerning software:3.4.7 CRU has been accused of the effective, if not deliberate, falsification of findings through deployment of “http://www.jgc.org/blog/2010/02/something-bit-confusing-from-ueacru.htmlnoreply@blogger.com (John Graham-Cumming)3tag:blogger.com,1999:blog-19303585.post-4833468070609683656Wed, 24 Feb 2010 23:20:00 +00002010-02-24T23:20:00.238Zclimate changeI'm told by a BBC journalist that the Met Office has said through their press office that the errors that were pointed out by Ilya Goz and I have been confirmed. The station errors are being incorrectly calculated (almost certainly because of a bug in the software) and that the Met Office is rechecking all the error data.I haven't heard directly from the Met Office yet; apparently the Met Officehttp://www.jgc.org/blog/2010/02/station-errors-in-crutem3-and-hadcrut3.htmlnoreply@blogger.com (John Graham-Cumming)2tag:blogger.com,1999:blog-19303585.post-36060017044546445Tue, 16 Feb 2010 09:03:00 +00002010-02-16T09:18:15.051Zclimate changepseudo-randomnessIn the print version of my Times article today there's been significant cutting to get it to fit into the space available. This is the magic work of sub-editors. Here's the full text of the article with the words that remained in the sub-edited version (which appeared in the paper):The history of science is filled with stories of amateur scientists who made significant contributions. In 1937 http://www.jgc.org/blog/2010/02/magic-of-sub-editors.htmlnoreply@blogger.com (John Graham-Cumming)7tag:blogger.com,1999:blog-19303585.post-6659916268056118297Mon, 15 Feb 2010 22:41:00 +00002010-02-16T16:36:48.785Zclimate changeHere's a major newspaper writing about what I found in the Met Office data:A science blogger has uncovered a catalogue of errors in Met Office records that form a central part of the scientific evidence for global warming.The mistakes, which led to the data from a large number of weather stations being discarded or misused, had been overlooked by professional scientists and were only discovered http://www.jgc.org/blog/2010/02/times-writes-up-my-met-office.htmlnoreply@blogger.com (John Graham-Cumming)0tag:blogger.com,1999:blog-19303585.post-1784711398180473707Mon, 15 Feb 2010 09:33:00 +00002010-02-15T13:12:39.233Zclimate changeThe following is a popular picture used by climate change skeptics to attempt to show that there's something seriously wrong with the the surface temperature record which is used to show that the world is getting hotter. It appears to show that two weather stations with Stevenson screens are situated right at the end of the runway of Rome Ciampino. It's not hard to put two and two together and http://www.jgc.org/blog/2010/02/climate-change-skepticism-youre-doing.htmlnoreply@blogger.com (John Graham-Cumming)21tag:blogger.com,1999:blog-19303585.post-1557216333591546314Sun, 14 Feb 2010 14:02:00 +00002010-02-14T17:54:58.623Zrants and ravesOne of the most depressing things about being a programmer is the realization that your time is not entirely spent creating new and exciting programs, but is actually spent eliminating all the problems that you yourself introduced.This process is called debugging. And on a daily basis every programmer must face that fact that as they write code, they write bugs. And when they find that their http://www.jgc.org/blog/2010/02/bad-workman-blames-his-tools.htmlnoreply@blogger.com (John Graham-Cumming)10tag:blogger.com,1999:blog-19303585.post-818960330886429024Sat, 13 Feb 2010 08:49:00 +00002010-02-13T09:02:28.909ZSome people seem to be very good at searching the web, others seem to be very poor at it. What differentiates them? I think it's unconcious knowledge of something called TF-IDF (or term frequency-inverse document frequency).If you clicked through to that Wikipedia link you were probably confronted by a bunch of mathematics, and since you are reading this you probably hit the back button as http://www.jgc.org/blog/2010/02/if-youre-searching-remember-your-tf-idf.htmlnoreply@blogger.com (John Graham-Cumming)6tag:blogger.com,1999:blog-19303585.post-9212609156875520571Fri, 12 Feb 2010 14:34:00 +00002010-02-12T14:35:56.988Zday jobmachine learningHere's something I wrote for the company blog.If you say the words 'machine learning' to people they either look confused or bored. Since the promise of Artificial Intelligence evaporated in the 1970s, machine intelligence seems to be one of those things that's a perpetual 20 years away.But computer industry insiders know that many forms of machine learning are at work all the time. The most http://www.jgc.org/blog/2010/02/so-you-think-machine-learning-is-boring.htmlnoreply@blogger.com (John Graham-Cumming)0tag:blogger.com,1999:blog-19303585.post-7555882076118280987Thu, 11 Feb 2010 08:58:00 +00002010-02-11T09:01:51.769Zpseudo-randomnessEmail developed from a file transfer based protocol to the widely used SMTP (which was created in 1982). As a protocol it leaves much to be desired: senders can be forged, it's a playground for spammers, the ability to send anything other than plain text (ASCII) messages had to be added with duct tape at a later date, its error messages are cryptic and it has few facilities to deal with the http://www.jgc.org/blog/2010/02/is-your-new-technology-crappy-enough.htmlnoreply@blogger.com (John Graham-Cumming)4tag:blogger.com,1999:blog-19303585.post-6825724884495347381Wed, 10 Feb 2010 12:40:00 +00002010-02-10T13:27:49.780Zpseudo-randomnessSo it's been a year. A year ago I moved house and didn't unpack the TV. It was a nice TV: a 42" flat panel display which when connected to the right sort of cable receiver could be used to watch broadcast TV. With the TV in a box, I never subscribed to any cable or satellite service. My house is without a TV receiver of any kind.And I don't miss it. Only occasionally do I get that urge to http://www.jgc.org/blog/2010/02/year-without-tv.htmlnoreply@blogger.com (John Graham-Cumming)13tag:blogger.com,1999:blog-19303585.post-4468750586208385075Wed, 10 Feb 2010 08:31:00 +00002010-02-10T10:14:51.798Zclimate changepseudo-randomnessIn 1937 an amateur American astronomer named Grote Reber completed construction of a 9 meter radio telescope in his back garden. By 1940 he had verified that there were radio signals coming from the heavens and by 1943 he had completed a radio frequency map of the sky. Reber, with his enormous hand-built dish, kick started radio astronomy and eventually sold his invention to the US http://www.jgc.org/blog/2010/02/rank-amateur.htmlnoreply@blogger.com (John Graham-Cumming)12tag:blogger.com,1999:blog-19303585.post-5971575011317197613Tue, 09 Feb 2010 15:44:00 +00002010-02-09T15:49:44.590Zpseudo-randomnessLufthansa is running a competition to name one of their A380 jets. You can submit your own entry here.My suggestion is Wing Kong. If you like it you can vote for me on their site.Wing Kong: it's big, it's powerful, it comes from a strange, dark place (well, Toulouse). It's also just a little bit romantic.http://www.jgc.org/blog/2010/02/wing-kong.htmlnoreply@blogger.com (John Graham-Cumming)1tag:blogger.com,1999:blog-19303585.post-2013295982842131436Mon, 08 Feb 2010 16:37:00 +00002010-02-08T16:57:10.938Zpseudo-randomnessI first got an email address with an Internet @ in it in 1986. It was jgc@prg.ox.ac.uk, or for those of you on JANET it was JGC@UK.OX.AC.PRG (happily I only briefly used bang paths). In 24 years I think there have been three major end-user innovations: address books, MIME and email searching.Address BooksInitially, I didn't need an email address book. Most of the people I was emailing were on http://www.jgc.org/blog/2010/02/24-years-of-email.htmlnoreply@blogger.com (John Graham-Cumming)3tag:blogger.com,1999:blog-19303585.post-1320505019984063981Sun, 07 Feb 2010 14:56:00 +00002010-02-08T11:34:59.685Zclimate changeOut of the blue I got a comment on my blog about CRUTEM3 station errors. The commenter wanted to know if I'd tried to verify them: I said I hadn't since not all the underlying data for CRUTEM3 had been released. The commenter (who I now know to be someone called Ilya Goz) correctly pointed out that although a subset had been released, for some years and some locations on the globe that subset http://www.jgc.org/blog/2010/02/something-odd-in-crutem3-station-errors.htmlnoreply@blogger.com (John Graham-Cumming)9tag:blogger.com,1999:blog-19303585.post-8148977687015708179Wed, 03 Feb 2010 13:35:00 +00002010-02-03T13:37:39.172Zpseudo-randomnessThe Times has kindly mentioned this blog as one of its Top 30 Science Blogs saying:John Graham-Cumming is one of the few people out there who makes the nuts and bolts of computer programming actually sound interesting. Expect anything from an analysis of the statistical likelihood of election fraud in the last Iranian election to the unveiling of flaws in the Met Office’s global climate http://www.jgc.org/blog/2010/02/compliment-from-times.htmlnoreply@blogger.com (John Graham-Cumming)0tag:blogger.com,1999:blog-19303585.post-8865306825100082269Fri, 29 Jan 2010 17:21:00 +00002010-01-29T17:27:04.386Zclimate changeThere's a new version of the Met Office land surface temperature record out with lots and lots more stations.Plus it includes corrections for all the problems I found with the data (they didn't make good on their promise to acknowledge me, sadly).But my handiwork is shown by the points in green:My two corrections: A and B.I'll run these through my own programs to see what they produce.http://www.jgc.org/blog/2010/01/new-version-of-crutem3-and-hadcrut3.htmlnoreply@blogger.com (John Graham-Cumming)10tag:blogger.com,1999:blog-19303585.post-8521940331731662041Thu, 28 Jan 2010 10:47:00 +00002010-01-28T11:19:29.235Zpseudo-randomnessNow, at last, I can reveal the top diet secrets that doctors have been keeping from you! Yes, this is how I lost an AMAZING 9.9kg (21.8 pounds) in just 6 months doing absolutely no exercise at all.Put down those weights, step off the StairMaster and follows these amazing simple steps to a better figure:EAT LESS, EAT WELLIn April 2008 I decided that my 82.5kg (181.9 pounds) was too much for my http://www.jgc.org/blog/2010/01/johns-amazing-diet-secrets-revealed.htmlnoreply@blogger.com (John Graham-Cumming)4