John Graham-Cumming

Which countries have the most beautiful women? (My deeply flawed analysis)

2008-05-15T04:46:00.003Z

So, I happened upon the Wikipedia page about the Miss World pageant and noticed that it had a list of winners by country. For example, India has won Miss World 5 times. But, of course, India has a very large population so you'd expect it to be able to churn out a few beauties. So, to get a better idea here is a population adjusted list of countries that have won Miss World:

Country	Wins	Pop.	Wins/Pop.	Normalized
Bermuda	1	66163	0.0000151141876879827	100.00%
Iceland	3	316252	0.00000948610601672084	62.76%
Grenada	1	110000	0.00000909090909090909	60.15%
Guam	1	173456	0.00000576515081634536	38.14%
Jamaica	3	2651000	0.000001131648434553	7.49%
Trinidad and Tobago	1	1305000	0.000000766283524904215	5.07%
Sweden	3	9182927	0.000000326693221017656	2.16%
Puerto Rico	1	3994259	0.000000250359328225836	1.66%
Austria	2	8316487	0.000000240486157195941	1.59%
Ireland	1	4339000	0.000000230467849734962	1.52%
Finland	1	5308208	0.000000188387493481793	1.25%
Venezuela	5	28199822	0.000000177306083705067	1.17%
Israel	1	7282000	0.000000137324910738808	0.91%
Netherlands	2	16408557	0.000000121887622415548	0.81%
Dominican Republic	1	9760000	0.000000102459016393443	0.68%
Czech Republic	1	10381130	0.0000000963286270377117	0.64%
Australia	2	21290000	0.0000000939408172851104	0.62%
Greece	1	11216708	0.0000000891527175353054	0.59%
Peru	2	28674757	0.0000000697477575834383	0.46%
UK	4	60487300	0.0000000661295842267716	0.44%
Argentina	2	40301927	0.0000000496254186555397	0.33%
South Africa	2	43700000	0.000000045766590389016	0.30%
Poland	1	38518241	0.0000000259617255107781	0.17%
France	1	64473140	0.0000000155103350015216	0.10%
Turkey	1	70586256	0.0000000141670639111387	0.09%
Egypt	1	80335036	0.0000000124478689472424	0.08%
Germany	1	82210000	0.0000000121639703199124	0.08%
Russia	1	142008838	0.00000000704181524251329	0.05%
Nigeria	1	148000000	0.00000000675675675675676	0.04%
US	2	304072000	0.00000000657738956562919	0.04%
Brazil	1	186757608	0.00000000535453420457174	0.04%
India	5	1132446000	0.00000000441522156464856	0.03%
China	1	1321851888	0.000000000756514409124179	0.01%

So, far and away, the top three are Bermuda, Iceland and Grenada. Given that Bermuda is the winner, and a tax-haven, and has a sub-tropical climate... Hamilton here I come!

The Spammers' Compendium finds a new home

2008-05-01T16:09:00.002Z

Shortly after I announced that I was getting out of anti-spam the folks at Virus Bulletin contacted me about taking over The Spammers' Compendium. I was delighted.

Today the transfer is complete and the new home is here. It will be maintained and updated by Virus Bulletin. Please send submissions to them.

Bookmark-based registration

2008-04-22T11:39:00.004Z

Recently, I've been learning Ruby on Rails and I can never learn anything unless I build something with it. I also recently read Programming Collective Intelligence and had a desire to use some of those algorithms too.

I'll post more about the actual web site I created another time; currently it's in alpha form running on Heroku. The web site is used for naming babies, the initial alpha-release is the girls' names only site EmilyOrEmma? which uses "Hot or Not" style voting and manages to incorporate in one page the Levenshtein distance, Metaphone algorithm and both item-based and user-based filtering.

But my biggest bug bear with baby naming web sites is the need to create an account. You can browse baby names all you want, but as soon as you want to do something like add a name to a list of favourites you are forced into registration. Or you don't have to register but anything you do is ephemeral.

For my site, I came up with a better solution: bookmark-based registration. If you visit the site you'll see at the bottom of every page a link to bookmark. This link is unique to you. It contains your user id and a hash which I used to prevent forgery.

Bookmark this link and you can return to your recommendations, saved names, etc. any time.

Interesting real-world Apache Problem

2008-04-08T08:07:00.002Z

I'm working with a large client who has a number of web servers behind a load balancer. This morning one Apache 1.3 had failed to come up on one of them. The client sends a SIGUSR1 to each Apache once an hour to force a graceful reload. This particular machine had operated correctly restarting Apache once per hour for 54 hours (since a recent reboot of the machine) and then died.

A quick look in the Apache error.log file showed the following:

module "mod_jk.c" could not be loaded, because the dynamic module limit was reached. Please increase DYNAMIC_MODULE_LIMIT and recompile.

Naturally I went looking for a problem with mod_jk which was the wrong place to look. Scrolling through the log file I noticed that every time Apache restarted we'd get the error:

Cannot remove module mod_include.c: not found in module list

This was where the real problem lay. A quick httpd -l showed that mod_include was compiled into the client's Apache and looking in the httpd.conf revealed that mod_include was also being loaded with LoadModule:

LoadModule includes_module modules/mod_include.so

When a module is both statically linked into Apache and dynamically loaded you run into a nasty problem: Apache doesn't complain when you start, but it will fail to unload the double loaded module on exit. So for every SIGUSR1 a single slot of the DYNAMIC_MODULE_LIMIT was used up. The default DYNAMIC_MODULE_LIMIT is 64 and with 10 real dynamic modules and a boot once per hour it took 54 hours to consume every slot in the module limit.

Removing the errorneous LoadModule fixed the problem.

Digg 3 Million

2008-04-04T10:40:00.003Z

A quick update on my previous estimate of Digg users shows that, as predicted, Digg passed the 3 million user mark during March, 2008.

Comparing this estimated data and data from the Digg API shows that around 20% of the 3m accounts are not active. I speculated before that these accounts had been banned for spamming or other activities (that's around 600,000 bad accounts).

Growth appears to be the same as before adding around 150,000 accounts per month.

Juxtaposition

2008-04-03T17:58:00.002Z

It's 3am and there's a crisis somewhere in the world

2008-04-03T10:00:00.002Z

If you follow US politics then you'll know that Hillary Clinton has a couple of ads that start with a 3am phone call to the Whitehouse. The first ad was intended as a slam against Barack Obama implying that he didn't have the experience to deal with such a crisis. The second is going up against John McCain claiming he doesn't want to do anything about the housing finance problem in the US.

You know if it's 3am and there's a crisis in the world there's only one place and only one man to call.

CTU and Jack Bauer.

First of all, he's already up. 3am is nothing to Jack. Hillary and McCain both look like they could use the sleep and Obama looks like he gets his beauty sleep every night. So, Jack's ready to go before any of them.

As much as you might think Obama is David Palmer (safe pair of hands), he's more like a Wayne Palmer (a slick little fighter) and you know what that means: gets blown up within five minutes of being president and then pops a brain vein and the evil VP has to take over. The only good thing to say about Obama is that he would call Jack.

Now, McCain might look like a Bauer type with his military background and heroic time spent as a PoW. But here's the difference. McCain was 5 years as a PoW, no one came to get him out so he can't be that valuable. Also, if Jack had been held hostage in North Vietnam for 5 years there wouldn't be a North Vietnam now. Because you can bet the life of the next random CTU cast member the Jack would have (a) escaped and (b) annihilated everyone involved.

So, that leaves Hillary. She can't tell sniper fire from a little girl with flowers. In fact she reminds me more and more of the evil Vice Presidents that pop up in every 24 trying to take power from the real president. Hell, she's probably even got backing from Phillip Bauer.

Only one word of warning: make sure it's a man that calls Jack. If it's a woman he's bound to have been involved with her, she'll turn out to be a double agent or her father will be evil, and Jack'll be distracted.

If it has to be a woman, make sure it's Chloe.

Multi-route (email and phone) self-aware phishing

2008-03-31T14:34:00.006Z

Today, I received the following email:
This communication was sent to safeguard your account against any unauthorized activity. Max Federal Credit Union is aware of new phishing e-mails that are circulating. These e-mails request consumers to click a link due to a compromise of a credit card account. You should not respond to this message. For your security we have deactivate your card. How to activate your card Call +1 (800)-xxx-9629 Our automated system allows you to quickly activate your card Card activation will take approximately one minute to complete.
Of course, I don't have an account with Max Federal Credit Union and this is obviously a phish. Notice that the English is quite right:

"For your security we have deactivate your card." and "You should not respond to this message." doesn't make sense in context.

What's more interesting is that the message itself warns you about phishing emails and asks you to call an 800 number.

If you call the 800 number an electronic voice reminds you again to never give your PIN, password or SSN in email and then proceeds to ask you for the card number, PIN, expiry date and CVV2. The assumption is that you've been warned twice not to do something in email, so it's OK by phone.

It's painful to see the phisher use the existence of phishing as a way to phish.

Names: Boys vs. Girls

2008-03-31T10:05:00.004Z

Using data from the 1990 US Census I was amazed to discover that 90% of the US male population has one of 1,219 first names, but 90% of the female population has one of 4,275. There are 3.5x as many female first names as male first names.

The top 10 male first names are: James, John, Robert, Michael, William, David, Richard, Charles, Joseph and Thomas (which account for 23.2% of the male population; 50% of the population have one of only 60 names).

The top 10 female first names are: Mary, Patricia, Linda, Barbara, Elizabeth, Maria, Susan, Margaret and Dorothy (which account for 10.7% of the female population; 50% of the population have on of 139 names).

You can also see that all the variety in names happens between the 80% and 90%. For males 80% of the population is covered by 27% of the names; for females 80% is covered by 19% of the names).

The large numbers of female names appears to be because there are lots of variants of female names compared to male names. A quick run through calculating the Levenshtein distance between names and selecting the 10 closest for each gives an average distance of:

Male: 2.62
Female: 2.01

So female names are more 'similar' than male names, hence the variety created by all these variants.

The other thing we can extract from this data is the prevalence of names beginning with certain letters and weight adjust based on the occurrence of each name.

Things are much more polarized when you look at trailing letters (for example, the trailing letter A is an almost sure sign that it's a woman; the opposite is true of D):

So combining the two it's possible to give a 'maleness' score (the blue part) to each final letter:

BOUTS: The Complete Song Parodies

2008-03-31T07:57:00.003Z

Back when my site UseTheSource was a "blog" (this was late 90s/early naughties) I wrote a number of song parodies. Someone emailed me and asked where they were.

So, here are my complete song parodies from back in 2001:

March 7, 2001: "Candle in the Wind"

Yahoo! was in trouble with banner ad sales falling, profits disappearing and the then CEO, Tim Koogle, was off to spend more time with his family.
Goodbye Tim Koogle Though I never Yahoo! at all You had the faith, to sell banners While those around you failed They crashed into the deadpool And they whispered into your brain You need to begin to charge For things you give for free And it seems to me you lived your life Like a candle in the wind Never knowing what to add next To your list of links And I would have liked to have told you But I was just a geek Your web brand burned out long before Your stock price ever did Jerry Yang was tough The toughest boss you ever had Softbank created a superstar And pain was the price you paid As the whole web died Oh CNET still hounded you All they had to say Was that Google was the site to use [Repeat chorus] Goodbye Tim Koogle From the young man on the DSL link Who sees you as something more successful More than just our long lost CEO [Repeat chorus]
March 22, 2001: "Don't Cry for Me Argentina"

Steve Jobs was back at Apple, the blue iMac was out, Microsoft was in big anti-trust trouble and had just invested $100m in Apple, Steve had bought out NeXT. But the future wasn't yet assured:
This won't be easy, you'll think it's strange. When we try to explain what we need that we now need your help after all that we've said. You won't believe us. All you'll see is Apple you once knew, although we've crashed down in the dumps begging for Microsoft cash. It didn't have to happen. We should have won Better software and patents than Bill Looking down on Windows, staying far from Sun. So we chose NeXT. Running aground, trying computers in blue. But nothing revived us at all. You never expected it too. Don't cry for us, William H. Gates. The truth is we're dead without you. We need your dollars We need Mac Office You need a rival, for your survival. And as for Fortune, and as for Time, We never invited them in though it appeared, to the world, they were all Steve desired. Even Adobe, they're making solutions for Windows right now The answer was here all the time. We need you, and hope you need us. [chorus] Have we begged too much? There's nothing more we can think of to say to you. But all you have to do, is look at us to know, we're through without you.
April 1, 2001: "I Just Called To Say I Love You"

In the midst of the crash, .coms were going out of business like crazy:
No IPO to celebrate No friends and family stocks and shares to give away No big opening No first day ping In fact here's just another ordinary day No Aeron chair No onsite chef No working Saturday until the site is done But what it is, is something blue Made up of these few words that I must say to you We just failed to get more funding We just failed to keep our doors open We just failed to get more funding And we need it just to avoid bankruptcy No free massage No free soda No caffeine trip to keep us working every night No dry cleaning No stock option Not even time for us to pack our things and leave No beanbag room No Maui trip No giving thanks to all that NASDAQ did for us But what it is, though old so new Grab what you can before your jobs right here are through [chorus] [chorus]
April 6, 2001: "Uptown Girl"

Ah, to be in love with a marketing .com girl:
.com girl She's been living in her .com world I bet she never had a software guy I bet her mama never told her why I'm gonna try for a .com girl She's been living in her wide web world As long as anyone in marketing can And now she's looking for a comp. sci. man That's what I am And when she knows what She wants from her time And when she wakes up And makes up her mind She'll see I'm not a nerd Just because I'm in love with a .com girl You know I've seen her in her online world She's getting tired of her high tech toys And all her presents from her VC boys She's got a choice .com girl You know I can't afford to buy her a Porsche But maybe someday when my stock cashes in She'll understand what kind of guy I've been And then I'll win And when she's walking She's using her Nokia And when she's talking She'll say that she's mine She'll say I'm not a nerd Just because I'm in love With a .com girl She's been living in her latte world As long as anyone in marketing can And now she's looking for a comp. sci. man That's what I am .com girl She's my .com girl You know I'm in love With a .com girl
April 13, 2001: "Gangsta's Paradise"

Linus Torvalds was the flavor of the day as one of the thorn's in Microsoft's side:
As I drive through the Valley of the Silicon Dream I take a look at my life and realize there's nothing left 'Cause I've been coding and debuggin' so long That even my manager thinks that my mind has gone But I ain't never crossed a man that didn't deserve it Linus treated like a punk, ya know that's unheard of Ya better watch how ya postin' And what ya codin' Or you Dr Tanenbaum'll be lined in chalk I really hate Minix and FreeBSD As they croak, I see myself in the pistol smoke Fool, I'm the kinda hacker script kiddies wanna be like On the Net in the night, writin' layers of the core code CHORUS: Been spending most our lives living in a Windows paradise Been spending most our lives living in a Windows paradise Keep spending most our lives living in Bill Gates' paradise Keep spending most our lives living in Bill Gates' paradise Look at the situation they got me facing I can't live a normal life, I was raised on the PC So I gotta be down with the kernel team Too much crazy Usenet posting got me chasing dreams I'm a educated fool with Posix on my mind Speak Swedish in my home and English on the phone I'm a loc'd out hacker, wrote my life story And my homies is down so don't arouse my anger Fool, death ain't nothin' but a heart beat away I'm livin' life, do or die, what can I say? I'm 28 now, but will I ever see 29? The way things is going, I don't know Tell me why are we so blind to see That Microsoft's a monopoly CHORUS CHORUS Tell me why are we so blind to see That Microsoft's a monopoly Tell me why are we so blind to see That Microsoft's a monopoly
April 18, 2001: "Copacabana"

Carly Fiorina was fighting for her life as she tried to merge HP and Compaq with Walter Hewlett attempting a proxy fight to stop her in the name of the family:
Her name was Carly, she was a VP With Lucent and AT&T and a degree from MIT She went to HP and wowed the board room And while she tried to be a star, sometimes went a bit too far And then September 4, Compaq became the score They were failing and needed each other Leaning drunks galore! At the HP, HP/Com-pa-q The merger that upset the family At the HP, HP/Com-pa-q David and William were always the fashion At the HP... they ran the show His name was Walter, his dad was famous He wasn't present for the board, but he wouldn't be ignored And what she pro-posed, "Dad would've hated" Then Walter went a bit too far, "Carly: time for au revoir!" And then the insults flew and careers were smashed in two There were ads and a lot of bankers, but just who screwed who? At the HP, HP/Com-pa-q The merger that upset the family At the HP, HP/Com-pa-q David and Walter are today the fashion At the HP... they run the show Her name is Carly, she was CEO But that was 30 weeks ago, when she used to run show Now she's a VC, but that's our Carly Still in the suit she used to wear, new blonde highlights in her hair She sits there so refined, and drinks to Walter's health She lost her job and she lost the proxy, now she enjoy's her wealth! At the HP, well just the HP The toughest job belongs to Walter At the HP, well just the HP William and David were always the fashion At the HP, don't buy the stock...

If you find these funny and can sing... feel free to set them to music and give me a laugh.

More 11:11 mystical nuttery

2008-03-29T16:34:00.003Z

Out of the blue I received an email about my post the other day about Benford's Law and 11:11:


every time i look at the clock the number add up to 11. 
how does that get explained

OK. Well, it turns out that that's pretty simple to explain: the sum 11 is the most common sum you'll see on a clock. The following graph shows the count for each sum of digits. You'll see that for a 12 hour clock the peak is at 11 and for a 24 hour clock the peak is at 12 with 11 being a close second.

For a 12 hour clock the probability that the sum of digits will be 128/1440 (or about 9% of the time). For a 24 hour clock it's 124/1440 (or about 9% of the time). So it's unsurprising that 11 comes up a lot here.

Another area of 11 craziness is airline seating. This is probably because people get freaked out by flying and look for patterns. Suppose you sit in economy on a British Airways long haul flight. You'll be sitting in a 747, 767 or 777. You then take your seat number add up the digits and then add the letter on using its place in the alphabet (e.g. sitting in 14F then you have 1 + 4 + 6 = 11). Using the British Airways seat maps you can compute the value of for each seat in economy:

On a 767 11 is the most frequently occurring sum, on a 747 it's 10 (with 11 close behind) and on a 777 11 is just beaten out by 12.

"Retiring" from anti-spam

2008-03-25T17:39:00.003Z

Today, I'm "retiring" from anti-spam work. Practically, that means the following:

No more updates to The Spammers' Compendium or Anti-spam Tool League Table pages. These remain on line, but are not being maintained.
I'm looking for a new leader for the POPFile project.
I'm no longer active on any anti-spam mailing lists.
I am leaving all anti-spam conference committees.
My anti-spam newsletter is no longer being published.

I will, however, be continuing with commercial anti-spam work where I have agreements currently in place with customers. No change to their support, terms or assistance.

The obvious question is why? For me, the interest just isn't there. The battle against spam continues but is now about trench warfare rather than creating new weapons. We'll continue to see innovation, but for any hacker it's the new, new thing that's important. For me, spam is yesterday's news. Watching companies squabble and refuse to cooperate, seeing a decline in quality at anti-spam conferences, and major companies essentially killing their consumer anti-spam means anti-spam just isn't where I want to be.

Of course, there are many really good people fighting spam out there. This post isn't meant to demean them.

Thank you to everyone who has supported what I've done over the last 7 years, and good luck!

Building a temperature probe for the OLPC XO-1 laptop

2008-03-22T12:38:00.004Z

I bought an OLPC XO-1 laptop through the G1G1 program and was intrigued to discover the Measure activity.

The measure activity uses the internal audio system to measure a value input on the microphone socket. With nothing connected this application reads the value of the internal microphone and displays a waveform. You can have fun just by whistling, speaking or singing with Measure running.

But since you can measure a voltage input into the microphone socket, it's possible to build sensors and connect them to th OLPC XO-1. On the Measure web site they mention building a simple temperature sensor using an LM35 temperature sensor that looks like this:

The LM35 can measure a temperature between 0 and 155 Celsius just by hooking it up to a 5v supply. It outputs 10mv per degree so a temperature of 20 Celsius corresponds to 0.200v.

Since the OLPC XO-1 has a USB port it's possible to get 5v from the laptop by hacking a USB connector, and connect 5v to the LM35 and then take the signal coming from the LM35 (the middle pin) and connect it to the microphone socket.

I did this by building two parts: a generic adapter which gives me 5v and a signal line out of a standard stereo 3.5mm jack:

The stereo jack is wired up so that the tip is +5v, the base is Gnd and the middle is the signal going to the microphone socket. The USB plug has only two wires connected (for +5v and Gnd), and the jack going to the microphone socket (which is mono) has the connected to the middle of the stereo jack, and the base is Gnd. All the grounds are joined together.

When plugged into the OLPC XO-1 it creates a generic connector for any other projects I might work on:

For the temperature sensor I simply connected the LM35 to a stereo socket with the correct connections to match up with the stereo jack plug. Then I created a probe with an old plastic pen and some waterproofing compound (so that I can do things like shove the probe in a cup of coffee without wetting the contacts on the LM35). Here it is:

Connect the two together and run the standard Measure activity and you can start to look at the output of the sensor and hence the temperature.

But there's a problem. The microphone input can only handle voltages in the range 0.3v to 1.9v (and my measurements of my OLPC XO-1 show this range to actually be 0.4v to 1.9v). So that means as is the probe can be used to measure temperatures in the range 40 Celsius to 155 Celsius. That low end is a bit high for the sorts of experimentation you can do at home (e.g. measure the temperature in the fridge, or a glass of cold water, or even the temperature inside your mouth).

So we need to scale the voltages coming from the sensor to fit better into the range that's readable by the laptop. The standard way to do that is with an operational amplifier which is used to add two voltages together: the voltage coming from the sensor and a reference voltage. Doing this will move the voltage up.

For that I used the LM1458 which in a single 8 pin package contains a pair of operational amplifiers.

Here's the circuit diagram:

The circuit has three parts: a voltage divider, a summing amplifier and an inverting amplifier.

Voltage divider: the reference voltage is created by taking the 5v available from the USB port and passing it through resistors R8 and R9. The voltage at the middle point of these two resistors is determined by the standard formula for a voltage divider of 5v * R9/(R8 + R9) = 5v * 1 / ( 10 + 1 ) = 0.45v. In my actual circuit with 1% tolerance resistors the measured voltage was 0.41v.

Summing amplifier: the middle portion of the circuit takes the two inputs and adds them together (and because of the nature of the circuit inverts the summed value). So its output going into R7 is -ve the sum of the reference voltage and the sensor voltage.

Inverting amplifier: the final part just inverts the voltage so that the output is +ve and in the range that the OLPC XO-1 can read.

One complexity is that this circuit requires +9v, Gnd and -9v to operate. I obtain that with a pair of 9v batteries linked together giving Gnd where the two are connected. Here's the final circuit with appropriate connectors to hook up to my existing probe and laptop adapter:

And here's what it looks like when it's all hooked together:

Now, this wouldn't be any fun without a bit of software and since the Measure activity can only display the voltage being presented (which is now a mixture of the sensor voltage and the reference voltage) what's needed as a new activity.

I found the developer documentation to be very hard to follow and I ended up hacking the existing Measure activity and renaming it Temperature.

The critical code is in the file drawWaveform.py where it reads self.avg (the value coming from the microphone input via the ADC) and scale it for display. I measured voltages coming from my probe for a couple of known temperatures and worked out a scale factor (The +32768 is because the self.avg ranges from -32768 to 32767):
layout.set_text("Temperature: %.1f C" % (0.00221833*(self.avg+32768)) )
Here's a screenshot of Temperature running on the laptop and measuring the ambient temperature in my office:

You can download my Temperature activity using the browser on your OLPC XO-1 to install it.

Sleeping with the enemy

2008-03-20T13:55:00.002Z

I loaded the dish washer:

My SO loaded the dish washer:

Who needs help?

First assume all new email is useless

2008-03-11T19:02:00.004Z

When I download email none of it goes in my Inbox. In fact, I don't have an Inbox. I work on the assumption that all new email is useless.

Many reports tell us that between 80% and 90% of all email is spam, so for starters only 10% to 20% is at all likely to be useful. Then, if you account for being on mailing lists, being CC:ed needlessly and receiving automatic updates such as order confirmations from Amazon.com, you'll see that almost all email is useless. Only a tiny fraction of the mail you receive is useful. And by useful I mean requiring action.

I use Thunderbird and my email folder structure looks like this:

When email arrives it is automatically sorted using POPFile into the folders: Family, GNU Make, Misc, polymail, POPFile and Spam. These six folders are the categories of mail that I receive:

POPFile: Since I wrote POPFile I get lots of mail about it and I use this is a general box for other open source projects I work on and anything else about anti-spam
polymail: Anything to do with my commercial product polymail and my consulting business
GNU Make: Anything to do with GNU Make or the company, Electric Cloud, that I co-founded
Family: Anything from my family
Misc: Order confirmations, airline tickets, PayPal statements, etc.
Spam: spam

POPFile uses Naive Bayesian text classification to automatically sort my email (with just a point and click interface for training) and then six rules (which never need updating) move the incoming mail based on POPFile's classification to one of those folders.

Of course, POPFile can be used to sort mail in any way you choose: my categories are unlikely to be yours. You might use POPFile to sort Work from Home from Spam. At least one journalist I know uses POPFile to sort Interesting from Boring from Spam so that he only gets to read interesting press releases.

When I identify mail that does need action taken I move it to the ACTION folder (which is the closest I've got to an Inbox). Moving mail there is a snap because I use the QuickMove extension for Thunderbird and have ALT-number keys mapped to each folder: one key press and the message is moved into or out of ACTION.

To keep on top of things I publish the number of items in my ACTION folder on my web site. Here's a live view over the last 24 hours. Currently, 9 items need dealing with.

My rules for managing email:

Assume that all new email is useless
Automatically sort email into folders on delivery
Take control of your Inbox: only you put email in it

Why Rails rules: continuous forward motion

2008-03-10T23:07:00.002Z

Lately I've been playing with Ruby on Rails and I'm impressed. Not by the documentation (I was pulling my hair out trying to map my copy of the Rails book that deals with 1.x to Rails 2.0 installed on my machine). Not by the screencasts, or by DHH being arrogant.

I'm impressed by the fact that Rails keeps you (or at least me) in continuous forward motion.

Yesterday sitting in an airport I decided to learn Rails. I had the two books (one on Ruby, one on Rails) which I'd read before, but I'd never actually coded anything. I had an idea for an application that was CRUD worthy.

Tonight, after a total of 4 hours of programming I have a working application in Rails that allows me to track health care expenses (appointments, bills, insurance reimbursements, payments, ...). Zero knowledge to working application in four hours isn't meant to illustrate my genius, it illustrates that Rails/Ruby is easy to learn and that the combination of generators and scaffolding keep you moving.

I've noticed in the past when working on apps that I'll come up against a difficult bit and go work on something easier ("Oh, I don't want to come up with foo-bar algorithm right now, I'll go design the buttons"). And the easier things are lower value.

Rails keeps me going after the functionality because it puts in place most of the functionality and then lets me evolve it. My application looks horrible (I've wasted no time on the CSS or HTML), but the functionality is there.

Some sleep and a little design work and it'll look like something.

Anyone else like access to a free application for health care expense management?

To the idiotic spammer posting comment spam on this site

2008-03-03T09:31:00.003Z

Since your name is two Chinese characters I'm going to address you as "Dude".

Dude,

Lately you've been posting comment spam on my blog for your World of Warcraft Gold. This is a little silly:

1. I'm fairly well known in anti-spam circles, did you really think I was going to let comment spam through on this site?

2. Comment moderation is turned on on this site. So your comment spam goes nowhere when I click the Discard button.

3. There has been a little some collateral damage from your World of Warcraft spamming. I accidentally killed two comments by Hypermechanic and I can't retrieve them. He/she wanted to say something useful about an old post:

You could do that like cameroid.com .
I guess in JAVA or .NET.

and

Cool I will hunt for it… This is a very sweet look app you have here. Even though you down play your role this is still brilliant.

Thank you for something new and useful.

Any sufficiently simple explanation is indistinguishable from magic

2008-02-28T13:44:00.004Z

Well, that's true if you are a fool.

Take for example the mystical belief that the number 11 or 11:11 is somehow significant. Uri Geller goes on about this on his web site. To quote from Geller's web site (and you'll find other similar thinking on many 11:11 web sites):

String theory is said to be the theory of everything. It is a way of describing every force and matter regardless of how large or small or weak or strong it is. There are a few eleven's that have been found in string theory.

I find this to be interesting since this theory is supposed to explain the universe! The first eleven that was noticed is that string theory has to have 11 parallel universes (discussed in the beginning of the "11.11" article) and without including these universes, the theory does not work.

The second is that Brian Greene has 11 letters in his name. For those of you who do not know, he is a physicist as well as the author of The Elegant Universe, which is a book explaining string theory. (His book was later made into a mini series that he hosted.) Another interesting find is that Isaac Newton (who's ideas kicked off string theory many years later) has 11 letters in his name as well as John Schwarz. Schwarz was one of the two men who worked out the anomalies in the theory. Plus, 1 person + 1 person = 2 people = equality.

Also, the two one's next to each other is 11. The two men had to find the same number (496) on both sides of the equation in order for the anomalies to be worked out, so the equation had to have equality! There were two matching sides to the equation as well because they ultimately got 496 on both sides. So, the 1 + 1 = 2 = equality applies for the equation as well.

I added a little bold type there because it amused me; pity that Mr Geller didn't look up the definition of equation before writing that line.

But key to this whole belief is that the number 11 keeps turning up at random. When I first read about this I looked up at the clock and it was 11:43. Whoa! Spooky!

But then I remembered Benford's Law. Benford's Law is essentially that in lots of real-life data the leading digit is 1 with a probability of about 30% (instead of the 10% you'd expect if the first digit was random from 0 through 9) and hence numbers beginning with 1 occur more often than numbers starting with any other digit.

A simple illustration is my clock experience. What's the probability that if you look at a clock at random that the first digit is a 1? Well it's more likely than any other number.

For a clock showing 12 hour time it cycles through: 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. A simple count will show you that the number 1 is the first digit for 8 out of the 24 hours and that all the other digits occur 2 times in 24 hours. So what's the probability that if I glance at a clock at random I'll see a 1 at the beginning? 8/24 or 1/3 of the time... which is Benford's Law.

Now, Benford's Law isn't restricted to time. It occurs all over the place (Wikipedia lists: electricity bills, street addresses, stock prices, population numbers, death rates, lengths of rivers, physical and mathematical constants) and so if you walk through life looking at random numbers you'll see numbers starting with a 1 more often than any other number. In 1988 a mathematician named Ted Hill showed why this is the case for many real-world systems.

But, what about 11? I hear you ask. Well if the first digit is more likely to be 1 than any other than it's clear that you are more likely to see numbers in the range 10 through 19 more than other two digit numbers, but a more interesting offshoot of Benford's Law is explained here.

Essentially as you walk through the digits of a number you are more likely to see a 1 than another digit, but that effect diminishes the longer the number gets. The probability that the the second digit is a 1 is about 11% (instead of the expected 10%) and given that the probability that the first digit is a 1 is 30% you are bound to come across 11 more frequently than you'd expect (if numbers were random).

So, it's no surprise that we see lots of 11s, and hence there's a simple explanation for all those 11s. Either that or I've been missing the call of the 11:11 Spirit Guardians all these years:

These 11:11 Wake-Up Calls on your digital clocks, mobile phones, VCR’s and microwaves are the "trademark" prompts of a group of just 1,111 fun-loving Spirit Guardians, or Angels. Once they have your attention, they will use other digits, like 12:34, or 2:22 to remind you of their presence. Invisible to our eyes, they are very real.

Would they hide me?

2008-02-27T06:48:00.005Z

This is pretty much exclusively a technical blog, but I was very struck by something legendary investor Warren Buffett said when answering the question "How do you define happiness and what about your life makes you most happy?":

I know a woman in her 80’s, a Polish Jew woman forced into a concentration camp with her family but not all of them came out. She says, “I am slow to make friends because when I look at people, I have one question in mind; would they hide me?” If you get to be my age, or younger for that matter, and have a lot of people that would hide you, then you can feel pretty good about how you’ve lived your life.

I know people on the Forbes 400 list whose children would not hide them. “He’s in the attic, he’s in the attic.” Some of them keep compensating by joining board seats or getting honorary degrees, but it doesn’t change the fact that no one will give a damn when they are gone. The most powerful force in the world is unconditional love. To hoard it is a terrible mistake in life. The more you try to give it away, the more you get it back.

The sum of the first n odd numbers is always a square

2008-02-14T08:35:00.003Z

I was staring at the checked pattern on the back of an airline seat the other day when I suddenly saw that the sum of the first n odd numbers is always a square. For example,


1
1 + 3 = 4
1 + 3 + 5 = 9
1 + 3 + 5 + 7 = 16

And, of course, it occurred to me that it would be nice to be able to prove it. There are lots of ways to do that. Firstly, this is just the sum of an arithmetic progression starting at a = 1 with a difference of d = 2. So the standard formula gives us:


sum_odd(n) = n(2a + (n-1)d)/2
           = n(2 + (n-1)2)/2
           = n(1 + n - 1)
           = n^2

So, the sum of the first n odd numbers is n^2.

But using standard formulae is annoying, so how about trying a little induction.


sum_odd(1) = 1

sum_odd(n+1) = sum_odd(n) + (2n + 1)
             = n^2 + 2n + 1
             = (n+1)^2

But back to the airline seat. Here's what I saw (I added the numbering, Lufthansa isn't kind enough to do that for you :-):

The other thing I noticed was this:

You can view the square as the sum of two simpler progressions (the sum of the first n numbers and the sum of the first n-1 numbers):


1 + 3 + 5 + 7 =
1 + 2 + 3 + 4 +
    1 + 2 + 3

And given that we know from Gauss the sum of the first n numbers if n(n+1)/2 we can easily calculate:


sum_odd(n) = sum(n) + sum(n-1)
           = n(n+1)/2 + (n-1)n/2
           = (n^2 + n + n^2 - n)/2
           = n^2

What do you do on long flights?

Tonight, I'm going to write myself an Aston Martin

2008-02-13T09:15:00.004Z

This is the story of my attempt to 'cheat' in an on-line spot-the-ball competition to win an Aston Martin. It's also the story of my failure, but you get free source code that implements automatic detection of image alteration using copy/paste or tools like the Clone Tool in Photoshop.

First, take a look at this photo:

Notice anything strange? In fact this image has been tampered with to cover up a truck. The truck is completely hidden by foliage. Here's the original:

Wouldn't it be nice to be able to detect that automatically? It is possible. Here's an image automatically generated by my code showing what was moved. All of the red was moved to the blue (or the other way around).

I was motivated to work on this program by greed (or at least my never-ending love of having a little flutter on things). Best of the Best runs spot-the-ball competitions in airports to win very expensive cars. But they also run the same competition online. That meant I could get my hands on the actual image used... could I process it to discover where the ball had been removed? (In reality, this isn't the right way to win because the actual ball position is not governed by where it actually was, but where a judge thinks it was).

Would it be cheating if I could? Apparently not, the competition rules say I should use my skill and judgment in determining the ball position. Surely, skill covers my programming ability.

So, I went looking for tampering algorithms and eventually came across Detection of Copy-Move Forgery in Digital Images written by Jessica Fridrich at SUNY Binghamton. The paper describes an algorithm for detecting just the sort of changes I thought I was looking for.

Unfortunately, I know nothing about image processing. Fortunately, the paper is written in a very clear style and a bit of Internet research enabled me to track down the knowledge I didn't have. (Also, thanks to Jessica for sending me the original images she used to test my implementation).

In brief the algorithm does the following:

Slide a 16x16 block across the entire image from left hand corner to bottom right hand corner. For each 16x16 block perform a discrete cosine transform (DCT) on it and then quantize the 16x16 block using an expanded version of the standard JPEG quantization matrix.
Each quantized DCT transformed block is stored in a matrix with one row per (x,y) position in the original image (the (x,y) being the upper left hand corner of the 16x16 block being examined).
The resulting matrix is lexicographically sorted and then rows that match in the matrix are identified. For each pair of matching rows (x1,y1) and (x2,y2) the shift vector (x1-x2,y1-y2) (normalized by swapping if necessary so that the first value is +ve) is computed and for each shift vector a count is kept of the number of times it is seen.
Finally the shift vectors with a count > some threshold are examined, the corresponding pair of positions in the image are found and the 16x16 blocks they represent are highlighted.

Here's another picture showing a golfing image that's been touched up to remove something from the grass:

To get access to image data I used the FreeImage library and wrote a small C program that implements Jessica's algorithm. You can download the source here; it's released to you under the GNU GPL.

The program has two key parameters that affect how the image is processed: the quality factor and the threshold.

The quality factor is a number used to 'blur' the image (actually it changes the quantization): the higher the factor the more blurring and hence more 16x16 blocks are likely to seem the same to the algorithm. Increasing the quality factor will tend to increase the false matches.

The threshold is simply the number of blocks that have to appear to have been copied together. This prevents us from seeing a single 16x16 block as evidence of copying. Increasing the threshold means ever larger groups of blocks have to be identified together before they are identified as copying.

Back at Best of the Best I grabbed the image for Supercar Competition (SC-272), cut out a section that I thought the ball had to be in (just to speed up processing) and ran the algorithm. After some parameter tweaking the algorithm came up only with what look like false matches to me (along the bar where it's all one color):

And, of course, that's not where the judge thought the ball was. So, I guess I won't be driving home in the V8 Vantage, but what geek needs that when they've got a cool piece of software that detects copy/move forgery in images?

Which leaves me with one question: how are spot-the-ball images generated? Is this an algorithm problem, a problem because they use JPG (which is already transformed) for their images, or are these images generated in some other way?

Interface to SQLite database in 23 lines of Arc

2008-02-12T21:28:00.000Z

One thing that the first release of Arc was missing was access to any sort of database, but that's easily remedied. Here are 23 lines of Arc code that provide access to a SQLite database:


(= db! 'nil)

(def db+ (name (o host "localhost") (o port 49153))
 (let (i o) (connect-socket host port)
   (db> o name)
   (if (db< i) (list i o))))

(def sql ((i o) q)
 (db> o q)
 (if (db< i) (readall i 200)))

(def db- (db)
 (map close db))

(def db> (o s)
 (write s o)
 (writec #\return o)
 (writec #\newline o)
 (flush-socket o))

(def db< (i)
 (= db! (read i))
 (iso db! 200))

The three functions you need to care about are db+ (get a connection to a named SQLite database), db- (close a connection to a database) and sql (execute a SQL query and return a list (or lists) of rows. There's also db! which contains the status of the last command (200 for OK, or 500 followed by a string explaining the error).

Here's a little Arc session creating a database, putting some data in it and then querying it. The database called test didn't exist at the start of this session:


arc> (= db (db+ "test"))
(#<input-port> #<output-port>)
arc> (sql db "create table foo (id integer primary key, text varchar(255))")
nil
arc> (sql db "select * from foo")
nil
arc> (sql db "insert into foo (text) values ('first');")
nil
arc> (sql db "select * from foo")
(("1" "first"))
arc> (sql db "insert into foo (text) values ('something else')")       
nil
arc> (sql db "select * from foo")
(("1" "first") ("2" "something else"))
arc> (db- db)
nil

To make this work I had to write a TCP server that wraps SQLite (it's just a small C program that you can get here). The C program listens on a port for connections from your Arc program and handles queries.

I did have to make a small patch to Arc itself (since arc0 doesn't contain any outgoing socket code). My patch adds the ability to make a TCP connection to a remote machine and to flush an output port (add this to your ac.scm):


(xdef 'connect-socket (lambda (host port)
       (let-values ([(in out) (tcp-connect host port)]) (list in out))))
(xdef 'flush-socket (lambda (s) (flush-output s)))

(Apologies if I have abused Scheme there, I'm a Scheme n00b)

All this code is released under the same license as Arc itself.

The leakiness of web mail

2008-02-12T08:59:00.000Z

Many people seem to use web mail systems like Hotmail or Yahoo! Mail as a way of providing anonymity. This is a mistake because all these systems leak the IP address of the machine the user is typing on!

Hotmail

Here are part of the headers of a message that a family member sent me from their Hotmail account:


Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
Received: from 134.151.225.153 by lw14fd.law14.hotmail.msn.com with HTTP;
X-Originating-IP: [134.151.225.153]

This leaks that original IP address (134.151.225.153) twice: once in an X-Originating-IP header and once in the first Received header which indicates that it was received from the same IP address using HTTP (i.e. using the web). A quick lookup shows that that IP address is in Birmingham, UK (which I happen to know is correct). So, if they were trying to keep their location secret, they've failed.

A whois lookup on that IP address tells me even more information, including that fact that is belongs to an Aston University. So, it's easy to conclude that this family member was student or staff at that university.

Yahoo! Mail

Yahoo! Mail leaks in a similar way. Here are part of the headers of a message I received from someone with what looks like a random email address and no name:


Received: from [193.95.59.139] by web25709.mail.ukl.yahoo.com via HTTP;

Geo locating that IP address shows me that the writer is in Tunisia.

Another Yahoo! Mail leak from an old colleague in California let's me track down their home city from their DSL line.


Received: from [67.102.112.112] by web14204.mail.yahoo.com via HTTP;

AOL Mail

Here are some headers from a message sent from an AOL web mail account that reveal that the sender is in Germany and looks like it gives away the name of the company that they are working for in the DNS name of the machine:


X-MB-Message-Source: WebUI
X-AOL-IP: 62.128.31.9

The X-AOL-IP gives the IP address of the machine that generated the message (i.e. where the web browser is running) and the helpful X-MB-Message_Source tells us they are using the web interface.

Earthlink

Here's an email I received from the editor of Wired who was using Earthlink:


Nice one! When I get off dialup from the French countryside, I'll blog
that...

Was he really in France?


X-Originating-IP: 213.11.198.147

Yes

Others

A search of my own email showed me that X-Originating-IP is a popular leak point (used by Inbox.com, kth.se, Network Solutions, MSN.com and others).

Google Mail and Hushmail

Neither Google Mail nor Hushmail appear to leak the IP address. They may include the IP address (for example, in the Message-ID) but it does not appear to be readily discoverable.

My first Arc project: a simple Wiki

2008-02-11T18:00:00.000Z

The only way to learn a programming language is to write something in it. So, I decided it was time to dig into Arc and my first project is a very simple Wiki.

Here's the source (wiki.arc):


; A wiki written in Arc (arc0)
;
; Copyright (c) 2008 John Graham-Cumming
;
; (load "wiki.arc")
; (wsv)
;
; Then go to http://localhost:8080/show

(load "web.arc")
(load "util.arc")

(= pagedir* "wiki/")

(def histfiles (page)
  (sort > (map [coerce _ 'int] (rem [is "current" _] (dir (pagepath page))))))

(def nexthist (page)
  (let h (histfiles page)
    (if h (++ (car h)) 0)))

(def pagepath (page)
  (string pagedir* (page 0) "/" (page 0) (page 1) "/" page ))

(def pagefile (page (o file))
  (string (pagepath page) "/" (or file "current")))

(def slurp (page (o file))
  (if 
  (let p (pagefile page file)
    (if (file-exists p) (readfile p)))))

(def upperlen (word)
  (len (keep upper word)))

(def is-wikilink (word)
  (if (alphas word)
    (if (~is (word 0) (downcase (word 0)))
      (>= (upperlen word) 2))))

(mac url-show (page)
  `(string "show?p=" ,page))

(mac url-edit (page)
  `(string "edit?p=" ,page))

(mac link-show (page text)
  `(link ,text (url-show ,page)))

(mac link-edit (page text)
  `(link ,text (url-edit ,page)))

(def wikify (word)
  (if (is-wikilink word)
    (if (file-exists (pagefile word))
      (link-show word word)
      (pr word)(link-edit word "?"))
    (pr word))
  (ws))

(mac spew-raw (page)
  `(spew ,page [pr _ " "]))

(mac spew-wiki (page (o file))
  `(spew ,page [wikify (string _)] ,file))

(def spew (page f (o file))
  (let p (pagepath page)
  (if (dir-exists p) 
    (map f (flat (map tokens (slurp page file))))
    (pr "This page does not yet exist."))))

(def squash (file body)
  (writefile1 body file))

(def save-page (req)
  (w/$ p
    (w/$ t
      (squash (pagefile p) t)
      (squash (string (pagepath p) "/" (nexthist p)) t))
    (url-show p)))

(mac mtime (f)
  `(datetime (file-mtime ,f)))

(def last-modified (page)
  (let f (pagefile page)
    (if (file-exists f) 
      (pr "Last modified: " (mtime f)))))

(mac show-page (page)
  `(whitepage
    (tag h1 (link-show ,page ,page))
    (spew-wiki ,page)
    (br 2)
    (hr)
    (last-modified ,page)
    (br)
    (link-edit ,page "[edit]")
    (ws)
    (link "[history]" (string "history?p=" ,page))))

(mac edit-page (page)
  `(whitepage
   (tag h1 (pr (string "Editing " ,page)))
   (arform save-page 
     (textarea "t" 25 80 (spew-raw ,page))
     (hidden "p" ,page)
     (br)
     (submit "Save"))
     (link-show ,page "[cancel]")
   (br 2)))

(def revision (page rev)
  (tag li
    (pr "Revision: " )
    (link rev (string "revision?p=" page "&r=" rev))
    (pr " modified " (mtime (string (pagepath page) "/" rev)))))

(mac history-page (page)
  `(whitepage
   (tag h1 (pr (string "History of " ,page)))
   (tag ul (map [revision ,page _] (histfiles ,page)))
   (hr)
   (link-show ,page (string "Back to " ,page))))

(mac revision-page (page rev)
  `(whitepage
    (tag h1 (pr "Showing revision " ,rev " of " ,page))
    (spew-wiki ,page ,rev)
    (br 2)
    (hr)
    (last-modified ,page)
    (br)
    (link-show ,page (string "Back to " ,page))))

(defop show req
  (w/$ p
    (if p
      (show-page ($ "p"))
      (show-page "HomePage"))))

(defop edit req
  (w/$ p
    (ensure-dir (pagepath p))
    (edit-page p)))

(defop history req
  (history-page ($ "p")))

(defop revision req
  (revision-page ($ "p") ($ "r")))

(def wsv ()
  (ensure-dir pagedir*)
  (asv))

It loads two helpers. The first contains common utilities that aren't really Wiki-related (util.arc):


(def alpha (c)
  (or (<= #\a c #\z) (<= #\A c #\Z)))

(def alphas (str)
  (is (keep alpha str) str))

(def upper (c)
  (is (upcase c) c))

(def datetime ((o time (seconds)))
  (let val (tostring
    (system (string "date -u -r " time " \"+%Y-%m-%d %H:%M\"")))
      (subseq val 0 (- (len val) 1))))

And the second contains enhancement to Arc's web/HTML handling (web.arc):


(mac hidden (name val)
  `(gentag input type 'hidden name ,name value ,val))

(mac hr ()
  `(gentag hr))

(mac ws ()
  `(pr " "))

(mac $ (r)
  `(arg req ,r))

(mac w/$ (r . body)
  `(with (,r ($ (string ',r))) ,@body))

In web.arc there are a couple of bits of syntax to make accessing form/URL arguments easier: ($ "p") (which gets the value of the argument p) and (w/$ p ...) which sets a variable called p to the value of the argument p and then evaluates the rest of the expression.

All this is released under the same license as Arc. (Since I have never programmed in Arc before, and it's been almost 20 years since I stopped coding in LISP or ML, I'd appreciate constructive comments).

PPP3 (final version) in Java and C

2008-02-11T08:38:00.000Z

Steve Gibson has released the final version of his PPP system: PPPv3 and so I've updated my code to be compatible.

Two versions of PPPv3 are available:

Both are released, as before, under the BSD license.