I don’t know if anybody else has noticed, but Spam filters have gotten a *lot* better since the 90′s. This is due to a dramatic leap forward in how our filters are designed. The technical design behind this improvement is fascinating, especially to a programmer, but rather than go into the real nitty-gritty, I’m going to give a real brief over-view, and then discuss some of the implications.
The fifty-thousand foot view of the new spam filters is this: They’ve gotten so much better because they’re no longer maintained by humans. Instead, a spam filter is initially given a training set of, say, 50,000 emails. These emails are flagged as ‘spam’ or ‘not spam’. The filter then chews through those emails, and stores every single individual word. As the messages are digested, every word present is ‘weighted’, which represents how useful that word is in determining whether a word is spam or not. By the end of the training, the filter can achieve an accuracy rating of 99.99%; on average, only one message in ten thousand will be flagged incorrectly. Then, every time you manually flag a message as ‘Spam’ or ‘Not Spam’, the message is processed and added to the global filters.
The existence of this filtering technology has basically eradicated spam on all services utilizing it, and it has done so without accidently deleting millions of genuinely useful emails at the same time. The filters constantly are constantly learning, which means spammers have a permanent uphill battle to get spam email through.
This filtering technology is wonderful; I love the fact that my gmail account almost never receives spam messages anymore, and I don’t have to constantly check my spam box to check for misrouted emails.
The legal implications are fairly staggering if this technology is expanded to other fields, however. Since email servers are already scanning every message coming in for spam, changing the filters to look for terrorist emails is a natural extension. I would be extremely surprised if the government is not already planning this type of software; it’s simply too effective to pass on for long, and with the Patriot Act, I imagine any legal hurdles would be minor at best.
What is a reasonable expectation of privacy in regards to email? Emails handled through a corporate server are usually considered public to the company; you can be fired for slandering someone, or for sending pornographic material. But should we really consider private emails to be government accessible? After all, the USPS can’t read our mail.
I think a big player in the whole spam filter / info gathering issue is Google. It’s really the 800lb gorilla here. Google gathers such a massive amount of information from its Search and Gmail services that it is difficult to fathom. And at least for now, Google has been able to avoid the bad press by their generally-accepted “Don’t Be Evil” credo and repeated statements that the data is kept anonymous. But the potential for abuse or use of the data (either by hackers, disgruntled Google employees, or the Feds) is not discussed much.
In the legal world in the U.S. at least, policy is often based on case law. In this case, I’m betting that a major player like Google might be the one to set legal precedent – good or bad. I certainly have appreciated Gmail’s exemplary spam filtering abilities, but I also realize the potential issues with data confidentiality.