Filtering's ding-dong fight with malicious spam

Attempts by governments and police to stop spam at its source have proved futile. But as the threat has evolved, so have the filtering techniques that help ensure spam never reaches the user, as Anthony Plewes reports.

No one knows just how much of the email traversing the internet is unwanted, unsolicited marketing communications. Estimates suggest spam accounts for between 70 per cent and 90 per cent of all email. While some experts say it has peaked, others believe the torrent will intensify.

The exact scale of the spam problem may be disputed but it is commonly agreed that each year it gets more invidious and criminal in nature. As outlined in the first article of this special report there are millions of infected computers forming botnets, whose primary purpose is to spew out countless messages urging us to buy Viagra, or providing the latest 'hot tip' in shares.

The modern spammer is either a member of an organised criminal gang or works for one, and he relies on the gullibility of consumers and sheer volume to sell his wares.

In some cases the connection to crime is explicit. The pump-and-dump stock spam, for example, is initiated by criminals who buy up stock of publicly traded companies with stolen credit card details.

Just 24 hours later, they dump the stock which they hope is trading at a higher price thanks to their pumping. The criminals' money is laundered with a tidy profit even before the credit card victim knows their card has been compromised.

The Spamhaus Project has been collecting information on known professional spam operations for years and its weekly top 10 most-wanted list of most prolific spammers is dominated by spam gangs from Russia and eastern Europe.

Stopping these people is proving impossible. Legislation has been a damp squib because most perpetrators of spam base themselves in regions where the law can't touch them.

Even the capture of long-time US spam pest Robert Solway was a Pyrrhic victory with no effect on spam volumes whatsoever.

Filtering out unwanted email

So if stopping spam at the source is ineffective, it needs to be intercepted before it reaches the user. Irrespective of what percentage of email is spam, the halcyon days when genuine emails outnumbered spam have long gone.

Fortunately, email filtering technology has improved so much that we can actually continue to use email as a communications tool.

Spam filtering uses a range of techniques to root out spam. These include looking for specific spam signatures, blocking certain IP address ranges from which spam is originating and using Bayesian and heuristic filters to identify spam.

Like much of the security industry, success in stopping spam ebbs and flows when each innovation from spammers is overcome by a new filtering technique from the security industry.

Guillaume Lovet, manager of the threat response team at Fortinet, says at the start of the ongoing battle spammers first used plain text but this was picked up by filters.

"They then used images, so we had to develop technology to extract the text from images. Next to fool filters that rely on word frequency, spammers obfuscated the text by inserting it within some decoy text," says Lovet.

"Now they are blending the two strategies so need to extract the meaning from text and images, and to stop us reading the image they are adding noise so it looks like a Captcha test [the textual recognition test that people need to take when signing up for accounts to prove they are not a machine]."

The upshot of this dizzying escalation is that much spam is becoming difficult to read and easily identified by users. This is having an effect on the spammers' click through rates (CTR), which - according to Lovet - have slumped to less than one in several million. Spammers need to get cleverer and use social engineering techniques seen in virus distribution to try and improve their CTR.

One approach is to create more sophisticated spam and this was demonstrated in 2007 with the success of PDF spam. Because they mimic real analyst reports, they are a convincing medium for stock spam.

For a three-month period in 2007, according to filtering company Ironport, the volume of PDF spam surpassed traditional image-based spam, accounting for tens of billions of messages on some days. Towards the end of 2007, spammers were also investigating the potential of MP3 and Excel files and prime candidates for 2008 are PowerPoint presentations or even Flash and video files.

The rise of malicious spam

Text-based spam, however, has not gone away and is the primary vehicle in the other key trend - the rise of malicious spam. These spam messages contain just a few words of text and a link to a web page, whose purpose is solely to infect the user with a Trojan.

Ironport's recent 2008 Internet Security Trends report estimated more than 80 per cent of spam now contains a URL. This is a dramatic change from previous spam that contained an actual call to action.

Email is not the only communications channel that is used by spammers. Although instant messaging (IM) has been relatively free of spam, this could change in 2008 if MessageLabs is to be believed. Its predictions for next year suggest that IM might be used like telemarketing.

Spammers would send out spam to 100 targets and wait for a response, if it is successful, then they will send out a next batch and if not, switch to a new message. Volumes would be much lower but there is a potential for a much better CTR.

One area where spam has made more of an impact is on the web in places such as comments on blogs and forums. Security company Websense predicts there will be a lot more of this in 2008, particularly in malicious spam to drive users to infected sites.

Automated agents can be used that aggregate RSS feeds and when a blog is updated, the spam can be sent automatically as a comment to the source and all that's needed is a link to an infected website.

The rise of malicious spam makes it even more important to try and stop the flood of unwanted communication. Some believe ISPs should do more to block spam being sent out.

Mark Sunner, chief security office at MessageLabs, says: "You wouldn't be expected to clean your own water - but in internet terms that's what you are expected to do."

"ISP are belching out the equivalent of raw sewage and saying: 'You sort it out, it's not our problem'. That has got to change," says Sunner.

But legislation and ISPs can only do so much, and users have to take action to protect themselves from spam and malware on all channels, including email, the web and IM.

This demands a multi-layered approach to spam protection encompassing filtering at various levels, such as the ISP, mail host, server and desktop.

Mark Deakin, manager for unified communications at Microsoft, says offloading the first load of filtering to a third party stops your mail server becoming overloaded. "The final layer of protection on the desktop will protect you against spam coming internally or through other channels," he says.

Deakin says: "Using different service providers' filters will also help you stop a higher proportion of spam as they will be updated at different times and are better at stopping different things."

Other techniques are also useful, adds Deakin, such as identifying if an IM user ID suddenly sends out huge numbers of messages.

Although warnings five years ago that email would become unusable because of the flood of spam have not been realised, there is little doubt that the medium is tarnished.

False positives continue to damage the effectiveness of email as a business communications channel, and users need take a more holistic approach to communications. However, the alternative comms channels are also potential vehicles for spam, and users and businesses must not let their guard slip.