My bayesian filter gets trained mostly by me hitting “this is spam” when spam makes it to my inbox. If I’m expecting an email “immediately” – something like a mailing list COI confirmation or email as part of buying something online – I’ll check my spam filter and move the mail to my inbox in the rare case it ended up there. Other than that I let it and spamassassin chug along with no tweaking.

I’m starting a data analysis project, based on my own inboxes, and as part of that I’m using some tools to look for false positives in my junk folders, and manually fixing anything that’s misclassified. I’ve been doing this for a couple of hours now, and I’ve found some interesting things.

1. Simple content filters work remarkably well out of the box, at least for my mail stream. Spectacularly well. There’s very little in the way of false positives. Very, very little.
2. Of those false positives there’s nothing I’d have been bothered about. It’s generic, unexciting junk mail.
3. Most of the systemic false positives seem to be correlated with the senders doing something bad. Heathrow Express, for instance, sent me mail every two weeks or so since I’d signed up. Then for no obvious reason they stopped sending for three months, then started sending again. Every mail they sent after that pause ended up in the junk folder, and I never missed them.
4. I get regular newsletters from ThinkGeek. Every one of those goes to the inbox. I occasionally get mails from them about my account (“you’ve got 420 geek points left”) that are kinda transactional, but not something I expect to see – and they all end up in the junk folder. Several other senders do the same thing, and get the same result.
5. Several companies have used tagged addresses to send me newsletters for a while, and also used them to send unsolicited facebook invites (to the tagged address, from facebook servers). The regular newsletters all go to the inbox, while the facebook invites all go to the junk folder. “Legitimate” facebook mail, meanwhile, keeps going to the inbox.
6. Apple send me a lot of newsletters – I’m a Mac and iPhone developer, I get their consumer newsletters, transactional stuff from our local store – lots and lots of newsletters. They all made it to the inbox except for one. The one that ended up in the junk folder was a one-off about recycling, and it wasn’t up to their usual design standards – it had ugly big green “call to action” headlines in it, very different to their usual clean design.
7. Just one sender hit the junk folder every time. The distinctive thing about their messages (apart from them not being something I missed) was that the plain text part of them was dreadful, just a bad lynx dump of the html section. Even a “No plain text for you! Go to this link!” would have been better.

I was surprised at how effective this simple content-based filtering setup had worked with little tuning other than hitting the this-is-spam button – both in it’s accuracy at removing spam while keeping a very low false positive rate, but also how well the false positives matched my judgement of “Meh. This mail isn’t interesting.”.

We spend a lot of time talking about the things you should do to make the mail relevant to the recipient – compelling content, consistent, predictable delivery schedules, clear consistent branding, use of a single consistent mail stream to communicate with a recipient rather than several different streams. Until I went through this exercise it wasn’t clear to me how much of an effect those things also have on fairly simple recipient-trained filters.

Generally IPs that the ISP has not seen traffic from before starts out with a slight negative reputation. If you think about all the new IPs that an ISP will see mail from on a daily basis, 99 out of 100 of those will be bot infected windows boxes. So they’re going to treat that mail very suspiciously. And, in the grand scheme of things, that mail is going to be spam a lot more than it’s not going to be spam.

Some ISPs put mail in the inbox and bulk foldering during the whitelisting process. Basically they’re looking to see if your recipients care enough about your mail to look for it in the bulk folder. This then feeds back to create the reputation of the IP address. There is another fairly major ISP that told me that when they’re seeing erratic data for an particular sender they will put some mail in bulk and some mail in the inbox and let the recipients tell the system which is more correct.

That’s what happens while you’re establishing a reputation on an IP. Once there is some history on the IP, things get a little different. At that point, IP reputation becomes unimportant in terms of bulk foldering. The ISP knows an IP has a certain level of reputation, and *all* their mail has that level of reputation. So bulk foldering is more related to content and reputation of the domains and URLs in the message.

The other reason IP reputation isn’t trumping domain / content reputation as much as it did in the past is that spammers stomped all over that. Affiliates, snowshoers, botnets, all those methods of sending spam made IP reputation less important and the ISPs had to find new ways to determine spam / not spam.

So if you’re seeing a lot of bulk foldering of mail, it’s unlikely there’s anything IP reputation based to do. Instead of worrying about IP reputation, focus instead on the content of the mail and see what you may need to do to improve the reputation of the domains and URLs (or landing pages) in the emails. While the content may not appear that different, the mere mention of “domain.com” where domain.com is seen in a lot of spam can trigger bulking.

1. I get 15 copies. There are a lot of spammers out there who buy and scrape addresses and don’t do even the simplest of de-duping. Send multiple copies to a single address, you’re probably spamming.
2. I get mail to a non-tagged address. I use tags for every signup, and have done since mid-1999 or so. If I get commercial email to a non-tagged address, I know it’s spam.
3. I get to a tagged address from someone it wasn’t given to. As above, the tags remind me who was given my email address. If mail comes into a tagged address and it wasn’t given to the sender, then I know the mail is spam.
4. I get mail to a poorly harvested address. Another subset of the tagged addresses, there are a couple badly written web harvesters out there which add random characters onto the end of my tags. So I get mail to -infon@ and -infonn@ and -infonnn@ addresses, and I know the sender is spamming.
5. Someone attempts to send mail to an address that never existed at my domain. Even better, I don’t have to receive the mail for this one. If you attempt to send mail to a non-existent user here, I know without even looking that the mail was not asked for.

Am I wrong?

That has a bunch of implications. If you run an affiliate programme where your affiliates use your URLs then spam sent by your affiliates can cause your (clean, opt-in, transactional) email to be treated as spam. If you send a newsletter with advertisers URLs in it then bad behaviour by other senders with the same advertisers can cause your email to be spam foldered. And, as we discussed yesterday, if spammers use the same URL shortener you do, that can cause your mail to be marked as spam.

Even if the hostname you use for your URLs is unique to you, if it resolves to the same IP address as a URL that’s being used in spam, that can cause delivery problems for you.

What does this mean when it comes to using URL shorteners (such as bit.ly, tinyurl.com, etc.) in email you send out? That depends on why you’re using those URL shorteners.

The URLs in the text/html parts of my message are big and ugly

Unless the URL you’re using is, itself, part of your brand identity then you really don’t need to make the URL in the HTML part of the message visible at all. Instead of using ‘<a href=”long_ugly_url”> long_ugly_url </a>’ or ‘<a href=”shortened_url”> shortened_url </a>’ use ‘<a href=”long_ugly_url”> friendly phrase </a>’.

(Whatever you do, don’t use ‘<a href=”long_ugly_url”> different_url </a>’, though – that leads to you falling foul of phishing filters).

The URLs in the text/plain parts of my message are big and ugly

The best solution is to fix your web application so that the URLs are smaller and prettier. That will make you seem less dated and clunky both when you send email, and when your users copy and paste links to your site via email or IM or twitter or whatever. “Cool” or “friendly” URLs are great for a lot of reasons, and this is just one. Tim Berners-Lee has some good thoughts on this, and AListApart has two good articles on how to implement them.

If you can’t do that, then using your own, branded URL shortener is the next best thing. Your domain is part of your brand – you don’t want to hide it.

I want to use a catchy URL shortener to enhance my brand

That’s quite a good reason. But if you’re doing that, you’re probably planning to use your own domain for your URL shortener (Google uses goo.gl, Word to the Wise use wttw.me, etc). That will avoid many of the problems with using a generic URL shortener, whether you implement it yourself or use a third party service to run it.

I want to hide the destination URL from recipients and spam filters

Then you’re probably spamming. Stop doing that.

I want to be able to track clicks on the link, using bit.ly’s neat click track reporting

Bit.ly does have pretty slick reporting. But it’s very weak compared to even the most basic clickthrough reporting an ESP offers. An ESP can tell you not just how many clicks you got on a link, but also which recipients clicked and how many clicks there were for all the links in a particular email or email campaign, and how that correlates with “opens” (however you define that).

So bit.ly’s tracking is great if you’re doing ad-hoc posts to twitter, but if you’re sending bulk email you (or your ESP) can do so much better.

I want people to have a short URL to share on twitter

Almost all twitter clients will abbreviate a URL using some URL shortener automatically if it’s long. Unless you’re planning on using your own branded URL shortener, using someone else’s will just hide your brand. It’s all probably going to get rewritten as t.co/UgLy in the tweet itself anyway.

If your ESP offers their own URL shortener, integrating into their reporting system for URLs in email or on twitter that’s great – they’ll be policing users of that just the same as users of their email service, so you’re unlikely to be sharing it with bad spammers for long enough to matter.

All the cool kids are using bit.ly, so I need to to look cool

This one I can’t help with. You’ll need to decide whether bit.ly links really look cool to your recipient demographic (Spoiler: probably not) and, if so, whether it’s worth the delivery problems they risk causing.

And, remember, your domain is part of your brand. If you’re hiding your domain, you’re hiding your branding.

So… I really do need a URL shortener. Now what?

It’s cheap and easy to register a domain for just your own use as a URL shortener. Simply by having your own domain, you avoid most of the problems. You can run a URL shortener yourself – there are a bunch of freely available packages to do it, or it’s only a few hours work for a developer to create from scratch.

Or you can use a third-party provider to run it for you. (Using a third-party provider does mean that you’re sharing the same IP address as other URL shorteners – but everyone you’re sharing with are probably people like you, running a private URL shortener, so the risk is much, much smaller than using a freely available public URL shortener service.)

These are fairly simple fixes for a problem that’s here today, and is going to get worse in the future.

1. Make a URL shorter
2. Track clicks on the URL
3. Hide the destination URL

Making URLs shorter was their original role, and it’s why they’re so common in media where the raw URL is visible to the recipient – instant messaging, twitter and other microblogs, and in plain text email where the “real” URL won’t fit on a single line.

From the moment they were invented they’ve been used to trick people to click on links to pages they’d rather not visit, from musical classics to less tasteful content. And, in just the same way, spammers quickly found that they were a good way to avoid content-based filters or to hide a suspicious looking target URL.

Inevitably, URL shorteners that are persistently abused by spammers (especially those where that’s done with the support of the URL shortener operator) start to be seen as a sign of spam, and email that uses them will be treated with suspicion by content-based spam filters and often sent to the spam folder.

bit.ly is probably the highest profile URL shortener, so it’s the one you’ll most likely see people trying to use in email. What effects does that have?

Now being “totally owned” by the Canadian Pharmacy gang, thousands of URLs being spammed with very slow takedowns. Not good.SpamHaus on bit.ly

bit.ly have been on SpamHaus’s radar for quite a while. They’re listed on the SBL multiple times. They’re listed in the DBL – SpamHaus’s newish domain based blacklist, intended for content-based filtering of email. All this means that emails that contain bit.ly URLs are increasingly likely to have serious delivery problems.

This isn’t unique to bit.ly: many other URL shorteners have similar problems – j.mp, su.pr, and others. Nor is it unique to SpamHaus: many other spam filters, public and private, are starting to treat common URL shorteners with suspicion.

Naive use of URL shorteners in your email will send it to the spam folder.

More about why you shouldn’t do that – and what you can do instead – tomorrow.

I take a look.

I’ve reserved a seat for you (and up to 2 guests from Word) at your choice of upcoming, complimentary lunch seminars that I will be hosting around the Bay Area …

Sure enough, it’s spam. And it was sent by a “senior account executive” at an ESP.

I’m pretty sure it was sent using harvested or epended data – they think my company name is “Word” rather than “Word to the Wise”, it was sent to my personal email address rather than my @wordtothewise.com one, and the postal address information I found they had on file was wildly wrong – “San Francisco, CA 94101″ isn’t quite as obviously fake as “Beverly Hills, CA 90210″, but it’s close. They probably bought a list from Jigsaw or one of their competitors.

There’s no pretense of permission – I didn’t recognize them, didn’t give them my email address, and have no interest in their offerings. If they’re purchasing lists this bad, that’s probably true of most of the recipients – and those recipients are going to consider it spam too.

That’s not all. It probably violates CAN-SPAM. There’s the deceptive subject line and also several problems with the unsubscription link that probably make it non-compliant. Any ISP postmaster, spam filter maintainer or blacklist volunteer who looks at the mail is not going to be impressed. Heck, one blacklist maintainer – one of the sane, responsible, professional ones – I mentioned it to thought it would be adequate grounds for adding the sender to their blacklist.

Yet the ESP is reasonably mainstream. They’re a MAAWG member, and it turns out I know their deliverability / isp relations manager. I’m pretty sure they don’t let their customers get away with this sort of thing – but internal (or “friends and family”) accounts don’t get the same sort of oversight as customers, much the same as most other companies.

The sales guy sent it through the ESPs production systems – it’s from one of their smarthosts, DKIM signed by the ESP and uses the ESPs click-tracking domain in the body of the message. So the spam is going to damage their reputation – IP, domain and social.

I’m betting that they’ll be seeing higher complaint rates and some delivery problems over the next few days, due to one sales guy sending spam using their systems. Longer term, and potentially more seriously, people in the email industry are likely to remember them as spammers and be less prone to be helpful or cut them slack when they make a mistake.

It’s Wednesday. Do you know where your sales staff are?

We don’t get much spam on Yahoo, mostly because we don’t give our email address out much. The only spam we really get is from <stockbroker website>, and that all goes to the spam folder. We use the site for checking stock quotes – it’s free, and we never see any of the spam they send.

A typical email marketer would look at that and object loudly to her use of the “S word” to describe their email – it’s mail the subscriber signed up and gave permission for, and they have an ongoing relationship with the sender, and they haven’t unsubscribed, and, and, and…

But a delivery expert will point out that none of that matters one jot. Sure, the sender has a figleaf of permission, because they convinced the recipient to “subscribe to their mailings” (even if that was via the threat of withholding a free web service if they didn’t sign up). And that does provide some legal protection.

But as far as delivering email to recipients inboxes, let alone receiving any ROI for an email campaign, it’s pretty much irrelevant. The recipient perceives the mail as spam, and describes it as such to other people – “<stockbroker company> sends spam” is not the image you want to have. The subscriber doesn’t read the email, doesn’t want the email, certainly doesn’t pull it out of the spam folder and may well be hitting the “this is spam” button for messages that end up in the inbox.

You’re certainly not getting any benefit at all from that subscriber, and their relationship to the mail you’re sending them – not opening or interacting with it, categorizing it as spam, etc – is teaching their ISPs spam filters that your mail is unwanted spam. The reputation of your domain, your content and your sending IP addresses will suffer, and your delivery rates to all your subscribers will suffer.

If you’re forcing someone to give you permission, it’s not permission-based marketing.

Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 554 554 5.7.1 The IP address of web site www.client.com [75.101.163.44] is listed at www.spamhaus.org (state 18).

What? An SBL listing? The client hadn’t done anything wrong, certainly nothing that would provoke the wrath of Spamhaus. And… they’re not sending email from 75.101.163.44 anyway, they’re sending it out through Google Apps. And, wait, www.client.com is listed on the SBL – the SBL lists IP addresses, not hostnames.

What is going on here?

We’ve mentioned in passing before that one of the good ways to filter mail based on content is to look for suspicious URLs in the message. One way of doing this is to use hostname-based blacklists, such as SURBL, URIBL or DBL. These list domain names that have been seen in spam (and pretty much only spam), and sending email with a listed hostname in it is a quick trip to blocksville at many ISPs.

Spammers, phishers especially, often cycle through domain names quickly in order to avoid the (manually maintained) hostname-based blacklists. They often host them at the same place, though, so if you look up the IP address the hostname resolves to you can use an IP based blacklist to see if the hostname is being used for spam or phishing related email payloads, and use that information to block the email. That’ll work even if the phishers use an entirely new domain for their websites, if it’s still hosted at the same place.

The SBL blacklist is commonly used in this way. It’s manually maintained and fairly hard to get on to, and finding URLs that resolve to addresses listed on the SBL in an email corresponds pretty strongly to the mail being unwanted. The folks who run the SBL are quite aware of this, and will commonly list IP addresses that are being used to host websites advertised in spam even if they never send email.

What happened in this case was that the client was hosting their website with Heroku, a perfectly respectable cloud-based ruby-on-rails web host. But Heroku use just three IP addresses for all their customers. And one of their customers was zapt.in, a URL shortener with a serious spam problem. Zapt.in caused problems for long enough, and didn’t respond to them for long enough, that their IP address was listed on the SBL.

That meant that all of Heroku’s customers were using an IP address listed on the SBL. Which, in turn, meant that any email those customers (or their affiliates or customers or…) sent that used the customers domain would be rejected by ISPs using the SBL-as-a-hostname-blacklist trick – which is a lot of large ISPs.

What can you do to avoid this? The ideal is to not host your website on a shared IP address (or in a /24 that’s littered with spam and phishing sites).

If you can’t do that – you really can’t move your main website to a more reputable host – then your next best option is to not use your main website in any of the email you send. You don’t want to hide the connection (because you don’t want to look like a snowshoe spammer who’s obfuscating their domain ownership), but you want the hostname to be different. A good way to do that is, if your main domain name is example.com and your website is www.example.com, is to use a subdomain for URLs in emails. click.example.com, maybe.

Host that subdomain somewhere else, on an IP address you have a bit more control over – an inexpensive VPS or web hosting provider, and just run a web redirector there that simply sends an http 302 redirect for any http://click.example.com/foo/bar/baz.html to http://example.com/foo/bar/baz.html. And use the click.example.com form of the URL for everything you use in your email – not just links, but also image tags and so on.

What if setting up your own redirector isn’t something you have the resources to do? Sign up for a web redirector or URL shortener service that’ll let you use your own domain name, like bit.ly Pro. That won’t give you as much control, or protection, as running your own redirector but it’s a lot better than running your website on a shared IP address.

One thing she doesn’t mention is that while bulk foldering can sometimes be the result of poor content, more often it’s the result of unengaged recipients. Think of bulk foldering this way: the ISP has some subscribers they’re pretty sure want your mail, so they’re not going to block your mail. But they’re pretty sure a lot of subscribers don’t want your mail so they’re not going to deliver it to the inbox.

The trick to getting mail moved out of the bulk folder is to get more people engaged with your email marketing. This is tough to do if they’re not actively checking their bulk folder for mail but there are some ways I’ve helped clients get mail into the inbox.

1. Contact customers through a method other than email. This works well to remind them to look for your mail. You can also tell them how to add your From: address to their address book which will override the ISP bulk foldering decision.
2. Segment your list and focus on engaged subscribers. Improving the reputation of your IP addresses and content will help improve delivery. Focus on those people who are highly engaged (clicks and opens) and stop mailing those people who aren’t reading your mail any way.
3. Create a re-engagement campaign with enticing subject lines. Your goal is to get unengaged recipients to actively seek out your email in their bulk folder and move it to the inbox. Having good subject lines (test, test, test) is critical to this effort. People just glance through their spamfolder and don’t open mail unless the subject line catches their interest. If you can catch their interest enough to have them open the mail and move it to their inbox then you’ll end up in their inbox by default.
4. Let go of your unengaged recipients. I know it’s difficult to admit that it’s over between you and some of your recipients. But, it really is over. The longer you hold on to them the more they’re going to hinder your relationship with other recipients. Dropping the long term unengaged off your list will help increase your reputation and inbox delivery.

None of these are quick fixes. When I’ve worked with clients to deal with long term bulk foldering problems it’s taken from 2 to 4 months to resolve the issue. In some cases it did require contacting the ISP to get them to recalculate the reputation, in other cases the ISPs changed their filtering without any contact needed.