Well, that got someone’s attention. The person managing their twitter account tweeted at me with an email address and a suggestion to send him my address so he could take care of it. I sent the mail as asked and even got a reply.

Unfortunately, the reply was “I clicked the unsubscribe link at the bottom of the message for you.”

I dunno, maybe his mouse is a magic mouse and, somehow, the click from that magic mouse will be more effective than a click from my not-magic mouse. I’m not holding out much hope, though. I have no doubt that my sales address will keep getting invited to raves in London long after I retire.

I disagree. Appending, and other non-permission based sending cause a lot of costs to trickle down on the ESP. Many of the large ESPs have teams of 8 or 10 people working to manage delivery, deal with blocks and keep the mail flowing. In fact, I once had a client say “We want to be as clean as ExactTarget” only to choke when I told them how many people are on the compliance and delivery team at ET.

That’s not even looking at the cost of a SBL listing. One company estimated the cost of a slightly less than 24 hour block at over $1,000,000 in lost opportunity costs and in actual staff costs to deal with the listing. I know of one Fortune 20 company who had to re-engineer their entire customer and prospect databases due to a blocklist. And, yeah, that one was actually due to an append. They did an append and the append not only added a “new” address to a record where the person had previously opted out, but that person worked at a major spam filtering company. They experienced a whole world of very expensive pain.

Many ESPs are actually making a sound business decision by refusing to deal with non-permission mail, whether it be a purchased list or an appended list. The sender does not have permission to send to the addresses. That causes all sorts of delivery problems, which costs the ESPs lots of money and staff time to deal with. Most marketers won’t actually pay for the resources they use when appending or buying lists. Then they blame the ESP when their mail ends up in the bulk folder or is blocked outright.

I don’t think many marketers fully integrate the cost of dealing with a poor list into their decisions. My tweet from earlier today “If you have to “ignore all the costs associated with complaints” to find a positive ROI on opt-out mail, is there really a positive ROI? is a paraphrase of one of the things I heard.

ESPs can’t avoid those costs, they’re stuck with them. Lowering those costs by requiring senders to only send to recipients who have given permission is a smart business decision. Marketers don’t pay those costs, but if they even acknowledged them I suspect that there would be a whole lot less sloppy email marketing.

This first post is about IP address reputation with some background on why IPs are so important and why ISPs focus so heavily on the sending IP.

Why IP addresses?

ISPs built reputation around IP addresses because it was one bit of data that malicious senders / spammers couldn’t forge. The connecting IP is a fundamental part of the network transaction and if you forge an IP then SMTP can’t work. Because that was the reliable data they had to work with, that’s what they used. Even now, when there are other kinds of data, the IP address is still the first thing the receiving MTA sees.

What is IP reputation?

IP reputation can best be summed up as “past performance is an indicator of future results.” In other words if recipients responded well to mail from an IP address in the past, then they’re likely to respond well to new mail from that IP address.

How is IP reputation measured?

While each spam filtering company and ISP have their own ways of calculating the reputation of an IP address, there are some similarities in what they measure.

• How many non-existent email addresses is this IP attempting to deliver to?
• How many abandoned email addresses is this IP attempting to deliver to?
• How many “known bad” email addresses (spamtraps) is this IP attempting to deliver to?
• How many recipients complain about receiving this mail?
• How many recipients complain about not receiving this mail?
• How respectful of my resources is this IP?
• Does this IP keep connections open for long periods of time?
• Does this IP retry deliveries too aggressively?
• Does this IP stop mailing addresses after receiving a “user unknown” message?
• Is this IP address configured as if the associated machine was infected by a virus?
• Is this IP address listed on blocklists we use?

That is by no means an exhaustive list of what ISPs measure. If they can measure it they’ve tried. If the measurement helps them separate spam mail from not-spam mail then they’re using it.

How fast does IP reputation change?

IP reputation is often measured over multiple time periods. ISPs can look at a 1 day, 7 day, 30 day and 90 day reputation. A good analogy is stock prices. Prices can be very volatile in the short term, but more consistent over the long term. A single bad day, where one or more reputation measurements go bad, may affect delivery that day or the next day but won’t damage an overall good reputation. Likewise, a few days of improved mail may not be sufficient to counter months of poor reputation.

How is IP reputation used?

Mail from IPs with a high reputation is accepted faster and at a higher rate than mail from IPs with a lower or unknown reputation.  IP reputation can also influence whether mail is delivered to the inbox or the bulk folder.

Key IP Reputation takeaways

• IP reputation is about how recipients react to mail from that IP. Happy, content recipients turn into good delivery.
• Brief changes (for good or bad) don’t necessarily ruin delivery over the long term.
• Steady improvements will result in improved reputation.
• It may takes as much time to change a reputation in one direction or another as it took to establish the reputation in the first place.

Next we’ll look at content reputation, how it’s measured and used.

EDIT: A version of this information is available at the Word to the Wise wiki

EDIT: This post was also shared at CircleID

Experian CheetahMail believes that opt-out email appending is no longer an acceptable practice, and that marketers should no longer use of this practice to acquire customer email addresses. EmailResponsibly

In my experience, appending causes major delivery problems. Of course, every time the issue comes up some marketer tells us who think it’s a bad idea that they successfully used appending and it worked and all the delivery problems are a figment.

Maybe the supporters will believe Ben and Experian / CheetahMail that appending is not a good thing to do. After all, Ben was a large proponent of the practice many years ago and Experian still sells appending services in some countries.

Sending mail without permission, which is what appending usually is, will cause delivery problems. Stick to real permission, not vague promises.

Not good. Just the sort of urgent, high-risk issue that ISP abuse desks really want to hear about. I sent email about it to the ISPs involved, including a copy of the original email. One of them went to iWeb, a big (tens of thousands of servers) hosting company.

This was the response:

<abuse@noc.privatedns.com>: host mott.privatedns.com[174.142.252.34] said: 554 rejected due to spam content (in reply to end of DATA command)

That’s iWeb’s main abuse address for their address space, as registered with ARIN. They even have a comment in their network registration that says “Please use abuse@noc.privatedns.com for abuse issues”.

For email related abuse (spam, malware email, botnets, phishing, viruses, …) almost all valid, actionable abuse reports will include a copy of the email involved. And that’s exactly the sort of content that content-based spam filters do their best to block. That means that putting content-based spam filters on your abuse or security role addresses will prevent you seeing most reports about abusive traffic coming from your network.

There are some companies that have an intentional policy of rejecting most spam reports sent to them so that their abuse metrics look better, and they don’t have to pay for abuse desk staff to handle the high volumes of abuse reports their customers provoke. “Mistakenly” putting spam filters on their abuse alias is one way of doing that – others include using non-standard abuse aliases, demanding reports come in only via web forms, requiring abuse reports be sent in non-human-writable formats while discarding all others, and many more. If you don’t want to behave responsibly it’s easy enough to dodge those reports.

Legitimate companies really want to know about abusive traffic sooner rather than later, so they can shut it down and mitigate the damage as quickly as possible. Email systems are complex, though, and it’s quite easy for an upgrade to spam filtering at a companies main mailserver to mistakenly by applied to abuse@ and security@ aliases – especially when spam filtering or email services are outsourced. And if you’re a company that uses dozens of domains it’s easy to lose track of where mail to abuse@ some of those domains ends up.

If you’re responsible for email, abuse or security at your organization it’s worth occasionally checking that your role accounts actually work. Find yourself a fairly obvious bit of spam, then forward it to your abuse@ role address (with a sentence or two telling your abuse desk that you’re just testing, and can they reply to your mail so you know they received it).

Real spam sent directly to abuse@ role addresses can be a severe problem, but content-based filtering is not the way to deal with it. One approach that we suggest to our Abacus users is to prioritize reports that mention a URL or an IP address on your network, so that legitimate, actionable reports will “bubble up” above any spam.

Bounce handling should be designed to appropriately manage your subscriber base. Delivery problems are the consequence if you don’t do that. They shouldn’t be the reason you bounce handle, though.

Context matters.

My experience tells me that senders that think about the impact of their sends can do things that “break the rules” while still being respectful of their subscribers and still see good delivery.

All of these things are great. I love hearing from readers, either in comments or in email. We have a valid (unfiltered) contact address here on the blog. My email address(es) aren’t difficult to find. I want to talk to people.

Sometimes some of the people who contact me do actually send spam. It’s bulk, it’s impersonal, it’s not about me or my perspective it’s about them trying to sell something (themselves, their newest product, their company) to anyone who is buying.

If it’s clear it’s a one off I’ll generally just move the mail out of my inbox and forget about it. Sometimes, though, there are hints that this is more than just a one time mail. The email will have an unsubscribe link, or it’s the third or fourth time I’ve gotten mail from that sender or it will be from a PR company. I deal with them in different ways. Sometimes I’ll offer a different email address that I route better, or I’ll just filter the mail based on some unique bit of the header.

The ones that really get me, though, are when the senders argue with me that I should feel special to get their bulk mail. “It was individually sent to you!” “I sent it because you’re such a great resource and wanted to say thank you!” But it was bulk mail, mail dozens of other people got (hint: the email / delivery industry is very small. we talk to each other all the time, if you send mail to more than one of us, we’re going to talk about it).

I have no problem with you inviting me to your event. Or telling me about the latest or greatest thing you wrote. I don’t even mind the occasional one-off bulk mail. But if you are sending mail to a specific person, put in the 20 seconds to personalize it and make it feel like it’s special for me.

A few moments to think and personalize before you send that email will make your recipient much more open to your pitch. This is as applicable to one off mail as it is to bulk.

My bayesian filter gets trained mostly by me hitting “this is spam” when spam makes it to my inbox. If I’m expecting an email “immediately” – something like a mailing list COI confirmation or email as part of buying something online – I’ll check my spam filter and move the mail to my inbox in the rare case it ended up there. Other than that I let it and spamassassin chug along with no tweaking.

I’m starting a data analysis project, based on my own inboxes, and as part of that I’m using some tools to look for false positives in my junk folders, and manually fixing anything that’s misclassified. I’ve been doing this for a couple of hours now, and I’ve found some interesting things.

1. Simple content filters work remarkably well out of the box, at least for my mail stream. Spectacularly well. There’s very little in the way of false positives. Very, very little.
2. Of those false positives there’s nothing I’d have been bothered about. It’s generic, unexciting junk mail.
3. Most of the systemic false positives seem to be correlated with the senders doing something bad. Heathrow Express, for instance, sent me mail every two weeks or so since I’d signed up. Then for no obvious reason they stopped sending for three months, then started sending again. Every mail they sent after that pause ended up in the junk folder, and I never missed them.
4. I get regular newsletters from ThinkGeek. Every one of those goes to the inbox. I occasionally get mails from them about my account (“you’ve got 420 geek points left”) that are kinda transactional, but not something I expect to see – and they all end up in the junk folder. Several other senders do the same thing, and get the same result.
5. Several companies have used tagged addresses to send me newsletters for a while, and also used them to send unsolicited facebook invites (to the tagged address, from facebook servers). The regular newsletters all go to the inbox, while the facebook invites all go to the junk folder. “Legitimate” facebook mail, meanwhile, keeps going to the inbox.
6. Apple send me a lot of newsletters – I’m a Mac and iPhone developer, I get their consumer newsletters, transactional stuff from our local store – lots and lots of newsletters. They all made it to the inbox except for one. The one that ended up in the junk folder was a one-off about recycling, and it wasn’t up to their usual design standards – it had ugly big green “call to action” headlines in it, very different to their usual clean design.
7. Just one sender hit the junk folder every time. The distinctive thing about their messages (apart from them not being something I missed) was that the plain text part of them was dreadful, just a bad lynx dump of the html section. Even a “No plain text for you! Go to this link!” would have been better.

I was surprised at how effective this simple content-based filtering setup had worked with little tuning other than hitting the this-is-spam button – both in it’s accuracy at removing spam while keeping a very low false positive rate, but also how well the false positives matched my judgement of “Meh. This mail isn’t interesting.”.

We spend a lot of time talking about the things you should do to make the mail relevant to the recipient – compelling content, consistent, predictable delivery schedules, clear consistent branding, use of a single consistent mail stream to communicate with a recipient rather than several different streams. Until I went through this exercise it wasn’t clear to me how much of an effect those things also have on fairly simple recipient-trained filters.

The article looks at opens, clicks and click through rates over 7 quarters (Q1 – Q4 2010; Q1 – Q3 2011) covering 330 million emails. I strongly suggest anyone interested go read the whole article.

The short version, though, is that the opt-in lists had more opens and more clicks than the opt-out lists. In some quarters it was double the number of opens and clicks.

This data is a strong indication that opt-in lists perform much better than even the best opt-out lists.

One of the best things about the document is the discussion of how spammers attempt to hide their identity. All too often I’ve been called in by ESPs to help them identify how a spammer got on their network and where their process failed. As filtering gets better at blocking spam, spammers are spending more and more time trying to steal good reputations to get their unwanted mail through.

Providers who follow these rules may still find themselves with spammers as customers, but the spammers will have to work harder to get on clean networks.