Is their increased vigilance pissing you off? If so, your anger is misplaced. They are reacting quite sensibly to market conditions apparently imposed by Spamhaus. Ken Magill

While I agree with Ken that the ESPs are reacting to market conditions. Where we disagree is the idea that these conditions are imposed by Spamhaus. I don’t think all the uptick in ESP enforcement and compliance activity is the result of Spamhaus’ actions. I believe that many of the mass market ISPs are changing how they detect unwanted mail, and are fine tuning filters to reduce the amount of unwanted mail that shows up in the inbox.

One of the big changes is better tools for handling huge data sets. Bigger ISPs handle billions of messages a week. Even just collecting and storing the mail is a giant task. Storing it in a useable form was almost out of the question. But over the last few years there have been significant improvements in the speed and affordability of hardware to handle very, very large datasets. Likewise, there have been algorithm and software improvements in mining that data for useful correlations.

In practical terms, ISPs and filtering companies like Spamhaus don’t have to focus on complaints or trap hits or “simple” measurements. They can draw complex correlations and look at mail in a way that was simply impossible 2 or 3 years ago. This means they can better identify senders who had previously been able to slide in under the filters.

Spamhaus rolled out tools to monitor their spam feeds in a different way and have been listing a lot more “legitimate” senders because of it. ISPs are rolling out tools to better filter “greymail” and keep users inboxes full of mail that the users actually want.

One of the trends I’m noticing is that direct marketers are getting more aggressive. Whether it’s a response to the years of recession or a response to the slowly warming economy, I can’t tell. But there are a lot of direct marketers who are no longer afraid to break the law. For instance, my cell phone is getting multiple telemarketing calls a week, despite being a cell and despite being on the do not call list. My inbox is full of unsolicited email carefully engineered to get past standard filters, much of which violates CAN SPAM. I’m even getting the occasional unsolicited fax.

The increase in listings by Spamhaus are one example of the filtering screws being tightened. But it’s not just Spamhaus that’s driving this; ISPs and filtering companies are also filtering more aggressively. I’m seeing a lot more emphasis being placed on content and a good IP reputation is no longer a ticket to the inbox. Content must be clean and recipients have to want mail for it to get into the inbox.

A bunch of them are still active, but there’s a good dozen or so that haven’t been updated in months. I realize I’m getting most of my current news from Twitter (or, Facebook) not from my actual RSS feeds.

So what email / marketing / delivery / internet security related blogs are people reading these days? What should I add to my list to keep up to date on the pulse of the email industry?

EDIT: apparently the Akismet filter I use went berserk with the multiple links in comments. I think I’ve pulled everything they caught incorrectly. If you tried to post and it’s not showing, drop me an email at the obvious place.

Apparently, some time back in 2002 someone opted in an address that didn’t belong to them to a marketing database. It may have been a hard to read scribble that was misread when the data was scanned (or typed) into the database. It could be that the person didn’t actually know their email address. There are a lot of ways spamtraps can end up on lists that don’t involve malice on the part of the sender.

But I can’t help thinking that mailing an address for 10 years, where the person has never ever responded might be a sign that the address isn’t valid. Or that the recipient might not want what you’re selling or, is not actually a potential customer.

I wrote a few weeks back about the difference between delivery and marketing. That has sparked conversations, including one where I discovered there are a lot of marketers out there that loathe and despise delivery people. But it’s delivery people who understand that not every email address is a potential purchaser. Our job is to make sure that mail to non-existent “customers” doesn’t stop mail from actually getting to actual potential customers.

Email doesn’t have an equivalent of “occupant” or “resident.” Email marketers need to pay attention to their data quality and hygiene. In the snail mail world, that isn’t true. My parents still get marketing mail addressed to me, and I’ve not lived in that house for 20+ years. Sure, it’s possible an 18 year old interested in virginia slims might move into that house at some point, and maybe that 20 years of marketing will pay off. It only costs a few cents to keep that address on their list and the potential return is there.

In email, though, sending mail to addresses that don’t have a real recipient there has the potential to hurt delivery to all other recipients on your list. Is one or two bad addresses going to be the difference between blocked and inbox? No, but the more abandoned addresses and non-existent recipients on a list there are on a list, the more likely filters will decide the mail isn’t really important or wanted.

The cost of keeping that address, one that will never, ever convert on a list may mean losing access to the inbox of actual, real, converting customers.

DMARC is working on a standard to allow senders to publish sending policies and receivers to act on those policies. Currently, senders who want receivers to not deliver unauthenticated email have to negotiate private agreements with the ISPs to make that happen. This is a way to expand the existing programs. Without a published standard, the overhead in managing individual agreements would quickly become prohibitive.

It is an anti-phishing technique built on top of current authentication processes. This is the “next step” in the process and one that most people involved in the authentication process were anticipating and planning for. I’m glad to see so many big players participating.

Yes, there are APIs that can be queried at some of the larger ISPs to identify if an account name is taken, but this doesn’t mean that there is an associated email address. Yes, senders can do a real time SMTP transaction, but ISPs are quick to block SMTP transactions that quit before DATA.

I decided to check out one service to see how accurate it was. I’m somewhat lucky in that I created a username at Yahoo Groups over a dozen years ago but never activated the associated email address. This means that the account is shown as taken and no one else can register that address at Yahoo. But the address doesn’t accept any mail.

The address verification for Yahoo addresses

There is a service that offers real time verification and allows potential customers to check an address on their website. I plugged my Yahoo address into their text box. They verified it as active and connected to all networks. Just to make sure I checked my existing Yahoo address as well, and that shows the same: connected to active online networks.

I next sent an email to both Yahoo accounts. Yahoo accepted mail to my working account but bounced mail to the Yahoo Groups only account.

Final-Recipient: rfc822; biskybabe@yahoo.com
Original-Recipient: rfc822;biskybabe@yahoo.com
Action: failed
Status: 5.0.0
Remote-MTA: dns; mta5.am0.yahoodns.net
Diagnostic-Code: smtp; 554 delivery error: dd This user doesn't
   have a yahoo.com account (biskybabe@yahoo.com) [-5] -
   mta1289.mail.ac4.yahoo.com

This tells me that for Yahoo addresses, Briteverify is using some sort of API call to identify whether or not an account name is taken. But just because an account name is taken doesn’t specifically mean that an account is a valid email address. It’s probably better than no verification, but usage of all real time verification isn’t going to help in all cases.

What about email accounts that don’t provide an API or a way to check the validity of an account? In that case it appears that they are using an aborted SMTP transaction. we tested

Jan 24 15:20:00 misc postfix/smtpd[28917]: connect from
   smtpout9.briteverify.com[107.20.232.98]
Jan 24 15:20:01 misc postfix/smtpd[28917]: NOQUEUE: reject:
   RCPT from smtpout9.briteverify.com[107.20.232.98]: 550 5.1.1
   <mu/er9w9kmbyg+s5uehqdxqe@blighty.com>: Recipient
   address rejected: User unknown in virtual alias table;
   from=<admin@origindata.com>
   to=<mu/er9w9kmbyg+s5uehqdxqe@blighty.com>
   proto=SMTP helo=<emailver.briteleads.com>
Jan 24 15:20:01 misc postfix/smtpd[28917]: lost connection after
   RCPT from smtpout9.briteverify.com[107.20.232.98]
Jan 24 15:20:01 misc postfix/smtpd[28917]: disconnect from
   smtpout9.briteverify.com[107.20.232.98]
Jan 24 15:20:01 misc postfix/smtpd[28915]: connect from
   smtpout7.briteverify.com[184.73.155.120]
Jan 24 15:20:01 misc postfix/smtpd[28915]: NOQUEUE: reject:
   RCPT from smtpout7.briteverify.com[184.73.155.120]: 550 5.1.1
   <aardvark@blighty.com>: Recipient address rejected: User
   unknown in virtual alias table; from=<admin@origindata.com>
   to=<aardvark@blighty.com> proto=SMTP
   helo=<emailver.briteleads.com>
Jan 24 15:20:01 misc postfix/smtpd[28915]: lost connection after
   RCPT from smtpout7.briteverify.com[184.73.155.120]
Jan 24 15:20:01 misc postfix/smtpd[28915]: disconnect from
   smtpout7.briteverify.com[184.73.155.120]

The verification service did correctly identify both addresses as invalid. However, this is exactly the kind of SMTP behaviour that is blocked by many places.

Real time address verification for 100% of addresses is incredibly difficult. As I demonstrated above, their use of testing APIs makes the assumption that everyone with a login at Yahoo (or google or other places) has an email address, but this isn’t necessarily true.

There are other assumptions that realtime address verification makes.

1. No one ever typos the left hand side of their email address into an address of another user at the site. This isn’t true, for instance, I entered a common typo of my email address into the form and the service verified it as accurate. It probably is a valid, deliverable account but that doesn’t mean that it’s a good address.
2. Spamtraps are always undeliverable addresses. This is not true and the above form did verify a spamtrap address that a friendly blocklist admin checked for me.
3. No one typos the right hand side of an address to a valid domain. This is not true. For instance, I know a number of spamtrap domains used by Trend Micro. The form validates addresses there and tells me I’m good to send.

I’m not trying to knock the real time address verification services, I think what they’re attempting to do is good. I think the glossy marketing, though, will lead senders into a false sense of security. Just because a 3rd party service tells you an address is deliverable, doesn’t mean that the address is deliverable or that the address is safe to mail.

I do think potential verification customers deserve to understand how the services work so that they can make good decisions about purchasing those services.

At the same time, I was internet friends with an employee of JPL. I’d met him, like I met many of my online acquaintances, through a pet related mailing list.

One Wednesday, The Onion published an article Mir Scientists Study Effects of Weightlessness on Mortal Terror. As this was the time when the Internet consisted of people banging rocks together, there was not an online link to Onion articles. But I was sure my friend at JPL, and all his friends, would appreciate the joke. That night I stayed late at the lab and typed the article into an email (with full credit to the Onion) and mailed it off to him.

As expected, the article garnered quite a few chuckles and was passed around to various folks inside JPL. What wasn’t expected was another friend, from totally different circles, sending me a copy of that same article 3 days later. Yes, in 1997 it took three days for information to be shared full circle on the Internet.

Information sharing is a whole lot quicker now, with things coming full circle in mere seconds. But that doesn’t make the information any more reliable and true. Take a recent article in ZDNet Research: Spammers actively harvesting emails from Twitter in real-time.

ZDNet links to a study published by Websense, claiming that email addresses on Twitter were available for harvesting.

That’s all well and good, but all ZDNet and Websense are saying is that email addresses are available for harvesting. I’ve not seen any evidence, yet, that spammers are harvesting and sending to them. This doesn’t, of course, mean they’re not, but it would be nice to see the spam email received at an address only shared on twitter.

Well, I have unique addresses and an un-spamfiltered domain. I went ahead and seeded a tagged address onto twitter. We’ll see if it gets harvested and spammers start sending to it. I’ll be sure to keep you updated.

The general response from the marketing side of things appeared to be that ISPs need to stop actually filtering marketing email. That would resolve the problems from the marketers perspective. I don’t necessarily think that will help. I believe if marketers had unfettered access to the inbox, most inboxes would be totally un-useable.

My thinking triggered other folks to consider delivery and marketing and what drives both. George Bilbrey, from Return Path, posted an article in Mediapost looking at why good delivery is an important part of a good marketing strategy.

George points out many marketers really do act as if delivery is separate and detrimental to good marketing.

I hear this with my clients and I hear this on discussion lists.  They think that the practices that drive high inbox placement rates are antithetical to return on their email marketing investment.

Exactly. I hear a lot of contempt for delivery consultants and good delivery practices from a lot of marketers. They claim our methods and our recommendations come from not understanding marketing. They flat out tell me that “we’re” manufacturing delivery problems by pointing out mail that users don’t want has poor delivery.

There are thousands of companies that have never heard of Return Path, or Word to the Wise, who don’t understand why their perfectly crafted marketing isn’t getting to the inbox. It’s because they don’t understand email and delivery. They want to do what works elsewhere, and those models don’t always map onto email.

And that’s why companies like Word to the Wise and Return Path exist.

I had some people ask me about the bills today and have been looking for explanations of the issues and why these laws are so problematic.

• Reddit has a good explanation for the non technical folks.
• Wikipedia has a number of good pages including: SOPA Initative, SOPA
• Infojustice has a copy of the letter sent to Congress by Vint Cerf and other DNS experts
• MAAWG published a technical analysis of the problems for ISPs
• Securityskeptic has a list of statements about SOPA

Over the years I’ve seen “the Internet” get upset about a lot of things. The idea of an Internet blackout has been tried again and again. This is one of the few efforts that has gotten major sites on board and may have an impact.

Is there anyway to keep information safe if it’s accessible from the internet? Some of my uber-security conscious friends would say no. I am beginning to believe them.