Not good. Just the sort of urgent, high-risk issue that ISP abuse desks really want to hear about. I sent email about it to the ISPs involved, including a copy of the original email. One of them went to iWeb, a big (tens of thousands of servers) hosting company.

This was the response:

<abuse@noc.privatedns.com>: host mott.privatedns.com[174.142.252.34] said: 554 rejected due to spam content (in reply to end of DATA command)

That’s iWeb’s main abuse address for their address space, as registered with ARIN. They even have a comment in their network registration that says “Please use abuse@noc.privatedns.com for abuse issues”.

For email related abuse (spam, malware email, botnets, phishing, viruses, …) almost all valid, actionable abuse reports will include a copy of the email involved. And that’s exactly the sort of content that content-based spam filters do their best to block. That means that putting content-based spam filters on your abuse or security role addresses will prevent you seeing most reports about abusive traffic coming from your network.

There are some companies that have an intentional policy of rejecting most spam reports sent to them so that their abuse metrics look better, and they don’t have to pay for abuse desk staff to handle the high volumes of abuse reports their customers provoke. “Mistakenly” putting spam filters on their abuse alias is one way of doing that – others include using non-standard abuse aliases, demanding reports come in only via web forms, requiring abuse reports be sent in non-human-writable formats while discarding all others, and many more. If you don’t want to behave responsibly it’s easy enough to dodge those reports.

Legitimate companies really want to know about abusive traffic sooner rather than later, so they can shut it down and mitigate the damage as quickly as possible. Email systems are complex, though, and it’s quite easy for an upgrade to spam filtering at a companies main mailserver to mistakenly by applied to abuse@ and security@ aliases – especially when spam filtering or email services are outsourced. And if you’re a company that uses dozens of domains it’s easy to lose track of where mail to abuse@ some of those domains ends up.

If you’re responsible for email, abuse or security at your organization it’s worth occasionally checking that your role accounts actually work. Find yourself a fairly obvious bit of spam, then forward it to your abuse@ role address (with a sentence or two telling your abuse desk that you’re just testing, and can they reply to your mail so you know they received it).

Real spam sent directly to abuse@ role addresses can be a severe problem, but content-based filtering is not the way to deal with it. One approach that we suggest to our Abacus users is to prioritize reports that mention a URL or an IP address on your network, so that legitimate, actionable reports will “bubble up” above any spam.

All of these announcements cause much consternation in the email marketing industry. Just today there was a long discussion on the Only Influencers list about the new Hotmail filtering. There was even some discussion about why the ISPs were doing this.

I think it’s pretty simple why they’re creating new tools: users are asking for them. The core of these new filters is ISPs reacting to consumer demand. They wouldn’t put the energy into development if their users didn’t want it. And many users do and will use priority inbox or the new Hotmail filtering.

Some people are concerned that marketing email will be less effective if mail is not in the inbox.

From a marketer’s standpoint, it is a challenge because we want to get our customers’ attention, and it is harder to do that if these messages are going into a separate box. Laura Santos

I think, though, a lot of email marketing (and direct marketing in general) relies on the consumer being lazy. That’s why negative options work so well. It’s not that the user is actually making a choice, it’s that they’re not making a choice, so the marketer chooses for them. In many cases the marketer controls the channel more than the target does.

This isn’t the case in email. Marketers don’t control the channel, the ISPs and end users do. This requires a shift in thinking in order to effectively use the channel.

I’m someone who filters all my newsletters to a non-standard inbox. I’ve done it for years now. It improves my workflow and actually means when I open that box I’m more receptive to the advertising. Folks sending me newsletters don’t get to interrupt me, they get my attention when I’m ready to give it to them.

I can see how this shift – from interruption based marketing to non-interruption based marketing can be difficult for marketers. It’s a new paradigm, one that is much more challenging than getting a lazy consumer to purchase.

Adapt or Die.

For the major ISPs, the people who plan, approve, design and monitor the filters usually want to maximize customer happiness. They want to deliver as much real mail as possible while blocking as much bad mail. Blocking real mail and letting through bad mail both result in unhappy customers and increase the ISP’s costs, either through customer churn or through support calls. And this is a process, filters are not static. ISPs roll out new filters all the time, sometimes they are an improvement and sometimes they’re not. When they’re not, they’re pulled out of production. This works both for positive filters like Return Path and negative filters like blocklists.

Then there is mail filtering that doesn’t have to do with spam. Business filters, for instance, often block non-business mail. Permission of the recipient often isn’t even a factor. Companies don’t often go out of their way to block personal mail, but if personal mail gets blocked (say the vacation plane ticket or the amazon receipt) they don’t often unblock it. But when you think about why a business provides email, it makes perfect sense. The business provides email to further its own business goals. Some personal usage is usually OK, but if someone notices and blocks personal email then it’s unlikely the business will unblock it, even if the employee opted in.

In the case of email filters, the free market does work. Different ISPs filter mail differently. Some people love Gmail’s filters. Other people think Hotmail has the best filtering. There are different standards for filtering, and that makes email stronger and more robust. Consumers have choices in their mail provider and spamfiltering.

There have been two symptoms I’ve been hearing about. One is an increase in bulk folder delivery for mail that previously was reliably hitting the inbox. The other is a bit more interesting. I’ve heard of 3 different mailers, with good reputations and very clean lists, that are seeing 4xx delays on some of their mail. The only consistency I, and my colleagues at some ESPs, have identified is that the mail is “bursty.”

The senders affected by this do send out mail daily, but the daily mail is primarily order confirmations or receipts or other transactional mails. They send bi-weekly newsletters, though, exploding their volume from a few tens of thousands up to hundreds of thousands. This seems to trigger Gmail to defer mail. It does get delivered eventually. It’s frustrating to try and deal with because neither side is really doing anything wrong, but good senders are seeing delivery delays.

For the bulk foldering, Bronto has a good blog post talking about the changes and offering some solid suggestions for how to deal with them. I’m also hearing from some folks who are reliable that Gmail may be rolling back some of the bulk foldering changes based on feedback from their users.

So if you’re seeing changes at Gmail, it’s not just you.

The judge referenced a couple specific deficiencies of Holomaxx’s claims in his dismissal.

Holomaxx alleges no facts in support of its conclusory claim that Yahoo!’s filtering program is faulty, nor does it identify an objective industry standard that Yahoo! fails to meet. While it suggests that Yahoo! is “using cheap and ineffective technologies to avoid the expense of appropriately tracking and eliminating only spam email,” it offers no factual support for these allegations. (Compl. ¶ 40). Nor does Holomaxx cite any legal authority for its claim that Yahoo! has a duty to discuss in detail the particular reasons for blocking Holomaxx’s communications or to provide a remedy for such blocking.

Holomaxx did their best to identify an objective standard, they really did.

9. The objective industry standard has been set forth by the Messaging Anti-Abuse Working Group (“MAAWG”), a group consisting of the largest internet service providers on the planet. YAHOO is a member of the MAAWG and has approved of the MAAWG’s industry standard for abuse desk responses but has failed to follow the MAAWG evidencing, among other things, a lack of good faith.

10. The MAAWG sets forth an objective industry standard protocol by which its members should respond to blocked senders, such as HOLOMAXX, requesting unblocking. The MAAWG protocol states that “the simplest option would be to grant unblocking when it is requested. This alleviates the necessity of judgment calls, arguments with senders or time-consuming research. The overwhelming majority of the vote was for this approach because if the sender had not fixed the issue that caused the block it would be reinstated quickly, so damage would be theoretically minimal.” In this case, YAHOO has breached this provision of the MAAWG in that, with regard to HOLOMAXX and YAHOO’s blocking of HOLOMAXX’s emails, YAHOO refuses to  unblock HOLOMAXX’s emails.

Their filing doesn’t actually reference the document in question, so I spent some time this morning looking for which document this was. It didn’t sound like anything familiar. I eventually identified the 2007 Abuse Desk Common Practices document published by the Collaboration Committee. In the 2nd paragraph it was obvious why Holomaxx didn’t include the document as an exhibit.

The intent of this document is to present options for abuse desk administrators for addressing common problems faced by abuse desks. This is not intended to represent an absolute set of best practices. It is our belief that what is “best” is frequently determined by the particular circumstances of the network/mailboxes served by the abuse desk.

So the “objective industry standard” that Holomaxx cites explicitly states this is a document that solely discusses options and how some abuse desks currently (in 2006 and 2007) handle inquiries from senders. In most cases, the abuse desk and the postmaster desk are separate, something I’m comfortable asserting about Yahoo’s setup.

In dismissing the wiretapping claim the judge cites specific problems with Holomaxx’s complaint.

Holomaxx alleges that Yahoo! “intentionally intercepted electronic communications sent by Holomaxx, in violation of 18 U.S.C. § 2510, et seq.,” (Compl. ¶ 49), and “intentionally used and disclosed the contents of such electronic communications sent by Holomaxx, while knowing 2or having reason to know that the information was obtained through the interception of those communications in violation of 18 U.S.C. § 2511.” (Id. at ¶ 50). It claims that Yahoo!intercepted e-mails to evaluate the “spam like characteristics” of the e-mails (Id. at ¶¶ 59-60). However, while the Court must “take all factual allegations in the complaint as true,” (Ashcroft v. Iqbal, 129 S.Ct.1937, 1949 (2009)), it is not required to accept a “legal conclusion couched as a factual allegation.” Bell Atl. Corp. v. Twombly, 550 U.S. 544, 555 (2007). Here, Holomaxx’s allegations are both conclusory and devoid of factual support. Holomaxx does not explain how Yahoo! “intercepted” its communications, or how Yahoo! “used and disclosed” the contents of those communications. Additional factual detail is necessary in order to permit the Court to make a meaningful assessment of the plausibility of the allegations.

Holomaxx tried to address this deficiency.

44. Upon Information and belief, YAHOO intercepts and scans the contents of HOLOMAXX’s emails in the following manner: YAHOO uses Bayesian filtering techniques which scans the content of HOLOMAXX’s email message body and headers for words and phrases associated with alleged spam messages. YAHOO then provides the incoming and intercepted HOLOMAXX email with a score before the HOLOMAXX email reaches the YAHOO receiving server. The receiving server, armed with the score procured from the illegal interception, then decides whether or not to accept the HOLOMAXX email or reject it as spam.

45. Upon information and belief, in conjunction with the Bayesian technique, YAHOO also intercepts HOLOMAXX emails before they reach YAHOO’s servers and uses a distributed catalogue of signatures to detect spam. These signatures are generated as a result of YAHOO user submissions in the past. The HOLOMAXX emails are intercepted and the contents scanned and compared to the catalogue of past signatures. This is known as a collaborative filtration system.

46. YAHOO did not have authorization from HOLOMAXX to access the contents of HOLOMAXX’s emails in the manner alleged herein, in transit between leaving HOLOMAXX’s server and before reaching [sic] MICROSOFT’s servers.

Anyone who knows anything about email can see the flaw here. Filters don’t run on the wire. They have to go through a server somewhere, and the only way Yahoo can apply filters is to have actually received the email. Anywhere Yahoo is going to be able to filter the mail is, objectively, a server Yahoo owns or is allowed to access.  I also don’t think these three paragraphs actually explain how the mail was intercepted. It makes a lot of assertions and throws around buzzwords, but doesn’t explain how this alleged interception happened.

Holomaxx also started claiming that Yahoo sends email advertising to its users and that Y! is blocking Holomaxx to drive Holomaxx out of business and make advertisers pay higher advertising rates to Yahoo. Their proof? Screenshots of a Yahoo inbox showing banner ads on the right hand side of the screen. I think it would really help Holomaxx’s case if they could manage to not fundamentally mis-represent simple things like this. They’re ads on a webpage, they’re not sent by email.

All in all, I’m not seeing much improvement in the amended complaint. I mean, Holomaxx tried to address the deficiencies, but they really can’t support their claims. Yahoo isn’t violating the wiretapping act, they’re not violating industry standards and Holomaxx is clearly grasping at straws to try and save their company. e360 tried the same thing against Comcast and ended up bankrupting themselves. Will the same thing happen to Holomaxx?

Anyone sending a lot of email has complained about spam filters and false positives at some point. But most people haven’t run a mailbox with no spam filters in front of it in recent years, so don’t have much of a feel for what an unfiltered mailbox looks like, how important filters are and how difficult their job is.

I run no transaction level filters in front of my mailbox, just content filters that route mail to one of several inboxes or a junk folder, so if I want to I can look at what unfiltered email looks like. I took data from all mail that was sent to me yesterday, and put it in a format that really shows the problem filters face and especially the difficulty of spotting which mail in the junk folder is a false positive.

An inbox with no filters looks like this.

Running a spam filter against it, simply categorizing each email as spam (pink) or not-spam (green) looks like this.

Even with the messages categorized as spam vs not-spam it’s hard to work out which messages are important and which aren’t, let alone where the false positives might be.

If I sort the categories by hand you get this – where you can see that out of 1200 or so mails about three quarters were spam. Of the three false positives two were bulk email that I didn’t care that I didn’t receive and only one was email that I considered important.

Anyhow, Mickey beat me to it and posted much of what I was going to say about Ken Magill’s response to a very small quote from Neil’s guest post on expiring email headers last week.

I, too, was at that meeting, and at many other meetings where marketers and the folks that run the ISP spam filters end up in the same room. I don’t think the marketers always understand what is happening inside the postmaster and filtering desks on a day to day basis at the ISPs. Legitimate marketing? It’s a small fraction of the mail they deal with. Ken claims that marketing pays the salaries of these employees and they’d be out of a job if marketing didn’t exist. Possibly, but only in the context that they are paid to keep their employers servers up and running so that the giant promises made by the marketing team of faster downloads and better online experiences actually happen.

If there wasn’t an internet and there weren’t servers to maintain, they’d have good jobs elsewhere. They’d be building trains or designing buildings or any of the thousands of other jobs that require smart technical people.

Ken has no idea what these folks running the filters and keeping your email alive deal with on a regular basis. They deal with the utter dregs and horrors of society. They are the people dealing with unrelenting spam and virus and phishing attacks bad enough to threaten to take down their networks and the networks of everyone else. They also end up dealing with law enforcement to deal with criminals. Some of what they do is deal with is unspeakable, abuse and mistreatment of children and animals. These are the folks who stand in front of the rest of us, and make the world better for all of us.

They should be thanked for doing their job, not chastised because they’re doing what the people who pay them expect them to be doing.

Yes, recipients want the mail they want. But, y’know, I bet they really don’t want all the bad stuff that the ISPs protect against. Ken took offense at a statement that he really shouldn’t have. ISPs do check their false positive rates on filtering, and those rates are generally less than 1% of all the email that they filter. Marketers should be glad they’re such a small part of the problem. They really don’t want to be a bigger part.

What worked today won’t work tomorrow. Spammers are forever evolving new techniques to get past spam filters. ISPs are forever evolving new techniques to stop them.

One of the current driving forces for spam filter development is focused on the individual recipients. Recipient wants and needs are king in the world of ISP mail filtering. Much of that is driven by the underlying business models of the free ISPs. They are selling eyeballs to their advertisers and that relies on keeping as many eyeballs around for as long as possible.

An early version of the recipient driven filtering was “add to your address book” where individual users could over ride ISP delivery decisions by actively adding a From: address to their address book. The ISPs have been refining this over time. For instance, if you reply to an email in some clients, you are prompted to add that address to your address books. If you take an email out of your bulk folder and move it to your inbox then that address is automatically added to your address book.

But the refinements haven’t stopped there. ISPs are now making smart decisions about what emails a particular recipient will want to receive. This raises a number of challenges to senders. How do you send email to ten thousand or a hundred thousand or a million people and make it relevant to all of them?

Smart senders will take the individual delivery challenge in stride. They will change along with the ISPs, to send mail that their recipients want to receive. Change is inevitable and required.

The last two weeks the culprit has been Yahoo. They seem to be making a lot of changes to their filtering schemes right at the busiest email marketing time of the year. Senders are increasing their volume trying to extract that last little bit of cash out of holiday shoppers, but they’re seeing unpredictable delivery results. What worked to get mail into the inbox a month ago isn’t working, or isn’t working as well, now.

Some of this could be holiday volume related. Many marketers have drastically increased their mail volume over the last few weeks. But I don’t think the whole issue is simply that there is more email marketing flowing into our mailboxes.

As I’ve been talking with folks, I have started to see a pattern and have some ideas of what may be happening. It seems a lot of the issue revolves around bulk foldering. Getting mail accepted by the MXs seems to be no different than it has been. The change seems to be based on the reputation of the URLs and domains in the email.

Have a domain with a poor reputation? Bulk. Have a URL seen in mail people aren’t interested in? Bulk. Have a URL pointing to a website with problematic content? Bulk.

In the past IPs that were whitelisted or had very good reputations could improve delivery of email with neutral or even borderline poor reputations. It seems that is no longer an effect senders can rely on. It may even be that Yahoo, and other ISPs, are going to start splitting IP reputation from content reputation. IP reputation is critical for getting mail in the door, and without a good IP reputation you’ll see slow delivery. But once the mail has been accepted, there’s a whole other level of filtering, most of it on the content and generally unaffected by the IP reputation.

I don’t think the changes are going to go away any time soon. I think they may be refined, but I do think that reputation on email content (particularly domains and URLs and target IP addresses) is going to play a bigger and bigger role in email delivery.

What, specifically, is going to happen at Yahoo? Only they can tell you and I’m not sure I have enough of a feel for the pattern to speculate about the future. I do think that it’s going to take a few weeks for things to settle down and be consistent enough that we can start to poke the black box and map how it works.

There was one comment from “The Other Barry” about complaints that I think bears highlighting.

Silly people.  High complaints means filters need to be more aggressive.   ”Not Spam” reports means the filters need to be less aggressive.  Low complaints means the filters are accurate.

The Other Barry is someone who has real world experience managing filtering for a large ISP.

In other comments, Steve White is excited to see ISP filtering come under judicial scrutiny. I’m not sure why, there is plenty of case law around filters already. There’s even US law stating ISPs can filter. But, hey, I’m sure some lawsuit from a company no one has ever heard of before will be sufficient to turn over 10 years of precedence.