Skip to content


State of the Industry

Over the last few weeks I’ve had a series of posts on the blog from various authors who are active in the email space.

I posted A very young industry commenting on the lack of experience among email marketers. I think that some of the conflict between ISPs and ESPs and receivers and marketers can be traced back to this lack of longevity and experience. Often there is only a single delivery expert at a company. These people often have delivery responsibilities dropped on them without any real training or warning. They have to rely on outside resources to figure out how to do their job and often that means leaning on ISPs for training.

JD Falk described how many at ISPs feel about this in his post With great wisdom…

we’re also tired with teaching the same thing to people with the same title, and feeling like the message never gets through. Part of what we’re saying is “It’s your industry, you’ve learned this stuff, now you teach ‘em.”

His comments are similar to comments I’ve heard from many people behind the spamfilters at ISPs and spam filtering companies. Some ESPs go through delivery folks almost yearly. Those new delivery folks then reach out to the ISPs and ask the same question their predecessor asked a year ago which, in turn, was the same question that their predecessor asked the year before that. Really, the ISPs don’t like repeating themselves like this. I keep telling them they should just cut and paste their previous answers until the questions change.

Finally, Phil Schott posted You must be present to win pointing out the lack of resources.

There’s no books on this stuff and you can’t go to school to get your BA in deliverability. All we’ve got is each other.

While he’s right and there are no books or school lessons, I’m not quite sure how useful they would be. Things in the email space change very fast and what was true a few months ago may not be true now. However, there are some resources for people who want to learn about delivery. I have been working on the delivery wiki as a place to categorize and clearly present information to people. MAAWG holds training sessions for senders at every conference (and they’re always looking for suggestions for what kind of training sessions people want). There are other conferences and meetings that offer delivery help.

I think there are other options as well. But the real solution is more involvement and more information sharing between delivery professionals. The knowledge is there and we can share it among ourselves. We don’t need to rely on the overworked and underpaid staff at the ISPs to teach the newbies how to do their jobs.

Tags: , , , , , , , .


You must be present to win

Guest post by Phil Schott

I often have the pleasure of putting my four year-old son to bed at night and I’m usually exhausted afterward. It’s a never-ending string of questions and admonishments that goes something like this,

“Daddy, is it a stay-at-home day tomorrow?

“No, Joe, tomorrow is a go-to-school day, it’s Tuesday. Joe, stop talking and go to sleep and please stop picking your nose.”

“Daddy, how long until the Easter bunny comes?”

“A few weeks. Now, go to sleep and stop picking your nose, Josef.”

“Dude, what did I say about picking your nose?”

“Sorry daddy, I can’t help it. It’s my job.”

“Daddy, When’s it going to be my birthday?”

“Joe, you’re not going to live to see your birthday if you don’t stop picking your nose and go to sleep.”

Lather, rinse, repeat for about 10-30 minutes every night. Same questions, same answers, always picking his nose.

In retrospect it seems funny and maybe sweet, but it never does at the time and the thought of doing it all over again tomorrow night makes me want to run out screaming.

However, I realize that if not me, who? Who’s going to tell Joe to stop picking his nose? Who’s going to answer his questions? I have to. It’s my job. If I want to be his dad, that’s what I’ve got to do. If not, then I don’t get to be his dad, I don’t get to be part of his life, and I don’t get to be part of my family.

There are folks in our industry just like Joe and me–those who never seem to get it, those who ask questions over and over, and those who tire of answering the same questions.

I’d like to thank those who answer those questions over and over. Folks like Al Iverson, JD Falk, Mickey Chandler, Greg Kraios, Ken Magill, Laura Atkins, Steve Atkins, Karen Balle, Annalivia Ford, and many others who deserve to be on this list.

I’ve only been in deliverability for a few years and I’d be nowhere if these folks hadn’t answered my dumb questions, posted their thoughts, shared their knowledge, and told me to stop picking my nose on occasion.

It pains me though to read from time to time the ranting of those in our industry who want to decry the dumb marketer, give up, and take their ball home. It’s a shame, but that’s their right and their decision. However, they then don’t get to be part of the community. They lose the effectiveness to tell a dumb marketer to stop picking his nose. They become a washed-up, has been, curmudgeon with no voice. Like with my four year-old son, if I want to be a part of the deliverability community I’ve got to stick it out and deal with it. You have to be present to win.

In her post, A very young industry, Laura Atkins of Word to the Wise quotes ExactTarget’s Joel Book as stating that less than 20% of those in email marketing have more than two years experience. Yes, it’s an industry full of four year-olds. If you’re one of those in the know are you going to bemoan this fact that’s beyond your control or are you going to work to make the community you’ve helped build a better place? You absolutely can choose to move on. We will miss you and I wish you the best of luck. But either keep helping out as you’ve expertly done or get out of the way. Don’t take cheap shots at those trying to do the right thing and trying to do some good work.

For those of you tired of answering the same inane questions you’re fooling yourself if you think the folks who really need to hear your message are reading. They’re not. And they’re going to keep on asking their inane questions until somebody helps them out. I choose to help them out. I choose to be part of the community. I choose to be present.

A big part of the issue is how daunting it can be to ask for help without the risk of appearing the fool. There are far too many folks in this business of deliverability who are more interested in proving how smart they are and selectively sharing knowledge than they are in helping raise the overall level of consciousness and enlightenment.

If you want the idiots and fools to go away then help them become something more. Help them like no one helped you when you started out. With much effort, time, and frustration, I could pick through five years of your blog posts to find the one bit of information I need, or you could give me the URL to the post that will reveal all. I’m not asking you to spoon feed me, I’m just asking for a little help. There’s no books on this stuff and you can’t go to school to get your BA in deliverability. All we’ve got is each other.

Phil Schott has been handling delivery and compliance for a major ESP for the last 3 and a half years.

Tags: , , , , , .


Troubleshooting email delivery

Mark Brownlow has a post up explaining how he discovered some problems with delivery at Gmail by digging deeper into his statistics. Mark goes through his thought process including his initial conjecture on what might be causing the problems and then how he looked at the data to see if his supposition fit the data.

I love this post. It is so refreshing to watch someone document how they asked questions, then looked at data to find out the answers. Too many people treat best practices in email delivery as a set of rules that are meant to be broken. Instead of actually asking questions and determining what is best for their market and their recipients they implement best practices.

Following best practices isn’t exactly a bad thing, the reason they’re best is because they’re easy to communicate practices that will not result in bad outcomes. But, they’re not always the ideal practices for a specific situation. Best practices are ones that work across a wide range of senders and situations. Blindly implementing best practices will not always result in the best outcome for each situation.

Mark’s post is a tutorial in the art of looking at email delivery. I think there is a need for more of those kinds of posts, explaining the process from identifying an email problem through to confirming that is actually the problem and then testing potential fixes. I’ll be posting troubleshooting guides here over the next few weeks and months. If you have an issue you think would be an interesting case study drop me an email and we’ll go through it.

Tags: , .


Which is better UTF-8 or ISO-?

Someone asked today on a mailing list whether they should be using UTF-8 or “ISO” encoding for sending email. What’s the best choice depends on some of the details of the situation, but here’s the answer I gave:

UTF-8 will work for pretty much anything, as it’s just an 8 bit encoding scheme for Unicode (which is supposed to be the one character encoding to rule them all). It’s well supported in most languages and development environments – Windows has been native UTF-16 under the covers since the mid 90s, for instance – and typical messages that use mainstream glyphs should render well from utf-8 in most western MUAs and browsers.

There are still a very few old or broken clients out there that will not handle UTF-8 well but (outside the asian language market, where there’s still some non-ASCII, non-Unicode legacy usage) they’re typically ones that don’t really handle any character set encoding well and the only thing safe to send to them is either plain ASCII or whichever ASCII superset their OS happens to support natively (which is probably an argument for sending Windows-1252 codepage, but not a terribly strong one).

The various extended ASCIIs (such as ISO-8859-*) will only work for messages that are written solely using characters from that character set. If you have even one character in a message that cannot be expressed in ISO-8859-1, then you can’t use ISO-8859-1 to send that message.

ISO-8859-1 (aka Latin1) is fairly sloppy in some respects – it has no apostrophe, nor single quotes, for instance – but it can handle an awful lot of languages, from Kurdish to Swahili. It can’t handle Dutch, Estonian, Finnish, Hungarian and Welsh particularly well, nor can it show the Euro symbol (ISO-8859-14 or -15 are needed for some characters there).

A common problem is that many people (and the software they write) think that Windows uses Latin1. It doesn’t, it uses Windows-1252. If you accept messages written on Windows, using the Windows-1252 code page, and throw them out on the wire as ISO-8859-1 what you end up with is not quite right. It mostly works, as the two codepages overlap quite a bit, but they have different glyphs in the 0×80-0×9f range. So if you use single or double quotes (“smart quotes”), or the Euro symbol, or ellipses, or bullet, or the trademark symbol in your message they’ll be garbled. This is so common that some mail clients and web browsers will actually treat a document that claims to be ISO-8859-1 as Windows-1252, but that’s a bug workaround and not something it’s really safe to rely on.

If you’re doing personalized messages, and you’re sending one of them to Győző and one of them to Eiður then you may have to use different character sets for the two messages. If you’re talking about Győző and personalizing it for Eiður then you might find things break horribly.

Someone probably has some concrete data on mail client character set support, broken down by region and language, but my understanding is that this is a reasonable approach:

  1. If you’re sending any entirely non-ASCII messages (asian languages, primarily) then it’s way more complex than it should be, and you need to find someone who really understands the deployment of BIG5 and ShiftJIS and so on if you want to dial in to perfection. If you don’t have someone like that, UTF-8 is your best bet.
  2. If your tool chain supports non-ASCII messages, and you want to choose a single encoding, go with UTF-8. If the UTF-8 you end up sending is entirely, or almost entirely, ASCII then this will render well even on the tiny fraction of mail clients that don’t support character sets. (This is what I’d do).
  3. If your tool chain supports non-ASCII, but almost all your messages are plain ASCII and you want to be clever, encode each message separately. That’ll mean storing the templates and mailmerge data as UTF-8 probably, then converting the resulting message to ASCII if possible. That’s actually a lot simpler to do than it sounds, as a valid UTF-8 message that can be converted to ASCII is already an ASCII message, so all you need to do is set the charset to UTF-8 if there’s any byte bigger than 127 in the message or to us-ascii otherwise.
  4. If your toolchain was written to support only truly 8 bit character sets, or if that’s all you ever expect to send, then ISO-8859-15 is about as good as you can do.

If you’re accepting messages via a web front end, or any other path that could lead to stuff being copy and pasted or uploaded from Windows, look out for the Windows-1252 issues, and convert the provided message to whatever format you’re storing templates as (probably utf-8).

Then, once you have the message, be careful to send it over the wire as 7 bit ASCII, probably using quoted-printable encoding, to make sure it isn’t thrown away or transcoded to base64 en-route.

Tags: , , .


Transitioning Yahoo bound email from Goodmail certification

In early February Yahoo announced they were no longer offering preferred delivery to Goodmail customers. By the end of March, Yahoo will have decommissioned the Goodmail specific mail handling servers. What does this mean for Goodmail customers who have no history of mail to the normal Yahoo mail exchanges? Will they have to go through an IP warmup period?

Thankfully, no, they won’t. IP addresses that have been delivering Goodmail certified mail are being transitioned across to the Yahoo whitelisting program. Just because customers are losing Goodmail certification does not mean they will lose all their sending history at Yahoo. This is very good news, as senders don’t have to give up all their sending history due to Yahoo’s decisions.

I have heard some grumbling from some delivery experts that the ‘pre-warmup’ isn’t meaningful or useful. I strongly disagree. The reason senders have to warm up IP addresses is because spammers are very good at finding unused addresses and exploiting them to send spam. The warmup period gives the receivers a way to evaluate the mailstream from a particular IP and determine if the mail is wanted without having to subject their users to excessive amounts of spam.

In this case, Yahoo knows that good senders will be moving from one set of mail exchangers to another. They have nothing to gain by forcing those senders to go through a warmup period. They know what the mailstreams look like and can special case them. This isn’t a benefit every sender gets, in fact losing established reputation is one of the major considerations when moving IP addresses, ESPs or certification services.

While current Goodmail customers are getting this benefit now, they will be subject to the same spam filtering other senders face at Yahoo. Failure to meet Yahoo’s thresholds for good email may result in loss of whitelisting, bulk foldering of email and rate limiting.

More detailed information about delivering to Yahoo is available on the Word to the Wise Delivery Wiki.

Tags: , , .


Yahoo decomissioning Goodmail MXs

Yahoo announced today that they would be decommissioning the Goodmail specific MX machines as of March 24. Goodmail customers should talk to Goodmail about necessary transition issues. On Yahoo’s end, my understanding is that they are working to make the transition as painless as possible for the customers of Goodmail.

This seems to be the final nail in the coffin for Goodmail at Yahoo.

I’ll have more next week on how senders can cope with the loss of Goodmail certification.

Tags: , , .


Improving the email interface

Want an improved email interface? Then build it.

There’s been an ongoing discussion about adding thumbs up / thumbs down style buttons to email clients. While I am dubious this is a useful feature or something that recipients will use, if there are others in the industry that think it would be useful then I strongly suggest they go ahead and create it.

In fact, there are a couple things that have been asked for in email interfaces that aren’t currently provided. Last October I blogged about adding an unsubscribe button to email clients.

Now, many email clients have not implemented an unsubscribe button, but I’m told it’s not difficult to write plugins for many common email clients (mail.app, thunderbird, outlook). I think that if a few senders should get together and write the unsubscribe plugins and make them freely available to recipients. If the tools are out there, and the recipients want them, then they’ll be used. There’s nothing stopping senders from creating the tools they want created. Hire a few developers and get it done. You’re the marketers, market the benefit to the recipients to use your tools to improve everyone’s life.

The same thing goes for a thumbs up / thumbs down mechanism. If you write it, and it is useful (and useable) then users will demand that the proprietary interfaces provide the same functionality.

In the blog post I initially made that suggestion one of the comments said

To suggest that email marketers ought to get together and write plug-ins for popular email clients in order to fix the problem misses the point – this is a feature that ought to be a standard part of the software/web interface, as a plug-in it’s subject to vagaries like incompatibilities when the software is upgraded (see how many Firefox plug-ins show errors immediately after an upgrade).

Which shows some gross misunderstanding of how feature development works. Every feature started off with someone saying the mail client needs to do X either because they wanted the feature or because their users were asking for it.

In the case of unsubscribe buttons, or thumbs up/down buttons, end users aren’t currently asking for the functionality. That’s not to say they wouldn’t use it if it was there, just that they aren’t asking for it. The way to get the functionality inserted as a standard part of the software/web interface, is to get users to ask for it. In order to get users to ask for it, the best way to start is to create a plug-in that they like and use. If they like it in their Outlook interface at work, then they’ll ask for it in their webmail interface at home.

Many plug-ins are written by a single coder as a hobby or functionality they wanted or needed for their own mail. This is why there are maintenance failures this is a hobby or labor of love and real life™ interfered in the upkeep or the functionality was no longer needed by the maintainer or any number of reasons. This can trivially be resolved by someone being paid to maintain the plug-in and keeping up with the new versions of mail clients throughout the development cycles.

Is this guaranteed to lead to the email interface improvements senders are asking for? Of course not, nothing is guaranteed. But I can assure you that actually creating the protocols and buttons and interfaces will lead to widespread adoption faster than simply waiting for someone else to do what you want.

Tags: , , , .


With great wisdom…

Guest Post by JD Falk

There was certainly some surprise in the room when I pointed out (yep, it was me) that Laura has been around since before there were ESPs. Part of it, I’m sure, was because Laura’s not particularly ancient — and part was because it’s a shock to realize that people sent and received email and everything was just fine long before the segment of the industry that you work in had even been imagined.

Since this was at MAAWG, there were quite a few people in the room who were involved before there were ESPs (I asked for a show of hands) — and it was interesting to see how many of them work for ESPs now. Commenting on Laura’s article “A very young industry,” Kent McGovern mentioned three — including Anne Mitchell, who made up the word “deliverability” not long after stepping down as the head lawyer for the first shared blacklist of email-sending IP addresses.

Just think about that. She was the head lawyer for the MAPS RBL before there was such a thing as deliverability. (I worked with her there; so did Laura.)

There are a lot of us who’ve been around that long, and most don’t work in the deliverability/marketing side of the industry. Nearly all of us have become cynical over the years; some were cynical to begin with. A few, sadly, have burned out entirely from the frustration of having the same arguments, same discussions, over and over and over.

I think some of the recent refrain calling for ESPs to pressure each other into better practices comes in part from that same frustration. Yes, bad practices are bad, but we’re also tired with teaching the same thing to people with the same title, and feeling like the message never gets through. Part of what we’re saying is “It’s your industry, you’ve learned this stuff, now you teach ‘em.”

And when you do, it does work — far more often than when we say it, because you speak the same language. There’s now a generation (for lack of a better term) of ESP & deliverability staff who weren’t around before there were ESPs, maybe not even before CAN-SPAM, but have learned many of the same things and undergone similar transformation. Who’d have thought that Jaren Angerbauer — quite possibly the nicest guy in the industry — would ever start sighing at those young whippersnappers like a cynical old anti-spammer? And Jaren’s not only teaching deliverabilitators; he’s also teaching college students, ensuring that they’ll know far more when they enter the work force than you or he did.

We old-timers once struggled with the idea that we must reach out — even to people we disagree with — and teach what we knew, learning along the way to put it into terms that marketers understand. It’s so much simpler to add to a blacklist and throw away they key, declaring “not my problem anymore.” But we did start teaching, and look how far we’ve come; we’re still doing it, and look how much further there is to go.

Now it’s time for the next generation to do the same. Stop looking to us, or to the ISPs, to solve the problems of your industry for you; we’re busy dealing with spam, as we should’ve been doing all along. Your colleagues’ cluelessness is exactly as impermanent as your own was, and can be overcome in the same ways. Whether you have fifteen or ten or five or merely two years of experience, you’ve found your way to this blog and read down to this line, and attained some measure of wisdom, and you can ease the passage for others.

When someone at a marketing conference says something that you know isn’t true, that you know will result in poor deliverability and industry ire, call them on it. Engage them in a dialogue. Teach, explain, cajole, push — because with great wisdom comes great responsibility.

It’s your turn.

J.D. Falk is Director of Product Strategy for Receiver Products at Return Path, which is not an ESP.

Tags: , , , , , .




Follow me on Twitter