I can say that users at webmail are easily as likely to click on a message that they have opened than users at pay-for ISPs.

Who else collects data on opens per ISP? And Monkeys, I’m looking at you, I know you have this data.

Under the new EEC definitions an open is measured when either a tracking pixel is displayed or a user clicks on any link in the email, including the unsubscribe link. Open rate is defined as the number of opens (either unique or total) divided by the number of accepted emails. Accepted emails equals the number of emails sent minus the number of emails rejected by the ISP for any reason.

The authors do caution, however, that even their measurements may under count the number of email subscribers that actually open or read an email. Some readers don’t load images or click on links but happily read and digest the content being sent. Others may not click on a link but actually visit a website or brick and mortar store to purchase something based on the email.

Overall, I think the definitions created by the S.A.M.E. group accurately reflect the things they want to measure within the limits of what is actually measurable. Their definitions won’t affect conversations in the short term, but are likely to drive change to standard terminology over the longer term. I do strongly encourage people to grab a copy of their document and see how their definitions compare with your current measurements.

If asked, most people will tell you that open rate is the number of emails that were opened by the recipient. The problem is that this isn’t actually true. Open rate is measured by the number of people that display an image in an email. Traditionally this has been a uniquely tagged 1×1 pixel, until some filters and mail clients stopped displaying 1×1 pixels. More recently, every image in an email is tagged, so opening one image would record as an open.

So open rate doesn’t actually tell a sender how many people opened and read an email. It really only records that an image in a particular email is loaded. It does not record when an email is opened. Some people don’t load images by default. Some people don’t load images at all, even when they open and actively read the text portion of the email.

Clearly, there are some uses for open rates. It can give a useful metric when comparing different forms of the same email (A/B testing) and when looking at user engagement over time. However, we have also recently seen that open rate is not predictive for click through rate.

Take, for instance, the email address I acquired in 1993. Yes, I still have this address. I stopped using it to sign up for lists in 1999 and stopped using it for most of the rest of my mail around 2001. This address, though, is on any number of spam mailing lists. The spam that gets through is usually sent by hard-core spammers. The ISP that hosts that mailbox uses Communigate Pro to filter mail, so much of the casual spam is filtered.

Generally, if I open an email (and load images or click through) on that account it is only in order to track down a spammer. For instance, I’m getting a lot of spam there from affiliates offering me the opportunity to purchase printing services for a very low price. I have actually been opening the mail, and clicking through. But I’m not clicking through because I’m interested in purchasing. I’m clicking through to see if my reports to abuse@ printer are resulting in any action against the spammers. (They’re not).

The thing is, though, I know that by clicking through on ads, I’ve now been promoted by the spammer to the “clicks on emails! it’s a live address!” list. Which only means I’m going to get more spam from them. Lucky me.

Using clicks and opens as a measure of engagement isn’t necessarily bad. But when using them you have to understand the limitations of the measurement and that what you may think it’s telling you isn’t actually what it’s telling you.

I have previously posted about my views on the best time to send email. There is no one best time to send email. In fact, my experience leads me to believe if someone said the best time to send email is at 4pm on Tuesday afternoon then 4pm on Tuesday afternoon would rapidly become the absolute worst time to send email.

It should come as no surprise, then, that I really like Mark’s #4 recommendation.

4. Make the question obsolete

An alternative option to second guessing the best time of day to send out your emails is to do away with the question entirely.

Eh?

Control of timing becomes less critical to success where the recipient either determines the time of send for you (a concept also applied in the one-to-one model described above). Or where the recipient is determined to seek your emails out, irrespective of when you send them.

Send relevant email that recipients want and they will seek you out. They’ll create filters to put your mail in their “must read” mailboxes, they’ll add your sending addresses to their address books, they’ll actively look for your mail and exactly when you send the mail will become less of a variable in response rates.

Researchers recently developed a model for email user behaviour that splits email users into two classes “e-mailaholics” that send, and presumably read, email all the time and “day labourers” that send, and presumably read, email during standard business hours. There is very little transition between groups, 75% of users stayed in the same usage group over the 2 years of the study.

What does this mean for senders? Senders need to know know how their recipients use email and which user group recipients are. By analyzing clicks and opens, senders can classify recipients and use that data to send mail that is more relevant and better targeted.

h/t arXiv blog at Technology Review

There has been an lot written about open rates recently, but there are two posts that stand out to me. One was the EEC’s post on renaming open rate to render rate and Mark Brownlow’s excellent post on what open rate does and does not measure. I’ve also weighed in on the subject.

Overall, I find open rates to be a very frustrating metric. Some senders, particularly those relatively new to email marketing, are so sure they know what open rate is and what it means, that they don’t take any time to actually understand the number. While the name “open rate” seems self explanatory, it’s actually not. Open rate is actually not a measure of how many recipients open an email. However, there are times where open rate is a useful metric for measuring a marketing program over time.

What is an open?
If asked, most people will tell you that open rate is the number of emails that were opened by the recipients. The problem is that this isn’t actually true. An open is counted when a tagged image in an email is rendered by the recipient’s email client. Not all mail clients render images by default, but the emails are still available for the recipient to read. If a user clicks on a link in an email that has not had an image rendered, some ESPs count that as an open as well as a click. In other cases, visiting a link in an email with no image rendered is just a click, no open is recorded.

What is the open rate?
Open rate is generally the percentage of email opens divided by some number representing the number of emails sent. Many senders use the number of emails sent minus the number of bounced emails, others use just the number of emails sent without factoring in the number of emails bounced.

Open rate is a secondary metric. While it does not measure the success, or failure, of a campaign directly, it can be used as a indicator for campaigns. Many people use open rate as a metric because it’s easy to measure. Direct metrics, such as clicks or average purchase or total purchase, may take days or even weeks to collect and analyze. Open rates can be calculated quickly and easily.

What the open rate isn’t
Open rate is not a measure of how many people opened a mail. It is not a measure of how many people read a mail. It really only records that an image in a particular email is loaded and, sometimes, that a link was clicked on. Open rates can be wildly different depending on how the sender measures opens and how the sender measures sends.

What senders use open rates for
To compare their open rates with industry averages
As I talked about above, this use of open rates is problematic at best. You cannot compare numbers, even when they have the same name, if the numbers were arrived at using different calculations. Open rate is not open rate and unless you know the underlying algorithm used you cannot compare two open rates. This is a poor use of open rate.

As a metric for advertising rates
Since a sender can manipulate the open rate by using different calculation methods, this is a good metric for the advertiser to use. It is not so great for the purchaser though, who is at the mercy of the sender’s metrics. There are contractual ways a purchaser can protect herself from an unscrupulous marketer, but only if she understands how open rate can be manipulated and takes steps to define what open rate is in use.

To judge the success of campaigns over time
A single open data point doesn’t mean very much, however, using consistently measured open rates a sender can measure trends. Open trends over time are one area that open rates can help senders judge the success, or failure, of a marketing campaign.

As one metric in A/B testing
Comparing open rates in A/B testing gives some indication of which campaigns recipients may be more interested in. As with trends over time, the lone measurement isn’t useful, but as a comparative metric, it may provide senders with insight into a particular mailing.

To judge the engagement of recipients
Over the long term, recipients who do not interact with a mailing become dead weight on the list. Too many non responders can hurt a sender’s reputation at an ISP. List hygiene, in the form of removing people who never open or click on an email, is an important part of reputation management.

As metrics for email campaigns go, open rate is limited in what it measures about an email campaign. However, as a quick way to measure trending or do head to head comparisons it is a useful metric.

Average e-mail click-through rates dropped 0.1% in the fourth quarter of 2008 from the third to 5.8%, the lowest ever recorded, according to a study released today by Epsilon. [...]

They are down from 6.1% in 2007 and 6.5% in 2006, according to the marketing services provider. [...]

According to the study, average open rates increased for the third quarter in a row to 20.9%, up almost 6% from the fourth quarter of last year.

The real factoid that jumped out at me about this is that it clearly demonstrates what a useless metric “open rate” is. As more and more people are, supposedly, reading commercial bulk email fewer and fewer of them are actually clicking on links.

What does this tell us? It tells us that open rate is not a way to tell you anything useful about an email campaign. If a sender can get more people looking at a particular email, and fewer people actually following through on the call to action, then clearly the problem is not that no one is seeing the email. There are lots of reasons why the clickthroughs might be decreasing, but it seems clear from the Epsilon study that simply getting more eyeballs will not fix anything.

Can we now stop using “open rate” to measure anything relevant?