Skip to content


IP Address reputation primer

There has been a lot of recent discussion and questions about reputation, content and delivery. I started to answer some of them, and then realized there weren’t any basic reference documents I could refer to when explaining the interaction. So I decided to write some.

This first post is about IP address reputation with some background on why IPs are so important and why ISPs focus so heavily on the sending IP.

Why IP addresses?

ISPs built reputation around IP addresses because it was one bit of data that malicious senders / spammers couldn’t forge. The connecting IP is a fundamental part of the network transaction and if you forge an IP then SMTP can’t work. Because that was the reliable data they had to work with, that’s what they used. Even now, when there are other kinds of data, the IP address is still the first thing the receiving MTA sees.

What is IP reputation?

IP reputation can best be summed up as “past performance is an indicator of future results.” In other words if recipients responded well to mail from an IP address in the past, then they’re likely to respond well to new mail from that IP address.

How is IP reputation measured?

While each spam filtering company and ISP have their own ways of calculating the reputation of an IP address, there are some similarities in what they measure.

  • How many non-existent email addresses is this IP attempting to deliver to?
  • How many abandoned email addresses is this IP attempting to deliver to?
  • How many “known bad” email addresses (spamtraps) is this IP attempting to deliver to?
  • How many recipients complain about receiving this mail?
  • How many recipients complain about not receiving this mail?
  • How respectful of my resources is this IP?
  • Does this IP keep connections open for long periods of time?
  • Does this IP retry deliveries too aggressively?
  • Does this IP stop mailing addresses after receiving a “user unknown” message?
  • Is this IP address configured as if the associated machine was infected by a virus?
  • Is this IP address listed on blocklists we use?

That is by no means an exhaustive list of what ISPs measure. If they can measure it they’ve tried. If the measurement helps them separate spam mail from not-spam mail then they’re using it.

How fast does IP reputation change?

IP reputation is often measured over multiple time periods. ISPs can look at a 1 day, 7 day, 30 day and 90 day reputation. A good analogy is stock prices. Prices can be very volatile in the short term, but more consistent over the long term. A single bad day, where one or more reputation measurements go bad, may affect delivery that day or the next day but won’t damage an overall good reputation. Likewise, a few days of improved mail may not be sufficient to counter months of poor reputation.

How is IP reputation used?

Mail from IPs with a high reputation is accepted faster and at a higher rate than mail from IPs with a lower or unknown reputation.  IP reputation can also influence whether mail is delivered to the inbox or the bulk folder.

Key IP Reputation takeaways

  • IP reputation is about how recipients react to mail from that IP. Happy, content recipients turn into good delivery.
  • Brief changes (for good or bad) don’t necessarily ruin delivery over the long term.
  • Steady improvements will result in improved reputation.
  • It may takes as much time to change a reputation in one direction or another as it took to establish the reputation in the first place.

Next we’ll look at content reputation, how it’s measured and used.

EDIT: A version of this information is available at the Word to the Wise wiki

EDIT: This post was also shared at CircleID

Tags: , , , , .


Can you verify email addresses in real time?

In a recent discussion about spamtraps and address lists and data collection a participant commented, “[E]very site should be utilizing a real-time email address hygiene and correction service on the front end.” He went on to explain that real time hygiene prevents undeliverable addresses and spamtraps and all sorts of list problems. I was skeptical to say the least.

Yes, there are APIs that can be queried at some of the larger ISPs to identify if an account name is taken, but this doesn’t mean that there is an associated email address. Yes, senders can do a real time SMTP transaction, but ISPs are quick to block SMTP transactions that quit before DATA.

I decided to check out one service to see how accurate it was. I’m somewhat lucky in that I created a username at Yahoo Groups over a dozen years ago but never activated the associated email address. This means that the account is shown as taken and no one else can register that address at Yahoo. But the address doesn’t accept any mail.

Sceenshot of verification

The address verification for Yahoo addresses

There is a service that offers real time verification and allows potential customers to check an address on their website. I plugged my Yahoo address into their text box. They verified it as active and connected to all networks. Just to make sure I checked my existing Yahoo address as well, and that shows the same: connected to active online networks.

I next sent an email to both Yahoo accounts. Yahoo accepted mail to my working account but bounced mail to the Yahoo Groups only account.

Final-Recipient: rfc822; biskybabe@yahoo.com
Original-Recipient: rfc822;biskybabe@yahoo.com
Action: failed
Status: 5.0.0
Remote-MTA: dns; mta5.am0.yahoodns.net
Diagnostic-Code: smtp; 554 delivery error: dd This user doesn't
   have a yahoo.com account (biskybabe@yahoo.com) [-5] -
   mta1289.mail.ac4.yahoo.com

This tells me that for Yahoo addresses, Briteverify is using some sort of API call to identify whether or not an account name is taken. But just because an account name is taken doesn’t specifically mean that an account is a valid email address. It’s probably better than no verification, but usage of all real time verification isn’t going to help in all cases.

What about email accounts that don’t provide an API or a way to check the validity of an account? In that case it appears that they are using an aborted SMTP transaction. we tested

Jan 24 15:20:00 misc postfix/smtpd[28917]: connect from
   smtpout9.briteverify.com[107.20.232.98]
Jan 24 15:20:01 misc postfix/smtpd[28917]: NOQUEUE: reject:
   RCPT from smtpout9.briteverify.com[107.20.232.98]: 550 5.1.1
   <mu/er9w9kmbyg+s5uehqdxqe@blighty.com>: Recipient
   address rejected: User unknown in virtual alias table;
   from=<admin@origindata.com>
   to=<mu/er9w9kmbyg+s5uehqdxqe@blighty.com>
   proto=SMTP helo=<emailver.briteleads.com>
Jan 24 15:20:01 misc postfix/smtpd[28917]: lost connection after
   RCPT from smtpout9.briteverify.com[107.20.232.98]
Jan 24 15:20:01 misc postfix/smtpd[28917]: disconnect from
   smtpout9.briteverify.com[107.20.232.98]
Jan 24 15:20:01 misc postfix/smtpd[28915]: connect from
   smtpout7.briteverify.com[184.73.155.120]
Jan 24 15:20:01 misc postfix/smtpd[28915]: NOQUEUE: reject:
   RCPT from smtpout7.briteverify.com[184.73.155.120]: 550 5.1.1
   <aardvark@blighty.com>: Recipient address rejected: User
   unknown in virtual alias table; from=<admin@origindata.com>
   to=<aardvark@blighty.com> proto=SMTP
   helo=<emailver.briteleads.com>
Jan 24 15:20:01 misc postfix/smtpd[28915]: lost connection after
   RCPT from smtpout7.briteverify.com[184.73.155.120]
Jan 24 15:20:01 misc postfix/smtpd[28915]: disconnect from
   smtpout7.briteverify.com[184.73.155.120]

The verification service did correctly identify both addresses as invalid. However, this is exactly the kind of SMTP behaviour that is blocked by many places.

Real time address verification for 100% of addresses is incredibly difficult. As I demonstrated above, their use of testing APIs makes the assumption that everyone with a login at Yahoo (or google or other places) has an email address, but this isn’t necessarily true.

There are other assumptions that realtime address verification makes.

  1. No one ever typos the left hand side of their email address into an address of another user at the site. This isn’t true, for instance, I entered a common typo of my email address into the form and the service verified it as accurate. It probably is a valid, deliverable account but that doesn’t mean that it’s a good address.
  2. Spamtraps are always undeliverable addresses. This is not true and the above form did verify a spamtrap address that a friendly blocklist admin checked for me.
  3. No one typos the right hand side of an address to a valid domain. This is not true. For instance, I know a number of spamtrap domains used by Trend Micro. The form validates addresses there and tells me I’m good to send.

I’m not trying to knock the real time address verification services, I think what they’re attempting to do is good. I think the glossy marketing, though, will lead senders into a false sense of security. Just because a 3rd party service tells you an address is deliverable, doesn’t mean that the address is deliverable or that the address is safe to mail.

I do think potential verification customers deserve to understand how the services work so that they can make good decisions about purchasing those services.

 

 

Tags: , , .


Information sharing and the Internet

Many years ago I was working at the UW-Madison. Madison is a great town, I loved it a lot. One of the good bits was this local satire paper called The Onion. This paper would show up around campus on Wednesdays. Our lab, like many university employees and students, looked forward to Wednesday and the new humor The Onion would bring to us.

At the same time, I was internet friends with an employee of JPL. I’d met him, like I met many of my online acquaintances, through a pet related mailing list.

One Wednesday, The Onion published an article Mir Scientists Study Effects of Weightlessness on Mortal Terror. As this was the time when the Internet consisted of people banging rocks together, there was not an online link to Onion articles. But I was sure my friend at JPL, and all his friends, would appreciate the joke. That night I stayed late at the lab and typed the article into an email (with full credit to the Onion) and mailed it off to him.

As expected, the article garnered quite a few chuckles and was passed around to various folks inside JPL. What wasn’t expected was another friend, from totally different circles, sending me a copy of that same article 3 days later. Yes, in 1997 it took three days for information to be shared full circle on the Internet.

Information sharing is a whole lot quicker now, with things coming full circle in mere seconds. But that doesn’t make the information any more reliable and true. Take a recent article in ZDNet Research: Spammers actively harvesting emails from Twitter in real-time.

ZDNet links to a study published by Websense, claiming that email addresses on Twitter were available for harvesting.

That’s all well and good, but all ZDNet and Websense are saying is that email addresses are available for harvesting. I’ve not seen any evidence, yet, that spammers are harvesting and sending to them. This doesn’t, of course, mean they’re not, but it would be nice to see the spam email received at an address only shared on twitter.

Well, I have unique addresses and an un-spamfiltered domain. I went ahead and seeded a tagged address onto twitter. We’ll see if it gets harvested and spammers start sending to it. I’ll be sure to keep you updated.

Tags: , , , .


Delivery and marketing, another view

In addition to posting some of my thoughts about how delivery and marketing have different and possible contradictory constraints, I asked folks on the Only Influencers list what they thought. They had some different perspectives, primarily being marketers. One person even welcomed me to the dark side.

The general response from the marketing side of things appeared to be that ISPs need to stop actually filtering marketing email. That would resolve the problems from the marketers perspective. I don’t necessarily think that will help. I believe if marketers had unfettered access to the inbox, most inboxes would be totally un-useable.

My thinking triggered other folks to consider delivery and marketing and what drives both. George Bilbrey, from Return Path, posted an article in Mediapost looking at why good delivery is an important part of a good marketing strategy.

George points out many marketers really do act as if delivery is separate and detrimental to good marketing.

I hear this with my clients and I hear this on discussion lists.  They think that the practices that drive high inbox placement rates are antithetical to return on their email marketing investment.

Exactly. I hear a lot of contempt for delivery consultants and good delivery practices from a lot of marketers. They claim our methods and our recommendations come from not understanding marketing. They flat out tell me that “we’re” manufacturing delivery problems by pointing out mail that users don’t want has poor delivery.

There are thousands of companies that have never heard of Return Path, or Word to the Wise, who don’t understand why their perfectly crafted marketing isn’t getting to the inbox. It’s because they don’t understand email and delivery. They want to do what works elsewhere, and those models don’t always map onto email.

And that’s why companies like Word to the Wise and Return Path exist.

Tags: , , , .


Cheetahmail on appending

Experian CheetahMail believes that opt-out email appending is no longer an acceptable practice, and that marketers should no longer use of this practice to acquire customer email addresses. EmailResponsibly

In my experience, appending causes major delivery problems. Of course, every time the issue comes up some marketer tells us who think it’s a bad idea that they successfully used appending and it worked and all the delivery problems are a figment.

Maybe the supporters will believe Ben and Experian / CheetahMail that appending is not a good thing to do. After all, Ben was a large proponent of the practice many years ago and Experian still sells appending services in some countries.

Sending mail without permission, which is what appending usually is, will cause delivery problems. Stick to real permission, not vague promises.

Tags: , , , .


The internet protests SOPA / PIPA

For those who don’t know, a number of major websites will be going offline tomorrow to protest SOPA and PIPA, including wordpress, reddit, Wikipedia and the cheezeburger sites. Tomorrow may be the most productive day ever on the modern internet. Google will also be linking to information about SOPA tomorrow.

I had some people ask me about the bills today and have been looking for explanations of the issues and why these laws are so problematic.

Over the years I’ve seen “the Internet” get upset about a lot of things. The idea of an Internet blackout has been tried again and again. This is one of the few efforts that has gotten major sites on board and may have an impact.

Tags: , .


SOPA and PIPA update

There is quite a bit of vocal opposition to the SOPA (Stop Online Piracy Act) making its way through the House of Representatives and PIPA (Protect Intellectual Property Act) making its way through the Senate. The opposition seems to have had an effect. I blogged about the bills late last year.

CNet reported today that the DNS provision was pulled from SOPA. This resolves one, but certainly not the only problem with SOPA. Also today, OpenCongress.org posted a letter from 6 co-sponsors of the Senate bill to Majority Leader Reid asking him to cancel the vote on PIPA.

Congratulations to everyone who worked so hard to make their voice heard by their elected representatives.

 

Tags: , , , .


Delivery versus marketing

I’ve been thinking lately that sometimes that what works for marketing doesn’t always work for delivery.

For instance in many areas of marketing repetition is key. Repeat a slogan and forge an association between the slogan and the product in the mind of the consumer. More repetition is better. Marketers can even go so far as using the same ad to drive consumer action. Television advertising is a prime example of this. Companies don’t create new content for every advertising slot, they create one or a few ads and then replay them over and over. The advertiser doesn’t even really care if the consumer consciously ignores the ads. The unconscious connection is still being made.

In the world of email delivery, though, having many or most recipients ignore advertising is the kiss of death. Too many unengaged users and filters decide that mail shouldn’t go into the inbox. These don’t even have to be ISP level filters, but Bayesian filters built into desktop mail clients.

Sending repetitive ads over email may be an effective marketing strategy, but may not be an effective delivery strategy.

Am I off base here and missing something? Tell me I’m wrong in the comments.

Tags: , , , .




Follow me on Twitter