This first post is about IP address reputation with some background on why IPs are so important and why ISPs focus so heavily on the sending IP.

Why IP addresses?

ISPs built reputation around IP addresses because it was one bit of data that malicious senders / spammers couldn’t forge. The connecting IP is a fundamental part of the network transaction and if you forge an IP then SMTP can’t work. Because that was the reliable data they had to work with, that’s what they used. Even now, when there are other kinds of data, the IP address is still the first thing the receiving MTA sees.

What is IP reputation?

IP reputation can best be summed up as “past performance is an indicator of future results.” In other words if recipients responded well to mail from an IP address in the past, then they’re likely to respond well to new mail from that IP address.

How is IP reputation measured?

While each spam filtering company and ISP have their own ways of calculating the reputation of an IP address, there are some similarities in what they measure.

• How many non-existent email addresses is this IP attempting to deliver to?
• How many abandoned email addresses is this IP attempting to deliver to?
• How many “known bad” email addresses (spamtraps) is this IP attempting to deliver to?
• How many recipients complain about receiving this mail?
• How many recipients complain about not receiving this mail?
• How respectful of my resources is this IP?
• Does this IP keep connections open for long periods of time?
• Does this IP retry deliveries too aggressively?
• Does this IP stop mailing addresses after receiving a “user unknown” message?
• Is this IP address configured as if the associated machine was infected by a virus?
• Is this IP address listed on blocklists we use?

That is by no means an exhaustive list of what ISPs measure. If they can measure it they’ve tried. If the measurement helps them separate spam mail from not-spam mail then they’re using it.

How fast does IP reputation change?

IP reputation is often measured over multiple time periods. ISPs can look at a 1 day, 7 day, 30 day and 90 day reputation. A good analogy is stock prices. Prices can be very volatile in the short term, but more consistent over the long term. A single bad day, where one or more reputation measurements go bad, may affect delivery that day or the next day but won’t damage an overall good reputation. Likewise, a few days of improved mail may not be sufficient to counter months of poor reputation.

How is IP reputation used?

Mail from IPs with a high reputation is accepted faster and at a higher rate than mail from IPs with a lower or unknown reputation.  IP reputation can also influence whether mail is delivered to the inbox or the bulk folder.

Key IP Reputation takeaways

• IP reputation is about how recipients react to mail from that IP. Happy, content recipients turn into good delivery.
• Brief changes (for good or bad) don’t necessarily ruin delivery over the long term.
• Steady improvements will result in improved reputation.
• It may takes as much time to change a reputation in one direction or another as it took to establish the reputation in the first place.

Next we’ll look at content reputation, how it’s measured and used.

EDIT: A version of this information is available at the Word to the Wise wiki

EDIT: This post was also shared at CircleID

ICANN allocated the last couple of blocks of general usage IPv4 addresses to APNIC earlier today.

There are just five usable blocks of addresses left, and they’re reserved by IANA policy for the final phase of IPv4 exhaustion, one for each RIR.

Like any other resource that’s been strip-mined to depletion that doesn’t mean that IP addresses are completely unavailable – there are still some in the delivery pipeline (the regional internet registries who handle allocation of addresses), ISPs have stockpiled addresses they don’t really need yet in anticipation of the exhaustion, and we can still recycle the addresses we’ve already got.

But there won’t be any new IPv4 addresses available.

What does that mean to you?

It’s going to affect your business in lots of ways over the next few years. And delaying thinking about it will just make it more expensive and more painful once you’re forced to pay attention.

IPv4 addresses are increasing in price

You’re going to find it increasingly difficult to get assigned new ones. It’s long past time to stop thinking of them as “effectively free”.

You’re going to need IPv6 production deployments fairly soon, and we’re well past the point of “it’s cheaper to wait, so that our vendors can do the IPv6 work that’s needed”. Delaying putting IPv6 prototypes into the field is going to get increasingly painful and expensive.

If your CTO and CFO are not concerned about this yet, they should be.

Stop wasting addresses

Dedicating even a single IP address to a customer is wasteful, if you don’t need that to handle the volume of email sent. You should be signing all your mail with DKIM today and building domain based reputation for your customers, as IPv4 based per-customer reputation is going away.

As you grow and gain more customers you’re going to have to start sharing v4 addresses between customers, which will share their IPv4-based reputation.

And fairly soon, you’ll want to start sending mail over IPv6 to some destinations. Odds are good that per-customer IPv6 reputation isn’t going to happen much. There’ll be some broad IPv6 based reputation to distinguish your outbounds from spammers and botnets, sure, but IPv6 based reputation down to a per-customer level isn’t something you’re likely to see much benefit from.

Assigning multiple IPv4 addresses to a single customer is ridiculously wasteful. You’re going to want to engineer away from any need to do that – and have your sales staff stop promising it – so that you can recycle those precious, precious IPv4 addresses for use elsewhere by other customers.

Your recipients are moving to IPv6

Big consumer access ISPs are some of the biggest consumers of IP addresses. They’re likely going to be moving to native IPv6 for end-users, combined with some sort of v6-to-v4 translation before anyone else.

That means that your web users, from home and mobile devices, will be trying to reach you via IPv6 sooner rather than later. Native v6 will mostly work better than v6-to-v4. So you want to be thinking about v6 access soon.

Have you got v6 space assigned from your ISP(s) yet? No? Ask them for it this week.

Do you have plans for making your image and click-tracking webservers v6-aware?

Everything you do with IPv4 you need to be able to do with IPv6

Does your address list management support tracking signups and confirmations from IPv6 users as well as IPv4?

Can you track opens and click throughs from IPv6 users?

How about reporting? Does your database and reporting engine support IPv6 addresses? Can you do geolocation for IPv6 addresses?

Does your smarthost vendor support preferentially routing mail via IPv6?

IPv6 is an opportunity

What would you do if someone were to offer you dedicated MX machines at Yahoo, Google and Hotmail. Machines that were much the same as their primary MXes, but lightly loaded with much more available capacity. How much would you be prepared to pay for access to them?

Sooner or later there’ll be IPv6-only MXes at those, and other ISPs. Will you be ready to use them?

Do your competitors offer an IPv6-ready system yet? How much of a business disadvantage will you be at once they do?

And finally

None of this is new. You’ve known for years that you need to be more frugal with IPv4 addresses and have dual-stack IPv6-capable services.

But it’s getting urgent.

I need multiple IP addresses in different locations so as to provide redundancy against blacklisting of my ISP

Why this is right
If you think that your email is likely to be blocked due to the reputation of your ISP then having a backup ISP makes some operational sense.

Why this is wrong
It’s just very, very wrong. Why are you, an email sending company, buying service from an ISP you expect to be blacklisted? ARE YOU NUTS? Regardless of how cheap the deal an ISP is offering you, if you think that their reputation is bad enough that you need to do your network engineering around the possibility that they’ll be blacklisted, DON’T BUY SERVICE FROM THEM!

Don't do this, or you'll make Godzilla facepalm

I need multiple IP addresses in different locations to provide redundancy against network outages

Why this is right
If all your traffic goes out via a single ISP and your connection to that ISP is eaten by a backhoe you’re  not going to be sending any email until that’s fixed. And from personal experience I know that can easily take two or three days for even minor fiber damage.

But be careful
You need to be sending email fairly consistently from an IP address in order to maintain a decent reputation for that mail source. If you treat a second location as a cold standby, only used when your main ISP breaks, expect to see serious delivery problems as you migrate across to it. Better to spread load across both locations, to keep both sets of addresses “warm” – but remember that that will halve the amount of traffic that a receiving ISP will see from any given IP address, which will change your decisions about whether to assign customers to a pool or not.

A better architecture
If all your production machines – smarthosts, webservers, databases – are hosted in a high quality datacenter run by an ISP with redundant connections to the Internet then you don’t need to worry about the redundancy yourself. If one of those connections is broken the ISP will route the traffic over a different connection to the same IP addresses mostly transparently. You won’t need to reconfigure anything, it’ll just keep working.

(Why don’t you just have redundant connections to servers hosted at your offices? First, it’s very expensive and time consuming to handle the mechanical aspects of ensuring that your two connections are really redundant, rather than being multiplexed onto the same fiber or running in the same conduit. Second, the smallest block of addresses you can multi-home in this way is 1000, and you can’t acquire those unless you’re already using more than 500 IP addresses efficiently.)

The weak point then is the connection between your offices and your datacenter that you need to administer the servers and provide customer support. But the IP addresses used for that don’t matter, so it’s easy and cheap to have a backup connection – even a cheap consumer cable or DSL connection. Or, if you’re a very small company, have your customer support folks use laptops and know which local coffeeshops and bars have free wifi.

Hello Kitty Backhoe is nearly as scary as Godzilla

I need multiple IP addresses per customer so as to manage filtering issues

Why this is right
If you have, for example, three dedicated IP addresses per customer and one of those IP addresses gets “randomly” blacklisted, then you can divert traffic to the other two IP addresses temporarily while you resolve the listing.

Why this is wrong
While there are many, many reasons why a source of email can be blocked there are far fewer that affect enough recipients that you really care. If a block affects one guy and one beagle in North Carolina, it’s not worth your operational concentration to care.

Of the blocks you might care about they’re almost all going to be based on recipient response, reputation and content. If one of those blocks affects an IP address you’re sending a mail stream from, it’s probably going to affect the other IP addresses you’re sending the same mail stream from really soon. So the right operational decision is almost always going to be to suspend mailing (either to one particular recipient ISP or to all recipients) until someone has had time to investigate the underlying issue.

Why else this is wrong
If your network engineering decisions are driven primarily by avoiding recipient email filters then you have a deeper business philosophy or customer vetting issue to consider.

Three heads don't help if they all do the same thing

I need multiple IP addresses per customer so that that customer can deliver mail in a timely manner

Why this is right

If your customer needs to deliver a message to 100,000 recipients within 30 minutes and 3% of them are at an ISP that only accepts 1000 emails per recipient per hour then you’re going to need at least 7 IP addresses dedicated to that delivery. See the previous post on throttling at ISPs.

Why this is wrong

Email is a store-and-forward protocol, more comparable to traditional postal mail than IM or SMS or fax. Delivery of the vast majority of a mailing list within thirty minutes is certainly possible, but relying on or obsessing about delivery of an entire list in that time period is unrealistic.

Why else this is wrong

A lot of spam is sent by botnets, networks of thousands of compromised machines that are used to send hundreds of thousands of copies of spam from a thousand different sources simultaneously. If you try and use a lot of source IP addresses to get a lot of copies of the same message delivered simultaneously then you’re likely to trigger anti-botnet measures. You need to have a history and a relationship with the receiving ISP to avoid  that – and if you have that, you shouldn’t need multiple IP addresses.

Yet another reason this is wrong

Delivery of 100% of an email list in a timeframe measured in minutes is not something you’re ever going to be able to guarantee without heroic measures. If your customer is prepared to pay for those heroic measures, you still won’t be able to get to 100% but you should talk to a consultant who might be able to help you get to 99% in those rare cases where it’s something that the recipients might care about. If they’re not prepared to pay for those heroic measures, why are you wasting your time?

Supersonic Delivery

I need many IP addresses so that I can work around ISP throttling limits

Why this is right: There are ISPs that limit the number of emails that can be sent from a particular IP address in a given time period to quite a low level, as low as 1000 emails per hour per IP address in some cases. If that’s a mid-sized ISP with perhaps a million users and an ESP customer is sending email to 5% of their user base (not unreasonable for some customers) then that would take more than two days to send a news letter to those recipients, assuming absolutely perfect management of send rate. With more realistic inefficiencies for send rate management it could easily be a week. Using multiple IP addresses to spread the traffic in that case seems perfectly reasonable, though it would make everyone involved happier if the ISPs used more reasonable rate limiting metrics, perhaps tied to sending IP address reputation rather than applied globally to all non-whitelisted senders.

Why this is wrong: Not every customer will be trying to send huge volumes of email to a receiving ISP that throttles inbound mail. For dedicated IP customers there’s no need to give them extra outbound IP addresses for this reason unless it actually becomes a real operational problem -  if it takes an hour or two for a  senders maildrop to reach the inbox at a particular ISP that’s not an operational problem. For pooled IP customers you may need to add IP addresses to deal with this, but only to the extent it’s needed to keep delivery times for pool customers to that recipient ISP reasonable. And two or three hours is not unreasonable.

Why else is this wrong: Naive throttling of this sort, combined with the obvious engineering changes needed for senders to work around it hurts everyone – senders, recipients, receiving ISPs. You might have to work around it in the short term, but in the longer term work with receiving ISPs to resolve the problem in a way that makes everyone happier, whether that be whitelisting your outbounds or moving to a reputation adaptive throttling approach. Of course, if they tell you that your mail is throttled because it has a poor reputation you need to fix that before doing anything else.

Tread light as a moth

I need lots of IP addresses so my MTAs can handle the volume of mail sent

Why this is right

One IP address per outbound smarthost is a sensible minimum. It is possible to set up multiple smarthosts behind a single IP address using a proxy server or reverse load balancer, and some organizations do that, but it makes it much harder to diagnose some sorts of operational problems. If the different smarthosts behind the proxy use different hostnames then the externally visible behaviour will be a single IP address HELOing as many different machines – which is behavior that is otherwise distinctive to spam sent from botets of infected machines, so will lead to mail being blocked. If, instead, they all use the same hostname then it’ll be hard to say which one of the cluster of smarthosts sent a message if there is a problem that needs to be diagnosed later.

So if you need 50 smarthosts to handle the volume of email you’re sending out, that’s really good justification for needing 50 IP addresses.

Why this is wrong

A typical smarthost can send an awful lot of email if it’s configured correctly. I tried to find some hard numbers on that, but  smarthost vendors don’t seem to publish their numbers much, and most of the trustworthy benchmarks I found published were from some years ago, when servers were a tenth of the speed they are now.

Do I trust all these numbers? Not necessarily, and they’re not directly comparable, but they all look quite plausible from a software engineers point of view. (It would be fascinating to get some performance numbers from vendors that are comparable, though. A smarthost-for-bulk-mail benchmark. Hmm….).

Edit: I’ve confirmed that Message Systems can exceed 1,500,000 deliveries an hour on a well provisioned server – and that’s while providing all the knobs and reports and monitoring that make managing delivery at any sort of volume much easier than a dumb smarthost.

But if you assume that you can send even 100,000 50k messages per hour from one smarthost, that’s more than enough mail to saturate a ten megabit connection, and to send seventy million emails / month – for each smarthost.

There are good reasons to have more smarthost capacity than you’re using, and to have more than one IP address per smarthost, but pure MTA capacity is hardly ever going to be justification for using more than a small handful of IP addresses to send mail from.

The monkey can't justify his building full of mailservers

I need at least one IP address per customer, to handle IP based reputation

Why this is right

While DKIM is gradually moving the main key for reputation tracking to a domain based token, right now the main key that is used to track reputation is the sending IP address.

If you have multiple customers sending mail of different quality using a different sending IP address for each of those customers means that the good customers will not be penalized by the poor behaviour of the bad customers. And, just as importantly, poor customers will not benefit from the behaviour of the good customers. This allows receivers to track sender reputation more accurately, and so delivery just wanted email to their recipients better. That makes everyone happy (other than the bad customers who deserve to be unhappy until they fix their practices).

Why this is wrong

Reputation is tied to sending IP address, but it’s also affected by volume of emails received from that IP address, and the consistency of volume. If a customer is only sending a few hundred emails a week to any given receiver ISP or they’re only mailing monthly then they won’t be able to maintain much of a positive reputation, simply because they’re too small to keep track of or because they mail so infrequently that each time they mail the receiving ISP will have forgotten about their previous mailings. In those cases the sender will be treated much the same as a new sender from a given IP address (neutral, at best, maybe poorly). For those cases a customer is likely to get better delivery rates if their mail is sent through an IP address pool that sends enough email overall to be noticed and tracked by receiving ISPs.

Another reason this is wrong

Reputation is tied to sending IP address, but receiving ISPs aren’t stupid and do recognize attempts to game the system. If you’re an ESP with a mix of good and bad customers then segregating the IP addresses they send from will not completely isolate the reputation of those customers from each other. The bad customers will drag your reputation as an ESP down more than the good customers will pull it up. And as your reputation as an ESP degrades it will pull down the reputation of your good customers much more than it will increase the reputation of your bad or unknown customers.

So segregating senders onto their own IP addresses doesn’t entirely separate their reputation from each other or from their ESP. And if you believe it does, you’re likely to make business decisions based on that misunderstanding that will badly affect your reputation and the delivery rates of your customers. Don’t fall into that trap.

Godzilla sneaks up on Tokyo

The ESP was talking about assigning a couple of dozen IP addresses to each customer, because they might be useful for spreading load and it would provide some flexibility for moving from one IP address to another if one should get blocked. And IP addresses are pretty much free. They were wrong.

The ISP was considering an application for 750 IP addresses from a new ESP customer. They assumed that there was no possible reason other than snowshoe spam for an email related customer to need that many IP addresses. While I suspect they may have been right about the specific potential customer, the general assumption was wrong.

I’ve seen a lot of reasons given by ESPs for why they need so many IP addresses:

1. I need at least one IP address per customer, to handle IP based reputation
2. I need many IP addresses so my MTAs can handle the volume of mail sent
3. I need many IP addresses so that I can work around ISP throttling limits
4. I need multiple IP addresses per customer so that that customer can deliver mail in a timely manner
5. I need multiple IP addresses per customer so as to manage filtering issues
6. I need multiple IP addresses in different locations to provide redundancy against network outages
7. I need multiple IP addresses in different locations so as to provide redundancy against blacklisting of my ISP

I’m planning on going through these in some future posts, pretty much like Godzilla goes through downtown Tokyo. Can you think of any I’ve missed? tell me in the comments!

Godzilla destroys Tokyo