I need multiple IP addresses in different locations so as to provide redundancy against blacklisting of my ISP

Why this is right
If you think that your email is likely to be blocked due to the reputation of your ISP then having a backup ISP makes some operational sense.

Why this is wrong
It’s just very, very wrong. Why are you, an email sending company, buying service from an ISP you expect to be blacklisted? ARE YOU NUTS? Regardless of how cheap the deal an ISP is offering you, if you think that their reputation is bad enough that you need to do your network engineering around the possibility that they’ll be blacklisted, DON’T BUY SERVICE FROM THEM!

Don't do this, or you'll make Godzilla facepalm

I need multiple IP addresses in different locations to provide redundancy against network outages

Why this is right
If all your traffic goes out via a single ISP and your connection to that ISP is eaten by a backhoe you’re  not going to be sending any email until that’s fixed. And from personal experience I know that can easily take two or three days for even minor fiber damage.

But be careful
You need to be sending email fairly consistently from an IP address in order to maintain a decent reputation for that mail source. If you treat a second location as a cold standby, only used when your main ISP breaks, expect to see serious delivery problems as you migrate across to it. Better to spread load across both locations, to keep both sets of addresses “warm” – but remember that that will halve the amount of traffic that a receiving ISP will see from any given IP address, which will change your decisions about whether to assign customers to a pool or not.

A better architecture
If all your production machines – smarthosts, webservers, databases – are hosted in a high quality datacenter run by an ISP with redundant connections to the Internet then you don’t need to worry about the redundancy yourself. If one of those connections is broken the ISP will route the traffic over a different connection to the same IP addresses mostly transparently. You won’t need to reconfigure anything, it’ll just keep working.

(Why don’t you just have redundant connections to servers hosted at your offices? First, it’s very expensive and time consuming to handle the mechanical aspects of ensuring that your two connections are really redundant, rather than being multiplexed onto the same fiber or running in the same conduit. Second, the smallest block of addresses you can multi-home in this way is 1000, and you can’t acquire those unless you’re already using more than 500 IP addresses efficiently.)

The weak point then is the connection between your offices and your datacenter that you need to administer the servers and provide customer support. But the IP addresses used for that don’t matter, so it’s easy and cheap to have a backup connection – even a cheap consumer cable or DSL connection. Or, if you’re a very small company, have your customer support folks use laptops and know which local coffeeshops and bars have free wifi.

Hello Kitty Backhoe is nearly as scary as Godzilla

I need multiple IP addresses per customer so as to manage filtering issues

Why this is right
If you have, for example, three dedicated IP addresses per customer and one of those IP addresses gets “randomly” blacklisted, then you can divert traffic to the other two IP addresses temporarily while you resolve the listing.

Why this is wrong
While there are many, many reasons why a source of email can be blocked there are far fewer that affect enough recipients that you really care. If a block affects one guy and one beagle in North Carolina, it’s not worth your operational concentration to care.

Of the blocks you might care about they’re almost all going to be based on recipient response, reputation and content. If one of those blocks affects an IP address you’re sending a mail stream from, it’s probably going to affect the other IP addresses you’re sending the same mail stream from really soon. So the right operational decision is almost always going to be to suspend mailing (either to one particular recipient ISP or to all recipients) until someone has had time to investigate the underlying issue.

Why else this is wrong
If your network engineering decisions are driven primarily by avoiding recipient email filters then you have a deeper business philosophy or customer vetting issue to consider.

Three heads don't help if they all do the same thing

I need multiple IP addresses per customer so that that customer can deliver mail in a timely manner

Why this is right

If your customer needs to deliver a message to 100,000 recipients within 30 minutes and 3% of them are at an ISP that only accepts 1000 emails per recipient per hour then you’re going to need at least 7 IP addresses dedicated to that delivery. See the previous post on throttling at ISPs.

Why this is wrong

Email is a store-and-forward protocol, more comparable to traditional postal mail than IM or SMS or fax. Delivery of the vast majority of a mailing list within thirty minutes is certainly possible, but relying on or obsessing about delivery of an entire list in that time period is unrealistic.

Why else this is wrong

A lot of spam is sent by botnets, networks of thousands of compromised machines that are used to send hundreds of thousands of copies of spam from a thousand different sources simultaneously. If you try and use a lot of source IP addresses to get a lot of copies of the same message delivered simultaneously then you’re likely to trigger anti-botnet measures. You need to have a history and a relationship with the receiving ISP to avoid  that – and if you have that, you shouldn’t need multiple IP addresses.

Yet another reason this is wrong

Delivery of 100% of an email list in a timeframe measured in minutes is not something you’re ever going to be able to guarantee without heroic measures. If your customer is prepared to pay for those heroic measures, you still won’t be able to get to 100% but you should talk to a consultant who might be able to help you get to 99% in those rare cases where it’s something that the recipients might care about. If they’re not prepared to pay for those heroic measures, why are you wasting your time?

Supersonic Delivery

I need many IP addresses so that I can work around ISP throttling limits

Why this is right: There are ISPs that limit the number of emails that can be sent from a particular IP address in a given time period to quite a low level, as low as 1000 emails per hour per IP address in some cases. If that’s a mid-sized ISP with perhaps a million users and an ESP customer is sending email to 5% of their user base (not unreasonable for some customers) then that would take more than two days to send a news letter to those recipients, assuming absolutely perfect management of send rate. With more realistic inefficiencies for send rate management it could easily be a week. Using multiple IP addresses to spread the traffic in that case seems perfectly reasonable, though it would make everyone involved happier if the ISPs used more reasonable rate limiting metrics, perhaps tied to sending IP address reputation rather than applied globally to all non-whitelisted senders.

Why this is wrong: Not every customer will be trying to send huge volumes of email to a receiving ISP that throttles inbound mail. For dedicated IP customers there’s no need to give them extra outbound IP addresses for this reason unless it actually becomes a real operational problem -  if it takes an hour or two for a  senders maildrop to reach the inbox at a particular ISP that’s not an operational problem. For pooled IP customers you may need to add IP addresses to deal with this, but only to the extent it’s needed to keep delivery times for pool customers to that recipient ISP reasonable. And two or three hours is not unreasonable.

Why else is this wrong: Naive throttling of this sort, combined with the obvious engineering changes needed for senders to work around it hurts everyone – senders, recipients, receiving ISPs. You might have to work around it in the short term, but in the longer term work with receiving ISPs to resolve the problem in a way that makes everyone happier, whether that be whitelisting your outbounds or moving to a reputation adaptive throttling approach. Of course, if they tell you that your mail is throttled because it has a poor reputation you need to fix that before doing anything else.

Tread light as a moth

I need lots of IP addresses so my MTAs can handle the volume of mail sent

Why this is right

One IP address per outbound smarthost is a sensible minimum. It is possible to set up multiple smarthosts behind a single IP address using a proxy server or reverse load balancer, and some organizations do that, but it makes it much harder to diagnose some sorts of operational problems. If the different smarthosts behind the proxy use different hostnames then the externally visible behaviour will be a single IP address HELOing as many different machines – which is behavior that is otherwise distinctive to spam sent from botets of infected machines, so will lead to mail being blocked. If, instead, they all use the same hostname then it’ll be hard to say which one of the cluster of smarthosts sent a message if there is a problem that needs to be diagnosed later.

So if you need 50 smarthosts to handle the volume of email you’re sending out, that’s really good justification for needing 50 IP addresses.

Why this is wrong

A typical smarthost can send an awful lot of email if it’s configured correctly. I tried to find some hard numbers on that, but  smarthost vendors don’t seem to publish their numbers much, and most of the trustworthy benchmarks I found published were from some years ago, when servers were a tenth of the speed they are now.

Do I trust all these numbers? Not necessarily, and they’re not directly comparable, but they all look quite plausible from a software engineers point of view. (It would be fascinating to get some performance numbers from vendors that are comparable, though. A smarthost-for-bulk-mail benchmark. Hmm….).

Edit: I’ve confirmed that Message Systems can exceed 1,500,000 deliveries an hour on a well provisioned server – and that’s while providing all the knobs and reports and monitoring that make managing delivery at any sort of volume much easier than a dumb smarthost.

But if you assume that you can send even 100,000 50k messages per hour from one smarthost, that’s more than enough mail to saturate a ten megabit connection, and to send seventy million emails / month – for each smarthost.

There are good reasons to have more smarthost capacity than you’re using, and to have more than one IP address per smarthost, but pure MTA capacity is hardly ever going to be justification for using more than a small handful of IP addresses to send mail from.

The monkey can't justify his building full of mailservers

I need at least one IP address per customer, to handle IP based reputation

Why this is right

While DKIM is gradually moving the main key for reputation tracking to a domain based token, right now the main key that is used to track reputation is the sending IP address.

If you have multiple customers sending mail of different quality using a different sending IP address for each of those customers means that the good customers will not be penalized by the poor behaviour of the bad customers. And, just as importantly, poor customers will not benefit from the behaviour of the good customers. This allows receivers to track sender reputation more accurately, and so delivery just wanted email to their recipients better. That makes everyone happy (other than the bad customers who deserve to be unhappy until they fix their practices).

Why this is wrong

Reputation is tied to sending IP address, but it’s also affected by volume of emails received from that IP address, and the consistency of volume. If a customer is only sending a few hundred emails a week to any given receiver ISP or they’re only mailing monthly then they won’t be able to maintain much of a positive reputation, simply because they’re too small to keep track of or because they mail so infrequently that each time they mail the receiving ISP will have forgotten about their previous mailings. In those cases the sender will be treated much the same as a new sender from a given IP address (neutral, at best, maybe poorly). For those cases a customer is likely to get better delivery rates if their mail is sent through an IP address pool that sends enough email overall to be noticed and tracked by receiving ISPs.

Another reason this is wrong

Reputation is tied to sending IP address, but receiving ISPs aren’t stupid and do recognize attempts to game the system. If you’re an ESP with a mix of good and bad customers then segregating the IP addresses they send from will not completely isolate the reputation of those customers from each other. The bad customers will drag your reputation as an ESP down more than the good customers will pull it up. And as your reputation as an ESP degrades it will pull down the reputation of your good customers much more than it will increase the reputation of your bad or unknown customers.

So segregating senders onto their own IP addresses doesn’t entirely separate their reputation from each other or from their ESP. And if you believe it does, you’re likely to make business decisions based on that misunderstanding that will badly affect your reputation and the delivery rates of your customers. Don’t fall into that trap.

Godzilla sneaks up on Tokyo

The ESP was talking about assigning a couple of dozen IP addresses to each customer, because they might be useful for spreading load and it would provide some flexibility for moving from one IP address to another if one should get blocked. And IP addresses are pretty much free. They were wrong.

The ISP was considering an application for 750 IP addresses from a new ESP customer. They assumed that there was no possible reason other than snowshoe spam for an email related customer to need that many IP addresses. While I suspect they may have been right about the specific potential customer, the general assumption was wrong.

I’ve seen a lot of reasons given by ESPs for why they need so many IP addresses:

1. I need at least one IP address per customer, to handle IP based reputation
2. I need many IP addresses so my MTAs can handle the volume of mail sent
3. I need many IP addresses so that I can work around ISP throttling limits
4. I need multiple IP addresses per customer so that that customer can deliver mail in a timely manner
5. I need multiple IP addresses per customer so as to manage filtering issues
6. I need multiple IP addresses in different locations to provide redundancy against network outages
7. I need multiple IP addresses in different locations so as to provide redundancy against blacklisting of my ISP

I’m planning on going through these in some future posts, pretty much like Godzilla goes through downtown Tokyo. Can you think of any I’ve missed? tell me in the comments!

Godzilla destroys Tokyo