This is what a valid CBL query looks like:

• “14.23.177.10.cbl.abuseat.org”

It’s just the IP address being queried (10.177.23.14) with the numbers reversed, with “.cbl.abuseat.org” added on the end. Not rocket science.

Here’s a tiny sample of some of the invalid queries:

• “70.46.6.10.abuseat.org”
• “202.204.219.10cbl.abuseat.org”
• “252.94.193.10.ns1-cbl.abuseat.org”
• “255.190.244.10 cbl.abuseat.org”
• “166.193.222.10#cbl.abuseat.org”
• “214.6.224.10.*@cbl.abuseat.org”
• “212.9.185.10.http://cbl.abuseat.org”
• “76.207.80.10.bl.abuseat.org”
• “185.124.73.10.cbb.abuseat.org”
• “201.54.179.10.cbl-xbl.abuseat.org”
• “54.191.254.10.opm.abuseat.org”
• “181.4.133.10.sbl-xbl.abuseat.org”
• “176.33.165.10.cbl.abuseat.orgcbl.abuseat.org”
• “101.126.133.10.cbl.abuseat.org:Mail from %IP% refused by blackhole site cbl.abuseat.org”

Those are just 15 of about 1800 different misconfigurations I have on file, just for queries to the CBL. I’ve seen similar things at other domains I host, and I’ve heard of just the same sort of thing from other people who own domains that are similar in some way to a domain used by a blacklist. It’s not unusual.

What happens when someone misconfigures a blacklist lookup in this way? Because of the way DNS based blacklists work the response to any of these invalid queries will be “no, that IP address isn’t listed”. So all these people are attempting to use the CBL to filter out spam and haven’t noticed that it’s never actually stopped any email. And all the time they’re doing this, they’re hammering my DNS servers (and many other peoples) with millions of pointless queries every day.

What can the DNS server operators do about that? Because of the way DNS works, blocking the broken queries will actually increase the amount of traffic they have to deal with by several times. Contacting all the people making the queries and pointing out the problem would be a huge task, and even when I have tracked down contact information and notified people by email I’ve never had a response and the problem has never been fixed.

So the only remaining option is to make the misconfiguration more obvious to the user – by responding to the invalid queries with “yes, that IP address is listed” and hoping that causing them to reject all the mail sent to their users will encourage them to fix their configuration. I check my nameserver statistics every so often and add “poison” entries for the more obvious misconfigurations I find. I did that for a bunch of misconfigurations manually yesterday, which will probably cause a lot of domains to reject a bunch of email they didn’t want to this morning.

There are fairly simple ways to make sure you’re querying a real blacklist – pretty much all of the legitimate blacklists include the IP address “127.0.0.2″ as a test entry. You can use that to check that a blacklist is live manually – if the blacklist domain is sbl.spamhaus.org then a dns lookup for “2.0.0.127.sbl.spamhaus.org” should return an answer (typically 127.0.0.2) while a dns lookup for “1.0.0.127.sbl.spamhaus.org” should return “not found” / “NXDOMAIN”. If either of those tests fails, the blacklist is broken in some way, and you shouldn’t use it.

The choice of 127.0.0.2 for the test entry wasn’t arbitrary: 127.0.0.2 is a “local” address that’s always available on machine, though it’s usually never used for anything. But you can use it – if you open a commandline on your mailserver you can run an SMTP transaction by hand (as I discussed yesterday) from 127.0.0.2 using “telnet -b 127.0.0.2 your.hostname 25″ (on Linux-ish systems, anyway – some other telnets use “-s” instead of “-b”). That way you can see whether you’re really rejecting based on a blacklists, and what error you’re giving. (It would be nice if every blacklist also had another test entry in 127.* as well as 127.0.0.2, so you could check them individually, but they don’t. Hint to blacklist operators.).

It’s very easy for spam filter authors to check those test entries once a day for each of the blacklists they were configured to use, and to disable the ones that failed. If you’re a postmaster who uses blacklists as part of your spam filter (and you probably should) you should check with the people who provide the filter whether it makes those checks – and if it doesn’t, ask them to add them. That will protect you from misconfigurations, blacklists being shut down, blacklists being abandoned and bought up by domain squatters and all sorts of other things that can cause you to lose a lot of mail.

When you’re running a transaction by hand you’re doing everything your mailserver would do to deliver an email, but you’re doing it yourself. That means that you get to see all the responses from the mailserver you’re sending the mail to, and also any delays or errors in much more detail than you can usually get from mailserver delivery logs.

I want to send some email to playingwithtelnet@gmail.com. There are two main steps to doing this – first I need to find out which mailserver I need to talk to to send mail to gmail, then I need to actually send the mail.

To find the mailserver I have to look up the MX record for gmail.com. From a unix / linux / mac command prompt you can do that like this (the bits you type are in orange):

platter:~ steve$ host -t mx gmail.com
gmail.com mail is handled by 5 gmail-smtp-in.l.google.com.
gmail.com mail is handled by 30 alt3.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 10 alt1.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 40 alt4.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 20 alt2.gmail-smtp-in.l.google.com.

Another option is to use a web-based dns lookup site such as Al Iverson’s xnnd.com or our emailstuff.org.

However you do the DNS query you’ll usually end up with one or more hostnames, each with an associated priority. When you’re sending email you’re supposed to try those servers in order of priority, lowest number first, so in this case we’re going to be talking to the server gmail-smtp.in.l.google.com.

If you’re trying to replicate a delivery failure that you’ve seen then you can skip this step and go straight to trying to deliver to the same mailserver you saw fail previously (though double checking that that is a real MX for the domain is a good idea).

You may sometimes not get any answers to the MX record lookup – if, and only if, that happens then you should send mail directly to the domain part of the email address (“gmail.com” in this example).

Now we know which mailserver we’re going to talk to, so we connect to it with telnet on port 25. You can do this from a unix / linux / mac command line, or from a Windows command prompt (though a Windows firewall may be configured to block you from doing this). Again, the bits you type are in orange, while the responses from the server are in blue

platter:~ steve$ telnet gmail-smtp-in.l.google.com 25
Trying 74.125.155.27...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP y27si6889009wfi.2
HELO platter.wordtothewise.com
250 mx.google.com at your service
MAIL FROM:<steve@blighty.com>
250 2.1.0 OK y27si6889009wfi.2
RCPT TO:<playingwithtelnet@gmail.com>
250 2.1.5 OK y27si6889009wfi.2
DATA
354  Go ahead y27si6889009wfi.2
From: <steve@blighty.com>
To: <playingwithtelnet@gmail.com>
Subject: Just a test email

The body of the mail goes here.
.
250 2.0.0 OK 1276638161 y27si6889009wfi.2
QUIT
221 2.0.0 closing connection y27si6889009wfi.2
Connection closed by foreign host.

A simple mail delivery like this goes through a number of steps. First you telnet to the server on port 25, wait for that to connect, then wait for the server to send you it’s “banner” (“220 mx.google.com ESMTP …”). Then you tell the server who you are with the “HELO” command, and wait for a response.

Then you tell it the email address you’re sending mail from with the “MAIL FROM” command – note that there’s a colon after MAIL FROM, and then your bare email address surrounded by angle brackets – and wait for a response. Next you tell it who you want to send mail to, with the RCPT TO command – again followed by a colon and the recipients bare email address surrounded by angle brackets.

(You can send mail to multiple recipients by repeating the RCPT TO command here).

Then you’re ready to send the email itself, so you send the “DATA” command to tell the server you’re about to do that, and wait for it to respond with something like “354 Go ahead”. Then you enter the email you want to send, consisting of the headers, a blank line, then the body of the message. To tell the server you’re done, send a period (“.”) on it’s own as the final line. You’ll typically want to have at least To:, From: and Subject: headers and a short message body for even a simple test, but if you’re considering content-related filtering then you’ll want to paste in a copy of a real email.

Once the server has accepted the email you send the command “QUIT” to tell it you’re done, and it will close the connection.

You can see that each response from the server starts with a three digit number. If that starts with a “2″ it means that it’s happy with what you just told it, if it starts with a “3″ it’s waiting for you to send the text of the email. If it starts with a “4″ or a “5″ it means it’s unhappy with what you just told it – maybe you typoed something, maybe it’s rejecting the mail delivery, maybe you tried commands in the wrong order – and the text after the number should give you some idea as to why.

If you typo something while you’re using telnet you can use backspace to delete and correct it while you’re on the same line. Once you’ve hit return, though, it’s already sent to the server and you can’t change it (and you might need to quite out and start over).

If you get confused at any point and need to quit out of telnet and start over you can use Ctrl-] (hold the ctrl key and press the close square bracket key) to get to a telnet> prompt, then type “quit”.

There’s some more sophisticated things you can do, but they’ll have to wait for another post.

Meanwhile, feel free to use playingwithtelnet@gmail.com to play with running an email transaction by hand.

Barry: I see your “too many recipients” and raise you a “DNS failure.”

Me: You’re joking.

Barry: “Unknown address error (’550′, ['REQUESTED ACTION NOT TAKEN: DNS FAILURE'])

Me: That seems pretty self explanatory. I would close the ticket with a “not a mail issue.”

Barry: It wasn’t a ticket. It was a direct mail to me by a very well known person on the sender side. You’d die if you knew who it was. But he didn’t send me anything useful, not even an IP address.

Me: You’re kidding? Please tell me you’re kidding. Please.

This is yet another example of people bothering Barry with questions that should be answerable by anyone who holds themselves up as a delivery expert. Really.

Barry is not your free consultant. Barry has a job and it does not involve troubleshooting problems on your end. Asking questions about stupid stuff like “too many recipients this session” or “DNS failure” is why most Barry’s don’t hand out their info to senders. They don’t want to be bothered with questions just because the sender is too stupid or lazy to do their own troubleshooting.

There are two things that come to mind immediately when I see this error message and two things that I would check before even considering contacting someone.

1. This is an internal DNS failure and the MX lookup on the sender’s side failed. The sender should do a manual DNS lookup and confirm they can get a MX record (or A) record for the recipient domain.
2. This is a DNS failure on the receivers side. A little harder to troubleshoot, but some ISPs check the DNS of the sending domain before accepting mail. Make sure that the domain exists in DNS and is answering queries promptly.

Once you have checked DNS and everything is OK you can move to the next step. Open up a telnet session to the mail server and do a manual SMTP session. Use the same Mail From: and Rcpt To: that generated the 550 you’re attempting to troubleshoot. You don’t need to do the whole session, just through Mail From: and Rcpt To:.

If the Mail From and Rcpt To: addresses are accepted by the receiver mail server, then go back into your MTA and resend the message that originally failed.

It works, you’re done. If not, go back and think about what else might cause a DNS failure, then test it. Same as you did above. Repeat.

EDIT: While writing the post, I heard back from Barry. The problem was that the sending domain did not exist in DNS. This issue would have been identified at the 2nd DNS check. No mail to Barry needed.