I’d really like to know who leaked an email address, and when.

All my inbound mail is sorted into “spam” and “not-spam” by a combination of SpamAssassin, some static sieve rules and a learning spam filter in my mail client. That makes it fairly easy for me to look at my “recent spam”. That’s a huge amount of data, though, something like 40,000 pieces of spam a month.

Finding the needle of interesting data in that haystack is going to take some automation. As I’ve mentioned before you can do quite a lot of useful work with a mix of some little perl scripts and some commandline tools.

I’m interested in the first time a tagged address started receiving spam, so I start off with a perl script that will take a directory full of emails, one per file, find the ones that were sent to a tagged address and print out that address and the time I received the email. I can’t rely on the Date: header, as that’s under the control of the spammer, and often bogus. But I can rely on the timestamp my server adds when it receives the email – and it records that in the first Received: header in the message.

#!/usr/bin/perl
use strict;
use Date::Parse;
 
foreach my $file (@ARGV) {
    open IF, $file or die "Failed to open '$file': $!\n";
    my @headers;
    while() {
        s/[\r\n]//g;
        last if /^$/;
	push @headers, '' unless /^\s/;
	$headers[$#headers] .= $_;
    }
    my $date;
    my $timestamp;
    foreach my $header (@headers) {
	if($header =~ /^Received:.*;([^(]+)/) {
	    $date = $1;
	    $timestamp = str2time($1);
	    last;
	}
    }
    # Replace this regex with something that 
    # matches your tagged addresses
    if(join(' ', @headers) =~ 
                 /(foo\+[a-z0-9]+\@[a-z.-]+)/) {
	print "$timestamp $1 $date\n";
    }
}

Dates and times are annoying to work with on the command line, so I also use the perl Date::Parse module to convert the timestamp in the received header into epoch time – the number of seconds since January 1st, 1970. I use some unix commandline magic to run this against my two spam mailboxes and dump the results in a file.

find spamassassin/ | xargs stamp-address.pl >>junk.txt
find junk/ | xargs stamp-address.pl >> junk.txt

The end result is one line per email, with the epoch time, the tagged email address and the original format of the date and time. Something like this:

1300731078 cpan-tag@addr  Mon, 21 Mar 2011 11:11:18 -0700 
1300731122 vmware-tag@addr Mon, 21 Mar 2011 11:12:02 -0700 
1300731122 vmware-tag@addr Mon, 21 Mar 2011 11:12:02 -0700 
1300732902 unicorn-tag@addr Mon, 21 Mar 2011 11:41:42 -0700

Next, I want to find the first occurrence of each tagged address.

#!/usr/bin/perl
use strict;
 
my %seen;
while(<>) {
    chomp;
    my ($stamp, $address) = split / /;
    unless(exists $seen{$address}) {
	print "$_\n";
	$seen{$address} = 1;
    }
}

I sort the list of addresses numerically, then use this script to display the first time each email address received spam:

sort -n <junk.txt | findfirst.pl

That reduces the amount of data enough that I can look at it by hand. What did I find? Several interesting things, but I’m just going to mention one here.

1299111914 casemate-tag@addr Wed, 2 Mar 2011 16:25:14 -0800 
1307104954 dell-tag@addr Fri, 3 Jun 2011 05:42:34 -0700 
1307104986 codefast-tag@addr Fri, 3 Jun 2011 05:43:06 -0700

Casemate and Codefast have only ever mailed me via iContact, so given iContact’s history it seems likely that those leaks were via iContact.

Dell, on the other hand, have mailed me directly and through several ESPs – and I don’t recall them using iContact. Looking at the timestamps (and the content of the spams) it’s clear that the Dell and Codefast tagged addresses were both sent spam for the first time as part of the same spamrun – so it’s almost certain that they leaked at the same time.

Looking for iContacts bounce domain (icpbounce.com) in my mailbox I do find that Dell used them briefly, on May 4th. So that’s pretty compelling evidence that iContact leaked all three addresses. (Which means my previous theory about Dell customer addresses leaking, based on misleading statements from Intervision, was wrong.)

There’s another thing that’s interesting… iContact has had a history of email breaches. The data I have here (and it’s matched by a couple of older data points, if I recall correctly) shows spam being sent to newly leaked addresses on the 2nd or 3rd of the month.

I wonder if iContact does a batch export to a subcontractor, or an offsite backup or something similar on the first of each month?

For large volumes of mail where I might want to dig down in a lot of detail or generate graphical or statistical reports I tend to use Abacus to slurp in and analyze all the emails, store them in a SQL database in an easy to handle format and then do the ad-hoc work from a SQL commandline. For smaller work, though, you can get a long way with unix commandline tools and some basic perl scripting.

This morning I received Ukrainian bride spam to a tagged address that I’d only given to one vendor, RedEnvelope, so that address has leaked to criminal spammers from somewhere. Looking at a couple of RedEnvelope’s emails I see they’re sending from a number of sources, so I decided to dig a little deeper.

I started by searching for all emails to that tagged address in my mail client, then copied all the matching emails to a newly created folder. Then I took a copy of that folder and split it into one file per email using a shell one-liner:

formail -ds sh -c 'cat >msg.$FILENO' <mbox

I’m interested in the IP address they were sent from, so I write a tiny perl script, getips.pl, that’ll look for the first Received line, and print out the IP address:

#!/usr/bin/perl
 
foreach my $file (@ARGV) {
    open IF, $file or die;
    while(<IF>) {
        chomp;
        last if /^$/;
        if(/^Received:.*\[(\d+\.\d+\.\d+\.\d+)\]/) {
            print "$1\n";
            last;
        }
    }
}

I use it, along with the standard tools “sort” and “uniq” to summarize the sending IP addresses:

./getips.pl msg.* | sort | uniq -c | sort -nr

That takes the list of IP addresses generated by the perl script, then sorts them (so identical IP addresses are adjacent to each other), then counts how many times each discrete email address is found, then sorts them most to fewest. If you want to see how it does that, play around with the command line, removing commands off the end one-by-one to see the intermediate data it produces.

The result looks like this:

  27 208.49.63.243
  20 208.49.63.242
  20 208.49.63.240
  15 208.49.63.245
  15 208.49.63.241
  11 208.49.63.244
   3 38.107.108.146
   2 38.107.108.149
   2 209.112.253.83
   1 89.230.132.139
   1 38.107.108.144
   1 209.112.253.90
   1 209.112.253.85

The IP address beginning with 89 is the Ukrainian bride spam itself. Of the remainder it’s easy to see that they come from three main groups of addresses – 208.49.63.0/24, 38.107.108.0/24 and 209.112.253.0/24.

The 208 and 209 ranges are both IP space owned by RedEnvelope, so that was mail sent by them directly (both transactional and advertising mail). The 38 range is space owned by the ESP Cheetahmail. (I’ll go into how I worked all that out in a future post).

What does this mean? It means that either a russian bride spammer just happened to guess the email address I’d created solely to give to RedEnvelope (incredibly unlikely in this particular case, due to how I handle tagged addresses) or they stole it from somewhere. There are four places they could plausibly have stolen it from:

1. My mail client, by compromising my laptop
2. My mail server, by compromising the machine or the staff who run it
3. RedEnvelope, by compromising their servers, employees or employee machines
4. Cheetahmail, by compromising their servers, employees or employee machines

I run my own mailserver, so I know exactly which email addresses it handles. There are many hundreds or thousands of email addresses which it handles, and to which mail has been sent legitimately (as well as countless billions of addresses that it would accept email to if you made them up, but which have never been used). If it had been compromised in any way, I would expect many of those email addresses to be sent spam as part of this spam run (it’s not at all unusual for 40 or 50 of my email addresses to receive copies of any given spam). But only the RedEnvelope-specific email address received the Ukrainian bride spam.

Similarly for my laptop. It has hundreds of email addresses in it’s mailboxes. If it had been compromised, I’d have expected to see this spam sent to many of those email addresses, and I don’t. If someone had stolen multiple email addresses of mine, I’d expect them to be sending spam to all of them, unless they were doing something clever and deceptive like spear-phishing – and Ukrainian bride spam isn’t clever, subtle or targeted.

That leaves just RedEnvelope or CheetahMail as likely sources of the stolen address. Conveniently, Laura also has an account with RedEnvelope, and also uses a tagged address with them. She’s seen no spam at all to her RedEnvelope-specific address. Doing the same analysis with the legitimate RedEnvelope mail she’s received to that address I get this:

   2 209.112.253.90
   2 209.112.253.85
   2 208.49.63.242
   2 208.49.63.241
   1 209.112.253.83
   1 208.49.63.245

There are only three significant differences between Laura’s account and mine. Mine was created in June 2010, while hers was created in December. Mine has been emailed via CheetahMail, hers hasn’t. And mine received russian bride spam, hers didn’t.

One possibility is that RedEnvelope were compromised prior to December, so only my address was taken. But if that were the case I’d have expected to see that address misused before today. It’s possible, but not the most likely explanation.

More likely is that CheetahMail were compromised some time in the past few days, and the email address was stolen from there.

This isn’t conclusive proof by any means, but if I were RedEnvelope or CheetahMail I’d be looking very closely at other reports of stolen addresses, to see if there are patterns of theft from RedEnvelope lists sent across multiple ESPs or compromises of data from multiple CheetahMail customers.

Last time I worked with the actual HTML design of emails (a long time ago), <head> was not really needed. Is this still true for the most part? Any reason why you still want to include <head> + meta, title tags in emails nowadays?

There are several bits of information in the <head> part of an HTML document that can affect the rendering of it – there’s the doctype, which will control the html rendering model, there’s often some css which will control the styling, and there’s often a meta tag that states what character set is used in the document.

That last one is interesting in the case of a piece of HTML that’s being sent as part of a MIME email – as MIME already has a perfectly good way of specifying the character set a message has, as part of the Content-Type header. I looked at a few bulk messages I’d received recently and, sure enough, most of them include the <head> section, and have a meta tag in there that defines the character set. All of them have a character set defined in the Content-Type header. Sometimes those character sets didn’t match:

Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7Bit

<html>
<head>
<title></title>
<meta http-equiv=”Content-Type” content=”text/html; charset=windows-1252″>
<meta name=”title” content=”New CS5.5 Web Premium” />a snippet from this mornings email

What happens when they don’t match? I don’t think it’s defined anywhere. Time for some empirical testing.

Testing! For Science!

I needed to create some test emails which would be visibly different depending on which character set the mail client decided to use. I picked out two character sets – ISO-8859-15 and ISO-8859-16, as they differ from each other and from ISO-8859-1 enough that I could differentiate them just by the way two characters were rendered.

The byte 0xfd renders as e-with-a-tail (ę) in ISO-8859-16 and as y-acute (ý) in the other two character sets, while 0xa4 renders as the generic currency symbol (¤) in ISO-8859-1 and as a euro symbol (€) in the other two. I included the characters in two different ways in each test message – once as a raw character in the body of the message (=a4 or =fd in quoted-printable format), and once as a numeric HTML entity (&#164; or &#253;).

This is what I found:

There are several things to see from this data. The simple one first – regardless of which character set I declared, and where I declared it, both mail clients rendered characters written as HTML numeric entities (“&#164;”) consistently in ISO-8859-1. (This isn’t really a surprise, as it’s how the HTML specs define them.)

Raw characters were much less consistent. Mail.app consistently used the character set declared in the MIME Content-Type header when it was set to something reasonable, and ignored the encoding in the HTML meta tag. Giving it an unreasonable character set in the Content-Type header caused it to render 0xfd as a double dagger (‡), which makes no sense at all in any character set I can find. Gmail managed to render the raw character in ISO-8859-15 correctly, but gave up and fell back to using ISO-8859-1 for everything else.

Conclusions

There are a few things we can conclude from this, I think, even though it really needs some comparisons with different mail clients, and some testing with other character sets (including unicode and some of the asian sets).

1. Don’t bother with putting HTML meta content-type tags in your HTML
2. Send your text/html parts as plain 7 bit ascii, using HTML entities for non-ascii characters
3. It might be less confusing to use named entities such as &copy; rather than numeric ones such as &#169;
4. If you’re generating numeric entities from user-generated input, be wary of input that’s not ISO-8859-1 or Windows-1252
5. Character set conversion is hard, lets go unicode

I’ve made the test emails I used available for download. From a unix prompt, with swaks installed, you can send them like this:

for i in charset*.eml ; do swaks –to your@email.address –from your@email.address –server your.email.server –data – <$i; done

Two factor authentication that uses an uncopyable physical device (such as a cellphone or a security token) as a second factor mitigates most of these threats very effectively. Weaker two factor authentication using digital certificates is a little easier to misuse (as the user can share the certificate with others, or have it copied without them noticing) but still a lot better than a password.

Security problems solved, then?

Two-factor authentication isn’t our savior. It won’t defend against phishing. It’s not going to prevent identity theft. It’s not going to secure online accounts from fraudulent transactions. It solves the security problems we had 10 years ago, not the security problems we have today.Bruce Schneier, April 2005

Password stealing attacks are still a risk – especially use of the same password on different services – but they’re not the main thrust of modern attacks, and haven’t been for years. Rather we’re seeing man-in-the-middle attacks and trojan attacks – these can be used very effectively as part of a targeted attack initiated by phishing or social engineering.

One form of a man-in-the-middle attack is to create a fake website that looks like your real website, and then to entice one of your users to go to the fake website instead of the real one. Your user then enters their password and the second factor from their securid fob, and the attacker uses that to log in to your website. Done well, the user will never notice – the attacker either gives them a fake error message and redirects them to your real login page or tunnels their transactions through to your website while also piggybacking their own transactions at the same time.

A trojan attack is similar, but the man-in-the-middle is hostile code actually running on the users computer.

Not just a theoretical attack

This isn’t just a theoretical attack. It’s fairly widespread, and probably underreported. One example from a couple of years ago is use of a trojan to steal half a million dollars from a local company, despite their banks use of one-time-password, securid style two factor authentication. Here’s another.

The accounts an ESP is protecting likely aren’t worth half a million dollars, so maybe bank-grade two factor authentication is good enough for them?

Another heavy user of two factor authentication is the online game World of Warcraft. They use a physical security fob or a smartphone app to generate one time passwords.

As we’ve mentioned before there’s a black market in stolen World of Warcraft accounts. They’re typically worth $8-$10 in bulk. And they’re being targeted by a key-logging trojan that intercepts the authentication data and passes it to the attacker, who then can take control of the account until they log out.

That means it can be cost-effective for an attacker to use a reasonably sophisticated keylogger trojan to take control of an account worth $10 for a couple of hours, which is bad news if you’re relying on your customers accounts not being that high value a target.

What value does 2FA have, then?

it won’t work for remote authentication over the Internet. I predict that banks and other financial institutions will spend millions outfitting their users with two-factor authentication tokens. Early adopters of this technology may very well experience a significant drop in fraud for a while as attackers move to easier targets, but in the end there will be a negligible drop in the amount of fraud and identity theft.Bruce Schneier, 2005

2FA is a decent way to improve password security. It’s easier and cheaper to require some form of 2FA than it is to train your users to use good passwords, and not to reuse passwords. And they can be part of a decent security approach – though the inconvenience and support overhead might exceed their value. But focusing on 2FA as a security solution won’t protect you from most current attack vectors, and can distract you and consume resources you could better spend on more effective approaches.

By concentrating on authenticating the individual rather than authenticating the transaction, banks are forced to defend against criminal tactics rather than the crime itself.Bruce Schneier, 2005

But two factor authentication is a great way to deal with some non-security related business problems, such as sharing of “flat fee” accounts by multiple users.

Two factor authentication is not a magic bullet for ESP security, and if it distracts you from implementing more effective (behaviour-based, rather than authentication based) security approaches then that narrow focus risks making your overall security worse.

Unless, that is, you’re defending solely against security threats from 1995.

In computer security terms authentication is proving who you are – when you enter a username and a password to access your email account you’re authenticating yourself to the system using a password that only you know.

Authentication (“who you are”) is the most visible part of computer access control, but it’s usually combined with two other A’s – authorization (“what you are allowed to do”) and accounting (“who did what”) to form an access control system.

And what are the two factors?

Two factor authentication means using two independent sources of evidence to demonstrate who you are. The idea behind it is that it means an attacker need to steal two quite different bits of information, with different weaknesses and attack vectors, in order to gain access. This makes the attack scenario much more complex and difficult for an attacker to carry out.

It’s important that the different factors are independent – requiring two passwords doesn’t count as 2FA, as an attack that can get the first password can just as easily get the second password. Generally 2FA requires the user to demonstrate their identity via two out of three broad ways:

1. Something the user knows – a password or a PIN
2. Something the user has – a key, an ID card, a phone number, a digital certificate or a physical token
3. Something the user is – such as a fingerprint

An everyday example of 2FA is using a cash machine or ATM. You insert your ATM card (something you have) and enter your PIN (something you know) to get access to your bank account. An attacker would have to both steal or copy your card and know your PIN to access your account. While a crooked waiter might be able to copy your card and someone could look over your shoulder to see your PIN, it’s much more difficult for an attacker to get both.

Most deployed 2FA systems work in much the same way. They require you to enter a password you know, and then to demonstrate that you have something in your possession – by having your computer present a digital certificate, or having you enter a number from a security token like those pictured above, or respond to an SMS message.

Security problems solved, then?

I’ll look at that tomorrow.

(Spoiler: No)

Yesterday I got mail from Marriott, telling me that “unauthorized third party gained access to a number of Epsilon’s accounts including Marriott’s email list.”. Great! Lets start looking for spam to my Marriott tagged address, or for phishing targeted at Marriott customers.

I hit what looks like paydirt this morning. Plausible looking mail with Marriott branding, nothing specific to me other than name and (tagged) email address.

It’s time to play Real. Or. Phish?

1. Branding and spelling is all good. It’s using decent stock photos, and what looks like a real Marriott logo.

All very easy to fake, but if it’s a phish it’s pretty well done. Then again, phishes often steal real content and just change out the links.

Conclusion? Real. Maybe.

2. The mail wasn’t sent from marriott.com, or any domain related to it. Instead, it came from “Marriott@marriott-email.com”.

This is classic phish behaviour – using a lookalike domain such as “paypal-billing.com” or “aolsecurity.com” so as to look as though you’re associated with a company, yet to be able to use a domain name you have full control of, so as to be able to host websites, receive email, sign with DKIM, all that sort of thing.

Conclusion? Phish.

3. SPF pass

Given that the mail was sent “from” marriott-email.com, and not from marriott.com, this is pretty meaningless. But it did pass an SPF check.

Conclusion? Neutral.

4. DKIM fail

Authentication-Results: m.wordtothewise.com; dkim=fail (verification failed; insecure key) header.i=@marriott-email.com;

As the mail was sent “from” marriott-email.com it should have been possible for the owner of that domain (presumably the phisher) to sign it with DKIM. That they didn’t isn’t a good sign at all.

Conclusion? Phish.

5. Badly obfuscated headers

From: =?iso-8859-1?B?TWFycmlvdHQgUmV3YXJkcw==?= <Marriott@marriott-email.com>
Subject: =?iso-8859-1?B?WW91ciBBY2NvdW50IJYgVXAgdG8gJDEwMCBjb3Vwb24=?=

Base 64 encoding of headers is an old spammer trick used to make them more difficult for naive spam filters to handle. That doesn’t work well with more modern spam filters, but spammers and phishers still tend to do it so as to make it harder for abuse desks to read the content of phishes forwarded to them with complaints. There’s no legitimate reason to encode plain ascii fields in this way. Spamassassin didn’t like the message because of this.

Conclusion? Phish.

6. Well-crafted multipart/alternative mail, with valid, well-encoded (quoted-printable) plain text and html parts

Just like the branding and spelling, this is very well done for a phish. But again, it’s commonly something that’s stolen from legitimate email and modified slightly.

Conclusion? Real, probably.

7. Typical content links in the email

Most of the content links in the email are to things like “http://marriott-email.com/16433acf1layfousiaey2oniaaaaaalfqkc4qmz76deyaaaaa”, which is consistent with the from address, at least. This isn’t the sort of URL a real company website tends to use, but it’s not that unusual for click tracking software to do something like this.

Conclusion? Neutral

8. Atypical content links in the email

We also have other links:

• http://bp.specificclick.net?pixid=99015955
• http://ad.yieldmanager.com/pixel?id=550897&id=95457&id=102672&id=515007&t=2
• http://ad.doubleclick.net/activity;src=3286198;type=mari1;cat=rwdemls;ord=1; num=[Random Number]?
• http://ib.adnxs.com/seg?add=1519
• http://action.mathtag.com/mm//MARI//red?nm=rwdemls&s0=&s1=& s2=&v0=&v1=&ampv2=&ri=[Random Number]
• http://media.fastclick.net/w/tre?ad_id=26033;evt=13686;cat1=14501;cat2=14505
• http://images.bfi0.com/creative/spacer.gif

(Those “[Random Number]” bits aren’t me hiding things. That’s literally what is in the email.)

That’s an awful lot of other servers this mail is going to try and contact when you read it. I’m pretty sure that most of those are tracking links (but how many legitimate emails that advertise a single company and which are sent directly by that company, need to use half a dozen independent affiliate tracking links?).

Conclusion? Doesn’t look terribly honest. Maybe some sort of affiliate scam rather than a phish, though.

9. Most of the links in the email go to marriott-email.com, but then immediately redirect to marriott.com.

This shows someone is tracking clicks, which is pretty common for mail sent via ESPs, so as to make click tracking information available to the client without the client having to do any work to capture data on their website.

Conclusion? Real.

10. The unsubscription link goes to a terrible page with a set of checkboxes, rather than providing a simple unsubscription button.

Conclusion? Sadly, that’s a sign that it’s real.

11. Sending network configuration

It was sent from a machine with reverse DNS of dmailer0112.dmx1.bfi0.com, but which claimed to be called dmx1.bfi0.com, not a valid hostname for the IP address it came from.

This is pretty common misconfiguration of the network that happens at larger ESPs with complex outbound smarthost farms. I’d expect a phisher not to have that sort of mistake if they were sending from their own machine or through a botnet. And while “dmx1.bfi0.com” could be an obscure end-user DSL, the reverse DNS of dmailer0112 looks like it’s a system intended to send email, not a botnet.

Conclusion? Real.

Final Conclusion

You’ve probably guessed by now. It’s real email, sent on behalf of Marriott Rewards through one of their ESPs. But if it takes me several minutes of groveling through the mail before I convince myself it’s real, what chance does a typical consumer have of telling the difference between a well targeted phishing email and a typical piece of commercial email?

So I’ve put together a quick cheat sheet, showing the structure of four common types of email, and how their MIME structure looks.

Simple plain text

Plain text with a PDF attachment

HTML with a plain text fallback

HTML with embedded images and plain text fallback

ICANN allocated the last couple of blocks of general usage IPv4 addresses to APNIC earlier today.

There are just five usable blocks of addresses left, and they’re reserved by IANA policy for the final phase of IPv4 exhaustion, one for each RIR.

Like any other resource that’s been strip-mined to depletion that doesn’t mean that IP addresses are completely unavailable – there are still some in the delivery pipeline (the regional internet registries who handle allocation of addresses), ISPs have stockpiled addresses they don’t really need yet in anticipation of the exhaustion, and we can still recycle the addresses we’ve already got.

But there won’t be any new IPv4 addresses available.

What does that mean to you?

It’s going to affect your business in lots of ways over the next few years. And delaying thinking about it will just make it more expensive and more painful once you’re forced to pay attention.

IPv4 addresses are increasing in price

You’re going to find it increasingly difficult to get assigned new ones. It’s long past time to stop thinking of them as “effectively free”.

You’re going to need IPv6 production deployments fairly soon, and we’re well past the point of “it’s cheaper to wait, so that our vendors can do the IPv6 work that’s needed”. Delaying putting IPv6 prototypes into the field is going to get increasingly painful and expensive.

If your CTO and CFO are not concerned about this yet, they should be.

Stop wasting addresses

Dedicating even a single IP address to a customer is wasteful, if you don’t need that to handle the volume of email sent. You should be signing all your mail with DKIM today and building domain based reputation for your customers, as IPv4 based per-customer reputation is going away.

As you grow and gain more customers you’re going to have to start sharing v4 addresses between customers, which will share their IPv4-based reputation.

And fairly soon, you’ll want to start sending mail over IPv6 to some destinations. Odds are good that per-customer IPv6 reputation isn’t going to happen much. There’ll be some broad IPv6 based reputation to distinguish your outbounds from spammers and botnets, sure, but IPv6 based reputation down to a per-customer level isn’t something you’re likely to see much benefit from.

Assigning multiple IPv4 addresses to a single customer is ridiculously wasteful. You’re going to want to engineer away from any need to do that – and have your sales staff stop promising it – so that you can recycle those precious, precious IPv4 addresses for use elsewhere by other customers.

Your recipients are moving to IPv6

Big consumer access ISPs are some of the biggest consumers of IP addresses. They’re likely going to be moving to native IPv6 for end-users, combined with some sort of v6-to-v4 translation before anyone else.

That means that your web users, from home and mobile devices, will be trying to reach you via IPv6 sooner rather than later. Native v6 will mostly work better than v6-to-v4. So you want to be thinking about v6 access soon.

Have you got v6 space assigned from your ISP(s) yet? No? Ask them for it this week.

Do you have plans for making your image and click-tracking webservers v6-aware?

Everything you do with IPv4 you need to be able to do with IPv6

Does your address list management support tracking signups and confirmations from IPv6 users as well as IPv4?

Can you track opens and click throughs from IPv6 users?

How about reporting? Does your database and reporting engine support IPv6 addresses? Can you do geolocation for IPv6 addresses?

Does your smarthost vendor support preferentially routing mail via IPv6?

IPv6 is an opportunity

What would you do if someone were to offer you dedicated MX machines at Yahoo, Google and Hotmail. Machines that were much the same as their primary MXes, but lightly loaded with much more available capacity. How much would you be prepared to pay for access to them?

Sooner or later there’ll be IPv6-only MXes at those, and other ISPs. Will you be ready to use them?

Do your competitors offer an IPv6-ready system yet? How much of a business disadvantage will you be at once they do?

And finally

None of this is new. You’ve known for years that you need to be more frugal with IPv4 addresses and have dual-stack IPv6-capable services.

But it’s getting urgent.

Since then I’ve come across another issue with click tracking links that’s not terribly obvious, and that you’re not that likely to come across, but if you do get hit by it could be very painful – phishing and malware filters in web browsers.

First, some background about how a lot of malware is distributed, what’s known as “drive-by malware”. This is where the hostile code infects the victims machine without them taking any action to download and run it, rather they just visit a hostile website and that website silently infects their computer.

The malware authors get people to visit the hostile website in quite a few different ways – email spam, blog comment spam, web forum spam, banner ads purchased on legitimate websites and compromised legitimate websites, amongst others.

That last one, compromised legitimate websites, is the type we’re interested in. The sites compromised aren’t usually a single, high-profile website. Rather, they tend to be a whole bunch of websites that are running some vulnerable web application – if there’s a security flaw in, for example, WordPress blog software then a malware author can compromise thousands of little blog sites, and embed malware code in each of them. Anyone visiting any of those sites risks being infected, and becoming part of a botnet.

Because the vulnerable websites are all compromised mechanically in the same way, the URLs of the infected pages tend to look much the same, just with different hostnames – http://example.com/foo/bar/baz.html, http://www.somewhereelse.invalid/foo/bar/baz.html and http://a.net/foo/bar/baz.html – and they serve up just the same malware (or, just as often, redirect the user to a site in russia or china that serves up the malware that infects their machine).

A malware filter operator might receive a report about http://example.com/foo/bar/baz.html and decide that it was infected with malware, adding example.com to a blacklist. A smart filter operator might decide that this might be just one example of a widespread compromise, and go looking for the same malware elsewhere. If it goes to http//a.net/foo/bar/baz.html and finds the exact same content, it’ll know that that’s another instance of the infection, and add a.net to the blacklist.

What does this have to do with clickthrough links?

Well, an obvious way to implement clickthrough links is to use a custom hostname for each customer (“click.customer.com“), and have all those pointing at a single clickthrough webserver. It’s tedious to setup the webserver to respond to each hostname as you add a new customer, though, so you decide to have the webserver ignore the hostname. That’ll work fine – if you have customer1 using a clickthrough link like http://click.customer1.com/123/456/789.html you’d have the webserver ignore “click.customer1.com” and just read the information it needs from “123/456/789.html” and send the redirect.

But that means that if you also have customer2, using the hostname click.customer2.com, then the URL http://click.customer2.com/123/456/789.html it will redirect to customer1′s content.

If a malware filter decides that http://click.customer1.com/123/456/789.html redirects to a phishing site or a malware download – either due to a false report, or due to the customers page actually being infected – then they’ll add click.customer1.com to their blacklist, meaning no http://click.customer1.com/ URLs will work. So far, this isn’t a big problem.

But if they then go and check http://click.customer2.com/123/456/789.html and find the same redirect, they’ll blacklist click.customer2.com, and so on for all the clickthrough hostnames of yours they know about. That’ll cause any click on any URL in any email a lot of your customers send out to go to a “This site may harm your computer!” warning – which will end up a nightmare even if you spot the problem and get the filter operators to remove all those hostnames from the blacklist within a few hours or a day.

Don’t let this happen to you. Make sure your clickthrough webserver pays attention to the hostname as well as the path of the URL.

Use different hostnames for different customers clickthrough links. And if you pick a link from mail sent by Customer A, and change the hostname of that link to the clickthrough hostname of Customer B, then that link should fail with an error rather than displaying Customer A’s content.

Redirection links are simple in concept – you include a link that points to your webserver in email that you send out, then when recipients click on it they end up at your webserver. Instead of displaying a page, though, your webserver sends what’s called a “302 redirect” to send the recipients web browser on to the real destination. How does your webserver know where to redirect to? There are several different ways, with different tradeoffs:

The simplest approach

The simplest sort of redirection link includes the final destination in the link itself – something like http://click.example.com/cnn.com/WORLD/. The webserver at click.example.com would simply strip off the first part of the link, and redirect to the remainder – cnn.com/WORLD/.

This is nice, because it’s fairly transparent to the recipient – when they hover over the link in their mail client or webmail it’ll be fairly clear where it’s going.

But it has several limitations. One is that you can’t really record very much data about the click – you know where it was redirecting to, but almost nothing else.

The bigger problem is that it’s very easy for a spammer to abuse – they can send out spam that has the link http://click.example.com/onlinepharmacy.ru/order.html, to hide their real link from spam filters, and your webserver will happily redirect recipients to go there. Or, worse, that can be used to redirect to a website hosting viruses. That can cause all sorts of problems for your reputation, up to and including having your redirection webserver blacklisted by antivirus and antiphishing organizations, meaning it’ll be blocked by many web browsers.

Add some metadata

Some of the things you might want to be able to record about a click would be which customers mail it was found in, which mailing campaign and which recipient it was sent to. This would let you do more sensible reporting and click-tracking, and also let you spot when a link is misused in some way (for example, thousands of clicks on a url that was sent to just one recipient).

That might look like http://click.example.com/123/456/789/cnn.com/WORLD/. Your webserver would strip off the first four parts, recording a click for customer 123, campaign 456 and recipient 789, then redirect to the remainder – cnn.com/WORLD/

This lets you do better reporting and is still fairly transparent to the recipient, but can still be abused in the same way.

Use a database

If you stored every link you wanted to redirect to in  a database you could simply store a unique key for each link – so you might record that key 2718 means http://cnn.com/WORLD/. Then the redirection URL might look like http://click.example.com/123/456/789/2718

This lets you do good reporting and is much more difficult for spammers to abuse (but not impossible – if the spammer signs up for a free or demo account on your system, then sends a test email to themselves, they can then reuse the links that they received in that mail).

But it’s fairly opaque to the recipient – they have no idea where the link will go. And it requires maintaining a database of every link you’ve ever used, for as long as it’s valuable (which could easily be several years if a recipient goes back to an old newsletter) and requires a database lookup for every click – which adds a fair bit of infrastructure you need to keep working 24/7 just to make links work.

Use a database and a cosmetic link

You could take the database format and add the final destination link on the end – like this http://click.example.com/123/456/789/2718/cnn.com/WORLD/ – and then just ignore everything after the url key (2718). That’ll work exactly the same way, but the final destination will be fairly transparent to the recipient.

This still can’t be abused by spammers, as if they try to use http://click.example.com/123/456/789/2718/mypharmacy.ru, it’ll still just redirect to http://cnn.com/WORLD/ as the only meaningful bit of the redirection link is the “2718“.

Cryptographically sign your links

A different approach is to record all the information you need in the link and to also add a cryptographic signature to prevent people from misusing it. This is much simpler than the word “cryptography” suggests, you just need to use a magic word (we’ll use “albatross”) and know about the md5() function.

You start off with the same destination string we used in Add some metadata – “/123/456/789/cnn.com/WORLD/“. Then you add the magic word on the end, to give “/123/456/789/cnn.com/WORLD/albatross“, and take the md5 “hash” of that. That’s some cryptographic black magic that’ll give you a string of letters and numbers that’s a “fingerprint” of that string. It’ll look something like “609a78b941bdf9f045cadcfa2e09d54c“. Then you combine that with the destination string to look like this:

http://click.example.com/609a78b941bdf9f045cadcfa2e09d54c/123/456/789/cnn.com/WORLD/

Then, when your webserver sees this link it splits it into the hash (609a78b941bdf9f045cadcfa2e09d54c) and destination string (/123/456/789/cnn.com/WORLD/). It then does exactly the same thing you did when you created the link – appends the magic word to the destination string to give “/123/456/789/cnn.com/WORLD/albatross” and takes the md5 hash of that string. If the result of that matches the hash in the link, it knows it’s a valid redirection link and it can record the click-tracking data and forward to the destination link. If the result doesn’t match it knows that the link has been tampered with, and can return an error page.

To generate the link in PHP would be something like this:

$destination = "/$customerid/$campaignid/$recipientid/$link";
$clicktrack = 'http://click.example.com/' . md5($destination . 'albatross') . $destination;

This is much cheaper to generate and validate than using a database, even a typical in-memory database.

Which to use?

Don’t use the simple approach – it’ll get abuse sooner or later and you’ll regret it. Any of the database or cryptographic approaches work just fine, though the cryptographic approach may be easier to scale up and maintain. The database approaches make it easier to disable a link, or direct it to somewhere else at a later point, in case of abuse or some other need.

What else is it good for?

You can use the same sort of approach to validate unsubscription links and VERP return paths for bounce handling. And “open tracking” using these sort of links for image URLs, if you find that a useful metric to offer.