Getting your e-mail? Anyone getting yours?

I get email services from a large and respected company, Everyone.net. This week SpamCop has been blocking their servers, and the tens of thousands of domains they represent. As a result, some of my e-mail to you from my domains has not been getting through, and you may not be getting valid e-mail from other trusted sources either.

To start with a bit of background, SpamCop is one of the popular Block List services. I call them “agencies” because some of them are more like “organizations”, groups of interested parties rather than “companies”. You’ll often see the initials RBL, DNSBL and others to identify these agencies. E-mail server software relies on one or more of these services to determine what is spam from what is legitimate. The services use databases to check the source server and domain, and (hopefully) intelligent software to check headers and content to make their determination. They assign every e-mail a score which you may or may not see in headers when you finally get your mail. The higher the score, the more “spammy” an item looks. The more spammy it seems the less chance it has of getting through to the intended recipient.

E-mail servers usually rely on several block lists to form a consensus of just how bad the source of a mail item might be. Some servers make their determinations based on one source – like SpamCop.

But SpamCop uses a human voting system to decide what looks suspicious. Here is how that works:
A Recipient1 gets an e-mail from an Originator1. Recipient1 clicks a button in their e-mail client to indicate it’s OK or bad. Now Originator1 sends a note to another Recipient2. SpamCop sees that Recipient1 flagged mail as spam, so the mail is bounced and Recipient2 is spared the burden of processing the spam.

SpamCop has been performing this valued service for many years.

However, what if Recipient1 just hit the wrong button? Now Recipient2 doesn’t get something they may be waiting for from Originator1 because someone they don’t know made a bad decision. To make this worse, it’s not just Originator1 that’s affected. Originator2 happens to be using the same service as Originator1. These people don’t know one another. But mail from Originator2 to some Recipient3 now gets bounced because someone else who is completely unrelated made a bad decision.

It’s possible that Originator1 was indeed a spammer and Recipient1 made a good decision that should spare others. Unfortunately SpamCop doesn’t just censor Originator1, they censor the server from which Originator1 sent their message. That server could support tens of thousands of domains, representing millions of individuals. Because some company sold e-mail services to someone who abused their Terms Of Service, everyone else associated with that company now suffers.

It’s time now to break and to think of a science fiction movie. A medical robot is supposed to scan for viruses and destroy them, helping to create a society of people who live in perfect health for hundreds of years. But it gets a little over-zealous and determines that human beings are just big bags of viruses, or perhaps it starts using other criteria to determine infection. So it goes on a rampage to destroy the infestation we call humanity. We’ve all seen this plot many times. Well, here we are. It’s reality on a different scale.

E-mail service companies in this position do their best to immediately verify reports and terminate accounts of anyone who has actually abused the services. Obviously it’s in their best interest to do so, and this is the reason why SpamCop and other block list services do what they do. If someone is hosting a server and they actually do support spammers, it’s to everyone’s advantage that the entire service be shutdown, rather than just one Originator or domain. Legitimate hosts don’t want to be associated with that riff-raff, so they really do try to keep their servers free from the occasional bad guy.

But what happens when an e-mail host does what they should: they eliminate a spammer, or determined that someone incorrectly flagged valid mail, and then they can’t get through to SpamCop or one of the other block list agencies? Until they are removed from the block list, everyone is suffering. This is what’s happening to me and lots of other people this week.

Let’s go back to sci-fi plots for a moment. Two countries with their finger on the button for nuclear war decide that this isn’t what they really want to do. They both agree to disarm. But wait! A communications fault occurred and the signal to disarm didn’t get through, or maybe that robot cut the critical Disarm wire. Uh oh, it looks like they’re going to have nukes dropping after all. Hey, it’s corny, but this is exactly what’s happening.

Why don’t I just move to another e-mail company? Sure, I could get huffy and threaten to take my business elsewhere. But this situation happens a lot. Every e-mail server host gets wrongly accused at one time or another, and many of the block list agencies at some time or another become very unresponsive. I’ve had services through a few companies and every one of them was hit like this. Each one of them claims to have a good rapport with the various BLs, to get by issues before anyone notices, or to prevent issues before they occur. Everyone.net does a very good job of this. Moving to another company isn’t going to get me anything better than what I have. My choice of an e-mail provider isn’t about whether or not problems like these are going to occur, it’s about how well a company is going to handle situations like this when they occur. I trust that this company represents my interests when things like this happen. That’s really all I ask.

So, while spammers (black hats) clutter our servers and e-mail bins with their trash, the block list people (white hats) can be just as responsible for sabotaging our communications infrastructure. They don’t do it intentionally – usually. Some years ago one of these services decided that they were not going to filter forum postings anymore, just so that people would have a better appreciation for the valuable service that they provide. This essentially resulting in a denial of service (DoS) attack on a large number of servers, effectively shutting down the communications of many companies and individuals.

The people who provide these services often do so for free to consumers. They are funded by various companies who benefit from the services they provide, often large e-mail hosts who understand there are costs associated with improving the quality of their services. Unfortunately we have become so dependent on the BL services that they can hold us hostage and at their mercy. A temper tantrum, an honest mistake, or a server breakdown at any one of these agencies can cripple services for millions of people. This has happened many many times over the last 15+ years, and it will certainly happen again.

What’s the solution?

As far as e-mail goes, we’re all dependent on a protocol that’s around 30 years old. E-mail was never designed to be used and abused as it has over the last 15 years or so. I don’t think the originators of the e-mail protocols had a thought in their minds about the sort of warfare that we’re talking about here. Rather than fix the problems, people would rather endure the cost of anti-spam and anti-virus protection, and slow servers and all of these other things. It costs over $20Billion per year to deal with spam and malware – that includes the degradation of our networks which often transport more spam than legitimate traffic. How much would it cost for servers and clients to switch to new software based on new protocols which aren’t subject to these problems?

One dynamic in play here is that people are actually using e-mail much less these days than they did in the past, at least for personal use. Personal users tend to post messages on social networks or via SMS/texting. Business users are relying more on IM/chat (Yahoo, MSN, ICQ), Skype, and SMS/texting. I’m not saying this is really a solution, just that the problem seems to be a little less severe per-capita than it might have been if we were all as reliant on e-mail as were about 5 years ago.

Another part of the problem is that we’re allowing the lowest-common-denominator of end-user to determine how the masses are going to communicate. What’s missing from the entire process is an evaluation of not just the Originator, but of the Recipient who may chronically vote all e-mail as spam.

Some people don’t know the difference between mail that they don’t like, perhaps from known sources, and Unsolicited Commercial E-mail (UCE) from unknown sources which deserve aggressive attention. Letting these people decide the nature of spam is like letting people who can’t drive get behind the wheel of a car – everyone else on the road is affected. If I could change this system, I would include the rank of the Recipient in the spam calculation. Someone who chronically flags good mail as bad will be less reliable than someone who gets it right all the time. The more they get it wrong, the more spam they’ll get, the less they will affect others, and the more inclined they will be to get it right as they go forward. Of course they need to be informed that their negative score is affecting their volume of mail. Call it draconian, but people need to be educated so that they can make better decisions – or we shouldn’t be allowing them to make decisions on behalf of the rest of us. (This could SO turn into a discussion of voting for political office, but no, I won’t go there.)

So what’s the bottom line? Well, now I hope you know more about how block lists work, and how they don’t. Perhaps you can see why you get more spam than someone else, or maybe more spam at some times than others. Unfortunately, no we’re not going to see a mass change of the SMTP/POP protocols. And no, we can’t stop people from flagging undesirable or even valid email as spam. But I think the better we understand these things the easier it is to deal with them, and to understand why that next best thing may or may not be better than what we have now.

Errrrrr, for the real bottom line – if you do rely on SpamCop, and we do determine that we have these problems – or if you also use Everyone.net, this solution may help for now: On your e-mail server, in SpamCop or another high-level filter, please set a white-list entry for the following IP addresses:

216.200.145.35
216.200.145.36
216.200.145.37
216.200.145.38

Those are the addresses from which our e-mail might come. If you define those as accepted addresses, your server “should” approve mail from my servers before SpamCop bounces it.

What about mail from someone else? Well, if you can whitelist my domain, sure, go for it. But if you whitelist all of the IP addresses, remember that Everyone.net does a good job of keeping spam and spammers out, so chances re slim that you’ll get anything bad from them. Or rather, chances are a lot better that mail through them is good than mail coming from most other servers out there.