[CLUE-Tech] If you administer a mail server, you might find this useful.

[William]

I do not disagree with your exploration of the concerns that mail administrators face.  The path
your prose follows seems to work its way toward a maturity that plateaus in the one main question:
 What do we ultimately do with spam?  I would like to make a few observations:

1. I agree that to make spamming inneffective (unprofitable) is key to stopping spammers.  The
first problem is that new spammers jump into the forray all the time, and they all seem to want to
learn the hard way.  If we can shut down one spammer, there will be more to replace them.  So,
whatever mechanism we find effective will have to stay in place for a while.

2. The second problem is that big spammers have other ways to make money, so simply not responding
to their messages isn't enough to stop them.  Here's an unfortunate story:

An acquaintance of mine became an inadvertant spammer when she payed for a marketer to help
advertise her business.  As a customer, she had no idea that she was spamming when all she did was
attach a piece of marketing software to her Act! customer database (not too bright, I agree).  The
software promised to help her maintain contact with her clients, and drive up new customers.  It
turned out the software was just a front-end for an SMTP engine that came with a huge database of
"purchased" e-mail addresses that just incorporated her database.  I helped shut her down quickly
when she called me for help when her ISP e-mail stopped working.  The unfortunate reality of it is
that the software probably uploaded her entire Act! database of e-mail addresses and names to the
software firm.  The point is:  You have to watch out for the possibility that spammers AREN'T just
making money off sending their bulk themselves.  They have other revenue systems, as well. 
Additionally, there are unfortunate suckers out there sending spam without realizing it, for
example, the myraid zombies and open relays out there.

3. Filtering by URL is one of many techniques already employed today.  However, this technique has
a very, very short life-span per URL.  Spammers today employ "disposable domains" like
http://asdfhhi.wnfivi.com/, making a black-list of URLs effective only for short periods of time.

4. IP based black-listing, as employed by people like me (in the popauth3 code), is set up such
that the black-list expires after a certain period of inactivity (from minutes to months,
depending on the situation).  So, while the idea that multiple non-spammers may share an IP with
an identified spammer is legitimate, it is not a long-term concern.  The IP black-list will expire
when the spammer stops using it to attack servers participating in the black-list.

5. Interesting notes on content-based filtering:
A. This is where the highest percentage of false-positives originate.
B. Spammers, as of late, have shot themselves in the foot by attempting to circumvent
content-based filtering by using permutations of words like viagra (i.e.: v1 at gr@, v1agra,
v.iag.ra, etc.).  I have found it is simple and very effective to ban such permutations while
allowing the actual undoctored word, viagra, to pass through.  I'm sure you know that "viagra" is
but one of many keywords that spammers are trying to pitch today, so my filters have become
stronger with the more different techniques spammers are attempting.

Note also:  Content-based filtering is *not* part of popauth3.

Spam senders are an affront to legitimate e-mail handlers, regardless where the spam is coming
from.  It is my opinion that the absolute most effective way to stop a spammer is to prevent them
from even seeing your server:  null-route them, but do so fairly.  Allow that null-route to expire
so see if they have given up, or the IP was handed over to someone else.

How effective is a spam message that can't even be delivered, even if you intend to filter it
after-the-fact?  Besides, even bulk software has to *try* to establish a connection and relay
mail.  The more mail servers out there that employ null-routing, the less effective bulk mail
software will become while it sits there chewing up time, waiting for a time-out.

I choose this more aggressive approach to stop spam, and my project reflects this attitude:  Stop
spam before it even reaches your server.

--- Angelo Bertolli <angelo at freeshell.org> wrote:
> On Fri, 4 Jun 2004, Timothy Klein wrote:
> 
> > Date: Fri, 4 Jun 2004 10:16:44 -0600
> > From: Timothy Klein <teece at silverklein.net>
> > Reply-To: clue-tech at clue.denver.co.us
> > To: clue-tech at clue.denver.co.us
> > Subject: Re: [CLUE-Tech] If you administer a mail server,
> >     you might find this useful.
> > 
> > On Friday 04 June 2004 10:05 am, Angelo Bertolli wrote:
> >> For example, with our users it's
> >> more important that they don't miss a valid important email, than it is
> >> that their spam is zero.  So spam gets through.
> >
> > Isn't that the case always?  Shouldn't a false positive be a thousand times
> > worse than a false negative, WRT spam?
> >
> > For what kind of users is it OK to throw a random email away once in a blue
> > moon?
> >
> > That's something I have always wondered about with ISP-based or centralized
> > SPAM filters.  I check the spam folder on my machine every couple days, and
> > it catches only spam, that I remember.  Once in a while it catches spam-like
> > commercial email that I had actually signed up for, but that is
> > understandable.  But what if I want that stuff from REI about sales, but my
> > ISP throws it away?
> >
> > Curious to hear from people that actually work on that end.
> >
> 
> 
> I won't claim to be an expert in preventing spam, but I have some 
> experience with administering a small (less than 1000) base of email 
> users, so I have some opinions.
> 
> I think the reality is that spam makes money.  That's the only reason why 
> people do it.  And they do it easily because the Internet is a shared 
> resource where they costs of this resource are quite distributed. 
> That's a shame--the only other shared resource I can think of that we have 
> created is the road system, and people could abuse that too if it was 
> profitable for them to do so.
> 
> But the point is, the Internet and sharing of information such as in email 
> is more valuable when it is open and people play nicely.  The spammers 
> don't.  But I also think that if they get us to make too many compromises 
> to the functionality of this shared resource, they have hurt us more than 
> just letting the spam through.
> 
> The real and only way to win against spammers is to make their efforts 
> ineffective--as in preventing them from making a revenue.  I think the 
> best way to do this is accept all messages for delivery, and then filter 
> them.  I'm not convinced that blocking messages for delivery is a good 
> technique.  I know this is very popular mostly because of the logic behind 
> not wanting them to waste your resources.  However, I think this only 
> results in making them fight harder to try to get their spam through.  I 
> think it's much better to let them think their crap is getting through, 
> only to find out one day that no one is willing to pay them to spam 
> anymore cause it didn't turn out to be effective.  Making spam 
> unprofitable is the way to win against spammers in the long run.
> 
> I don't like tarpits or IP-based blocking.  I think some of this is 
> acceptable, for example blocking known dialup IPs.  But for the most part, 
> from a philosophical point of view it makes the assumption that either 
> there is only one person sending from point X and they're a spammer, or 
> all users from point X are spammers.  And tarpits are really just an 
> exercise in satisfying revenge, as they take up both your own resources 
> (though not many), your time to set up the tarpit, and they are also 
> implemented as IP-based blocking.
> 
> So how do you filter?  Well ideally, I'd say that if everyone used a mail 
> client which could filter spam, that would be the best thing for the long 
> run.  This allows the end user to decide what they consider spam, and 
> puts the responsibility on the user to take care of their own mail. 
> However, it's not realistic that users will know how to filter their mail 
> properly, not to mention the fact that the software really isn't available 
> for them to use.  (Does Outlook/Express do any sort of mail filtering?) 
> You have to remember that the kinds of people who pay for things 
> advertised in spam are also mostly the kinds of people who probably won't 
> bother to turn on their filter (for whatever reason).
> 
> So the judgement lands on the administrator of the mail server on how to 
> handle spam.  Personally, my first and most important concern is to stop 
> email viruses.  Email viruses can be used to spam, and they can turn 
> legitimate users (their computers) into spammers, allowing spam to 
> authenticate using the infected computer's SMTP setup, and even making it 
> look like your own server should be blocked.  Knowing what I know about 
> our network, this means I can block attachments based on extensions.
> 
> Then when you want to block non-viral spam, you have to choose something 
> to pick to filter on.  In general analyzing email headers is not a good 
> way to go IMHO.  Following strictly the rules set by the RFCs doesn't stop 
> spam, and creating "extra rules" such as checking to see if "the version 
> of Outlook you claim to be from can send this kind of multi-part message" 
> is both easy to spoof, and creates extra assumptions that other may not be 
> aware of.
> 
> I think spam should be filtered based on content.  So how do you choose 
> the content?  Maybe block everything that has the word "viagra" in it? 
> This means that any legitimate emails about viagra won't get through. 
> Actually going back to the original point that spam is for making money, 
> filter based on that particular connection to how they get their money.  I 
> would say that usually this is an http link.  Once you have identified a 
> link in an email that goes to a page which is advertised by spammers, 
> filter it out.  They don't deserve to make money off of it.
> 
> 
> I'm sure a lot of people will disagree with me on some points--I'll keep 
> an open mind.
> 
> 
> 
> Angelo


=====
William Kimball, Jr.
"Programming is an art form that fights back!"  =)


	
		
__________________________________
Do you Yahoo!?
Friends.  Fun.  Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/