Auto whitelist refinements.

Milter/__init__.py

@@ -44,7 +44,7 @@ class Milter:
     for i in msg: print i,
     print
 
-  def connect(self,hostname,unused,hostaddr):
+  def connect(self,hostname,family,hostaddr):
     "Called for each connection to sendmail."
     self.log("connect from %s at %s" % (hostname,hostaddr))
     return CONTINUE

NEWS

@@ -1,5 +1,6 @@
 Here is a history of user visible changes to Python milter.
 
+0.8.4	Auto-whitelist recipients of outgoing email.
 0.8.3   Keep screened honeypot mail, but optionally discard honeypot only mail.
 	spf_accept_fail option for braindead SPF senders 
 	  (treats fail like softfail)

TODO

@@ -1,10 +1,10 @@
+Train honeypot with auto-whitelisted mail as ham.  This should result in
+a totally self training auto screener.  Rescind whitelist for banned 
+extensions, in case sender is infected.
+
 Find rfc2822 policy for MFROM quoting.
 
-Use /etc/mail/access for domain specific SPF policies.
+Support explicit errors for SPF policy in access file:
-
-SPF-Fail:	REJECT
-SPF-Softfail:	OK
-SPF-Neutral:	OK
 SPF-Neutral:aol.com	ERROR:"550 AOL mail must get SPF PASS"
 
 Defer TEMPERROR in SPF evaluation - give precedence to security
@@ -16,9 +16,6 @@ Option to add Received-SPF header, but never reject on SPF.
 Create null config that does nothing - except maybe add Received-SPF
 headers.  Many admins would like to turn features on one at a time.
 
-Auto whitelist based on outgoing email - perhaps with magic subject
-or recipient prefix.
-
 Can't output messages with malformed rfc822 attachments.
 
 Move milter,Milter,mime,spf modules to pymilter
@@ -33,7 +30,8 @@ check spam keywords with character classes, e.g.
 
 Implement RRS - a backdoor for non-SRS forwarders.  User lists non-SRS 
 forwarder accounts, and a util provides a special local alias for the
-user to give to the forwarder.  Alias only works for mail from that
+user to give to the forwarder.  (Or user just adds arbitrary alias
+unique to that forwarder to a database.)  Alias only works for mail from that
 forwarder.  Milter gets forwarder domain from alias and uses it to
 SPF check forwarder.
 

bms.py

@@ -1,6 +1,9 @@
 #!/usr/bin/env python
 # A simple milter that has grown quite a bit.
 # $Log$
+# Revision 1.31  2005/10/14 01:14:08  customdesigned
+# Auto whitelist feature.
+#
 # Revision 1.30  2005/10/12 16:36:30  customdesigned
 # Release 0.8.3
 #
@@ -666,20 +669,21 @@ class AddrCache(object):
 	  l = time.strptime(ts,AddrCache.time_format)
 	  t = time.mktime(l)
 	  if t > too_old:
-	    cache[rcpt] = None
+	    cache[rcpt.lower()] = None
 	except:
-	  cache[ln.strip()] = None
+	  cache[ln.strip().lower()] = None
     except IOError: pass
 
   def has_key(self,sender):
-    return self.cache.has_key(sender)
+    return self.cache.has_key(sender.lower())
 
   def __getitem__(self,sender):
-    return self.cache[sender]
+    return self.cache[sender.lower()]
 
   def __setitem__(self,sender,res):
-    cached = sender in self.cache
+    lsender = sender.lower()
-    self.cache[sender] = res
+    cached = lsender in self.cache
+    self.cache[lsender] = res
     if not cached and not res:
       s = time.strftime(AddrCache.time_format,time.localtime())
       print >>open(self.fname,'a'),sender,s # log who we sent DSNs to
@@ -1017,9 +1021,9 @@ class bmsMilter(Milter.Milter):
       return Milter.TEMPFAIL
     self.add_header('Received-SPF',q.get_header(res,receiver))
     self.spf = q
-    if self.dspam and not self.internal_connection and res == 'pass':
+    if self.dspam and res == 'pass' and auto_whitelist.has_key(self.canon_from):
-      if auto_whitelist.has_key(self.canon_from):
       self.dspam = False
+      self.log("WHITELIST",self.canon_from)
     return Milter.CONTINUE
 
   # hide_path causes a copy of the message to be saved - until we
@@ -1358,8 +1362,12 @@ class bmsMilter(Milter.Milter):
 		  	force_result=dspam.DSR_ISSPAM)
 		  self.log("HONEYPOT:",rcpt)
 		return Milter.DISCARD
-	      #if not self.dspam:
+	      if not self.dspam:
-	        # FIXME: tag, but force as ham
+	        # Sender whitelisted: tag, but force as ham.  
+		# User can change if actually spam.
+	        txt = ds.check_spam(user,txt,self.recipients,
+			force_result=dspam.DSR_ISINNOCENT)
+	      else:
 		txt = ds.check_spam(user,txt,self.recipients)
 	      if not txt:
 	        # DISCARD if quarrantined for any recipient.  It

logmsgs.html

@@ -0,0 +1,76 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.1 Final//EN">
+<html>
+<head>
+<title>Python Milter Log Documentation</title>
+<style>
+DT              { font-weight: bolder; padding-top: 1em }
+</style>
+</head><body>
+
+<h1> Milter Log Documentation </h1>
+
+The milter log has a variety of "tags" in it that indicate what it did.
+
+<dl>
+<dt> DSPAM: honeypot SCREENED
+<dd> message was quarantined to the honeypot quarantine
+
+<dt> REJECT: hello SPF: fail 550 access denied
+<dt> REJECT: hello SPF: softfail 550 domain in transition
+<dt> REJECT: hello SPF: neutral 550 access neither permitted nor denied
+<dd> message was rejected because there was an SPF policy for the
+HELO name, and it did not pass.
+
+<dt> CBV: sender-17-44662668-643@bluepenmagic.com
+<dd> we performed a call back verification
+
+<dt> dspam
+<dd> dspam identifier was added to the message
+
+<dt> REJECT: spam from self: jsconnor.com
+<dd> message was reject because HELO was us (jsconnor.com)
+
+<dt> INNOC: richh
+<dd> message was used to update richh's dspam dictionary
+
+<dt> HONEYPOT: michaelb@jsconnor.com
+<dd> message was sent to a honeypot address (michaelb@jsconnor.com), the 
+message was added to the honeypot dspam dictionary as spam
+
+<dt> REJECT: numeric hello name: 63.217.19.146
+<dd> message was rejected because helo name was invalid (numeric)
+
+<dt> eom
+<dd> message was successfully received
+
+<dt> TEMPFAIL: CBV: 450 No MX servers available
+<dd> we tried to do a call back verification but could not look up 
+MX record, we told the sender to try again later
+
+<dt> CBV: info@emailpizzahut.com (cached)
+<dd> call back verification was needed, we had already done it recently
+
+<dt> abort after 0 body chars
+<dd> sender hung up on us
+
+<dt> REJECT: SPF fail 550 SPF fail: see 
+  http://openspf.com/why.html?sender=m.hendersonxk@163.net&ip=213.47.161.100
+<dd> message was reject because its sender's spf policy said to
+
+<dt> REJECT: Subject: Cialis - No prescription needed!
+<dd> message was rejected because its subject contained a bad expression
+
+<dt> DSPAM: tonyc tonyc@jsconnor.com
+<dd> message was sent to tonyc@jsconnor.com and it was identified as spam 
+and placed in the tonyc dspam quarantine
+
+<dt> REJECT: CBV: 550 calvinalstonis@ix.netcom.com...User unknown
+<dt> REJECT: CBV: 553 sorry, that domain isn't in my list
+<dt> REJECT: CBV: 554 delivery error: dd This user doesn't have an account 
+<dd> message was rejected because call back verification gave us a fatal
+error
+</dl>
+
+Please add more tags to this list if you know of any. Thanks.
+</body>
+</html>

milter.html

@@ -24,7 +24,7 @@ ALT="Viewable With Any Browser" BORDER="0"></A>
   Stuart D. Gathman</a><br>
 This web page is written by Stuart D. Gathman<br>and<br>sponsored by
 <a href="http://www.bmsi.com">Business Management Systems, Inc.</a> <br>
-Last updated Aug 28, 2005</h4>
+Last updated Oct 12, 2005</h4>
 
 See the <a href="faq.html">FAQ</a> | <a href="http://sourceforge.net/project/showfiles.php?group_id=139894">Download now</a> |
 <a href="/mailman/listinfo/pymilter">Subscribe to mailing list</a> |
@@ -47,10 +47,25 @@ efficient and secure.  I recommend upgrading.
 
 <h2> Recent Changes </h2>
 
-Python milter is being moved to 
+Python milter has been moved to 
 <a href="http://sourceforge.net/projects/pymilter/">pymilter Sourceforge
 project</a> for development and release downloads.
 <p>
+Release 0.8.3 uses the standard logging module, and supports configuring
+more detailed SPF policy via the sendmail access map.  SMTP AUTH connections
+are considered INTERNAL.  Preventing forgery between internal domains is
+just a matter of specifying the user-domain map - I'll define something
+for the next version.  We now send DSNs when mail is quarantined (rejecting
+if DSN fails) and for SPF syntax errors (PermError).  There is an
+experimental option to add a Sender header when it is missing and the From
+domain doesn't match the MAIL FROM domain.  Next release, we may start
+renaming and replacing an existing Sender header when neither it nor the
+From domain matches MAIL FROM.  Since bogus MAIL FROMs are rejected
+(to varying degrees depending on the configured SPF policy), and
+both Sender and From and displayed by default in many email clients,
+this provides some phishing protection without rejecting mail based
+on headers.
+<p>
 Release 0.8.2 has changes to <a href="http://openspf.net">SPF</a> to bring it
 in line with the newly official RFC.  It adds 
 <a href="http://ses.codeshare.ca/">SES</a>
@@ -245,6 +260,8 @@ content filtering.  SPF checking
 requires <a href="http://pydns.sourceforge.net/">
 pydns</a>.  Configuration documentation is currently included as comments
 in the <a href="milter.cfg">sample config file</a> for the bms.py milter.
+See also the <a href="HOWTO">HOWTO</a> and <a href="logmsgs.html">
+Milter Log Message Tags</a>.
 <p>
 Python milter is under GPL.  The authors can probably be convinced to
 change this to LGPL if needed.