This commit was manufactured by cvs2svn to create tag 'milter-0_8_8'.

Sprout from master 2007-07-25 19:04:44 UTC Stuart Gathman <stuart@gathman.org> 'Fixes from test on EL5.'
Cherrypick from bmsi 2005-05-31 18:23:49 UTC Stuart Gathman <stuart@gathman.org> 'Development changes since 0.7.2':
    rejects.py
    rhsbl.m4
    sample.py
    test/amazon
    test/big5
    test/bounce
    test/bounce1
    test/bound
    test/honey
    test/missingboundary
    test/samp1
    test/spam44
    test/spam7
    test/spam8
    test/test1
    test/test8
    test/virus1
    test/virus13
    test/virus2
    test/virus3
    test/virus4
    test/virus5
    test/virus6
    test/virus7
    testsample.py

COPYING

@@ -1,8 +1,8 @@
 		    GNU GENERAL PUBLIC LICENSE
 		       Version 2, June 1991
 
- Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
- 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+                       59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
  Everyone is permitted to copy and distribute verbatim copies
  of this license document, but changing it is not allowed.
 
@@ -15,7 +15,7 @@ software--to make sure the software is free for all its users.  This
 General Public License applies to most of the Free Software
 Foundation's software and to any other program whose authors commit to
 using it.  (Some other Free Software Foundation software is covered by
-the GNU Lesser General Public License instead.)  You can apply it to
+the GNU Library General Public License instead.)  You can apply it to
 your programs, too.
 
   When we speak of free software, we are referring to freedom, not
@@ -55,7 +55,7 @@ patent must be licensed for everyone's free use or not licensed at all.
 
   The precise terms and conditions for copying, distribution and
 modification follow.
-
+
 		    GNU GENERAL PUBLIC LICENSE
    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
 
@@ -110,7 +110,7 @@ above, provided that you also meet all of these conditions:
     License.  (Exception: if the Program itself is interactive but
     does not normally print such an announcement, your work based on
     the Program is not required to print an announcement.)
-
+
 These requirements apply to the modified work as a whole.  If
 identifiable sections of that work are not derived from the Program,
 and can be reasonably considered independent and separate works in
@@ -168,7 +168,7 @@ access to copy from a designated place, then offering equivalent
 access to copy the source code from the same place counts as
 distribution of the source code, even though third parties are not
 compelled to copy the source along with the object code.
-
+
   4. You may not copy, modify, sublicense, or distribute the Program
 except as expressly provided under this License.  Any attempt
 otherwise to copy, modify, sublicense or distribute the Program is
@@ -225,7 +225,7 @@ impose that choice.
 
 This section is intended to make thoroughly clear what is believed to
 be a consequence of the rest of this License.
-
+
   8. If the distribution and/or use of the Program is restricted in
 certain countries either by patents or by copyrighted interfaces, the
 original copyright holder who places the Program under this License
@@ -278,7 +278,7 @@ PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
 POSSIBILITY OF SUCH DAMAGES.
 
 		     END OF TERMS AND CONDITIONS
-
+
 	    How to Apply These Terms to Your New Programs
 
   If you develop a new program, and you want it to be of the greatest
@@ -303,9 +303,10 @@ the "copyright" line and a pointer to where the full notice is found.
     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     GNU General Public License for more details.
 
-    You should have received a copy of the GNU General Public License along
+    You should have received a copy of the GNU General Public License
-    with this program; if not, write to the Free Software Foundation, Inc.,
+    along with this program; if not, write to the Free Software
-    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
 
 Also add information on how to contact you by electronic and paper mail.
 
@@ -335,5 +336,5 @@ necessary.  Here is a sample; alter the names:
 This General Public License does not permit incorporating your program into
 proprietary programs.  If your program is a subroutine library, you may
 consider it more useful to permit linking proprietary applications with the
-library.  If this is what you want to do, use the GNU Lesser General
+library.  If this is what you want to do, use the GNU Library General
 Public License instead of this License.

CREDITS

@@ -7,8 +7,6 @@ real, usable Python extension.
 
 Other contributors (in random order):
 
-Dwayne Litzenberger, B.A.Sc.
-  for library_dirs patch to compile on Debian 
 Dave MacQuigg 
   for noticing that smfi_insheader wasn't supported, and creating
   a template to help first time pymilter users create their own milter.

HOWTO

@@ -35,13 +35,13 @@ wish to install pydspam.
 For basic pymilter you'll need:
 
 python-2.4
-milter-0.8.10
+milter-0.8.7
 sendmail-8.13.x	(with milter support enabled)
 
 and for SPF you'll need:
 
-pydns-2.3.3-2.4
+pydns-2.3.0-2.4
-pyspf-2.0.5-1.py24
+pyspf-2.0.3-2.py24
 
 and for SRS you'll need:
 
@@ -65,7 +65,7 @@ Start milter and pysrs with "service milter start", "service pysrs start".
 Tail /var/log/milter/milter.log while SMTP clients connect to your
 sendmail instance.  This should show you what the milter is doing.
 
-By default, milter-0.8.10 rejects on SPF fail.
+By default, milter-0.8.7 rejects on SPF fail.
 
 	Step four.  Tweaking the basic config.
 

MANIFEST.in

@@ -16,7 +16,6 @@ include bms.py
 include spf.py
 include cid2spf.py
 include spfquery.py
-include ban2zone.py
 include test.py
 include sample.py
 include milter-template.py

Milter/__init__.py

@@ -184,10 +184,8 @@ def runmilter(name,socketname,timeout = 0):
     print "Removing %s" % fname
     try:
       os.unlink(fname)
-    except os.error, x:
+    except:
-      import errno
+      pass
-      if x.errno != errno.ENOENT:
-        raise milter.error(x)
 
   # The default flags set include everything
   # milter.set_flags(milter.ADDHDRS)

Milter/cache.py

@@ -10,14 +10,6 @@
 # CBV results.
 #
 # $Log$
-# Revision 1.8  2007/09/03 16:18:45  customdesigned
-# Delete unparseable timestamps when loading address cache.  These have
-# arisen because of failure to parse MAIL FROM properly.   Will have to
-# tighten up MAIL FROM parsing to match RFC.
-#
-# Revision 1.7  2007/01/25 22:47:26  customdesigned
-# Persist blacklisting from delayed DSNs.
-#
 # Revision 1.6  2007/01/19 23:31:38  customdesigned
 # Move parse_header to Milter.utils.
 # Test case for delayed DSN parsing.
@@ -74,17 +66,13 @@ class AddrCache(object):
       for ln in fp:
 	try:
 	  rcpt,ts = ln.strip().split(None,1)
-          try:
 	  l = time.strptime(ts,AddrCache.time_format)
 	  t = time.mktime(l)
 	  if t < too_old:
 	    changed = True
 	    continue
 	  cache[rcpt.lower()] = (t,None)
-          except:       # unparsable timestamp - likely garbage
+	except:
-            changed = True
-            continue
-	except: # manual entry (no timestamp)
 	  cache[ln.strip().lower()] = (now,None)
 	wfp.write(ln)
       if changed:
@@ -94,10 +82,8 @@ class AddrCache(object):
     except IOError:
       lock.unlock()
 
-  def has_precise_key(self,sender):
+  def has_key(self,sender):
-    """True if precise sender is cached and has not expired.  Don't
+    "True if sender is cached and has not expired."
-    try looking up wildcard entries.
-    """
     try:
       lsender = sender and sender.lower()
       ts,res = self.cache[lsender]
@@ -105,16 +91,15 @@ class AddrCache(object):
       if not ts or ts > too_old:
         return True
       del self.cache[lsender]
-    except KeyError: pass
-    return False
-
-  def has_key(self,sender):
-    "True if sender is cached and has not expired."
-    if self.has_precise_key(sender):
-      return True
       try:
 	user,host = sender.split('@',1)
-      return self.has_precise_key(host)
+	return self.has_key(host)
+      except ValueError:
+        pass
+    except KeyError:
+      try:
+	user,host = sender.split('@',1)
+	return self.has_key(host)
       except: pass
     return False
 

Milter/dns.py

@@ -1,88 +0,0 @@
-# provide a higher level interface to pydns
-
-import DNS
-from DNS import DNSError
-
-MAX_CNAME = 10
-
-def DNSLookup(name, qtype):
-    try:
-        req = DNS.DnsRequest(name, qtype=qtype)
-        resp = req.req()
-        #resp.show()
-        # key k: ('wayforward.net', 'A'), value v
-        # FIXME: pydns returns AAAA RR as 16 byte binary string, but
-        # A RR as dotted quad.  For consistency, this driver should
-        # return both as binary string.
-        return [((a['name'], a['typename']), a['data']) for a in resp.answers]
-    except IOError, x:
-        raise DNSError, str(x)
-
-class Session(object):
-  """A Session object has a simple cache with no TTL that is valid
-   for a single "session", for example an SMTP conversation."""
-  def __init__(self):
-    self.cache = {}
-
-  # We have to be careful which additional DNS RRs we cache.  For
-  # instance, PTR records are controlled by the connecting IP, and they
-  # could poison our local cache with bogus A and MX records.  
-
-  SAFE2CACHE = {
-    ('MX','A'): None,
-    ('MX','MX'): None,
-    ('CNAME','A'): None,
-    ('CNAME','CNAME'): None,
-    ('A','A'): None,
-    ('AAAA','AAAA'): None,
-    ('PTR','PTR'): None,
-    ('TXT','TXT'): None,
-    ('SPF','SPF'): None
-  }
-
-
-  def dns(self, name, qtype, cnames=None):
-    """DNS query.
-
-    If the result is in cache, return that.  Otherwise pull the
-    result from DNS, and cache ALL answers, so additional info
-    is available for further queries later.
-
-    CNAMEs are followed.
-
-    If there is no data, [] is returned.
-
-    pre: qtype in ['A', 'AAAA', 'MX', 'PTR', 'TXT', 'SPF']
-    post: isinstance(__return__, types.ListType)
-    """
-    result = self.cache.get( (name, qtype) )
-    cname = None
-
-    if not result:
-        safe2cache = Session.SAFE2CACHE
-        for k, v in DNSLookup(name, qtype):
-            if k == (name, 'CNAME'):
-                cname = v
-            if (qtype,k[1]) in safe2cache:
-                self.cache.setdefault(k, []).append(v)
-        result = self.cache.get( (name, qtype), [])
-    if not result and cname:
-        if not cnames:
-            cnames = {}
-        elif len(cnames) >= MAX_CNAME:
-            #return result    # if too many == NX_DOMAIN
-            raise DNSError('Length of CNAME chain exceeds %d' % MAX_CNAME)
-        cnames[name] = cname
-        if cname in cnames:
-            raise DNSError, 'CNAME loop'
-        result = self.dns(cname, qtype, cnames=cnames)
-    return result
-
-DNS.DiscoverNameServers()
-
-if __name__ == '__main__':
-  import sys
-  s = Session()
-  for n,t in zip(*[iter(sys.argv[1:])]*2):
-    print n,t
-    print s.dns(n,t)

Milter/dsn.py

@@ -5,12 +5,6 @@
 # Send DSNs, do call back verification,
 # and generate DSN messages from a template
 # $Log$
-# Revision 1.15  2007/09/24 20:13:26  customdesigned
-# Remove explicit spf dependency.
-#
-# Revision 1.14  2007/03/03 18:19:40  customdesigned
-# Handle DNS error sending DSN.
-#
 # Revision 1.13  2007/01/04 18:01:11  customdesigned
 # Do plain CBV when template missing.
 #
@@ -25,22 +19,22 @@
 #
 
 import smtplib
+import spf
 import socket
 from email.Message import Message
 import Milter
 import time
-import dns
 
-def send_dsn(mailfrom,receiver,msg=None,timeout=600,session=None):
+def send_dsn(mailfrom,receiver,msg=None,timeout=600):
   """Send DSN.  If msg is None, do callback verification.
      Mailfrom is original sender we are sending DSN or CBV to.
      Receiver is the MTA sending the DSN.
      Return None for success or (code,msg) for failure."""
   user,domain = mailfrom.split('@')
-  if not session: session = dns.Session()
   try:
-    mxlist = session.dns(domain,'MX')
+    q = spf.query(None,None,None)
-  except dns.DNSError:
+    mxlist = q.dns(domain,'MX')
+  except spf.TempError:
     return (450,'DNS Timeout: %s MX'%domain)	# temp error
   if not mxlist:
     mxlist = (0,domain),	# fallback to A record when no MX
@@ -92,41 +86,23 @@ def send_dsn(mailfrom,receiver,msg=None,timeout=600,session=None):
       return (450,'No MX response within %f minutes'%(timeout/60.0))
   return (450,'No MX servers available')	# temp error
 
-class Vars: pass
+def create_msg(q,rcptlist,origmsg=None,template=None):
-
+  "Create a DSN message from a template.  Template must be '\n' separated."
-# NOTE: Caller can pass an object to create_msg that in a typical milter
-# collects things like heloname or sender anyway.
-def create_msg(v,rcptlist=None,origmsg=None,template=None):
-  """Create a DSN message from a template.  Template must be '\n' separated.
-     v - an object whose attributes are used for substitutions.  Must
-       have sender and receiver attributes at a minimum.
-     rcptlist - used to set v.rcpt if given
-     origmsg - used to set v.subject and v.spf_result if given
-     template - a '\n' separated string with python '%(name)s' substitutions.
-  """
   if not template:
     return None
-  if hasattr(v,'perm_error'):
+  heloname = q.h
-    # likely to be an spf.query, try translating for backward compatibility
+  sender = q.s
-    q = v
+  connectip = q.i
-    v = Vars()
+  receiver = q.r
+  sender_domain = q.o
+  result = q.result
+  perm_error = q.perm_error
+  rcpt = '\n\t'.join(rcptlist)
+  try: subject = origmsg['Subject']
+  except: subject = '(none)'
   try:
-      v.heloname = q.h
+    spf_result = origmsg['Received-SPF']
-      v.sender = q.s
+  except: spf_result = None
-      v.connectip = q.i
-      v.receiver = q.r
-      v.sender_domain = q.o
-      v.result = q.result
-      v.perm_error = q.perm_error
-    except: v = q
-  if rcptlist:
-    v.rcpt = '\n\t'.join(rcptlist)
-  if origmsg:
-    try: v.subject = origmsg['Subject']
-    except: v.subject = '(none)'
-    try:
-      v.spf_result = origmsg['Received-SPF']
-    except: v.spf_result = None
 
   msg = Message()
 
@@ -136,19 +112,18 @@ def create_msg(v,rcptlist=None,origmsg=None,template=None):
   hdrs,body = template.split('\n\n',1)
   for ln in hdrs.splitlines():
     name,val = ln.split(':',1)
-    msg.add_header(name,(val % v.__dict__).strip())
+    msg.add_header(name,(val % locals()).strip())
-  msg.set_payload(body % v.__dict__)
+  msg.set_payload(body % locals())
   # add headers if missing from old template
   if 'to' not in msg:
-    msg.add_header('To',v.sender)
+    msg.add_header('To',sender)
   if 'from' not in msg:
-    msg.add_header('From','postmaster@%s'%v.receiver)
+    msg.add_header('From','postmaster@%s'%receiver)
   if 'auto-submitted' not in msg:
     msg.add_header('Auto-Submitted','auto-generated')
   return msg
 
 if __name__ == '__main__':
-  import spf
   q = spf.query('192.168.9.50',
   'SRS0=pmeHL=RH==stuart@example.com',
   'red.example.com',receiver='mail.example.com')

Milter/greylist.py

@@ -1,74 +0,0 @@
-import time
-import shelve
-import thread
-import logging
-import urllib
-
-log = logging.getLogger('milter.greylist')
-
-def quoteAddress(s):
-  '''Quote an address so that it's safe to store in the file-system.
-  Address can either be a domain name, or local part.
-  Returns the quoted address.'''
-
-  s = urllib.quote(s, '@_-+~!.%')
-  if s.startswith('.'): s = '%2e' + s[1:]
-  return s
-
-class Record(object):
-  __slots__ = ( 'firstseen', 'lastseen', 'umis', 'cnt' )
-
-  def __init__(self):
-    now = time.time()
-    self.firstseen = now
-    self.lastseen = now
-    self.cnt = 0
-    self.umis = None
-
-class Greylist(object):
-
-  def __init__(self,dbname,grey_time=10,grey_expire=4,grey_retain=36):
-    self.ignoreLastByte = False
-    self.greylist_time = grey_time * 60         # minutes
-    self.greylist_expire = grey_expire * 3600   # hours
-    self.greylist_retain = grey_retain * 24 * 3600   # days
-    self.dbp = shelve.open(dbname,'c',protocol=2)
-    self.lock = thread.allocate_lock()
-  
-  def check(self,ip,sender,recipient):
-    "Return number of allowed messages for greylist triple."
-    sender = quoteAddress(sender)
-    recipient = quoteAddress(recipient)
-    key = ip + ':' + sender + ':' + recipient
-    self.lock.acquire()
-    try:
-      dbp = self.dbp
-      try:
-        r = dbp[key]
-        now = time.time()
-        if now > r.lastseen + self.greylist_retain:
-          # expired
-          log.debug('Expired greylist: %s',key)
-          r = Record()
-        elif now < r.firstseen + self.greylist_time:
-          # still greylisted
-          log.debug('Early greylist: %s',key)
-          #r = Record()
-          r.lastseen = now
-        elif r.cnt or now < r.firstseen + self.greylist_expire:
-          # in greylist window or active
-          r.lastseen = now
-          r.cnt += 1
-          log.debug('Active greylist(%d): %s',r.cnt,key)
-        else:
-          # passed greylist window
-          log.debug('Late greylist: %s',key)
-          r = Record()
-        dbp[key] = r
-      except:
-        r = Record()
-        dbp[key] = r
-      dbp.sync()
-    finally:
-      self.lock.release()
-    return r.cnt

Milter/utils.py

@@ -4,8 +4,6 @@ import socket
 import email.Errors
 from fnmatch import fnmatchcase
 from email.Header import decode_header
-#import email.Utils
-import rfc822
 
 ip4re = re.compile(r'^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$')
 
@@ -42,44 +40,6 @@ def iniplist(ipaddr,iplist):
       return True
   return False
 
-def parseaddr(t):
-  """Split email into Fullname and address.
-
-  >>> parseaddr('user@example.com')
-  ('', 'user@example.com')
-  >>> parseaddr('"Full Name" <foo@example.com>')
-  ('Full Name', 'foo@example.com')
-  >>> parseaddr('spam@spammer.com <foo@example.com>')
-  ('spam@spammer.com', 'foo@example.com')
-  >>> parseaddr('God@heaven <@hop1.org,@hop2.net:jeff@spec.org>')
-  ('God@heaven', 'jeff@spec.org')
-  >>> parseaddr('Real Name ((comment)) <addr...@example.com>')
-  ('Real Name', 'addr...@example.com')
-  >>> parseaddr('a(WRONG)@b')
-  ('WRONG', 'a@b')
-  """
-  #return email.Utils.parseaddr(t)
-  res = rfc822.parseaddr(t)
-  # dirty fix for some broken cases
-  if not res[0]:
-    pos = t.find('<')
-    if pos > 0 and t[-1] == '>':
-      addrspec = t[pos+1:-1]
-      pos1 = addrspec.rfind(':')
-      if pos1 > 0:
-        addrspec = addrspec[pos1+1:]
-      return rfc822.parseaddr('"%s" <%s>' % (t[:pos].strip(),addrspec))
-  if not res[1]:
-    pos = t.find('<')
-    if pos > 0 and t[-1] == '>':
-      addrspec = t[pos+1:-1]
-      pos1 = addrspec.rfind(':')
-      if pos1 > 0:
-        addrspec = addrspec[pos1+1:]
-      return rfc822.parseaddr('%s<%s>' % (t[:pos].strip(),addrspec))
-  return res
-    
-
 def parse_addr(t):
   """Split email into user,domain.
 

NEWS

@@ -1,6 +1,4 @@
-See pymilter.spec for recent history.
+Here is a history of user visible changes to Python milter.
-
-Here is a history of older changes to Python milter.
 0.8.8	move AddrCache, parse_addr, iniplist, parse_header to Milter package
 	fix plock for missing source and can't change owner/group
 	add sample spfmilter.py milter

TODO

@@ -1,21 +1,4 @@
-The recent feature to let a REJECT policy for SPF None be overridden
+Don't match dynamic ptr in bestguess.
-by whitelisting is working for CSI and CMS.  However, there could be
-a sender that we want to REJECT even when whitelisted - because they
-normally get a guessed PASS.  Need another policy name - or else just
-add them to local SPF so they won't ever get 'None'.
-
-When policy is OK, do not use cbv_cache for blacklist.
-
-Add postmaster option or general rcpt list to dsn.  Can send dsn to
-user and postmaster on the same connection.
-
-Check ESMTP NOTIFY before sending real DSNs.  Just use CBV if DSNs are
-not wanted.
-
-Support CBV to local domains and cache results so that invalid users
-can be rejected without maintaining valid user lists.
-
-Now that we blacklist IPs for too many bad rcpts, delay SPF until RCPT TO.
 
 When content filtering is not installed, reject BLACKLISTed MFROM
 immediately.  There is no use waiting until EOM.
@@ -27,8 +10,7 @@ MTA.  The mail is flagged external, so we don't list example.com in
 internal_domains (or we would get "spam from self").  But, if we try to do a
 CBV, we get "fraudulent MX", because the MX is ourself!  So we need to 
 avoid doing CBV on such domains.  Currently, we try to make sure the SPF
-policies don't do CBV.  The real solution is for users to use SMTP AUTH,
+policies don't do CBV.
-but some of them are stubborn.
 
 We now don't check internal domains for incoming mail if there is an
 SPF record.
@@ -93,6 +75,10 @@ Whitelisted senders from trusted relay get PROBATION.  Need to extracted
 SPF result from headers - and in the case of mail internal to relay
 (e.g. bmsi.com), supply 'pass' result.  
 
+For selected domains, check rcpts via CBV before accepting mail.  Cache
+results.  This will kick out dictonary attacks against a mail domain
+behind a gateway sooner.
+
 Add auto-blacklisted senders to blacklist.log with timestamp.
 Add emails blacklisted via CBV so that they are remembered across milter
 restarts.
@@ -109,6 +95,8 @@ e.g. verizon.net).
 Allow verified hostnames for trusted_relay.  E.g. HELO name that
 passes SPF.
 
+Table of sendmail macros for documentation.
+
 When do we get two hello calls?  STARTTLS is one reason.
 
 Option: accept mail from auto-whitelisted senders even with spf-fail,
@@ -190,18 +178,6 @@ Need a test module to feed sample messages to a milter though a live
 sendmail and SMTP.  The mockup currently used is probably not very accurate,
 and doesn't test the threading code.
 
-DONE Table of sendmail macros for documentation.  In API docs on milter.org.
-
-DONE For selected domains, check rcpts via CBV before accepting mail.  Cache
-results.  This will kick out dictonary attacks against a mail domain
-behind a gateway sooner.
-
-DONE Convert DSN to REJECT unless sender gets SPF pass or best guess pass.  Make
-configurable by SPF result with NOTSPAM policy (reject or deliver without DSN).
-Maybe policy should be NODSN - still verify sender with CBV.
-
-DONE Add parseaddr test case for 'foo@bar.com <baz@barf.biz>'
-
 DONE Require signed MFROM for all incoming bounces when signing all outgoing
 mail - except from trusted relays.
 
@@ -248,4 +224,3 @@ data structure as autowhitelist.log.
 
 DONE Backup copies for outgoing/incoming mail.
 
-DONE Don't match dynamic ptr in bestguess.

ban2zone.py

@@ -1,15 +0,0 @@
-#!/usr/bin/python2.4
-
-import socket
-import sys
-from glob import glob
-
-banned_ips = set(socket.inet_aton(ip) 
-    for fn in sys.argv[1:]
-    for ip in open(fn))
-banned_ips = list(banned_ips)
-banned_ips.sort()
-for ip in banned_ips:
-  a = socket.inet_ntoa(ip).split('.')
-  a.reverse()
-  print "%s\tIN A 127.0.0.2"%('.'.join(a))

bms.py

@@ -1,77 +1,6 @@
 #!/usr/bin/env python
 # A simple milter that has grown quite a bit.
 # $Log$
-# Revision 1.133  2008/10/09 18:44:54  customdesigned
-# Skip greylisting for good reputation.
-#
-# Revision 1.132  2008/10/09 00:55:13  customdesigned
-# Don't reset greylist timer on early retries.
-#
-# Revision 1.131  2008/10/08 04:57:28  customdesigned
-# Greylisting
-#
-# Revision 1.130  2008/10/02 03:19:00  customdesigned
-# Delay strike3 REJECT and don't reject if whitelisted.
-# Recognize vacation messages as autoreplies.
-#
-# Revision 1.129  2008/09/09 23:24:56  customdesigned
-# Never ban a trusted relay.
-#
-# Revision 1.128  2008/09/09 23:08:16  customdesigned
-# Wasn't reading banned_ips
-#
-# Revision 1.127  2008/08/25 18:32:22  customdesigned
-# Handle missing gossip_node so self tests pass.
-#
-# Revision 1.126  2008/08/18 17:47:57  customdesigned
-# Log rcpt for SRS rejections.
-#
-# Revision 1.125  2008/08/06 00:52:38  customdesigned
-# CBV policy sends no DSN.  DSN policy sends DSN.
-#
-# Revision 1.124  2008/08/05 18:04:06  customdesigned
-# Send quarantine DSN to SPF PASS only.
-#
-# Revision 1.123  2008/07/29 21:59:29  customdesigned
-# Parse ESMTP params
-#
-# Revision 1.122  2008/05/08 21:35:56  customdesigned
-# Allow explicitly whitelisted email from banned_users.
-#
-# Revision 1.121  2008/04/10 14:59:35  customdesigned
-# Configure gossip TTL.
-#
-# Revision 1.120  2008/04/02 18:59:14  customdesigned
-# Release 0.8.10
-#
-# Revision 1.119  2008/04/01 00:13:10  customdesigned
-# Do not CBV whitelisted addresses.  We already know they are good.
-#
-# Revision 1.118  2008/01/09 20:15:49  customdesigned
-# Handle unquoted fullname when parsing email.
-#
-# Revision 1.117  2007/11/29 14:35:17  customdesigned
-# Packaging tweaks.
-#
-# Revision 1.116  2007/11/01 20:09:14  customdesigned
-# Support temperror policy in access.
-#
-# Revision 1.115  2007/10/10 18:23:54  customdesigned
-# Send quarantine DSN to SPF pass (official or guessed) only.
-# Reject blacklisted email too big for dspam.
-#
-# Revision 1.114  2007/10/10 18:07:50  customdesigned
-# Check porn keywords in From header field.
-#
-# Revision 1.113  2007/09/25 16:37:26  customdesigned
-# Tested on RH7
-#
-# Revision 1.112  2007/09/13 14:51:03  customdesigned
-# Report domain on reputation reject.
-#
-# Revision 1.111  2007/07/25 17:14:59  customdesigned
-# Move milter apps to /usr/lib/pymilter
-#
 # Revision 1.110  2007/07/02 03:06:10  customdesigned
 # Ban ips on bad mailfrom offenses as well as bad rcpts.
 #
@@ -223,20 +152,17 @@ import gc
 import anydbm
 import Milter.dsn as dsn
 from Milter.dynip import is_dynip as dynip
-from Milter.utils import \
+from Milter.utils import iniplist,parse_addr,parse_header,ip4re,addr2bin
-        iniplist,parse_addr,parse_header,ip4re,addr2bin,parseaddr
 from Milter.config import MilterConfigParser
-from Milter.greylist import Greylist
 
 from fnmatch import fnmatchcase
-from email.Utils import getaddresses
+from email.Utils import getaddresses,parseaddr
 
 # Import gossip if available
 try:
   import gossip
   import gossip.client
   import gossip.server
-  gossip_node = None
 except: gossip = None
 
 # Import pysrs if available
@@ -318,7 +244,6 @@ hello_blacklist = ()
 smart_alias = {}
 dspam_dict = None
 dspam_users = {}
-dspam_train = {}
 dspam_userdir = None
 dspam_exempt = {}
 dspam_whitelist = {}
@@ -340,7 +265,6 @@ supply_sender = False
 access_file = None
 timeout = 600
 banned_ips = set()
-greylist = None
 
 logging.basicConfig(
         stream=sys.stdout,
@@ -353,7 +277,6 @@ milter_log = logging.getLogger('milter')
 def read_config(list):
   cp = MilterConfigParser({
     'tempdir': "/var/log/milter/save",
-    'datadir': "/var/log/milter",
     'socket': "/var/run/milter/pythonsock",
     'timeout': '600',
     'scan_html': 'no',
@@ -373,7 +296,6 @@ def read_config(list):
   })
   cp.read(list)
   if cp.has_option('milter','datadir'):
-      print "chdir:",cp.get('milter','datadir')
       os.chdir(cp.get('milter','datadir'))
 
   # milter section
@@ -451,7 +373,6 @@ def read_config(list):
   dspam_users = cp.getaddrdict('dspam','dspam_users')
   dspam_userdir = cp.getdefault('dspam','dspam_userdir')
   dspam_screener = cp.getlist('dspam','dspam_screener')
-  dspam_train = set(cp.getlist('dspam','dspam_train'))
   dspam_reject = cp.getlist('dspam','dspam_reject')
   dspam_internal = cp.getboolean('dspam','dspam_internal')
   if cp.has_option('dspam','dspam_sizelimit'):
@@ -499,7 +420,7 @@ def read_config(list):
     banned_users = cp.getlist('srs','banned_users')
 
   if gossip:
-    global gossip_node, gossip_ttl
+    global gossip_node
     if cp.has_option('gossip','server'):
       server = cp.get('gossip','server')
       host,port = gossip.splitaddr(server)
@@ -509,25 +430,6 @@ def read_config(list):
       for p in cp.getlist('gossip','peers'):
         host,port = gossip.splitaddr(p)
         gossip_node.peers.append(gossip.server.Peer(host,port))
-    if cp.has_option('gossip','ttl'):
-      gossip_ttl = cp.getint('gossip','ttl')
-    else:
-      gossip_ttl = 1
-
-  # greylist section
-  if cp.has_option('greylist','dbfile'):
-    global greylist
-    grey_db = cp.getdefault('greylist','dbfile')
-    grey_days = 36
-    if cp.has_option('greylist','retain'):
-      grey_days = cp.getint('greylist','retain')
-    grey_expire = 4
-    if cp.has_option('greylist','expire'):
-      grey_expire = cp.getint('greylist','expire')
-    grey_time = 10
-    if cp.has_option('greylist','time'):
-      grey_time = cp.getint('greylist','expire')
-    greylist = Greylist(grey_db,grey_time,grey_expire,grey_days)
 
 def findsrs(fp):
   lastln = None
@@ -551,12 +453,6 @@ def findsrs(fp):
         lastln = ln
         break
 
-def param2dict(str):
-  pairs = [x.split('=',1) for x in str]
-  for e in pairs:
-    if len(e) < 2: e.append(None)
-  return dict([(k.upper(),v) for k,v in pairs])
-
 class SPFPolicy(object):
   "Get SPF policy by result from sendmail style access file."
   def __init__(self,sender):
@@ -629,12 +525,6 @@ class SPFPolicy(object):
       policy = 'REJECT'
     return policy
 
-  def getTempErrorPolicy(self):
-    policy = self.getPolicy('spf-temperror:')
-    if not policy:
-      policy = 'REJECT'
-    return policy
-
   def getPassPolicy(self):
     policy = self.getPolicy('spf-pass:')
     if not policy:
@@ -644,8 +534,11 @@ class SPFPolicy(object):
 from Milter.cache import AddrCache
 
 cbv_cache = AddrCache(renew=7)
-auto_whitelist = AddrCache(renew=60)
+cbv_cache.load('send_dsn.log',age=30)
+auto_whitelist = AddrCache(renew=30)
+auto_whitelist.load('auto_whitelist.log',age=120)
 blacklist = AddrCache(renew=30)
+blacklist.load('blacklist.log',age=60)
 
 class bmsMilter(Milter.Milter):
   """Milter to replace attachments poisonous to Windows with a WARNING message,
@@ -768,13 +661,12 @@ class bmsMilter(Milter.Milter):
 
   def offense(self,inc=1):
     self.offenses += inc
-    if self.offenses > 3 and not self.trusted_relay:
+    if self.offenses > 3:
       try:
         ip = addr2bin(self.connectip)
         if ip not in banned_ips:
           banned_ips.add(ip)
           print >>open('banned_ips','a'),self.connectip
-          self.log("BANNED IP:",self.connectip)
       except: pass
     return Milter.REJECT
 
@@ -783,9 +675,6 @@ class bmsMilter(Milter.Milter):
   # of each message.
   def envfrom(self,f,*str):
     self.log("mail from",f,str)
-    #param = param2dict(str)
-    #self.envid = param.get('ENVID',None)
-    #self.mail_param = param
     self.fp = StringIO.StringIO()
     self.tempname = None
     self.mailfrom = f
@@ -796,7 +685,6 @@ class bmsMilter(Milter.Milter):
     self.dspam = True
     self.whitelist = False
     self.blacklist = False
-    self.greylist = False
     self.reject_spam = True
     self.data_allowed = True
     self.delayed_failure = None
@@ -829,9 +717,9 @@ class bmsMilter(Milter.Milter):
 
     self.user = self.getsymval('{auth_authen}')
     if self.user:
-      # Very simple SMTP AUTH policy by default:
+      # Very simple SMTP AUTH policy by defaul:
       #   any successful authentication is considered INTERNAL
-      # Detailed authorization policy is configured in the access file below.
+      # FIXME: configure allowed MAIL FROM by user
       self.internal_connection = True
       self.log(
         "SMTP AUTH:",self.user, self.getsymval('{auth_type}'),
@@ -898,14 +786,12 @@ class bmsMilter(Milter.Milter):
       return Milter.REJECT
     self.umis = None
     self.spf = None
-    self.policy = None
     if not (self.internal_connection or self.trusted_relay)     \
         and self.connectip and spf:
       rc = self.check_spf()
       if rc != Milter.CONTINUE:
         if rc != Milter.TEMPFAIL: self.offense()
         return rc
-      self.greylist = True
     else:
       rc = Milter.CONTINUE
     # FIXME: parse Received-SPF from trusted_relay for SPF result
@@ -913,20 +799,16 @@ class bmsMilter(Milter.Milter):
     hres = self.spf and self.spf_helo
     # Check whitelist and blacklist
     if auto_whitelist.has_key(self.canon_from):
-      self.greylist = False
       if res == 'pass' or self.trusted_relay:
         self.whitelist = True
         self.log("WHITELIST",self.canon_from)
       else:
         self.dspam = False
         self.log("PROBATION",self.canon_from)
-      if res not in ('permerror','softfail'):
-        self.cbv_needed = None
     elif cbv_cache.has_key(self.canon_from) and cbv_cache[self.canon_from] \
         or domain in blacklist:
-      # FIXME: don't use cbv_cache for blacklist if policy is 'OK'
       if not self.internal_connection:
-        self.offense(inc=2)
+        self.offense()
         if not dspam_userdir:
           if domain in blacklist:
             self.log('REJECT: BLACKLIST',self.canon_from)
@@ -937,25 +819,12 @@ class bmsMilter(Milter.Milter):
             self.log('REJECT:',desc)
             self.setreply('550','5.7.1',*desc.splitlines())
           return Milter.REJECT
-        self.greylist = False   # don't delay - use spam for training
         self.blacklist = True
         self.log("BLACKLIST",self.canon_from)
     else:
-      # REJECT delayed until after checking whitelist
-      if self.policy == 'REJECT':
-          self.log('REJECT: no PTR, HELO or SPF')
-          self.setreply('550','5.7.1',
-    "You must have a valid HELO or publish SPF: http://www.openspf.org ",
-    "Contact your mail administrator IMMEDIATELY!  Your mail server is ",
-    "severely misconfigured.  It has no PTR record (dynamic PTR records ",
-    "that contain your IP don't count), an invalid or dynamic HELO, ",
-    "and no SPF record."
-          )
-          return self.offense() # ban ip if too many bad MFROMs
       global gossip
       if gossip and domain and rc == Milter.CONTINUE \
-          and not (self.internal_connection or self.trusted_relay) \
+          and not (self.internal_connection or self.trusted_relay):
-          and gossip_node:
         if self.spf and self.spf.result == 'pass':
           qual = 'SPF'
         elif res == 'pass':
@@ -977,7 +846,6 @@ class bmsMilter(Milter.Milter):
             self.reputation = int(a[-2])
             self.confidence = int(a[-1])
             self.umis = umis
-            self.from_domain = domain
             # We would like to reject on bad reputation here, but we
             # need to give special consideration to postmaster.  So
             # we have to wait until envrcpt().  Perhaps an especially
@@ -987,8 +855,6 @@ class bmsMilter(Milter.Milter):
               self.setreply('550','5.7.1',
                 'Your domain has been sending nothing but spam')
               return Milter.REJECT
-            if self.reputation > 40 and self.confidence > 1:
-              self.greylist = False
         except:
           gossip = None
           raise
@@ -1012,22 +878,13 @@ class bmsMilter(Milter.Milter):
       res,code,txt = q.check()
     q.result = res
     if res in ('unknown','permerror') and q.perm_error and q.perm_error.ext:
-      self.cbv_needed = (q,'permerror') # report SPF syntax error to sender
+      self.cbv_needed = (q,res) # report SPF syntax error to sender
       res,code,txt = q.perm_error.ext   # extended (lax processing) result
       txt = 'EXT: ' + txt
     p = SPFPolicy(q.s)
     # FIXME: try:finally to close policy db, or reuse with lock
-    if res in ('error','temperror'):
-      if self.need_cbv(p.getTempErrorPolicy(),q,'temperror'):
-        self.log('TEMPFAIL: SPF %s %i %s' % (res,code,txt))
-        self.setreply(str(code),'4.3.0',txt,
-          'We cannot accept your email until the DNS server for %s' % q.o,
-          'is operational for TXT record queries.'
-        )
-        return Milter.TEMPFAIL
-      res,code,txt = 'none',250,'EXT: ignoring DNS error'
     hres = None
-    if res != 'pass':
+    if res not in ('pass','error','temperror'):
       if self.mailfrom != '<>':
         # check hello name via spf unless spf pass
         h = spf.query(self.connectip,'',self.hello_name,receiver=receiver)
@@ -1071,13 +928,27 @@ class bmsMilter(Milter.Milter):
       if self.missing_ptr and ores == 'none' and res != 'pass' \
                 and hres != 'pass':
         # this bad boy has no credentials whatsoever
-        res = 'none'
         policy = p.getNonePolicy()
-        if policy in ('CBV','DSN'):
+        if policy == 'CBV':
+          if self.mailfrom != '<>':
+            self.cbv_needed = (q,ores)  # accept, but inform sender via DSN
 	  self.offenses = 3    # ban ip if any bad recipient
-        self.need_cbv(policy,q,'strike3')
+        elif policy != 'OK':
+          self.log('REJECT: no PTR, HELO or SPF')
+          self.setreply('550','5.7.1',
+    "You must have a valid HELO or publish SPF: http://www.openspf.org ",
+    "Contact your mail administrator IMMEDIATELY!  Your mail server is ",
+    "severely misconfigured.  It has no PTR record (dynamic PTR records ",
+    "that contain your IP don't count), an invalid or dynamic HELO, ",
+    "and no SPF record."
+          )
+          return Milter.REJECT
     if res in ('deny', 'fail'):
-      if self.need_cbv(p.getFailPolicy(),q,'fail'):
+      policy = p.getFailPolicy()
+      if policy == 'CBV':
+        if self.mailfrom != '<>':
+          self.cbv_needed = (q,res)
+      elif policy != 'OK':
         self.log('REJECT: SPF %s %i %s' % (res,code,txt))
         self.setreply(str(code),'5.7.1',txt)
         # A proper SPF fail error message would read:
@@ -1085,7 +956,11 @@ class bmsMilter(Milter.Milter):
         # "forged.org" in the sender address.  Contact <postmaster@forged.org>.
         return Milter.REJECT
     if res == 'softfail':
-      if self.need_cbv(p.getSoftfailPolicy(),q,'softfail'):
+      policy = p.getSoftfailPolicy()
+      if policy == 'CBV':
+        if self.mailfrom != '<>':
+          self.cbv_needed = (q,res)
+      elif policy != 'OK':
         self.log('REJECT: SPF %s %i %s' % (res,code,txt))
         self.setreply('550','5.7.1',
           'SPF softfail: If you get this Delivery Status Notice, your email',
@@ -1096,7 +971,12 @@ class bmsMilter(Milter.Milter):
         )
         return Milter.REJECT
     if res == 'neutral':
-      if self.need_cbv(p.getNeutralPolicy(),q,'neutral'):
+      policy = p.getNeutralPolicy()
+      if policy == 'CBV':
+        if self.mailfrom != '<>':
+          self.cbv_needed = (q,res)
+          # FIXME: this makes Received-SPF show wrong result
+      elif policy != 'OK':
         self.log('REJECT: SPF neutral for',q.s)
         self.setreply('550','5.7.1',
           'mail from %s must pass SPF: http://openspf.org/why.html' % q.o,
@@ -1108,7 +988,11 @@ class bmsMilter(Milter.Milter):
         )
         return Milter.REJECT
     if res in ('unknown','permerror'):
-      if self.need_cbv(p.getPermErrorPolicy(),q,'permerror'):
+      policy = p.getPermErrorPolicy()
+      if policy == 'CBV':
+        if self.mailfrom != '<>':
+          self.cbv_needed = (q,res)
+      elif policy != 'OK':
         self.log('REJECT: SPF %s %i %s' % (res,code,txt))
         # latest SPF draft recommends 5.5.2 instead of 5.7.1
         self.setreply(str(code),'5.5.2',txt,
@@ -1116,6 +1000,10 @@ class bmsMilter(Milter.Milter):
           'We cannot accept mail from %s until this is corrected.' % q.o
         )
         return Milter.REJECT
+    if res in ('error','temperror'):
+      self.log('TEMPFAIL: SPF %s %i %s' % (res,code,txt))
+      self.setreply(str(code),'4.3.0',txt)
+      return Milter.TEMPFAIL
     kv = {}
     if hres and q.h != q.o:
       kv['helo_spf'] = hres
@@ -1131,15 +1019,6 @@ class bmsMilter(Milter.Milter):
   # track header mods separately from body mods - so use only
   # in emergencies.
   def envrcpt(self,to,*str):
-    try:
-      param = param2dict(str)
-      self.notify = param.get('NOTIFY','FAILURE,DELAY').upper().split(',')
-      if 'NEVER' in self.notify: self.notify = ()
-      #self.rcpt_param = param
-    except:
-      self.log("REJECT: invalid PARAM:",to,str)
-      self.setreply('550','5.7.1','Invalid SRS signature')
-      return Milter.REJECT
     # mail to MAILER-DAEMON is generally spam that bounced
     if to.startswith('<MAILER-DAEMON@'):
       self.log('REJECT: RCPT TO:',to,str)
@@ -1166,7 +1045,6 @@ class bmsMilter(Milter.Milter):
               self.log("srs rcpt:",newaddr)
             self.dspam = False    # verified as reply to mail we sent
             self.blacklist = False
-            self.greylist = False
             self.delayed_failure = False
           except:
             if not (self.internal_connection or self.trusted_relay):
@@ -1179,12 +1057,9 @@ class bmsMilter(Milter.Milter):
                 self.setreply('550','5.7.1','Invalid SES signature')
                 return Milter.REJECT
               # reject for certain recipients are delayed until after DATA
-	      if auto_whitelist.has_precise_key(self.canon_from):
-		self.log("WHITELIST: DSN from",self.canon_from)
-	      else:
               if srs_reject_spoofed \
-                  and user.lower() not in ('postmaster','abuse'):
+                  and not user.lower() in ('postmaster','abuse'):
-                  return self.forged_bounce(to)
+                return self.forged_bounce()
               self.data_allowed = not srs_reject_spoofed
 
         if not self.internal_connection and domain in private_relay:
@@ -1221,10 +1096,9 @@ class bmsMilter(Milter.Milter):
             return Milter.REJECT
           self.dspam = False
         if userl != 'postmaster' and self.umis    \
-          and self.reputation < -50 and self.confidence > 3:
+          and self.reputation < -50 and self.confidence > 1:
-          domain = self.from_domain
           self.log('REJECT: REPUTATION, rcpt to',to,str)
-          self.setreply('550','5.7.1','%s has been sending mostly spam'%domain)
+          self.setreply('550','5.7.1','Your domain has been sending mostly spam')
           return Milter.REJECT
 
         if domain in hide_path:
@@ -1235,18 +1109,8 @@ class bmsMilter(Milter.Milter):
     except:
       self.log("rcpt to",to,str)
       raise
-    if self.greylist and greylist and self.canon_from:
-      # no policy for trusted or internal
-      rc = greylist.check(self.connectip,self.canon_from,canon_to)
-      if rc == 0:
-        self.log("GREYLIST:",self.connectip,self.canon_from,canon_to)
-        self.setreply('451','4.7.1',
-          'Greylisted: http://projects.puremagic.com/greylisting/',
-          'Please retry in %.1f minutes'%(greylist.greylist_time/60.0))
-        return Milter.TEMPFAIL
-      self.log("GREYLISTED: %d"%rc)
-      
     self.log("rcpt to",to,str)
+      
     self.smart_alias(to)
     # get recipient after virtusertable aliasing
     #rcpt = self.getsymval("{rcpt_addr}")
@@ -1308,14 +1172,7 @@ class bmsMilter(Milter.Milter):
           # original sender (encoded in Message-ID) is blacklisted
 
     elif lname == 'from':
-      fname,email = parseaddr(val)
+      name,email = parseaddr(val)
-      # check for porn keywords
-      lval = fname.lower().strip()
-      for w in porn_words:
-        if lval.find(w) >= 0:
-          self.log('REJECT: %s: %s' % (name,val))
-          self.setreply('550','5.7.1','Watch your language')
-          return Milter.REJECT
       if email.lower().startswith('postmaster@'):
         # Yes, if From header comes last, this might not help much.
         # But this is a heuristic - if MTAs would send proper DSNs in
@@ -1336,9 +1193,9 @@ class bmsMilter(Milter.Milter):
         return Milter.REJECT
     return Milter.CONTINUE
 
-  def forged_bounce(self,rcpt='-'):
+  def forged_bounce(self):
     if self.mailfrom != '<>':
-      self.log("REJECT: bogus DSN",rcpt)
+      self.log("REJECT: bogus DSN")
       self.setreply('550','5.7.1',
         "I do not accept normal mail from %s." % self.mailfrom.split('@')[0],
         "All such mail has turned out to be Delivery Status Notifications",
@@ -1346,7 +1203,7 @@ class bmsMilter(Milter.Milter):
         "you need to.  Use another MAIL FROM if you need to send me mail."
       )
     else:
-      self.log('REJECT: bounce with no SRS encoding',rcpt)
+      self.log('REJECT: bounce with no SRS encoding')
       self.setreply('550','5.7.1',
         "I did not send you that message. Please consider implementing SPF",
         "(http://openspf.org) to avoid bouncing mail to spoofed senders.",
@@ -1367,10 +1224,9 @@ class bmsMilter(Milter.Milter):
         if gossip and self.umis:
           gossip_node.feedback(self.umis,1)
         return rc
-    elif self.whitelist_sender:
+    elif self.whitelist_sender and lname == 'subject':
         # check for AutoReplys
-      if (lname == 'subject' and reautoreply.match(val)) \
+        if reautoreply.match(val):
-	or (lname == 'user-agent' and val.lower().startswith('vacation')):
           self.whitelist_sender = False
           self.log('AUTOREPLY: not whitelisted')
 
@@ -1557,12 +1413,6 @@ class bmsMilter(Milter.Milter):
             elif not self.internal_connection or dspam_internal:
               if len(txt) > dspam_sizelimit:
                 self.log("Large message:",len(txt))
-                if self.blacklist:
-                  self.log('REJECT: BLACKLISTED')
-                  self.setreply('550','5.7.1',
-                        '%s has been blacklisted.'%self.canon_from)
-                  self.fp = None
-                  return Milter.REJECT
                 return False
               if user == 'honeypot' and Dspam.VERSION >= '1.1.9':
                 keep = False    # keep honeypot mail
@@ -1574,12 +1424,9 @@ class bmsMilter(Milter.Milter):
                     return False
                   if self.spf and self.mailfrom != '<>':
                     # check that sender accepts quarantine DSN
-                    if self.spf.result == 'pass':
                     msg = mime.message_from_file(StringIO.StringIO(txt))
                     rc = self.send_dsn(self.spf,msg,'quarantine')
                     del msg
-                    else:
-                      rc = self.send_dsn(self.spf)
                     if rc != Milter.CONTINUE:
                       return rc 
                   ds.check_spam(user,txt,self.recipients,quarantine=True,
@@ -1597,13 +1444,8 @@ class bmsMilter(Milter.Milter):
               elif self.blacklist:
                 txt = ds.check_spam(user,txt,self.recipients,
                         force_result=dspam.DSR_ISSPAM)
-              elif user in dspam_train:
-                txt = ds.check_spam(user,txt,self.recipients)
               else:
-                txt = ds.check_spam(user,txt,self.recipients,classify=True)
+                txt = ds.check_spam(user,txt,self.recipients)
-                if txt:
-                  self.add_header("X-DSpam-Score",'%f' % ds.probability)
-                  return False
               if not txt:
                 # DISCARD if quarrantined for any recipient.  It
                 # will be resent to all recipients if they submit
@@ -1634,7 +1476,7 @@ class bmsMilter(Milter.Milter):
           ds.check_spam(screener,txt,self.recipients,
                   force_result=dspam.DSR_ISINNOCENT)
           return False
-        if self.reject_spam and self.spf.result != 'pass':
+        if self.reject_spam:
           self.log("DSPAM:",screener,
                 'REJECT: X-DSpam-Score: %f' % ds.probability)
           self.setreply('550','5.7.1','Your Message looks spammy')
@@ -1644,18 +1486,12 @@ class bmsMilter(Milter.Milter):
         if self.spf and self.mailfrom != '<>':
           # check that sender accepts quarantine DSN
           self.fp.seek(0)
-	  if self.spf.result == 'pass' or self.cbv_needed:
           msg = mime.message_from_file(self.fp)
-	    if self.spf.result == 'pass':
           rc = self.send_dsn(self.spf,msg,'quarantine')
-	    else:
-	      rc = self.do_needed_cbv(msg)
-	    del msg
-	  else:
-	    rc = self.send_dsn(self.spf)
           if rc != Milter.CONTINUE:
             self.fp = None
             return rc
+          del msg
         if not ds.check_spam(screener,txt,self.recipients,classify=True):
           self.fp = None
           return Milter.DISCARD
@@ -1698,23 +1534,6 @@ class bmsMilter(Milter.Milter):
         quarantine=False)
     self.log("TRAINSPAM:",screener,'X-Dspam-Score: %f' % ds.probability)
 
-  def do_needed_cbv(self,msg):
-    q,template_name = self.cbv_needed
-    rc = self.send_dsn(q,msg,template_name)
-    self.cbv_needed = None
-    return rc
-
-  def need_cbv(self,policy,q,tname):
-    self.policy = policy
-    if policy == 'CBV':
-      if self.mailfrom != '<>' and not self.cbv_needed:
-        self.cbv_needed = (q,None)
-    elif policy == 'DSN':
-      if self.mailfrom != '<>' and not self.cbv_needed:
-        self.cbv_needed = (q,tname)
-    elif policy != 'OK': return True
-    return False
-
   def eom(self):
     if not self.fp:
       return Milter.ACCEPT      # no message collected - so no eom processing
@@ -1835,12 +1654,22 @@ class bmsMilter(Milter.Milter):
       except Milter.error:
         self.addheader(name,val)        # older sendmail can't insheader
 
-    # Do not send CBV to internal domains (since we'll just get
+    # do not send CBV to internal domains (since we'll just get
-    # the "Fraudulent MX" error).  Whitelisted senders clearly do not
+    # the "Fraudulent MX" error).
-    # need CBV.  However, whitelisted domains might (to discover 
-    # bogus localparts).  Need a way to tell the difference.
     if self.cbv_needed and not self.internal_domain:
-      rc = self.do_needed_cbv(msg)
+      q,res = self.cbv_needed
+      if res == 'softfail':
+        template_name = 'softfail'
+      elif res in ('fail','deny'):
+        template_name = 'fail'
+      elif res in ('unknown','permerror'):
+        template_name = 'permerror'
+      elif res == 'neutral':
+        template_name = 'neutral'
+      else:
+        template_name = 'strike3'
+      rc = self.send_dsn(q,msg,template_name)
+      self.cbv_needed = None
       if rc == Milter.REJECT:
         # Do not feedback here, because feedback should only occur
         # for messages that have gone to DATA.  Reputation lets us
@@ -1908,24 +1737,22 @@ class bmsMilter(Milter.Milter):
       out.close()
     return Milter.TEMPFAIL
 
-  def send_dsn(self,q,msg=None,template_name=None):
+  def send_dsn(self,q,msg,template_name):
     sender = q.s
     cached = cbv_cache.has_key(sender)
     if cached:
       self.log('CBV:',sender,'(cached)')
       res = cbv_cache[sender]
     else:
-      m = None
-      if template_name:
       fname = template_name+'.txt'
       try:
         template = file(template_name+'.txt').read()
-          m = dsn.create_msg(q,self.recipients,msg,template)
         self.log('CBV:',sender,'Using:',fname)
-        except IOError: pass
+      except IOError:
-      if not m:
+        template = None
-        self.log('CBV:',sender,'PLAIN (%s)'%q.result)
+        self.log('CBV:',sender,'PLAIN')
-      else:
+      m = dsn.create_msg(q,self.recipients,msg,template)
+      if m:
         if srs:
           # Add SRS coded sender to various headers.  When (incorrectly)
           # replying to our DSN, any of these which are preserved
@@ -1949,10 +1776,7 @@ class bmsMilter(Milter.Milter):
         return Milter.TEMPFAIL
       cbv_cache[sender] = res
       self.log('REJECT:',desc)
-      try:
       self.setreply('550','5.7.1',*desc.splitlines())
-      except TypeError:
-        self.setreply('550','5.7.1',"Callback failure")
       return Milter.REJECT
     cbv_cache[sender] = res
     return Milter.CONTINUE
@@ -1979,7 +1803,6 @@ def main():
       return
   try:
     from glob import glob
-    global banned_ips
     banned_ips = set(addr2bin(ip) 
         for fn in glob('banned_ips*')
         for ip in open(fn))
@@ -2001,10 +1824,6 @@ def main():
 if __name__ == "__main__":
   read_config(["/etc/mail/pymilter.cfg","milter.cfg"])
       
-  cbv_cache.load('send_dsn.log',age=30)
-  auto_whitelist.load('auto_whitelist.log',age=120)
-  blacklist.load('blacklist.log',age=60)
-      
   if dspam_dict:
     import dspam        # low level spam check
   if dspam_userdir:

doc/changes.ht

@@ -2,47 +2,6 @@ Title: Recent Changes
 
 <h2> Recent Changes </h2>
 
-<h3> 0.8.10 </h3>
-
-SRS rejections now log the recipient.  
-I have finally implemented plain CBV (no DSN).  The CBV policy
-will do a plain CBV from now on, and the DSN policy is required
-if you want to send a DSN.
-I started checking the MAIL FROM fullname (human readable part
-of an email) for porn keywords.  There is now a banned IP database.
-IPs are banned for too many bad MAIL FROMs or RCPT TOs, and remain banned
-for 7 days.
-
-<h3> 0.8.9 </h3>
-
-I use the <code>%ifarch</code> hack to build milter and milter-spf
-packages as noarch, while pymilter is built as native.
-
-I removed the spf dependency from dsn.py, so pymilter can be used without
-installing pyspf, and added a Milter.dns module to let python milters do
-general DNS lookups without loading pyspf.
-
-<h3> 0.8.8 </h3>
-
-Programs do not belong in the /var/log directory.  I moved the
-milter apps to /usr/lib/pymilter.  Since having the programs and
-data in the same directory is convenient for debugging, it will
-still use an executable present in the datadir.
-
-Several general utility classes and functions are now in the Milter package
-for possible use by other python milters.  In addition to the trivial example
-milter, a simple SPF only milter is included as a realistic example.
-
-The spec file now build 3 RPMs:
-
-<ul>
-<li> pymilter is the milter module and Milter package for use by all python
-     milters.
-<li> milter is the all-singing, all-dancing python milter application, with
-     supporting <code>/etc/init.d</code>, logrotate and other scripts.
-<li> milter-spf is the simple SPF only milter application.
-</ul>
-
 <h3> 0.8.7 </h3>
 
 The spf module has been moved to the 

doc/credits.ht

@@ -5,7 +5,7 @@ Title: Credits
 <a href="mailto:Jim Niemira <urmane@urmane.org>">Jim Niemira</a>
 wrote the original C module and some quick
 and dirty python to use it.  
-<a href="http://gathman.org/vitae">Stuart D. Gathman</a>
+<a href="mailto:Stuart Gathman <stuart@bmsi.com>">Stuart D. Gathman</a>
 took that kludge and added threading and context objects to it, wrote a proper
 OO wrapper (Milter.py) that handles attachments, did lots of testing, packaged
 it with distutils, and generally transformed it from a quick hack to a

doc/links.h

@@ -11,7 +11,7 @@
 <li><a href="credits.html">CREDITS</a>
 <li><a href="http://sourceforge.net"><img src="http://sflogo.sourceforge.net/sflogo.php?group_id=139894&amp;type=1" width="88" height="31" border="0" alt="SourceForge.net Logo" /></a>
 <h3>Links</h3>
-<li><a href="https://www.milter.org/developers/api/index">C&nbsp;API</a>
+<li><a href="http://www.milter.org/milter_api/api.html">C&nbsp;API</a>
 <li><a href="http://www.milter.org/">Milter.Org</a>
 <li><a href="http://www.python.org/">Python.Org</a>
 <li><a href="http://www.sendmail.org/">Sendmail.Org</a>

doc/milter.ht

@@ -13,19 +13,14 @@ ALT="Viewable With Any Browser" BORDER="0"></A>
 </map>
 </P>
 
-<table rules="none">
+<img src="Maxwells.gif" alt="Maxwell's Daemon: pymilter mascot" align=left>
-<tr><td>
-<img src="Maxwells.gif" alt="Maxwell's Daemon: pymilter mascot" align="top">
-Mascot by <a href="http://alphard.ethz.ch/hafner/lebl.htm">Christian Hafner</a>
-</td>
-<td>
 <h1 align=center>Sendmail Milters in Python</h1>
 <h4 align=center>by <a href="mailto:%75%72%6D%61%6E%65%40%6E%65%75%72%61l%61%63%63%65%73%73%2E%63%6F%6D">Jim Niemira</a>
   and <a href="mailto:%73%74%75%61%72%74%40%62%6D%73%69%2E%63%6F%6D">
   Stuart D. Gathman</a><br>
 This web page is written by Stuart D. Gathman<br>and<br>sponsored by
 <a href="http://www.bmsi.com">Business Management Systems, Inc.</a> <br>
-Last updated Aug 26, 2008</h4>
+Last updated Mar 30, 2007</h4>
 
 See the <a href="faq.html">FAQ</a> | <a href="http://sourceforge.net/project/showfiles.php?group_id=139894">Download now</a> |
 <a href="http://bmsi.com/mailman/listinfo/pymilter">Subscribe to mailing list</a> |
@@ -36,7 +31,7 @@ See the <a href="faq.html">FAQ</a> | <a href="http://sourceforge.net/project/sho
 <a href="//www.python.org">
 <img src="python55.gif" align=left alt="A Python"></a>
 <a href="//www.sendmail.org/">Sendmail</a> introduced a
-<a href="https://www.milter.org/developers/api/index"> new API</a> beginning with version 8.10 -
+<a href="http://www.milter.org/milter_api/api.html"> new API</a> beginning with version 8.10 -
 libmilter.  The milter module for <a href="//www.python.org">Python</a>
 provides a python interface to libmilter that exploits all its features.
 <p>
@@ -44,11 +39,7 @@ Sendmail 8.12 officially releases libmilter.
 Version 8.12 seems to be more robust, and includes new privilege
 separation features to enhance security.  Even better, sendmail 8.13
 supports socket maps, which makes <a href="pysrs.html">pysrs</a> much more
-efficient and secure.   Sendmail 8.14 finally supports modifying 
+efficient and secure.  I recommend upgrading.
-MAIL FROM via the milter API.  Unfortunately, I haven't gotten around
-to supporting that yet in python milter.
-</td></tr>
-</table>
 
 <h3><a name=overview>Overview</a></h3>
 
@@ -57,7 +48,7 @@ href="#milter">milters</a>, and the beginnings of a general purpose mail
 filtering system written in Python.
 <p>
 At the lowest level, the 'milter' module provides a thin wrapper around the
-<a href="https://www.milter.org/developers/api/index">
+<a href="http://www.milter.org/milter_api/api.html">
 sendmail libmilter API</a>.  This API lets you register callbacks for 
 a number of events in the
 <a href="http://www.cs.concordia.ca/~group/fig/public/email/relay/milter+ruleset-checks.html">process of sendmail receiving a message via SMTP</a>.  
@@ -130,7 +121,7 @@ copy mail.
 <li> For more ideas, check the <a href="//www.milter.org">Milter Web Page</a>.
 </menu>
 
-<a href="https://www.milter.org/developers/api/index">
+<a href="http://www.milter.org/milter_api/api.html">
 Documentation</a> for the C API is provided with sendmail.  Miltermodule
 provides a thin python wrapper for the C API.  Milter.py provides a simple
 OO wrapper on top of that.
@@ -211,8 +202,60 @@ href="http://www.duh.org/cvsweb.cgi/~checkout~/pmilter/doc/milter-protocol.txt?r
 <h3> Confirmed Installations </h3>
 
 Please <a href="mailto:%73%74%75%61%72%74%40%62%6D%73%69%2E%63%6F%6D">email</a>
-me if you do <i>not</i> successfully install milter.  The confirmed
+me if you successfully install milter on a system not mentioned below.
-installations are too numerous to list at this point.
+<p>
+<table>
+<tr>
+<th>Operating System</th> <th>Compiler</th> <th>Python</th> <th>Sendmail</th>
+<th>milter</th>
+<tr>
+<td>Mandrake 8.0</td><td>gcc-3.0.1</td><td>2.1.1</td><td>8.12.0</td>
+<td>0.3.3</td><tr>
+<td>Mandrake 8.0</td><td>gcc-2.96</td><td>2.0</td><td>8.11.2</td>
+<td>0.3.6</td><tr>
+<td>RedHat 6.2</td><td>egcs-1.1.2</td><td>2.2.2</td><td>8.11.6</td>
+<td>0.5.4</td><tr>
+<td>RedHat 7.1</td><td>gcc-2.96</td><td>?</td><td>8.12.1</td>
+<td>0.3.5</td><tr>
+<td>RedHat 7.3</td><td>gcc-2.96</td><td>2.2.2</td><td>8.11.6</td>
+<td>0.5.5</td><tr>
+<td>RedHat 7.3</td><td>gcc-2.96</td><td>2.3.3</td><td>8.13.1</td>
+<td>0.7.2</td><tr>
+<td>RedHat 7.3</td><td>gcc-2.96</td><td>2.4.1</td><td>8.13.5</td>
+<td>0.8.4</td><tr>
+<td>RedHat 8.0</td><td>gcc-3.2</td><td>2.2.1</td><td>8.12.6</td>
+<td>0.5.2</td><tr>
+<td>RedHat 9.0</td><td>gcc-3.2.2</td><td>2.4.1</td><td>8.13.1</td>
+<td>0.8.2</td><tr>
+<td>RedHat EL3</td><td>gcc-3.2.3</td><td>2.4.1</td><td>8.13.5</td>
+<td>0.8.4</td><tr>
+<td>Debian Linux</td><td>gcc-2.95.2</td><td>2.1.1</td><td>8.12.0</td>
+<td>0.3.7</td><tr>
+<td>Debian Linux</td><td>gcc-3.2.2</td><td>2.2.2</td><td>8.12.7</td>
+<td>0.5.4</td><tr>
+<td>AIX-4.1.5</td><td>gcc-2.95.2</td><td>2.1.1</td><td>8.11.5</td>
+<td>0.3.3</td><tr>
+<td>AIX-4.1.5</td><td>gcc-2.95.2</td><td>2.1.1</td><td>8.12.1</td>
+<td>0.3.4</td><tr>
+<td>AIX-4.1.5</td><td>gcc-2.95.2</td><td>2.1.3</td><td>8.12.3</td>
+<td>0.4.2</td><tr>
+<td>AIX-4.1.5</td><td>gcc-2.95.2</td><td>2.4.1</td><td>8.13.1</td>
+<td>0.8.4</td><tr>
+<td>Slackware 7.1</td><td>?</td><td>?</td><td>8.12.1</td>
+<td>0.3.8</td><tr>
+<td>Slackware 9.0</td><td>gcc-3.2.2</td><td>2.2.3</td><td>8.12.9</td>
+<td>0.5.4</td><tr>
+<td>OpenBSD</td><td>?</td><td>2.3.3?</td><td>8.13.1?</td>
+<td>0.7.2</td><tr>
+<td>SuSE 7.3</td><td>gcc-2.95.3</td><td>2.1.1</td><td>8.12.2</td>
+<td>0.3.9</td><tr>
+<td>FreeBSD</td><td>gcc-2.95.3</td><td>2.2.1</td><td>8.12.3</td>
+<td>0.4.0</td><tr>
+<td>FreeBSD</td><td>gcc-2.95.3</td><td>2.2.2</td><td>?</td>
+<td>0.5.5</td><tr>
+<td>FreeBSD 4.4</td><td>gcc-2.95.3</td><td>?</td><td>8.12.10</td>
+<td>0.6.6</td>
+</table>
 
 <h2> Enough Already! </h2>
 

doc/policy.ht

@@ -4,7 +4,8 @@ Title: Python Milter Mail Policy
 
 These are the policies implemented by the <code>bms.py</code> milter
 application.  The milter and Milter modules do not implement any policies
-by themselves.  
+by themselves.  Eventually, I'll get the bms.py milter moved to its
+own package.
 
 <h3> Classify connection </h3>
 
@@ -76,174 +77,161 @@ altered accordingly.
 
 <h2> SPF check </h2>
 
-The MAIL FROM, connect IP, and HELO name are checked against
+Finally, the MAIL FROM, connect IP, and HELO name are checked against
-any SPF records published via DNS for the alleged sender (MAIL FROM)
+any SPF records published via DNS for the alleged sender (MAIL FROM).
-to determine the official SPF policy result.
+If there is no SPF record, we check for a local substitute under the
-The offical SPF result is then logged in the Received-SPF header field,
+domain defined in the <code>[spf]delegate</code> configuration.
-but certain results are subjected to further processing to create
+Further checks depend on the result.
-an effective result for policy purposes.
+
-<p>
+<table border=1>
-If the official result is 'none', we try to turn it into an effective result of
+<tr><th>NONE</th><td>
-'pass' or 'fail'.  First, we check for a local substitute SPF record
+If there is no SPF record (official or delegated), then we 
-under the domain defined in the <code>[spf]delegate</code> configuration.  
+initiate a "three strikes and your out" regime, which looks for
-It is often useful to add local SPF records for correspondents that are
+<b>some</b> form of validated identification.
-too clueless to add their own.  If there is no local substitute, we use a "best
+<ol>
-guess" SPF record of "v=spf1 a/24 mx/24 ptr" for MAIL FROM or "v=spf1 a/24
+<li>We try a "best guess" SPF record of "v=spf1 a/24 mx/24 ptr".  If this
-mx/24" for HELO.  In addition, a HELO that is a subdomain of MAIL FROM and
+    passes, good.  
-resolves to the connect IP results in an effective result of 'pass'.
+<li> We try to validate the HELO name. First check for an SPF record.
-<p>
+   Otherwise, check whether the connect IP matches any A record for
-If there is no local SPF record, and the effective result is still not
+   the HELO name, or any A record for any MX name for the HELO name,
-'pass', we check for either a valid HELO name or a valid PTR record for
+   or is at least in the same /24 subnet as any of the above.
-the connect IP.  A valid HELO or PTR cannot look like a dynamic name
+   (In other words, a HELO SPF "best guess" of "v=spf1 a/24 mx/24".)
-as determined by the heuristic in <code>Milter.dynip</code>.
+   If so, good.  We consider the HELO validated.  If the HELO SPF
-<p>
+   check fails, we reject the email.
-If HELO has an SPF record, and the result is anything but pass, we reject
+</ol>
-the connection:
 <pre>
 2005Jul30 19:45:16 [93991] connect from [221.200.41.54] at ('221.200.41.54', 3581) EXTERNAL DYN
 2005Jul30 19:45:18 [93991] hello from adelphia.net
 2005Jul30 19:45:19 [93991] mail from <wendy.stubbsua@link-it.com> ()
 2005Jul30 19:45:19 [93991] REJECT: hello SPF: fail 550 access denied
 </pre>
-Note that HELO does not have any forwarding issues like MAIL FROM, and so
+<ol>
-any result other than 'pass' or 'none' should be treated like 'fail'.
+<li> If there is a validated PTR name, and it doesn't look
+   like a dynamic name, good.  We consider the connection validated.
+</ol>
+If any of the above can be validated, we continue on.
+If none of the above can be validated, and the <code>[SPF]reject_noptr</code>
+option is true, we reject the message immediately with the explanation
+that we need some form of valid identification before we accept an email.
+If <code>[SPF]reject_noptr</code> is false, we flag the message as
+needing Call Back Validation.  
+The Call Back Valildation sends a DSN to the purported sender informing
+them of the lack of identification.  If the message is legitimate, the
+sender needs to know that their email setup is broken and should be corrected.
+If the message is forged, the sender is informed of the forgery, 
+and their need to publish an SPF record or at least use a valid HELO name.
+If the purported sender does not accept the DSN,
+then the message is rejected.  The CBV status is cached to avoid
+annoying the purported sender with too many DSNs.  Currently, the DSN
+is repeated to the same sender once per month.
 <p>
-Only if nothing about the SMTP envelope can be validated does the effective
+In this example, although 3com.com has no SPF record, we assume that 
-result remain 'none.  I call this the "3 strikes" rule.
+any legitimate mail from them will at least have a valid HELO or PTR.
-<p>
+<pre>
-If the official result is 'permerror' (a syntax error in the sender's
+2005Jul30 23:52:03 [96777] connect from [222.252.233.200] at ('222.252.233.200', 29934) EXTERNAL DYN
-policy), we use the 'lax' option in pyspf to try various heuristics to guess 
+2005Jul30 23:52:03 [96777] hello from 3mail.3com.com
-what they really meant.  For instance, the invalid mechanism "ip:1.2.3.4" is
+2005Jul30 23:52:04 [96777] mail from <etec_nic_family@3mail.3com.com> ()
-treated as "ip4:1.2.3.4".  The result of lax processing is then used
+2005Jul30 23:52:04 [96777] REJECT: no PTR, HELO or SPF
-as the effective result for policy purposes.
+</pre>
-<p>
+</td></tr>
-With an effective SPF result in hand, we consult the sendmail access
-database to find our receiver policy for the sender.  
 
-<table border=1>
+<tr><th>PASS</th><td>
-<tr><th>REJECT</th><td>
+A pass result normally lets the email continue on, but the domain is 
-Reject the sender with a 550 5.7.1 SMTP code.  The SMTP rejection
+tracked for reputation (and may be blocked), and may skip content scanning if
-includes a detailed description of the problem.
+it matches a whitelist.
+<pre>
+2005Jul24 17:44:26 [2104] mail from <gnucash-devel-bounces@gnucash.org> ('SIZE=4410',)
+2005Jul24 17:44:26 [2104] Received-SPF: pass (mail.bmsi.com: domain of gnucash.org
+	designates 204.107.200.65 as permitted sender)
+	client-ip=204.107.200.65; envelope-from=gnucash-devel-bounces@gnucash.org; helo=cvs.gnucash.org;
+</pre>
 </td></tr>
-<tr><th>CBV</th><td>
+
-Do a Call Back Validation by connecting to an MX of the sender
+<tr><th>NEUTRAL</th><td>
-and checking that using the sender as the RCPT TO is not rejected.
+A neutral result normally lets the email continue on, but the domain is not
-We quit the CBV connection before actualling sending a message.
+tracked for reputation or matched against any whitelists. 
-If the CBV is rejected, our SMTP connection is rejected with the
+Highly forged domains listed in <code>[SPF]reject_neutral</code> are
-same error code and message.  CBV results are cached.
+rejected.
+<pre>
+2005Jul24 17:41:37 [2070] connect from cp500627-a.dbsch1.nb.home.nl at ('84.27.225.3', 3465) EXTERNAL
+2005Jul24 17:41:37 [2070] hello from cp500627-a.dbsch1.nb.home.nl
+2005Jul24 17:41:38 [2070] mail from <nwarjejkw@yahoo.com> ()
+2005Jul24 17:41:38 [2070] REJECT: SPF neutral for nwarjejkw@yahoo.com
+</pre>
 </td></tr>
-<tr><th>DSN</th><td>
+
-Do a Call Back Validation by connecting to an MX of the sender
+<tr><th>SOFTFAIL</th><td>
-and checking that using the sender as the RCPT TO is not rejected.
+A softfail result normally lets the email continue on, but the domain is not
-Unlike a CBV, we continue on to data and send a detailed message
+tracked for reputation or matched against any whitelists.  Furthermore,
-explaining the problem.  This can be useful for reporting PermError
+the message is flagged as needing Call Back Validation,
-or SoftFail to the sender.  Keep in mind that for any result other
+and the highly forged domains listed in <code>[SPF]reject_neutral</code> are
-than 'pass', the sender could be forged, and your DSN could annoy the
+rejected as well.  
-wrong person.  However, a SoftFail result is requesting such feedback
+<p>
-for debugging and a PermError result needs to be fixed by the sender ASAP
+At present, we also require a valid HELO or PTR to avoid rejecting 
-whether forged or not.  DSN results are cached so that senders are
+a softfail.  But this should probably change to only require a
-annoyed only weekly.
+successful CBV.
+<p>
+The Call Back Valildation sends a DSN to the purported sender informing
+them of the softfail.  If the message is legitimate, the sender needs
+to know about the softfail so that their email setup can be corrected.
+If the message is forged, the sender is informed of the forgery, confirming
+that SPF is protecting their reputation and encouraging a rapid transition
+to a strict policy.  If the purported sender does not accept the DSN,
+then the message is rejected.  The CBV status is cached to avoid
+annoying the purported sender with too many DSNs.  Currently, the DSN
+is repeated to the same sender once per month.
+<pre>
+2005Jul24 15:41:33 [801] mail from <Aitp@horafeliz.com> ()
+2005Jul24 15:41:33 [801] Received-SPF: softfail (mail.bmsi.com: transitioning domain of horafeliz.com
+	does not designate 221.184.83.185 as permitted sender)
+	client-ip=221.184.83.185; envelope-from=Aitp@horafeliz.com;
+	helo=p8185-ipad30funabasi.chiba.ocn.ne.jp;
+2005Jul24 15:41:33 [801] rcpt to <david@example.com> ()
+2005Jul24 15:41:35 [801] Subject: Microsoft, Adobe, Macromedia, Corel software. Up to 80% discount.
+2005Jul24 15:41:35 [801] X-Mailer: Microsoft Outlook, Build 10.0.2605
+2005Jul24 15:41:35 [801] CBV: Aitp@horafeliz.com
+2005Jul24 15:41:38 [801] REJECT: CBV: 550 <Aitp@horafeliz.com>: User unknown
+</pre>
 </td></tr>
-<tr><th>OK</th><td>
+
-Accept the sender.  The message may still be rejected via reputation
+<tr><th>FAIL</th><td>
-or content filtering.
+The message is rejected with a reference the SPF why page.
+<pre>
+2005Jul30 19:53:27 [94070] connect from [212.70.52.16] at ('212.70.52.16', 3192) EXTERNAL DYN
+2005Jul30 19:53:27 [94070] hello from winzip.com
+2005Jul30 19:53:27 [94070] mail from <dan@winzip.com> ()
+2005Jul30 19:53:27 [94070] REJECT: SPF fail 550 SPF fail:
+	see http://openspf.com/why.html?sender=dan@winzip.com&ip=212.70.52.16
+</pre>
 </td></tr>
+
+<tr><th>PERMERROR</th><td>
+Permanent errors were called "unknown", and are still show that way
+in the log.  The message is rejected.  Previously, we enabled "lax" parsing
+of the SPF record, but rejecting is better because it informs the
+sender about their problem.  The next milter version will 
+look for a local substitute SPF record (as for a missing SPF record)
+before rejecting.  This will inform the sender of their problem, but
+also let the receiver install a temporary workaround.
+<pre>
+2005Jul24 18:05:37 [2312] mail from <b-mihdbcgaacaa-becibijh-000-@msg.euxiphipops.com> () 
+2005Jul24 18:05:37 [2312] REJECT: SPF unknown 550 SPF Permanent Error:
+	include mechanism missing domain: include
+</pre>
+The SPF record for msg.euxiphipops.com looked like this at the time of the
+above error:
+<pre>
+msg.euxiphipops.com	TXT "v=spf1 mx ptr a include"
+</pre>
+</td></tr>
+
+<tr><th>TEMPERROR</th><td>
+Temporary errors result in a 451 "Try again later" response.  The sender
+should retry the message at a later time.
+<pre>
+2005Jul24 07:33:13 [29846] mail from <quickenloans@rate.quicken.com> ('SIZE=73775', 'BODY=8BITMIME')
+2005Jul24 07:33:43 [29846] TEMPFAIL: SPF error 450 SPF Temporary Error: DNS Timeout
+</pre>
+</td></tr>
+
 </table>
-
-<h3> SPF policy syntax </h3>
-
-First, the full sender is checked:
-<pre>
-SPF-Fail:abeb@adelphia.net     DSN
-</pre>
-This says to accept mail from that adelphia.net user despite the
-SPF fail, but only after annoying them with a DSN about their ISP's broken
-policy. 
-<p>
-If there is no match on the full sender, the domain is checked:
-<pre>
-SPF-Neutral:aol.com     REJECT
-</pre>
-This says to reject mail from AOL with an SPF result of neutral.
-This means AOL users can't use their AOL address with another mail service
-to send us mail.  This is good because the other mail service is 
-likely a badly configured greeting card site or a virus.
-<p>
-Finally, a default policy for the result is checked.  While there are program
-defaults, you should have defaults in the access database for SPF results:
-<pre>
-SPF-Neutral:            CBV
-SPF-Softfail:           DSN
-SPF-PermError:          DSN
-SPF-TempError:          REJECT
-SPF-None:               REJECT
-SPF-Fail:               REJECT
-SPF-Pass:               OK
-</pre>
-
-<h2> Reputation </h2>
-
-If the sender has not been rejected by this point, and if a GOSSiP server is
-configured, we consult GOSSiP for the reputation score of the sender and
-SPF result.  The score is a number from -100 to 100 with a confidence
-percentage from 0 to 100.  A really bad reputation (less than -50 with
-confidence greater than 3) is rejected.  Note that the reputation is tracked
-independently for each SPF result and sender combination.  So aol.com:neutral
-might have a really bad reputation, while aol.com:pass would be ok.
-Furthermore, when a sender finally publishes an SPF policy and starts
-getting SPF pass, their reputation is effectively reset.
-
-<h2> Whitelists and Blacklists </h2>
-
-The administrator can whitelist or blacklist senders and sending domains by
-appending them to <code>${datadir}/auto_whitelist.log</code> or
-<code>${datadir}/blacklist.log</code> respectively.  In addition,
-recipients of internal senders (except for automatic replies like vacation
-messages and return receipts) are automatically whitelisted for 60 days, and
-senders that fail CBV or DSN checks are automatically blacklisted for 30 days.
-Whitelisted and blacklisted senders are used to automatically train the
-bayesian content filter before being delivered or rejected, respectively.
-<p>
-Real Soon Now users will be able to maintain their own whitelist and
-blacklist that applies only when they are the recipient.
-
-<h2> Content Filter </h2>
-
-Most messages have been rejected or delivered by now, but spammers
-are always finding new places to send their junk from.  For instance,
-we get around 10000 emails a day, of which around 500 are first time
-spam senders.  A bayesian filter is trained by the whitelists and
-blacklists, and scores the message.  What is likely spam is either
-rejected or quarantined.  If the sender is an effective SPF pass,
-then they get a DSN notifying them that their message has been
-quarantined.  (A DSN failure gets the sender auto blacklisted.)
-Else, if the reject_spam option is set, the message is rejected.  
-Otherwise, a CBV is done (failure gets the sender auto blacklisted)
-and the message is silently quarantined.
-<p>
-Normally, you don't want email messages to silently disappear into
-a black hole, so you should set the reject_spam option.  However,
-if you don't want your correspondent's email to get rejected, you can
-check your quarantine frequently instead.
-
-<h3> Honeypot </h3>
-
-You can also blacklist recipients by listing them as aliases of the
-'honeypot' dspam user.  These are collectively called
-the honeypot.  Any email to these recipients is used to train the
-spam filter as spam and chalk up a reputation demerit for the sender, then
-discarded.  It might be a good idea to blacklist the sender if it has SPF pass
-as well, but I'm afraid of accidents.
-
-<h3> Reputation </h3>
-
-Reputation is tracked by sending domain and effective SPF result.
-The GOSSiP server tracks the spam/ham status of the last 1024 messages
-for each domain:result combination.  When the server is queried during
-the SMTP envelope phase (MAIL FROM), it also queries any configured
-peers, and the scores are combined.  Domains with a history of spam for
-a given SPF result are rejected at MAIL FROM.  The GOSSiP system has
-a command line utility to reset (delete) a reputation for cases where a
-sender that was infected with malware is repaired.  In addition,
-the confidence score of a reputation decays with time, so a bad sender
-will eventually be able to try again without manual intervention.

fail.txt

@@ -1,35 +0,0 @@
-To: %(sender)s
-From: postmaster@%(receiver)s
-Subject: SPF fail (EMAIL FORGERY)
-Auto-Submitted: auto-generated (configuration error)
-
-This is an automatically generated Delivery Status Notification.
-
-*** WARNING! YOU ARE SENDING FROM AN UNAUTHORIZED LOCATION ***
-
-The email administrator for '%(sender_domain)' (YOUR administrator)
-has FORBIDDEN you to send email from this location.  IMMEDIATELY contact your
-email administrator and follow his instructions to properly send mail.
-
-THIS IS A WARNING MESSAGE ONLY.
-
-YOU DO *NOT* NEED TO RESEND YOUR MESSAGE.
-
-Delivery to the following recipients has been delayed.
-
-       %(rcpt)s
-
-Subject: %(subject)s
-Received-SPF: %(spf_result)s
-
-Your sender policy indicated that the above email was forged.
-Because we believe your policy is in error, we have accepted the
-email anyway.  Please ask your email administrator to review
-your SPF policy.  You may also have neglected to follow your
-postmaster's instructions for configuring outgoing email.
-
-If you need further assistance, please do not hesitate to contact me.
-
-Kind regards,
-Stuart D Gathman
-postmaster@%(receiver)s

milter.cfg

@@ -1,6 +1,4 @@
 [milter]
-# the directory with log and data files
-datadir = /var/log/milter
 # the socket used to communicate with sendmail.  Must match sendmail.cf
 socket=/var/run/milter/pythonsock
 # where to save original copies of defanged and failed messages
@@ -27,10 +25,7 @@ internal_connect = 192.168.0.0/16,127.*
 ;trusted_relay = 1.2.3.4, 66.12.34.56
 
 # Relaying to these domains is allowed from internal connections only.
-# You might want to restrict aol.com, for instance, so that stupid
+;private_relay = mycorp.com
-# users don't forward their spam to aol for filtering and get your MTA 
-# blacklisted by aol.
-;private_relay = aol.com, yahoo.com
 
 # Reject external senders with hello names no legit external sender would use.
 # SPF will do this also, but listing your own domain and mailserver here
@@ -62,7 +57,7 @@ porn_words = penis, breast, pussy, horse cock, porn, xenical, diet pill, d1ck,
 	p-e-n-i-s, hydrocodone, vicodin, xanax, vicod1n, x@nax, diazepam,
 	v1@gra, xan@x, cialis, ci@lis, frëe, xãnax, valíum, vãlium, via-gra,
 	x@n3x, vicod3n, penís, c0d1n, phentermine, en1arge, dip1oma, v1codin,
-	valium, rolex, sexual, fuck, adv1t, vgaira, medz, acai berry
+	valium, rolex, sexual, fuck, adv1t
 # reject mail with these case sensitive strings in the subject
 spam_words = $$$, !!!, XXX, FREE, HGH
 # attachments with these extensions will be replaced with a warning
@@ -191,11 +186,6 @@ blind = 1
 
 # Map email addresses and aliases to dspam users
 ;dspam_users=david,goliath,spam,falsepositive
-# List dspam users which train on all delivered messages, as opposed to
-# "train on error" which trains only when a spam or falsepositive is reported.
-# Training mode will build the dictionary faster, but requires close attention
-# so as not to miss any spam or false positives.
-;dspam_train=goliath
 ;david=david@foocorp.com,david.yelnetz@foocorp.com,david@bar.foocorp.com
 ;goliath=giant@foocorp.com,goliath.philistine@foocorp.com
 # address to forward spam to.  milter will process these and not deliver
@@ -221,15 +211,7 @@ blind = 1
 # Use a dedicated GOSSiP server.  If not specified, a local database
 # will be used.
 ;server=host:11900
-# To include peers of a peer in reputation, set ttl=2
-;ttl=1
 # If a local database is used, also consult these GOSSiP servers about 
 # domains.  Peer reputation is also tracked as to how often they
 # agree with us, and weighted accordingly.
 ;peers=host1:port,host2
-
-[greylist]
-dbfile=greylist.db
-grey_time=5     # mins (Google retries in 5 mins)
-grey_expire=6   # hours (some legit sites don't retry for 6 hours)
-grey_retain=36  # days (keep "first monday" type mailings on file)

milter.rc

@@ -23,7 +23,7 @@ pidof() {
 # Source function library.
 . /etc/rc.d/init.d/functions
 
-[ -x /usr/lib/pymilter/start.sh ] || exit 0
+[ -x /var/log/milter/start.sh ] || exit 0
 
 RETVAL=0
 prog="milter"
@@ -36,7 +36,7 @@ start() {
 		mkdir -p /var/run/milter
 		chown mail:mail /var/run/milter
 	fi
-	daemon --check milter --user mail /usr/lib/pymilter/start.sh milter bms
+	daemon --check milter --user mail /var/log/milter/start.sh milter bms
 	RETVAL=$?
 	echo
 	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/milter
@@ -46,7 +46,7 @@ start() {
 stop() {
 	# Stop daemons.
 	echo -n "Shutting down $prog: "
-	killproc -d 9 milter
+	killproc milter
 	RETVAL=$?
 	echo
 	[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/milter

milter.rc7

@@ -23,7 +23,7 @@ pidof() {
 # Source function library.
 . /etc/rc.d/init.d/functions
 
-[ -x /usr/lib/pymilter/start.sh ] || exit 0
+[ -x /var/log/milter/start.sh ] || exit 0
 
 RETVAL=0
 prog="milter"
@@ -36,7 +36,7 @@ start() {
 		mkdir -p /var/run/milter
 		chown mail:mail /var/run/milter
 	fi
-	daemon --check milter --user mail /usr/lib/pymilter/start.sh milter bms
+	daemon --check milter --user mail /var/log/milter/start.sh milter bms
 	RETVAL=$?
 	echo
 	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/milter

milter.spec

@@ -1,10 +1,6 @@
-# This spec file contains 2 noarch packages in addition to the pymilter
+%define name pymilter
-# module.  To compile all three on 32-bit Intel, use:
+%define version 0.8.8
-# rpmbuild -ba --target=i386,noarch pymilter.spec
+%define release 1
-
-%define __python python2.4
-%define version 0.8.11
-%define release 1%{?dist}.py24
 # what version of RH are we building for?
 %define redhat7 0
 
@@ -22,58 +18,82 @@
 %define sysvinit milter.rc
 %endif
 # RH9, other systems (single ps line per process)
+%ifos Linux
+%define python python
+%else
+%define python python
+%endif
 %ifos aix4.1
 %define libdir /var/log/milter
 %else
-%define libdir %{_libdir}/pymilter
+%define libdir /usr/lib/pymilter
 %endif
 
-%ifarch noarch
+Summary: Python interface to sendmail milter API
-Name: milter
+Name: %{name}
-Group: Applications/System
-Summary:  BMS spam and reputation milter
 Version: %{version}
 Release: %{release}
-Source: pymilter-%{version}.tar.gz
+Source: %{name}-%{version}.tar.gz
 #Patch: %{name}-%{version}.patch
 License: GPL
 Group: Development/Libraries
 BuildRoot: %{_tmppath}/%{name}-buildroot
+Prefix: %{_prefix}
 Vendor: Stuart D. Gathman <stuart@bmsi.com>
+Packager: Stuart D. Gathman <stuart@bmsi.com>
 Url: http://www.bmsi.com/python/milter.html
-Requires: %{__python} >= 2.4, pyspf >= 2.0.4, pymilter
+Requires: %{python} >= 2.4, sendmail >= 8.13
 %ifos Linux
 Requires: chkconfig
 %endif
+BuildRequires: %{python}-devel >= 2.4, sendmail-devel >= 8.13
+
+%description
+This is a python extension module to enable python scripts to
+attach to sendmail's libmilter functionality.  Additional python
+modules provide for navigating and modifying MIME parts, sending
+DSNs, and doing CBV.
+
+%package -n milter
+Group: Applications/System
+Summary:  BMS spam and reputation milter
+Requires: pyspf >= 2.0.4
 
 %description -n milter
-A complex but effective spam filtering, SPF checking, greylisting,
+A complex but effective spam filtering, SPF checking, and reputation tracking
-and reputation tracking mail application.  It uses pydspam if installed for
+mail application.  It uses pydspam if installed for bayesian filtering.
-bayesian filtering.
 
 %package spf
 Group: Applications/System
 Summary:  BMS spam and reputation milter
-Requires: pyspf >= 2.0.4, pymilter
+Requires: pyspf >= 2.0.4
-Obsoletes: pymilter-spf < 0.8.10
 
 %description spf
 A simple mail filter to add Received-SPF headers and reject forged mail.
-Rejection policy is configured via sendmail access file and can be
+Rejection policy is configured via sendmail access file.
-tailored by domain.
 
 %prep
-%setup -q -n pymilter-%{version}
+%setup
 #patch -p0 -b .bms
 
+%build
+%if %{redhat7}
+  LDFLAGS="-s"
+%else # Redhat builds debug packages after 7.3
+  LDFLAGS="-g"
+%endif
+env CFLAGS="$RPM_OPT_FLAGS" LDFLAGS="$LDFLAGS" %{python} setup.py build
+
 %install
 rm -rf $RPM_BUILD_ROOT
+%{python} setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
+grep '.pyc$' INSTALLED_FILES | sed -e 's/c$/o/' >>INSTALLED_FILES
 mkdir -p $RPM_BUILD_ROOT/var/log/milter
 mkdir -p $RPM_BUILD_ROOT/etc/mail
 mkdir $RPM_BUILD_ROOT/var/log/milter/save
 mkdir -p $RPM_BUILD_ROOT%{libdir}
 cp *.txt $RPM_BUILD_ROOT/var/log/milter
-cp -p bms.py spfmilter.py ban2zone.py $RPM_BUILD_ROOT%{libdir}
+cp bms.py spfmilter.py $RPM_BUILD_ROOT%{libdir}
 cp milter.cfg $RPM_BUILD_ROOT/etc/mail/pymilter.cfg
 cp spfmilter.cfg $RPM_BUILD_ROOT/etc/mail
 
@@ -85,7 +105,7 @@ cat >$RPM_BUILD_ROOT/etc/logrotate.d/milter <<'EOF'
   compress
 }
 /var/log/milter/banned_ips {
-  rotate 7
+  rotate 3
   daily
   copytruncate
 }
@@ -107,14 +127,23 @@ find /var/log/milter/save -mtime +7 | xargs $R rm
 EOF
 chmod a+x $RPM_BUILD_ROOT/etc/cron.daily/milter
 
-%ifnos aix4.1
+%ifos aix4.1
+cat >$RPM_BUILD_ROOT%{libdir}/start.sh <<'EOF'
+#!/bin/sh
+cd /var/log/milter
+# uncomment to enable sgmlop if installed
+#export PYTHONPATH=/usr/local/lib/python2.1/site-packages
+exec /usr/local/bin/python bms.py >>milter.log 2>&1
+EOF
+%else
+cp start.sh $RPM_BUILD_ROOT%{libdir}
 mkdir -p $RPM_BUILD_ROOT/etc/rc.d/init.d
 cp %{sysvinit} $RPM_BUILD_ROOT/etc/rc.d/init.d/milter
 cp spfmilter.rc $RPM_BUILD_ROOT/etc/rc.d/init.d/spfmilter
 ed $RPM_BUILD_ROOT/etc/rc.d/init.d/milter <<'EOF'
 /^python=/
 c
-python="%{__python}"
+python="%{python}"
 .
 w
 q
@@ -122,13 +151,23 @@ EOF
 ed $RPM_BUILD_ROOT/etc/rc.d/init.d/spfmilter <<'EOF'
 /^python=/
 c
-python="%{__python}"
+python="%{python}"
 .
 w
 q
 EOF
-%endif	# aix4.1
+ed $RPM_BUILD_ROOT%{libdir}/start.sh <<'EOF'
+/^python=/
+c
+python="%{python}"
+.
+w
+q
+EOF
+%endif
+chmod a+x $RPM_BUILD_ROOT%{libdir}/start.sh
 
+mkdir -p $RPM_BUILD_ROOT/var/run/milter
 mkdir -p $RPM_BUILD_ROOT/usr/share/sendmail-cf/hack
 cp -p rhsbl.m4 $RPM_BUILD_ROOT/usr/share/sendmail-cf/hack
 
@@ -140,7 +179,7 @@ mkssys -s milter -p %{libdir}/start.sh -u 25 -S -n 15 -f 9 -G mail || :
 if [ $1 = 0 ]; then
   rmssys -s milter || :
 fi
-%else # not aix4.1
+%else
 %post -n milter
 #echo "pythonsock has moved to /var/run/milter, update /etc/mail/sendmail.cf"
 /sbin/chkconfig --add milter
@@ -157,9 +196,17 @@ fi
 if [ $1 = 0 ]; then
   /sbin/chkconfig --del spfmilter
 fi
-%endif # aix4.1
+%endif
 
-%files 
+%clean
+rm -rf $RPM_BUILD_ROOT
+
+%files -f INSTALLED_FILES
+%defattr(-,root,root)
+%doc README HOWTO ChangeLog NEWS TODO CREDITS sample.py milter-template.py
+%config %{libdir}/start.sh
+
+%files -n milter
 %defattr(-,root,root)
 /etc/logrotate.d/milter
 /etc/cron.daily/milter
@@ -172,14 +219,13 @@ fi
 %dir /var/log/milter
 %dir /var/log/milter/save
 %config %{libdir}/bms.py
-%config %{libdir}/ban2zone.py
+%{libdir}/bms.py?
 %config(noreplace) /var/log/milter/strike3.txt
 %config(noreplace) /var/log/milter/softfail.txt
 %config(noreplace) /var/log/milter/fail.txt
 %config(noreplace) /var/log/milter/neutral.txt
 %config(noreplace) /var/log/milter/quarantine.txt
 %config(noreplace) /var/log/milter/permerror.txt
-%config(noreplace) /var/log/milter/temperror.txt
 %config(noreplace) /etc/mail/pymilter.cfg
 /usr/share/sendmail-cf/hack/rhsbl.m4
 
@@ -190,97 +236,7 @@ fi
 %config(noreplace) /etc/mail/spfmilter.cfg
 /etc/rc.d/init.d/spfmilter
 
-%else # not noarch
-
-%define name pymilter
-Summary: Python interface to sendmail milter API
-Name: %{name}
-Version: %{version}
-Release: %{release}
-Source: %{name}-%{version}.tar.gz
-#Patch: %{name}-%{version}.patch
-License: GPL
-Group: Development/Libraries
-BuildRoot: %{_tmppath}/%{name}-buildroot
-Vendor: Stuart D. Gathman <stuart@bmsi.com>
-Url: http://www.bmsi.com/python/milter.html
-Requires: %{__python} >= 2.4, sendmail >= 8.13
-BuildRequires: %{__python}-devel >= 2.4, sendmail-devel >= 8.13
-
-%description
-This is a python extension module to enable python scripts to
-attach to sendmail's libmilter functionality.  Additional python
-modules provide for navigating and modifying MIME parts, sending
-DSNs, and doing CBV.
-
-%prep
-%setup -q
-#patch -p0 -b .bms
-
-%build
-%if %{redhat7} 
-  LDFLAGS="-s"
-%else # Redhat builds debug packages after 7.3
-  LDFLAGS="-g"
-%endif
-env CFLAGS="$RPM_OPT_FLAGS" LDFLAGS="$LDFLAGS" %{__python} setup.py build
-
-%install
-rm -rf $RPM_BUILD_ROOT
-%{__python} setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
-mkdir -p $RPM_BUILD_ROOT/var/run/milter
-mkdir -p $RPM_BUILD_ROOT%{libdir}
-%ifos aix4.1
-cat >$RPM_BUILD_ROOT%{libdir}/start.sh <<'EOF'
-#!/bin/sh
-cd /var/log/milter
-exec /usr/local/bin/python bms.py >>milter.log 2>&1
-EOF
-%else # not aix4.1
-cp start.sh $RPM_BUILD_ROOT%{libdir}
-ed $RPM_BUILD_ROOT%{libdir}/start.sh <<'EOF'
-/^python=/
-c
-python="%{__python}"
-.
-w
-q
-EOF
-%endif
-chmod a+x $RPM_BUILD_ROOT%{libdir}/start.sh
-%if !%{redhat7}
-#grep '.pyc$' INSTALLED_FILES | sed -e 's/c$/o/' >>INSTALLED_FILES
-%endif
-
-# start.sh is used by spfmilter and milter, and could be used by
-# other milters running on redhat
-%files -f INSTALLED_FILES
-%defattr(-,root,root)
-%doc README HOWTO ChangeLog NEWS TODO CREDITS sample.py milter-template.py
-%config %{libdir}/start.sh
-%dir %attr(0755,mail,mail) /var/run/milter
-
-%endif # noarch
-
-%clean
-rm -rf $RPM_BUILD_ROOT
-
 %changelog
-* Mon Aug 25 2008 Stuart Gathman <stuart@bmsi.com> 0.8.10-2
-- /var/run/milter directory must be owned by mail
-* Mon Aug 25 2008 Stuart Gathman <stuart@bmsi.com> 0.8.10-1
-- log rcpt for SRS rejections
-- improved parsing into email and fullname (still 2 self test failures)
-- implement no-DSN CBV, reduce full DSNs
-- check for porn words in MAIL FROM fullname
-- ban IP for too many bad MAIL FROMs or RCPT TOs
-- temperror policy in access
-- no CBV for whitelisted MAIL FROM except permerror, softfail
-- Allow explicitly whitelisted email from banned_users.
-- configure gossip TTL
-* Mon Sep 24 2007 Stuart Gathman <stuart@bmsi.com> 0.8.9-1
-- Use ifarch hack to build milter and milter-spf packages as noarch
-- Remove spf dependency from dsn.py, add dns.py
 * Fri Jan 05 2007 Stuart Gathman <stuart@bmsi.com> 0.8.8-1
 - move AddrCache, parse_addr, iniplist to Milter package
 - move parse_header to Milter.utils
@@ -296,7 +252,6 @@ rm -rf $RPM_BUILD_ROOT
 - SPF moved to pyspf RPM
 - wiretap archive option
 - Do plain CBV if missing template
-- SMTP AUTH policy in access
 * Tue May 23 2006 Stuart Gathman <stuart@bmsi.com> 0.8.6-2
 - Support CBV timeout
 - Support fail template, headers in templates

miltermodule.c

@@ -1,20 +1,19 @@
 /* Copyright (C) 2001  James Niemira (niemira@colltech.com, urmane@urmane.org)
- * Portions Copyright (C) 2001,2002,2003,2004,2005,2006,2007
+ * Portions Copyright (C) 2001,2002,2003,2004  Stuart Gathman (stuart@bmsi.com)
- *   Stuart Gathman (stuart@bmsi.com)
  * 
- * This program is free software: you can redistribute it and/or modify it
+ * This program is free software; you can redistribute it and/or
- * under the terms of the GNU General Public License as published by the
+ * modify it under the terms of the GNU General Public License
- * Free Software Foundation, either version 2 of the License, or (at your
+ * as published by the Free Software Foundation; either version 2
- * option) any later version.
+ * of the License, or (at your option) any later version.
  * 
- * This program is distributed in the hope that it will be useful, but
+ * This program is distributed in the hope that it will be useful,
- * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * General Public License for more details.
+ * GNU General Public License for more details.
  * 
- * You should have received a copy of the GNU General Public License along
+ * You should have received a copy of the GNU General Public License
- * with this program; if not, write to the Free Software Foundation, Inc.,
+ * along with this program; if not, write to the Free Software
- * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
  *
  * milterContext object and thread interface contributed by
  * 	Stuart D. Gathman <stuart@bmsi.com>
@@ -35,9 +34,6 @@ $ python setup.py help
      libraries=["milter","smutil","resolv"]
 
  * $Log$
- * Revision 1.10  2006/02/12 02:00:42  customdesigned
- * Resolve FIXME for wrap_close.
- *
  * Revision 1.9  2005/12/23 21:46:36  customdesigned
  * Compile on sendmail-8.12 (ifdef SMFIR_INSHEADER)
  *

quarantine.txt

@@ -22,6 +22,19 @@ their quarantined mail and may notice your message.  If your message is
 important, please contact them via other means.  You may also try sending
 them a simple plain text message.
 
+If you never sent the above message, then your domain, %(sender_domain)s,
+was forged - i.e. used without your knowlege or authorization by
+someone attempting to steal your mail identity.  This is a very
+serious problem, and you need to provide authentication for your
+SMTP (email) servers to prevent criminals from forging your
+domain.  The simplest step is usually to publish an SPF record
+with your Sender Policy.  
+
+For more information, see: http://www.openspf.org
+
+Your mail admin needs to publish a strict SPF record so that I can reject
+those forgeries instead of bugging you with them.
+
 If you need further assistance, please do not hesitate to contact me.
 
 Kind regards,

setup.py

@@ -6,7 +6,6 @@ from distutils.core import setup, Extension
 # on slackware and debian, leave it out entirely.  It depends
 # on how libmilter was built by the sendmail package.
 libs = ["milter", "smutil"]
-libdirs = ["/usr/lib/libmilter"]    # needed for Debian
 
 # patch distutils if it can't cope with the "classifiers" or
 # "download_url" keywords
@@ -16,7 +15,7 @@ if sys.version < '2.2.3':
   DistributionMetadata.download_url = None
 
 # NOTE: importing Milter to obtain version fails when milter.so not built
-setup(name = "pymilter", version = '0.8.11',
+setup(name = "pymilter", version = '0.8.8',
 	description="Python interface to sendmail milter API",
 	long_description="""\
 This is a python extension module to enable python scripts to
@@ -34,7 +33,6 @@ sending DSNs or doing CBVs.
 	packages = ['Milter'],
 	ext_modules=[
 	  Extension("milter", ["miltermodule.c"],
-            library_dirs=libdirs,
 	    libraries=libs,
 	    # set MAX_ML_REPLY to 1 for sendmail < 8.13
 	    define_macros = [ ('MAX_ML_REPLY',32) ]

start.sh

@@ -10,5 +10,7 @@ else
   cd /usr/lib/pymilter
 fi
 
-${python} ${script}.py &
+cd /var/log/milter
+exec >>${appname}.log 2>&1
+${python} ${appname}.py &
 echo $! >/var/run/milter/${appname}.pid

temperror.txt

@@ -1,33 +0,0 @@
-To: %(sender)s
-From: postmaster@%(receiver)s
-Subject: Critical DNS configuration error
-Auto-Submitted: auto-generated (configuration error)
-
-This is an automatically generated Delivery Status Notification.
-
-THIS IS A WARNING MESSAGE ONLY.
-
-YOU DO *NOT* NEED TO RESEND YOUR MESSAGE.
-
-Delivery to the following recipients has been delayed.
-
-	%(rcpt)s
-
-Subject: %(subject)s 
-Received-SPF: %(spf_result)s
-
-Your DNS server is not responding to TXT queries.  In other words,
-it is BROKEN.  You need to get somebody to fix it ASAP.  We
-are attempting to do TXT queries to see if you have an SPF record.
-
-See http://openspf.org
-
-We are sending you this message to alert you to the fact that
-you have problems with your DNS.
-
-If you need further assistance, please do not hesitate to
-contact me again.
-
-Kind regards,
-
-postmaster@%(receiver)s

testbms.py

@@ -44,7 +44,7 @@ class TestMilter(bms.bmsMilter):
       self._msg[field] = value
     self.headerschanged = True
 
-  def addheader(self,field,value,idx=-1):
+  def addheader(self,field,value):
     if not self._body:
       raise IOError,"addheader not called from eom()"
     self.log('addheader: %s=%s' % (field,value))