This commit was manufactured by cvs2svn to create tag 'pymilter-0_8_10'.

Sprout from master 2008-08-25 22:03:24 UTC Stuart Gathman <stuart@gathman.org> 'Fix /var/run/milter owner' Cherrypick from bmsi 2005-05-31 18:23:49 UTC Stuart Gathman <stuart@gathman.org> 'Development changes since 0.7.2': rejects.py rhsbl.m4 sample.py test/amazon test/big5 test/bounce test/bounce1 test/bound test/honey test/missingboundary test/samp1 test/spam44 test/spam7 test/spam8 test/test1 test/test8 test/virus1 test/virus13 test/virus2 test/virus3 test/virus4 test/virus5 test/virus6 test/virus7 testsample.py
Fix /var/run/milter owner
2008-08-25 22:03:25 +00:00 · 2008-08-25 22:03:24 +00:00 · 2008-08-25 22:02:39 +00:00 · 2008-08-25 22:00:46 +00:00 · 2008-08-25 21:41:18 +00:00 · 2008-08-25 20:00:51 +00:00
48 changed files with 3516 additions and 2879 deletions
@@ -1,8 +1,8 @@
 		    GNU GENERAL PUBLIC LICENSE
 		       Version 2, June 1991

- Copyright (C) 1989, 1991 Free Software Foundation, Inc.
-                       59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.

@@ -15,7 +15,7 @@ software--to make sure the software is free for all its users.  This
 General Public License applies to most of the Free Software
 Foundation's software and to any other program whose authors commit to
 using it.  (Some other Free Software Foundation software is covered by
-the GNU Library General Public License instead.)  You can apply it to
+the GNU Lesser General Public License instead.)  You can apply it to
 your programs, too.

  When we speak of free software, we are referring to freedom, not
@@ -55,7 +55,7 @@ patent must be licensed for everyone's free use or not licensed at all.

  The precise terms and conditions for copying, distribution and
 modification follow.
-
+
 		    GNU GENERAL PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

@@ -110,7 +110,7 @@ above, provided that you also meet all of these conditions:
    License.  (Exception: if the Program itself is interactive but
    does not normally print such an announcement, your work based on
    the Program is not required to print an announcement.)
-
+
 These requirements apply to the modified work as a whole.  If
 identifiable sections of that work are not derived from the Program,
 and can be reasonably considered independent and separate works in
@@ -168,7 +168,7 @@ access to copy from a designated place, then offering equivalent
 access to copy the source code from the same place counts as
 distribution of the source code, even though third parties are not
 compelled to copy the source along with the object code.
-
+
  4. You may not copy, modify, sublicense, or distribute the Program
 except as expressly provided under this License.  Any attempt
 otherwise to copy, modify, sublicense or distribute the Program is
@@ -225,7 +225,7 @@ impose that choice.

 This section is intended to make thoroughly clear what is believed to
 be a consequence of the rest of this License.
-
+
  8. If the distribution and/or use of the Program is restricted in
 certain countries either by patents or by copyrighted interfaces, the
 original copyright holder who places the Program under this License
@@ -278,7 +278,7 @@ PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
 POSSIBILITY OF SUCH DAMAGES.

 		     END OF TERMS AND CONDITIONS
-
+
 	    How to Apply These Terms to Your New Programs

  If you develop a new program, and you want it to be of the greatest
@@ -303,10 +303,9 @@ the "copyright" line and a pointer to where the full notice is found.
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

-    You should have received a copy of the GNU General Public License
-    along with this program; if not, write to the Free Software
-    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
-
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

 Also add information on how to contact you by electronic and paper mail.

@@ -336,5 +335,5 @@ necessary.  Here is a sample; alter the names:
 This General Public License does not permit incorporating your program into
 proprietary programs.  If your program is a subroutine library, you may
 consider it more useful to permit linking proprietary applications with the
-library.  If this is what you want to do, use the GNU Library General
+library.  If this is what you want to do, use the GNU Lesser General
 Public License instead of this License.
@@ -7,6 +7,8 @@ real, usable Python extension.

 Other contributors (in random order):

+Dwayne Litzenberger, B.A.Sc.
+  for library_dirs patch to compile on Debian 
 Dave MacQuigg 
  for noticing that smfi_insheader wasn't supported, and creating
  a template to help first time pymilter users create their own milter.
@@ -0,0 +1,214 @@
+# Revision 1.69  2006/11/04 22:09:39  customdesigned
+# Another lame DSN heuristic.  Block PTR cache poisoning attack.
+#
+# Revision 1.68  2006/10/04 03:46:01  customdesigned
+# Fix defaults.
+#
+# Revision 1.67  2006/10/01 01:44:06  customdesigned
+# case_sensitive_localpart option, more delayed bounce heuristics,
+# optional smart_alias section.
+#
+# Revision 1.66  2006/07/26 16:42:26  customdesigned
+# Support CBV timeout
+#
+# Revision 1.65  2006/06/21 22:22:00  customdesigned
+# Handle multi-line headers in delayed dsns.
+#
+# Revision 1.64  2006/06/21 21:12:04  customdesigned
+# More delayed reject token headers.
+# Don't require HELO pass for CBV.
+#
+# Revision 1.63  2006/05/21 03:41:44  customdesigned
+# Fail dsn
+#
+# Revision 1.61  2006/05/17 21:28:07  customdesigned
+# Create GOSSiP record only when connection will procede to DATA.
+#
+# Revision 1.60  2006/05/12 16:14:48  customdesigned
+# Don't require SPF pass for white/black listing mail from trusted relay.
+# Support localpart wildcard for white and black lists.
+#
+# Revision 1.59  2006/04/06 18:14:17  customdesigned
+# Check whitelist/blacklist even when not checking SPF (e.g. trusted relay).
+#
+# Revision 1.58  2006/03/10 20:52:49  customdesigned
+# Use re to recognize failure DSNs.
+#
+# Revision 1.57  2006/03/07 20:50:54  customdesigned
+# Use signed Message-ID in delayed reject to blacklist senders
+#
+# Revision 1.56  2006/02/24 02:12:54  customdesigned
+# Properly report hard PermError (lax mode fails also) by always setting
+# perm_error attribute with PermError exception.  Improve reporting of
+# invalid domain PermError.
+#
+# Revision 1.55  2006/02/17 05:04:29  customdesigned
+# Use SRS sign domain list.
+# Accept but do not use for training whitelisted senders without SPF pass.
+# Immediate rejection of unsigned bounces.
+#
+# Revision 1.54  2006/02/16 02:16:36  customdesigned
+# User specific SPF receiver policy.
+#
+# Revision 1.53  2006/02/12 04:15:01  customdesigned
+# Remove spf dependency for iniplist
+#
+# Revision 1.52  2006/02/12 02:12:08  customdesigned
+# Use CIDR notation for internal connect list.
+#
+# Revision 1.51  2006/02/12 01:13:58  customdesigned
+# Don't check rcpt user list when signed MFROM.
+#
+# Revision 1.50  2006/02/09 20:39:43  customdesigned
+# Use CIDR notation for trusted_relay iplist
+#
+# Revision 1.49  2006/01/30 23:14:48  customdesigned
+# put back eom condition
+#
+# Revision 1.48  2006/01/12 20:31:24  customdesigned
+# Accelerate training via whitelist and blacklist.
+#
+# Revision 1.47  2005/12/29 04:49:10  customdesigned
+# Do not auto-whitelist autoreplys
+#
+# Revision 1.46  2005/12/28 20:17:29  customdesigned
+# Expire and renew AddrCache entries
+#
+# Revision 1.45  2005/12/23 22:34:46  customdesigned
+# Put guessed result in separate header.
+#
+# Revision 1.44  2005/12/23 21:47:07  customdesigned
+# Move Received-SPF header to top.
+#
+# Revision 1.43  2005/12/09 16:54:01  customdesigned
+# Select neutral DSN template for best_guess
+#
+# Revision 1.42  2005/12/01 22:42:32  customdesigned
+# improve gossip support.
+# Initialize srs_domain from srs.srs config property.  Should probably
+# always block unsigned DSN when signing all.
+#
+# Revision 1.41  2005/12/01 18:59:25  customdesigned
+# Fix neutral policy.  pobox.com -> openspf.org
+#
+# Revision 1.40  2005/11/07 21:22:35  customdesigned
+# GOSSiP support, local database only.
+#
+# Revision 1.39  2005/10/31 00:04:58  customdesigned
+# Simple implementation of trusted_forwarder list.  Inefficient for
+# more than 1 or 2 entries.
+#
+# Revision 1.38  2005/10/28 19:36:54  customdesigned
+# Don't check internal_domains for trusted_relay.
+#
+# Revision 1.37  2005/10/28 09:30:49  customdesigned
+# Do not send quarantine DSN when sender is DSN.
+#
+# Revision 1.36  2005/10/23 16:01:29  customdesigned
+# Consider MAIL FROM a match for supply_sender when a subdomain of From or Sender
+#
+# Revision 1.35  2005/10/20 18:47:27  customdesigned
+# Configure auto_whitelist senders.
+#
+# Revision 1.34  2005/10/19 21:07:49  customdesigned
+# access.db stores keys in lower case
+#
+# Revision 1.33  2005/10/19 19:37:50  customdesigned
+# Train screener on whitelisted messages.
+#
+# Revision 1.32  2005/10/14 16:17:31  customdesigned
+# Auto whitelist refinements.
+#
+# Revision 1.31  2005/10/14 01:14:08  customdesigned
+# Auto whitelist feature.
+#
+# Revision 1.30  2005/10/12 16:36:30  customdesigned
+# Release 0.8.3
+#
+# Revision 1.29  2005/10/11 22:50:07  customdesigned
+# Always check HELO except for SPF pass, temperror.
+#
+# Revision 1.28  2005/10/10 23:50:20  customdesigned
+# Use logging module to make logging threadsafe (avoid splitting log lines)
+#
+# Revision 1.27  2005/10/10 20:15:33  customdesigned
+# Configure SPF policy via sendmail access file.
+#
+# Revision 1.26  2005/10/07 03:23:40  customdesigned
+# Banned users option.  Experimental feature to supply Sender when
+# missing and MFROM domain doesn't match From.  Log cipher bits for
+# SMTP AUTH.  Sketch access file feature.
+#
+# Revision 1.25  2005/09/08 03:55:08  customdesigned
+# Handle perverse MFROM quoting.
+#
+# Revision 1.24  2005/08/18 03:36:54  customdesigned
+# Don't innoculate with SCREENED mail.
+#
+# Revision 1.23  2005/08/17 19:35:27  customdesigned
+# Send DSN before adding message to quarantine.
+#
+# Revision 1.22  2005/08/11 22:17:58  customdesigned
+# Consider SMTP AUTH connections internal.
+#
+# Revision 1.21  2005/08/04 21:21:31  customdesigned
+# Treat fail like softfail for selected (braindead) domains.
+# Treat mail according to extended processing results, but
+# report any PermError that would officially result via DSN.
+#
+# Revision 1.20  2005/08/02 18:04:35  customdesigned
+# Keep screened honeypot mail, but optionally discard honeypot only mail.
+#
+# Revision 1.19  2005/07/20 03:30:04  customdesigned
+# Check pydspam version for honeypot, include latest pyspf changes.
+#
+# Revision 1.18  2005/07/17 01:25:44  customdesigned
+# Log as well as use extended result for best guess.
+#
+# Revision 1.17  2005/07/15 20:25:36  customdesigned
+# Use extended results processing for best_guess.
+#
+# Revision 1.16  2005/07/14 03:23:33  customdesigned
+# Make SES package optional.  Initial honeypot support.
+#
+# Revision 1.15  2005/07/06 04:05:40  customdesigned
+# Initial SES integration.
+#
+# Revision 1.14  2005/07/02 23:27:31  customdesigned
+# Don't match hostnames for internal connects.
+#
+# Revision 1.13  2005/07/01 16:30:24  customdesigned
+# Always log trusted Received and Received-SPF headers.
+#
+# Revision 1.12  2005/06/20 22:35:35  customdesigned
+# Setreply for rejectvirus.
+#
+# Revision 1.11  2005/06/17 02:07:20  customdesigned
+# Release 0.8.1
+#
+# Revision 1.10  2005/06/16 18:35:51  customdesigned
+# Ignore HeaderParseError decoding header
+#
+# Revision 1.9  2005/06/14 21:55:29  customdesigned
+# Check internal_domains for outgoing mail.
+#
+# Revision 1.8  2005/06/06 18:24:59  customdesigned
+# Properly log exceptions from pydspam
+#
+# Revision 1.7  2005/06/04 19:41:16  customdesigned
+# Fix bugs from testing RPM
+#
+# Revision 1.6  2005/06/03 04:57:05  customdesigned
+# Organize config reader by section.  Create defang section.
+#
+# Revision 1.5  2005/06/02 15:00:17  customdesigned
+# Configure banned extensions.  Scan zipfile option with test case.
+#
+# Revision 1.4  2005/06/02 04:18:55  customdesigned
+# Update copyright notices after reading article on /.
+#
+# Revision 1.3  2005/06/02 02:09:00  customdesigned
+# Record timestamp in send_dsn.log
+#
+# Revision 1.2  2005/06/02 01:00:36  customdesigned
+# Support configurable templates for DSNs.
@@ -1,3 +1,10 @@
+On Sun, 11 Feb 2007, Rick Saul wrote:
+
+> Stuart I was planning to move to centos4.4 in a couple of weeks anyway... 
+> Your advice of where to go from here. 
+
+Oh - you are asking for a howto.
+
 	Step one.  Which DSPAM is right for you?

 The DSPAM project makes dspam part of the LDA (Local Delivery Agent).
@@ -28,39 +35,42 @@ wish to install pydspam.
 For basic pymilter you'll need:

 python-2.4
-milter-0.8.2	(the RH9 rpm should work on Fedora Core - let me know)
+milter-0.8.10
 sendmail-8.13.x	(with milter support enabled)

 and for SPF you'll need:

-pydns-2.3.0-2.4
+pydns-2.3.3-2.4
+pyspf-2.0.5-1.py24

 and for SRS you'll need:

-pysrs-0.30.9-1.py24
+pysrs-0.30.11-1.py24

 I'm pretty sure you will want to have SPF and SRS available.

 	Step three.  Activate basic milter.

-Activate the basic milter by editing /etc/mail/sendmail.mc and adding:
+Activate the basic milter and pysrs by editing /etc/mail/sendmail.mc and adding:

+define(`NO_SRS_FILE',`/etc/mail/no-srs-mailers')dnl
+dnl define(`NO_SRS_FROM_LOCAL')dnl
+HACK(`pysrs',`/var/run/milter/pysrs')dnl
 INPUT_MAIL_FILTER(`pythonfilter', `S=local:/var/run/milter/pythonsock, F=T, T=C:5m;S:20s;R:5m;E:5m')

 You can then "make sendmail.cf" and restart sendmail.

+Start milter and pysrs with "service milter start", "service pysrs start".
+
 Tail /var/log/milter/milter.log while SMTP clients connect to your
 sendmail instance.  This should show you what the milter is doing.

-By default, milter-0.8.2 rejects on SPF fail, except for listed domains
-(that are known to be broken).  Some admins don't like that, and 0.8.3 will use
-the /etc/mail/access database to configure SPF responses.  For now,
-if you don't like SPF, you can disable spf by replacing "import spf"
-with "spf = None" around line 285 in /var/log/milter/bms.py.
+By default, milter-0.8.10 rejects on SPF fail.

 	Step four.  Tweaking the basic config.

-Most pymilter configuration is in /etc/mail/pymilter.cfg.
+Most pymilter configuration is in /etc/mail/pymilter.cfg.  To activate
+changes, "service milter restart".

 By default, milter scans attachments for executable extensions.  You can
 turn this off by setting banned_exts to the empty list.  There are options
@@ -76,7 +86,9 @@ should also run pymilter with similar policies.  (But this isn't
 needed for initial testing.)

 Configure internal_connect with subnets of your internal SMTP clients.
-Internal connections skip SPF testing and other policies.
+Internal connections skip SPF testing and other policies.  You will
+likely need to set this to allow outgoing mail if you have
+an SPF policy already.

 Configure internal_domains with domains used by your internal SMTP clients.
 If they attempt to use any other domain, the attempt is blocked and the
@@ -116,7 +128,7 @@ The CBV policy requires a valid HELO name.  If the EHLO name is
 RFC2822 compliant, then a DSN is sent to the alleged sender.  The 
 template for the DSN is selected according to the SPF result:

-Fail:		softfail.txt
+Fail:		fail.txt
 SoftFail:	softfail.txt
 Neutral:	neutral.txt
 PermError:	permerror.txt
@@ -134,3 +146,9 @@ SRS config

 pydspam config
 wiretap config
+
+-- 
+	      Stuart D. Gathman <stuart@bmsi.com>
+    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
+"Confutatis maledictis, flammis acribus addictis" - background song for
+a Microsoft sponsored "Where do you want to go from here?" commercial.
@@ -4,11 +4,12 @@ include NEWS
 include HOWTO
 include CREDITS
 include README
+include ChangeLog
 include MANIFEST.in
 include testsample.py
 include testmime.py
+include testutils.py
 include testbms.py
-include testdspam.py
 include rejects.py
 include report.py
 include bms.py
@@ -17,6 +18,10 @@ include cid2spf.py
 include spfquery.py
 include test.py
 include sample.py
+include milter-template.py
+include spfmilter.py
+include spfmilter.rc
+include spfmilter.cfg
 include test/*
 include doc/*
 include Milter/*.py
@@ -9,7 +9,7 @@ import milter
 import thread

 from milter import ACCEPT,CONTINUE,REJECT,DISCARD,TEMPFAIL,	\
-  set_flags, setdbg, setbacklog, settimeout, \
+  set_flags, setdbg, setbacklog, settimeout, error,	\
  ADDHDRS, CHGBODY, ADDRCPT, DELRCPT, CHGHDRS,	\
  V1_ACTS, V2_ACTS, CURR_ACTS

@@ -184,8 +184,10 @@ def runmilter(name,socketname,timeout = 0):
    print "Removing %s" % fname
    try:
      os.unlink(fname)
-    except:
-      pass
+    except os.error, x:
+      import errno
+      if x.errno != errno.ENOENT:
+        raise milter.error(x)

  # The default flags set include everything
  # milter.set_flags(milter.ADDHDRS)
@@ -0,0 +1,158 @@
+# Email address list with expiration
+#
+# This class acts like a map.  Entries with a value of None are persistent,
+# but disappear after a time limit.  This is useful for automatic whitelists
+# and blacklists with expiration.  The persistent store is a simple ascii
+# file with sender and timestamp on each line.  Entries can be appended
+# to the store, and will be picked up the next time it is loaded.
+#
+# Entries with other values are not persistent.  This is used to hold failed
+# CBV results.
+#
+# $Log$
+# Revision 1.8  2007/09/03 16:18:45  customdesigned
+# Delete unparseable timestamps when loading address cache.  These have
+# arisen because of failure to parse MAIL FROM properly.   Will have to
+# tighten up MAIL FROM parsing to match RFC.
+#
+# Revision 1.7  2007/01/25 22:47:26  customdesigned
+# Persist blacklisting from delayed DSNs.
+#
+# Revision 1.6  2007/01/19 23:31:38  customdesigned
+# Move parse_header to Milter.utils.
+# Test case for delayed DSN parsing.
+# Fix plock when source missing or cannot set owner/group.
+#
+# Revision 1.5  2007/01/11 19:59:40  customdesigned
+# Purge old entries in auto_whitelist and send_dsn logs.
+#
+# Revision 1.4  2007/01/11 04:31:26  customdesigned
+# Negative feedback for bad headers.  Purge cache logs on startup.
+#
+# Revision 1.3  2007/01/08 23:20:54  customdesigned
+# Get user feedback.
+#
+# Revision 1.2  2007/01/05 23:33:55  customdesigned
+# Make blacklist an AddrCache
+#
+# Revision 1.1  2007/01/05 21:25:40  customdesigned
+# Move AddrCache to Milter package.
+#
+
+# Author: Stuart D. Gathman <stuart@bmsi.com>
+# Copyright 2001,2002,2003,2004,2005 Business Management Systems, Inc.
+# This code is under the GNU General Public License.  See COPYING for details.
+
+import time
+from plock import PLock
+
+class AddrCache(object):
+  time_format = '%Y%b%d %H:%M:%S %Z'
+
+  def __init__(self,renew=7,fname=None):
+    self.age = renew
+    self.cache = {}
+    self.fname = fname
+
+  def load(self,fname,age=0):
+    "Load address cache from persistent store."
+    if not age:
+      age = self.age
+    self.fname = fname
+    cache = {}
+    self.cache = cache
+    now = time.time()
+    lock = PLock(self.fname)
+    wfp = lock.lock()
+    changed = False
+    try:
+      too_old = now - age*24*60*60	# max age in days
+      try:
+        fp = open(self.fname)
+      except OSError:
+        fp = ()
+      for ln in fp:
+	try:
+	  rcpt,ts = ln.strip().split(None,1)
+          try:
+            l = time.strptime(ts,AddrCache.time_format)
+            t = time.mktime(l)
+            if t < too_old:
+              changed = True
+              continue
+            cache[rcpt.lower()] = (t,None)
+          except:       # unparsable timestamp - likely garbage
+            changed = True
+            continue
+	except: # manual entry (no timestamp)
+	  cache[ln.strip().lower()] = (now,None)
+	wfp.write(ln)
+      if changed:
+	lock.commit(self.fname+'.old')
+      else:
+        lock.unlock()
+    except IOError:
+      lock.unlock()
+
+  def has_precise_key(self,sender):
+    """True if precise sender is cached and has not expired.  Don't
+    try looking up wildcard entries.
+    """
+    try:
+      lsender = sender and sender.lower()
+      ts,res = self.cache[lsender]
+      too_old = time.time() - self.age*24*60*60	# max age in days
+      if not ts or ts > too_old:
+        return True
+      del self.cache[lsender]
+    except KeyError: pass
+    return False
+
+  def has_key(self,sender):
+    "True if sender is cached and has not expired."
+    if self.has_precise_key(sender):
+      return True
+    try:
+      user,host = sender.split('@',1)
+      return self.has_precise_key(host)
+    except: pass
+    return False
+
+  __contains__ = has_key
+
+  def __getitem__(self,sender):
+    try:
+      lsender = sender.lower()
+      ts,res = self.cache[lsender]
+      too_old = time.time() - self.age*24*60*60	# max age in days
+      if not ts or ts > too_old:
+	return res
+      del self.cache[lsender]
+      raise KeyError, sender
+    except KeyError,x:
+      try:
+	user,host = sender.split('@',1)
+	return self.__getitem__(host)
+      except ValueError:
+        raise x
+
+  def addperm(self,sender,res=None):
+    "Add a permanent sender."
+    lsender = sender.lower()
+    if self.has_key(lsender):
+      ts,res = self.cache[lsender]
+      if not ts: return		# already permanent
+    self.cache[lsender] = (None,res)
+    if not res:
+      print >>open(self.fname,'a'),sender
+    
+  def __setitem__(self,sender,res):
+    lsender = sender.lower()
+    now = time.time()
+    self.cache[lsender] = (now,res)
+    if not res and self.fname:
+      s = time.strftime(AddrCache.time_format,time.localtime(now))
+      print >>open(self.fname,'a'),sender,s # log refreshed senders
+
+  def __len__(self):
+    return len(self.cache)
@@ -0,0 +1,59 @@
+from ConfigParser import ConfigParser
+
+class MilterConfigParser(ConfigParser):
+
+  def __init__(self,defaults={}):
+    ConfigParser.__init__(self)
+    self.defaults = defaults
+
+  # The defaults provided by ConfigParser show up in all sections,
+  # which screws up iterating over all options in a section.
+  # Worse, passing "defaults" with vars= overrides the config file!
+  # So we roll our own defaults.
+  def get(self,sect,opt):
+    if not self.has_option(sect,opt) and opt in self.defaults:
+      return self.defaults[opt]
+    return ConfigParser.get(self,sect,opt)
+    
+  def getlist(self,sect,opt):
+    if self.has_option(sect,opt):
+      return [q.strip() for q in self.get(sect,opt).split(',')]
+    return []
+
+  def getaddrset(self,sect,opt):
+    if not self.has_option(sect,opt):
+      return {}
+    s = self.get(sect,opt)
+    d = {}
+    for q in s.split(','):
+      q = q.strip()
+      if q.startswith('file:'):
+        domain = q[5:].lower()
+	d[domain] = d.setdefault(domain,[]) + open(domain,'r').read().split()
+      else:
+	user,domain = q.split('@')
+	d.setdefault(domain.lower(),[]).append(user)
+    return d
+  
+  def getaddrdict(self,sect,opt):
+    if not self.has_option(sect,opt):
+      return {}
+    d = {}
+    for q in self.get(sect,opt).split(','):
+      q = q.strip()
+      if self.has_option(sect,q):
+        l = self.get(sect,q)
+	for addr in l.split(','):
+	  addr = addr.strip()
+	  if addr.startswith('file:'):
+	    fname = addr[5:]
+	    for a in open(fname,'r').read().split():
+	      d[a] = q
+	  else:
+	    d[addr] = q
+    return d
+
+  def getdefault(self,sect,opt,default=None):
+    if self.has_option(sect,opt):
+      return self.get(sect,opt)
+    return default
@@ -0,0 +1,88 @@
+# provide a higher level interface to pydns
+
+import DNS
+from DNS import DNSError
+
+MAX_CNAME = 10
+
+def DNSLookup(name, qtype):
+    try:
+        req = DNS.DnsRequest(name, qtype=qtype)
+        resp = req.req()
+        #resp.show()
+        # key k: ('wayforward.net', 'A'), value v
+        # FIXME: pydns returns AAAA RR as 16 byte binary string, but
+        # A RR as dotted quad.  For consistency, this driver should
+        # return both as binary string.
+        return [((a['name'], a['typename']), a['data']) for a in resp.answers]
+    except IOError, x:
+        raise DNSError, str(x)
+
+class Session(object):
+  """A Session object has a simple cache with no TTL that is valid
+   for a single "session", for example an SMTP conversation."""
+  def __init__(self):
+    self.cache = {}
+
+  # We have to be careful which additional DNS RRs we cache.  For
+  # instance, PTR records are controlled by the connecting IP, and they
+  # could poison our local cache with bogus A and MX records.  
+
+  SAFE2CACHE = {
+    ('MX','A'): None,
+    ('MX','MX'): None,
+    ('CNAME','A'): None,
+    ('CNAME','CNAME'): None,
+    ('A','A'): None,
+    ('AAAA','AAAA'): None,
+    ('PTR','PTR'): None,
+    ('TXT','TXT'): None,
+    ('SPF','SPF'): None
+  }
+
+
+  def dns(self, name, qtype, cnames=None):
+    """DNS query.
+
+    If the result is in cache, return that.  Otherwise pull the
+    result from DNS, and cache ALL answers, so additional info
+    is available for further queries later.
+
+    CNAMEs are followed.
+
+    If there is no data, [] is returned.
+
+    pre: qtype in ['A', 'AAAA', 'MX', 'PTR', 'TXT', 'SPF']
+    post: isinstance(__return__, types.ListType)
+    """
+    result = self.cache.get( (name, qtype) )
+    cname = None
+
+    if not result:
+        safe2cache = Session.SAFE2CACHE
+        for k, v in DNSLookup(name, qtype):
+            if k == (name, 'CNAME'):
+                cname = v
+            if (qtype,k[1]) in safe2cache:
+                self.cache.setdefault(k, []).append(v)
+        result = self.cache.get( (name, qtype), [])
+    if not result and cname:
+        if not cnames:
+            cnames = {}
+        elif len(cnames) >= MAX_CNAME:
+            #return result    # if too many == NX_DOMAIN
+            raise DNSError('Length of CNAME chain exceeds %d' % MAX_CNAME)
+        cnames[name] = cname
+        if cname in cnames:
+            raise DNSError, 'CNAME loop'
+        result = self.dns(cname, qtype, cnames=cnames)
+    return result
+
+DNS.DiscoverNameServers()
+
+if __name__ == '__main__':
+  import sys
+  s = Session()
+  for n,t in zip(*[iter(sys.argv[1:])]*2):
+    print n,t
+    print s.dns(n,t)
@@ -4,110 +4,50 @@

 # Send DSNs, do call back verification,
 # and generate DSN messages from a template
+# $Log$
+# Revision 1.15  2007/09/24 20:13:26  customdesigned
+# Remove explicit spf dependency.
+#
+# Revision 1.14  2007/03/03 18:19:40  customdesigned
+# Handle DNS error sending DSN.
+#
+# Revision 1.13  2007/01/04 18:01:11  customdesigned
+# Do plain CBV when template missing.
+#
+# Revision 1.12  2006/07/26 16:37:35  customdesigned
+# Support timeout.
+#
+# Revision 1.11  2006/06/21 21:07:11  customdesigned
+# Include header fields in DSN template.
+#
+# Revision 1.10  2006/05/24 20:56:35  customdesigned
+# Remove default templates.  Scrub test.
+#

 import smtplib
-import spf
 import socket
 from email.Message import Message
 import Milter
+import time
+import dns

-nospf_msg = """Subject: Critical mail server configuration error
-
-This is an automatically generated Delivery Status Notification.
-
-THIS IS A WARNING MESSAGE ONLY.
-
-YOU DO *NOT* NEED TO RESEND YOUR MESSAGE.
-
-Delivery to the following recipients has been delayed.
-
-	%(rcpt)s
-
-Subject: %(subject)s 
-
-Someone at IP address %(connectip)s sent an email claiming
-to be from %(sender)s.  
-
-If that wasn't you, then your domain, %(sender_domain)s,
-was forged - i.e. used without your knowlege or authorization by
-someone attempting to steal your mail identity.  This is a very
-serious problem, and you need to provide authentication for your
-SMTP (email) servers to prevent criminals from forging your
-domain.  The simplest step is usually to publish an SPF record
-with your Sender Policy.  
-
-For more information, see: http://spfhelp.net
-
-I hate to annoy you with a DSN (Delivery Status
-Notification) from a possibly forged email, but since you
-have not published a sender policy, there is no other way
-of bringing this to your attention.
-
-If it *was* you that sent the email, then your email domain
-or configuration is in error.  If you don't know anything
-about mail servers, then pass this on to your SMTP (mail)
-server administrator.  We have accepted the email anyway, in
-case it is important, but we couldn't find anything about
-the mail submitter at %(connectip)s to distinguish it from a
-zombie (compromised/infected computer - usually a Windows
-PC).  There was no PTR record for its IP address (PTR names
-that contain the IP address don't count).  RFC2821 requires
-that your hello name be a FQN (Fully Qualified domain Name,
-i.e. at least one dot) that resolves to the IP address of
-the mail sender.  In addition, just like for PTR, we don't
-accept a helo name that contains the IP, since this doesn't
-help to identify you.  The hello name you used,
-%(heloname)s, was invalid.
-
-Furthermore, there was no SPF record for the sending domain
-%(sender_domain)s.  We even tried to find its IP in any A or
-MX records for your domain, but that failed also.  We really
-should reject mail from anonymous mail clients, but in case
-it is important, we are accepting it anyway.
-
-We are sending you this message to alert you to the fact that
-
-Either - Someone is forging your domain.
-Or - You have problems with your email configuration.
-Or - Possibly both.
-
-If you need further assistance, please do not hesitate to
-contact me again.
-
-Kind regards,
-
-postmaster@%(receiver)s
-"""
-
-softfail_msg = """Subject: SPF softfail (POSSIBLE FORGERY)
-
-This is an automatically generated Delivery Status Notification.
-
-THIS IS A WARNING MESSAGE ONLY.
-
-YOU DO *NOT* NEED TO RESEND YOUR MESSAGE.
-
-Delivery to the following recipients has been delayed.
-
-       %(rcpt)s
-
-Subject: %(subject)s
-Received-SPF: %(spf_result)s
-"""
-
-def send_dsn(mailfrom,receiver,msg=None):
+def send_dsn(mailfrom,receiver,msg=None,timeout=600,session=None):
  """Send DSN.  If msg is None, do callback verification.
     Mailfrom is original sender we are sending DSN or CBV to.
     Receiver is the MTA sending the DSN.
     Return None for success or (code,msg) for failure."""
  user,domain = mailfrom.split('@')
-  q = spf.query(None,None,None)
-  mxlist = q.dns(domain,'MX')
+  if not session: session = dns.Session()
+  try:
+    mxlist = session.dns(domain,'MX')
+  except dns.DNSError:
+    return (450,'DNS Timeout: %s MX'%domain)	# temp error
  if not mxlist:
    mxlist = (0,domain),	# fallback to A record when no MX
  else:
    mxlist.sort()
  smtp = smtplib.SMTP()
+  toolate = time.time() + timeout
  for prior,host in mxlist:
    try:
      smtp.connect(host)
@@ -145,51 +85,83 @@ def send_dsn(mailfrom,receiver,msg=None):
      pass		# any other error, try next MX
    except socket.error:
      pass		# MX didn't accept connections, try next one
+    except socket.timeout:
+      pass		# MX too slow, try next one
    smtp.close()
+    if time.time() > toolate:
+      return (450,'No MX response within %f minutes'%(timeout/60.0))
  return (450,'No MX servers available')	# temp error

-def create_msg(q,rcptlist,origmsg=None,template=None):
-  "Create a DSN message from a template.  Template must be '\n' separated."
-  heloname = q.h
-  sender = q.s
-  connectip = q.i
-  receiver = q.r
-  sender_domain = q.o
-  result = q.result
-  perm_error = q.perm_error
-  rcpt = '\n\t'.join(rcptlist)
-  try: subject = origmsg['Subject']
-  except: subject = '(none)'
-  try:
-    spf_result = origmsg['Received-SPF']
-  except: spf_result = None
+class Vars: pass
+
+# NOTE: Caller can pass an object to create_msg that in a typical milter
+# collects things like heloname or sender anyway.
+def create_msg(v,rcptlist=None,origmsg=None,template=None):
+  """Create a DSN message from a template.  Template must be '\n' separated.
+     v - an object whose attributes are used for substitutions.  Must
+       have sender and receiver attributes at a minimum.
+     rcptlist - used to set v.rcpt if given
+     origmsg - used to set v.subject and v.spf_result if given
+     template - a '\n' separated string with python '%(name)s' substitutions.
+  """
+  if not template:
+    return None
+  if hasattr(v,'perm_error'):
+    # likely to be an spf.query, try translating for backward compatibility
+    q = v
+    v = Vars()
+    try:
+      v.heloname = q.h
+      v.sender = q.s
+      v.connectip = q.i
+      v.receiver = q.r
+      v.sender_domain = q.o
+      v.result = q.result
+      v.perm_error = q.perm_error
+    except: v = q
+  if rcptlist:
+    v.rcpt = '\n\t'.join(rcptlist)
+  if origmsg:
+    try: v.subject = origmsg['Subject']
+    except: v.subject = '(none)'
+    try:
+      v.spf_result = origmsg['Received-SPF']
+    except: v.spf_result = None

  msg = Message()

-  msg.add_header('To',sender)
-  msg.add_header('From','postmaster@%s'%receiver)
-  msg.add_header('Auto-Submitted','auto-generated (configuration error)')
  msg.add_header('X-Mailer','PyMilter-'+Milter.__version__)
  msg.set_type('text/plain')

-  if not template:
-    if spf_result and spf_result.startswith('softfail'):
-      template = softfail_msg
-    else:
-      template = nospf_msg
-  hdrs,body = template.split('\n',1)
+  hdrs,body = template.split('\n\n',1)
  for ln in hdrs.splitlines():
    name,val = ln.split(':',1)
-    msg.add_header(name,(val % locals()).strip())
-  msg.set_payload(body % locals())
-
+    msg.add_header(name,(val % v.__dict__).strip())
+  msg.set_payload(body % v.__dict__)
+  # add headers if missing from old template
+  if 'to' not in msg:
+    msg.add_header('To',v.sender)
+  if 'from' not in msg:
+    msg.add_header('From','postmaster@%s'%v.receiver)
+  if 'auto-submitted' not in msg:
+    msg.add_header('Auto-Submitted','auto-generated')
  return msg

 if __name__ == '__main__':
+  import spf
  q = spf.query('192.168.9.50',
-  'SRS0=pmeHL=RH=bmsi.com=stuart@bmsi.com',
-  'bmsred.bmsi.com',receiver='mail.bmsi.com')
-  msg = create_msg(q,['charlie@jsconnor.com'],None,None)
+  'SRS0=pmeHL=RH==stuart@example.com',
+  'red.example.com',receiver='mail.example.com')
+  q.result = 'softfail'
+  q.perm_error = None
+  msg = create_msg(q,['charlie@example.com'],None,
+"""From: postmaster@%(receiver)s
+To: %(sender)s
+Subject: Test
+
+Test DSN template
+"""
+  )
  print msg.as_string()
  # print send_dsn(f,msg.as_string())
-  print send_dsn(q.s,'mail.bmsi.com',msg.as_string())
+  # print send_dsn(q.s,'mail.example.com',msg.as_string())
@@ -44,6 +44,8 @@ def is_dynip(host,addr):
  True
  >>> is_dynip('[1.2.3.4]','1.2.3.4')
  True
+  >>> is_dynip('c-71-63-151-151.hsd1.mn.comcast.net','71.63.151.151')
+  True
  """
  if host.startswith('[') and host.endswith(']'):
    return True
@@ -54,7 +56,7 @@ def is_dynip(host,addr):
    h = host
    m = ip3.findall(host)
    if m:
-      g = map(int,m)
+      g = map(int,m)[:4]
      ia3 = (ia[1:],ia[:3])
      if g[-3:] in ia3: return True
      if g[0] == ia[3] and g[1:3] == ia[:2]: return True
@@ -0,0 +1,66 @@
+# Author: Stuart D. Gathman <stuart@bmsi.com>
+# Copyright 2001 Business Management Systems, Inc.
+# This code is under the GNU General Public License.  See COPYING for details.
+
+import os
+from time import sleep
+
+class PLock(object):
+  "A simple /etc/passwd style lock,update,rename protocol for updating files."
+  def __init__(self,basename):
+    self.basename = basename
+    self.fp = None
+
+  def lock(self,lockname=None,mode=0660,strict_perms=False):
+    "Start an update transaction.  Return FILE to write new version."
+    self.unlock()
+    if not lockname:
+      lockname = self.basename + '.lock'
+    self.lockname = lockname
+    try:
+      st = os.stat(self.basename)
+      mode |= st.st_mode
+    except OSError: pass
+    u = os.umask(0002)
+    try:
+      fd = os.open(lockname,os.O_WRONLY+os.O_CREAT+os.O_EXCL,mode)
+    finally:
+      os.umask(u)
+    self.fp = os.fdopen(fd,'w')
+    try:
+      os.chown(self.lockname,-1,st.st_gid)
+    except:
+      if strict_perms:
+	self.unlock()
+	raise
+    return self.fp
+
+  def wlock(self,lockname=None):
+    "Wait until lock is free, then start an update transaction."
+    while True:
+      try:
+        return self.lock(lockname)
+      except OSError:
+        sleep(2)
+
+  def commit(self,backname=None):
+    "Commit update transaction with optional backup file."
+    if not self.fp:
+      raise IOError,"File not locked"
+    self.fp.close()
+    self.fp = None
+    if backname:
+      try:
+	os.remove(backname)
+      except OSError: pass
+      os.link(self.basename,backname)
+    os.rename(self.lockname,self.basename)
+
+  def unlock(self):
+    "Cancel update transaction."
+    if self.fp:
+      try:
+        self.fp.close()
+      except: pass
+      self.fp = None
+      os.remove(self.lockname)
@@ -0,0 +1,17 @@
+# Author: Stuart D. Gathman <stuart@bmsi.com>
+# Copyright 2005 Business Management Systems, Inc.
+# This code is under the GNU General Public License.  See COPYING for details.
+
+# The localpart of SMTP return addresses is often signed.  The format
+# of the signing is application specific and doesn't concern us -
+# except that we wish to extract some sort of fixed string from
+# the variable signature which represents the "source" of the message.
+
+def unsign(s):
+  """Attempt to unsign localpart and return original email.
+  No attempt is made to verify the signature.
+  >>> unsign('SRS0=8Y3CZ=3U=jsconnor.com=bills@bmsi.com')
+  'bills@jsconnor.com'
+  """
+  # not implemented yet
+  return s
@@ -0,0 +1,125 @@
+import re
+import struct
+import socket
+import email.Errors
+from fnmatch import fnmatchcase
+from email.Header import decode_header
+#import email.Utils
+import rfc822
+
+ip4re = re.compile(r'^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$')
+
+# from spf.py
+def addr2bin(str):
+  "Convert a string IPv4 address into an unsigned integer."
+  return struct.unpack("!L", socket.inet_aton(str))[0]
+
+MASK = 0xFFFFFFFFL
+
+def cidr(i,n):
+  return ~(MASK >> n) & MASK & i
+
+def iniplist(ipaddr,iplist):
+  """Return whether ip is in cidr list
+  >>> iniplist('66.179.26.146',['127.0.0.1','66.179.26.128/26'])
+  True
+  >>> iniplist('127.0.0.1',['127.0.0.1','66.179.26.128/26'])
+  True
+  >>> iniplist('192.168.0.45',['192.168.0.*'])
+  True
+  """
+  ipnum = addr2bin(ipaddr)
+  for pat in iplist:
+    p = pat.split('/',1)
+    if ip4re.match(p[0]):
+      if len(p) > 1:
+	n = int(p[1])
+      else:
+        n = 32
+      if cidr(addr2bin(p[0]),n) == cidr(ipnum,n):
+        return True
+    elif fnmatchcase(ipaddr,pat):
+      return True
+  return False
+
+def parseaddr(t):
+  """Split email into Fullname and address.
+
+  >>> parseaddr('user@example.com')
+  ('', 'user@example.com')
+  >>> parseaddr('"Full Name" <foo@example.com>')
+  ('Full Name', 'foo@example.com')
+  >>> parseaddr('spam@spammer.com <foo@example.com>')
+  ('spam@spammer.com', 'foo@example.com')
+  >>> parseaddr('God@heaven <@hop1.org,@hop2.net:jeff@spec.org>')
+  ('God@heaven', 'jeff@spec.org')
+  >>> parseaddr('Real Name ((comment)) <addr...@example.com>')
+  ('Real Name', 'addr...@example.com')
+  >>> parseaddr('a(WRONG)@b')
+  ('WRONG', 'a@b')
+  """
+  #return email.Utils.parseaddr(t)
+  res = rfc822.parseaddr(t)
+  # dirty fix for some broken cases
+  if not res[0]:
+    pos = t.find('<')
+    if pos > 0 and t[-1] == '>':
+      addrspec = t[pos+1:-1]
+      pos1 = addrspec.rfind(':')
+      if pos1 > 0:
+        addrspec = addrspec[pos1+1:]
+      return rfc822.parseaddr('"%s" <%s>' % (t[:pos].strip(),addrspec))
+  if not res[1]:
+    pos = t.find('<')
+    if pos > 0 and t[-1] == '>':
+      addrspec = t[pos+1:-1]
+      pos1 = addrspec.rfind(':')
+      if pos1 > 0:
+        addrspec = addrspec[pos1+1:]
+      return rfc822.parseaddr('%s<%s>' % (t[:pos].strip(),addrspec))
+  return res
+    
+
+def parse_addr(t):
+  """Split email into user,domain.
+
+  >>> parse_addr('user@example.com')
+  ['user', 'example.com']
+  >>> parse_addr('"user@example.com"')
+  ['user@example.com']
+  >>> parse_addr('"user@bar"@example.com')
+  ['user@bar', 'example.com']
+  >>> parse_addr('foo')
+  ['foo']
+  """
+  if t.startswith('<') and t.endswith('>'): t = t[1:-1]
+  if t.startswith('"'):
+    if t.endswith('"'): return [t[1:-1]]
+    pos = t.find('"@')
+    if pos > 0: return [t[1:pos],t[pos+2:]]
+  return t.split('@')
+
+def parse_header(val):
+  """Decode headers gratuitously encoded to hide the content.
+  """
+  try:
+    h = decode_header(val)
+    if not len(h) or (not h[0][1] and len(h) == 1): return val
+    u = []
+    for s,enc in h:
+      if enc:
+        try:
+	  u.append(unicode(s,enc))
+	except LookupError:
+	  u.append(unicode(s))
+      else:
+	u.append(unicode(s))
+    u = ''.join(u)
+    for enc in ('us-ascii','iso-8859-1','utf8'):
+      try:
+	return u.encode(enc)
+      except UnicodeError: continue
+  except UnicodeDecodeError: pass
+  except LookupError: pass
+  except email.Errors.HeaderParseError: pass
+  return val
@@ -1,5 +1,32 @@
-Here is a history of user visible changes to Python milter.
+See pymilter.spec for recent history.

+Here is a history of older changes to Python milter.
+0.8.8	move AddrCache, parse_addr, iniplist, parse_header to Milter package
+	fix plock for missing source and can't change owner/group
+	add sample spfmilter.py milter
+	private_relay config option
+0.8.7	Move spf module to pyspf
+	Prevent PTR cache poisoning
+	More lame bounce heuristics
+	Do plain CBV when template is missing
+0.8.6	Support CBV timeout
+	Support fail template, headers in templates
+	Create GOSSiP record only when connection will procede to DATA.
+	More SPF lax heuristics
+	Don't require SPF pass for white/black listing mail from trusted relay.
+	Support localpart wildcard for white and black lists.
+	Delay reject of unsigned RCPT for postmaster and abuse only
+	Fix dsn reporting of hard permerror
+	Resolve FIXME for wrap_close in miltermodule.c
+	Add Message-ID to DSNs
+	Use signed Message-ID in delayed reject to blacklist senders
+	Auto-train via blacklist and auto-whitelist
+	Don't check userlist for signed MFROM
+	Accept but skip DSPAM training for whitelisted senders without SPF PASS
+	Report GC stats 
+	Support CIDR matching for IP lists
+	Support pysrs sign feature
+	Support localpart specific SPF policy in access file
 0.8.5	Simple trusted_forwarder implementation.
 	Fix access_file neutral policy
 	Move Received-SPF header to beginning of headers
@@ -42,7 +42,7 @@ Quick Installation
 1. Build and install Sendmail, enabling libmilter (see libmilter/README).
 2. Build and install Python, enabling threading.
 3. Install this module: python setup.py --help
-4. Add these two lines to sendmail.cf:
+4. Add these two lines to sendmail.cf[*]:

 O InputMailFilters=pythonfilter
 Xpythonfilter,        S=local:/home/username/pythonsock
@@ -51,9 +51,17 @@ Xpythonfilter,        S=local:/home/username/pythonsock
 Note that milters should almost certainly not run as root.

 That's it.  Incoming mail will cause the milter to print some things, and
-some email will be rejected (see the "header" method).  Edit and play.  See
-bms.py for an example milter used in production.
+some email will be rejected (see the "header" method).  Edit and play.  
+See spfmilter.py for a functional SPF milter, or see bms.py for an complex
+milter used in production.

+[*] This is for a quick test.  Your sendmail.cf in most distros will get
+overwritten whenever sendmail.mc is updated.  To make a milter permanent,
+add something like:
+
+INPUT_MAIL_FILTER(`pythonfilter', `S=local:/home/username/pythonsock, F=T, T=C:5m;S:20s;R:5m;E:5m')
+
+to sendmail.mc instead.

 Not-so-quick Installation
 -------------------------
@@ -90,8 +98,10 @@ some options associated with it.  In this case, we have the "S" option, which
 names the socket that sendmail will use to communicate with this particular
 milter.  This milter's socket is a unix-domain socket in the filesystem.
 See libmilter/README for the definitive list of options.
+
 NB: The name is specified in two places: here, in sendmail's cf file, and
 in the milter itself.  Make sure the two match.
+
 NB: The above lines can be added in your .mc file with this line:

 INPUT_MAIL_FILTER(`pythonfilter', `S=local:/home/username/pythonsock')
@@ -1,8 +1,125 @@
-Require signed MFROM for all incoming bounces when signing all outgoing mail -
-except from trusted relays.
+Check ESMTP NOTIFY before sending real DSNs.  Just use CBV if DSNs are
+not wanted.
+
+Support CBV to local domains and cache results so that invalid users
+can be rejected without maintaining valid user lists.
+
+Now that we blacklist IPs for too many bad rcpts, delay SPF until RCPT TO.
+
+When content filtering is not installed, reject BLACKLISTed MFROM
+immediately.  There is no use waiting until EOM.
+
+Configuration is problematic when handling incoming, but not outgoing mail.
+The problem comes when alice@example.com sends mail to bill@example.com,
+and we are the MX for example.com, but alice is sending from some other
+MTA.  The mail is flagged external, so we don't list example.com in
+internal_domains (or we would get "spam from self").  But, if we try to do a
+CBV, we get "fraudulent MX", because the MX is ourself!  So we need to 
+avoid doing CBV on such domains.  Currently, we try to make sure the SPF
+policies don't do CBV.  The real solution is for users to use SMTP AUTH,
+but some of them are stubborn.
+
+We now don't check internal domains for incoming mail if there is an
+SPF record.
+
+On the other hand, if alice is sending internally, or with SMTP AUTH, she
+*does* need the domain to be in internal_domains.  The solution to that 
+is to use the new SMTP AUTH access configuration to specify which domains
+can be used by smtp AUTH (by user if desired).
+
+It would be cleaner if CBV would know which domains we have agreed to
+be MX for. Some ideas for external connections:
+
+a) check access file for To:example.com	RELAY
+b) check mailertable
+c) check mx_domains config list
+d) if there is an SPF record, don't check internal_domains
+   (let SPF block unauthorized machines)
+
+But that still doesn't handle the roaming user, who won't use SMTP
+AUTH, but sends through some hotel MTA.  Maybe we don't want to support
+him?
+
+When setting up pydspam, both sender and rcpt must resolve to dspam users
+for falsepositive recognition.  Usually, this means adding 
+honeypot@mail.example.com to alias list for honeypot in pymilter.cfg.
+This needs to be documented.  I was caught by it setting up a new site.
+
+Add signature (x-sig=AB7485f=TS) to Received-SPF, so it can be used
+to blacklist sources of delayed DSNs.
+
+rcpt-addr may let us know when a recipient is unknown.  That should count
+against reputation.
+
+Need to use wildcards in blacklist.log: *.madcowsrecord.net
+Need to exclude emails like !*-admin@example.com in whitelist_sender.
+Need to exclude robot users from autowhitelist.  Don't want to have to
+list all users, so implement something like !*-admin@bmsi.com,@bmsi.com.
+
+GOSSiP feedback from user training is ignored because UMIS has already been
+removed from queue.  Maybe keep UMIS in queue, and add method to 
+alter last feedback for ID.
+
+Generate DSNs according to RFC 3464
+
+Get temperror policy from access file.
+
+Reporting explanation for failure should show source if sender
+provided explanation.
+
+Bug in Auto-whitelist.  Recent Auto-whitelist doesn't override expired entry.
+
+SPF permerror diagnostics should include corrected mechanism.
+
+Delay SPF check until RCPT TO.  Cache result to avoid repeating
+for multiple RCPT.  This avoids overhead for invalid RCPT, and
+allows for per RCPT local policy.
+
+Check SPF for outgoing mail (including local policy for internal addresses).
+This could also solve the second part of the mail from relay problem below.
+
+Whitelisted senders from trusted relay get PROBATION.  Need to extracted
+SPF result from headers - and in the case of mail internal to relay
+(e.g. bmsi.com), supply 'pass' result.  
+
+Add auto-blacklisted senders to blacklist.log with timestamp.
+Add emails blacklisted via CBV so that they are remembered across milter
+restarts.
+
+Make all dictionaries work like honeypot.  Do not train as ham unless
+whitelisted.  Train on blacklisted messages, or spam feedback.  This
+can be called Train On Error.  Should be possible to startup
+with training on everything to get dictionary built fast, then switch
+to train on error to minimize labor.
+
+Allow unsigned DSNs from selected domains (that don't accept signed MFROM,
+e.g. verizon.net).
+
+Allow verified hostnames for trusted_relay.  E.g. HELO name that
+passes SPF.
+
+When do we get two hello calls?  STARTTLS is one reason.
+
+Option: accept mail from auto-whitelisted senders even with spf-fail,
+but do not update dspam.  This can be done for individual senders or domains 
+using the access file.
+
+pysrs: SRS doesn't get applied to proper recipients when there are
+multiple recipients.  This requires debugging cf scripts - yuk.
+
+auto_whitelist false_positives from quarantine - perhaps only when
+user selects special button (use special header to communicate
+that from dspamcgi.py to milter.)
+
+Use send_dsn.log for blacklist also.  AddrCache needs localpart
+wildcard (e.g. empty localpart).
+
+Quarantined mail is missing headers modified/added by milter after
+checking dspam.

 Send DSN for permerror before processing extended result.  An additional
-DSN may be sent based on extended result.
+DSN may be sent based on extended result.  Send permerror DSN to
+postmaster@sending_domain.

 Rescind whitelist for banned extensions, in case sender is infected.

@@ -16,9 +133,6 @@ SPF-Neutral:aol.com	ERROR:"550 AOL mail must get SPF PASS"
 Defer TEMPERROR in SPF evaluation - give precedence to security
 (only defer for PASS mechanisms).

-Option to add Received-SPF header, but never reject on SPF.
-  I think the above will handle this.
-
 Create null config that does nothing - except maybe add Received-SPF
 headers.  Many admins would like to turn features on one at a time.

@@ -47,8 +161,6 @@ a separate process.  However, a significant amount of memory is wasted
 for each additional Python VM, and communication between milters
 is cumbersome (e.g., adding mail headers, writing external files).

-Backup copies for outgoing/incoming mail.
-
 Copy incoming wiretap mail, even though sendmail alias works perfectly
 for the purpose, to avoid having to change two configs for a wiretap.

@@ -66,3 +178,63 @@ embarrass yourself), and also removing Received headers with hidepath.
 Need a test module to feed sample messages to a milter though a live 
 sendmail and SMTP.  The mockup currently used is probably not very accurate,
 and doesn't test the threading code.
+
+DONE Table of sendmail macros for documentation.  In API docs on milter.org.
+
+DONE For selected domains, check rcpts via CBV before accepting mail.  Cache
+results.  This will kick out dictonary attacks against a mail domain
+behind a gateway sooner.
+
+DONE Convert DSN to REJECT unless sender gets SPF pass or best guess pass.  Make
+configurable by SPF result with NOTSPAM policy (reject or deliver without DSN).
+Maybe policy should be NODSN - still verify sender with CBV.
+
+DONE Add parseaddr test case for 'foo@bar.com <baz@barf.biz>'
+
+DONE Require signed MFROM for all incoming bounces when signing all outgoing
+mail - except from trusted relays.
+
+DONE Added Message-ID header to DSN with SRS signed sender.  When seen on
+incoming rfc ignorant failure message, blacklist sender.
+
+DONE Option to add Received-SPF header, but never reject on SPF.
+  I think the above will handle this.
+
+DONE Received-SPF header field should show identity that was checked.
+
+DONE When training with spam, REJECT after data so that mistakenly blacklisted
+senders at least get an error.
+
+DONE Milter won't start when it can't change permissions on *.lock to match
+*.log.  Should maybe ignore that error - the effect will be to set
+the permissions to default.
+
+DONE Milter won't start when a whitelist/blacklist file is missing.
+
+DONE Delayed failure detection should parse From header to find email address.
+
+DONE When bms.py can't find templates, it passes None to dsn.create_msg(),
+which uses local variable as backup, which no longer exist.  Do plain
+CBV in that case instead.
+
+DONE Find and use X-GOSSiP: header for SPAM: and FP: submissions.  Would need
+to keep tags longer.
+
+DONE Parse incoming 3464 DSNs for "Action: failed" to recognize delayed
+failures.  This works regardless of Subject.
+
+DONE Reports PROBATION even when rejecting message (works, but confusing in
+log).
+
+DONE Delayed_failure detection needs to handle multi-line header fields.  
+Also, delayed_failure should be recognized when addressed to
+postmaster@helodomain 
+
+DONE DSN for Permerror shows 'None' for error under some condition.
+
+DONE Allow blacklisted emails as well as domains in blacklist.log.  Use same
+data structure as autowhitelist.log.  
+
+DONE Backup copies for outgoing/incoming mail.
+
+DONE Don't match dynamic ptr in bestguess.
@@ -1,153 +0,0 @@
-#!/usr/bin/python2.3
-
-# Convert a MS Caller-ID entry (XML) to a SPF entry
-#
-# (c) 2004 by Ernesto Baschny
-# (c) 2004 Python version by Stuart Gathman
-#
-# Date: 2004-02-25
-# Version: 1.0
-#
-# Usage:
-#  ./cid2spf.pl "<ep xmlns='http://ms.net/1'>...</ep>"
-#
-# Note that the 'include' directives will also have to be checked and
-# "translated". Future versions of this script might be able to get a
-# domain name as an argument and "crawl" the DNS for the necessary
-# information.
-#
-# A complete reverse translation (SPF -> CID) might be impossible, since
-# there are no way to handle:
-# - PTR and EXISTS mechanism 
-# - MX mechanism with an different domain as argument
-# - macros
-# 
-# References:
-# http://www.microsoft.com/mscorp/twc/privacy/spam_callerid.mspx
-# http://spf.pobox.com/
-#
-# Known bugs:
-# - Currently it won't handle the exclusions provided in the A and R
-#   tags (prefix '!'). They will show up "as-is" in the SPF record
-# - I really haven't read the MS-CID specs in-depth, so there are probably
-#   other bugs too :)
-#
-# Ernesto Baschny <ernst@baschny.de>
-#
-
-import xml.sax
-import spf
-
-# -------------------------------------------------------------------------
-class CIDParser(xml.sax.ContentHandler):
-  "Convert a MS Caller-ID entry (XML) to a SPF entry"
-
-  def __init__(self,q=None):
-    self.spf = []
-    self.action = '-all'
-    self.has_servers = None
-    self.spf_entry = None
-    if q:
-      self.spf_query = q
-    else:
-      self.spf_query = spf.query(i='127.0.0.1', s='localhost', h='unknown')
-
-  def startElement(self,tag,attr):
-      if tag == 'm':
-	if self.has_servers != None and not self.has_servers:
-	  raise ValueError(
-    "Declared <noMailServers\> and later <m>, this CID entry is not valid."
-	  )
-	self.has_servers = True
-      elif tag == 'noMailServers':
-	if self.has_servers:
-	  raise ValueError(
-    "Declared <m> and later <noMailServers\>, this CID entry is not valid."
-	  )
-	self.has_servers = False
-      elif tag == 'ep':
-	if attr.has_key('testing') and attr.getValue('testing') == 'true':
-	  # A CID with 'testing' found:
-	  # From the MS-specs:
-	  #  "Documents in which such attribute is present with a true
-	  #  value SHOULD be entirely ignored (one should act as if the
-	  #  document were absent)"
-	  # From the SPF-specs:
-	  #  "Neutral (?): The SPF client MUST proceed as if a domain did
-	  #  not publish SPF data."
-	  # So we set SPF action to "neutral":
-	  self.action = '?all'
-      elif tag == 'mx':
-	  # The empty MX-tag, same as SPF's MX-mechanism
-	  self.spf.append('mx')
-      self.tag = tag
-
-  def characters(self,text):
-	tag = self.tag
-	# Remove starting and trailing spaces from text:
-	text = text.strip()
-
-	if tag == 'a' or tag == 'r':
-	    # The A and R tags from MS-CID are both handled by the 
-	    # ipv4/6-mechanisms from SPF:
-	    if text.find(':') < 0:
-	      mechanism = 'ip4'
-	    else:
-	      mechanism = 'ip6'
-	    self.spf.append(mechanism + ':' + text)
-	elif tag == 'indirect':
-	    # MS-CID's indirect is "sort of" the include from SPF:
-	    # Not really true, because the <indirect> tag from MS-CID also 
-	    # provides a fallback in case the included domain doesn't provide
-	    # _ep-records: The inbound MX-servers of the included domains
-	    # are added to the list of allowed outgoing mailservers for the
-	    # domain that declared the _ep-record with the <indirect> tag.
-	    # In SPF you would use the 'mx:domain' to handle this, but this
-	    # wouldn't depend on referred domain having or not SPF-records.
-	    cid_xml = self.cid_txt(text)
-	    if cid_xml:
-	      p = CIDParser()
-	      xml.sax.parseString(cid_xml,p)
-	      if p.has_servers != False:
-		self.spf += p.spf
-	    else:
-	      self.spf.append('mx:' + text)
-
-  def cid_txt(self,domain):
-    q = self.spf_query
-    domain='_ep.' + domain
-    a = q.dns_txt(domain)
-    if not a: return None
-    if a[0].lower().startswith('<ep ') and a[-1].lower().endswith('</ep>'):
-      return ''.join(a)
-    return None
-
-  def endElement(self,tag):
-      if tag == 'ep':
-	# This is the end... assemble what we've got
-	spf_entry = ['v=spf1']
-	if self.has_servers != False:
-	  spf_entry += self.spf
-	spf_entry.append(self.action)
-	self.spf_entry = ' '.join(spf_entry)
-
-  def spf_txt(self,cid_xml):
-    if not cid_xml.startswith('<'):
-      cid_xml = self.cid_txt(cid_xml)
-      if not cid_xml: return None
-    # Parse the beast. Any XML-problem will be reported by xlm.sax
-    self.spf_entry = None
-    xml.sax.parseString(cid_xml,self)
-    return self.spf_entry
-
-if __name__ == '__main__':
-  import sys
-  if len(sys.argv) < 2:
-    print >>sys.stderr, \
-      """Usage: %s "<ep xmlns='http://ms.net/1'>...</ep>" """ % sys.argv[0]
-    sys.exit(1)
-
-  cid_xml = sys.argv[1]
-
-  p = CIDParser()
-  print p.spf_txt(cid_xml)
@@ -2,6 +2,55 @@ Title: Recent Changes

 <h2> Recent Changes </h2>

+<h3> 0.8.10 </h3>
+
+SRS rejections now log the recipient.  
+I have finally implemented plain CBV (no DSN).  The CBV policy
+will do a plain CBV from now on, and the DSN policy is required
+if you want to send a DSN.
+I started checking the MAIL FROM fullname (human readable part
+of an email) for porn keywords.  There is now a banned IP database.
+IPs are banned for too many bad MAIL FROMs or RCPT TOs, and remain banned
+for 7 days.
+
+<h3> 0.8.9 </h3>
+
+I use the <code>%ifarch</code> hack to build milter and milter-spf
+packages as noarch, while pymilter is built as native.
+
+I removed the spf dependency from dsn.py, so pymilter can be used without
+installing pyspf, and added a Milter.dns module to let python milters do
+general DNS lookups without loading pyspf.
+
+<h3> 0.8.8 </h3>
+
+Programs do not belong in the /var/log directory.  I moved the
+milter apps to /usr/lib/pymilter.  Since having the programs and
+data in the same directory is convenient for debugging, it will
+still use an executable present in the datadir.
+
+Several general utility classes and functions are now in the Milter package
+for possible use by other python milters.  In addition to the trivial example
+milter, a simple SPF only milter is included as a realistic example.
+
+The spec file now build 3 RPMs:
+
+<ul>
+<li> pymilter is the milter module and Milter package for use by all python
+     milters.
+<li> milter is the all-singing, all-dancing python milter application, with
+     supporting <code>/etc/init.d</code>, logrotate and other scripts.
+<li> milter-spf is the simple SPF only milter application.
+</ul>
+
+<h3> 0.8.7 </h3>
+
+The spf module has been moved to the 
+<a href="http://cheeseshop.python.org/pypi/pyspf">pyspf</a> package.
+Download <a href="http://sourceforge.net/project/showfiles.php?group_id=139894&package_id=191419">here</a>.
+
+<h3> 0.8.6 </h3>
+
 Python milter has been moved to 
 <a href="http://sourceforge.net/projects/pymilter/">pymilter Sourceforge
 project</a> for development and release downloads.
@@ -5,7 +5,7 @@ Title: Credits
 <a href="mailto:Jim Niemira <urmane@urmane.org>">Jim Niemira</a>
 wrote the original C module and some quick
 and dirty python to use it.  
-<a href="mailto:Stuart Gathman <stuart@bmsi.com>">Stuart D. Gathman</a>
+<a href="http://gathman.org/vitae">Stuart D. Gathman</a>
 took that kludge and added threading and context objects to it, wrote a proper
 OO wrapper (Milter.py) that handles attachments, did lots of testing, packaged
 it with distutils, and generally transformed it from a quick hack to a
@@ -2,8 +2,24 @@ Title: Python Milter FAQ

 <h1> Python Milter <a name=faq>FAQ</a> </h1>

+<menu>
+<li> <a href="#compiling">Compiling Python Milter</a>
+<li> <a href="#running">Running Python Milter</a>
+<li> <a href="#spf">Using SPF</a>
+<li> <a href="#srs">Using SRS</a>
+</menu>
+
 <ol>
-<h3> Compiling Python Milter </h3>
+
+<h3> <a name="compiling">Compiling Python Milter </a> </h3>
+
+<li> Q. I have tried to download the current milter code and my virus scan
+traps several viruses in the download.
+<p>  A. The milter source includes a number of deactivated viruses in
+the test directory.  All but the first and last lines of the base64
+encoded virus data has been removed.  I suppose I should randomize
+the first and last lines as well, since pymilter just deletes executables,
+and doesn't look for signatures.
 <li> Q. I have installed sendmail from source, but Python milter won't
 compile.
 <p>  A. Even though libmilter is officially supported in sendmail-8.12, 
@@ -20,8 +36,23 @@ RedHat 7.2?
 <p>  A. RedHat forgot to include the header in the RPM.  See the
 <a href="requirements.html#rh72">RedHat 7.2 requirements</a>.
 <p>
+<li> Q. Python milter compiles ok, but I get an error like this when
+	I try to import the milter module:
+<pre>
+ImportError: /usr/lib/python2.4/site-packages/milter.so: undefined symbol: smfi_setmlreply
+</pre>
+<p>  A. Your libmilter.a is from sendmail-8.12 or earlier.  You need
+	sendmail-8.13 or later to support setmlreply.  You can disable
+	setmlreply by changing setup.py.  Change:
+<pre>
+            define_macros = [ ('MAX_ML_REPLY',32) ]
+</pre>
+in setup.py to
+<pre>
+            define_macros = [ ('MAX_ML_REPLY',1) ]
+</pre>

-<h3> Running Python Milter </h3>
+<h3> <a name="running">Running Python Milter </a></h3>

 <li> Q. The sample.py milter prints a message, then just sits there.
 <pre>
@@ -171,10 +202,16 @@ The <code>internal_domains</code> option is simplistic, it assumes all
 valid senders of the domains are internal.  SPF provides a much more general
 check of IP and MAIL FROM for external email.  Pymilter should soon
 have a local policy feature for more general checking of internal mail.
+<li> Q. <code>mail_archive</code> isn't working.  Or I don't understand how
+	it's suppose to work.  I have
+	<code>mail_archive = /var/mail/mail_archive</code>
+	in <code>pymilter.cfg</code> but nothing ever gets dumped into
+	<code>/var/mail/mail_archive</code>.
+<p>  A. The 'mail' user needs to have write access.  Permission failures
+	should be logged as a traceback in milter.log if it doesn't.

-<h3> Using SPF </h3>
+<h3> <a name="spf">Using SPF </a></h3>

-<a name="spf">
 <li> Q. So how do I use the SPF support?  The sample.py milter doesn't seem
        to use it.
 <p>  A. The bms.py milter supports spf.  The RedHat RPMs will set almost
@@ -194,5 +231,63 @@ everything up for you.  For other systems:
     logfiles and a simple cron script using <code>find</code> to clean
     <code>tempdir</code>.
 </ol>
+	In CVS, there is <code>spfmilter.py</code>.  Run that as a service,
+	and it does just SPF.  It uses the sendmail <code>access</code>
+	file to configure SPF responses just like <code>bms.py</code>, but
+	supports only REJECT and OK.
+<li> Q. The SPF DSN is sent at least once for domains that don't publish a SPF.
+	How do I stop this behavior?
+<p>  A. The SPF response is controlled by <code>/etc/mail/access</code>
+	(actually the file you specify with <code>access_file</code> in
+	the <code>[spf]</code> section of <code>pymilter.cfg</code>).  
+	Responses are OK, CBV, and REJECT.  CBV sends the DSN.
+<p>
+You can change the defaults.  For instance, I have:
+<pre>
+SPF-None:	REJECT
+SPF-Neutral:	CBV
+SPF-Softfail:	CBV
+SPF-Permerror:	CBV
+</pre>
+I have best_guess = 1, so SPF none is converted to PASS/NEUTRAL for policy
+lookup, and 3 strikes (no PTR, no HELO, no SPF) becomes "SPF NONE" for local
+policy purposes (the Received-SPF header always shows the official SPF
+result.)
+<p>
+You can change the default for specific domains:
+<pre>
+# these guys aren't going to pay attention to CBVs anyway...
+SPF-None:cia.gov	REJECT
+SPF-None:fbi.gov	REJECT
+SPF-Neutral:aol.com	REJECT
+SPF-Softfail:ebay.com	REJECT
+</pre>
+
+<h3> <a name="srs">Using SRS </a></h3>
+
+<li> Q. The SRS part doesn't seem to work as whenever I try to start
+	<code>/etc/init.d/pysrs</code>, I get this in
+	<code>/var/log/milter/pysrs.log</code>:
+<pre>
+ConfigParser.NoOptionError: No option 'fwdomain' in section: 'srs'
+</pre>
+<p>  A. You need to specify the forward domain - i.e. the domain you want
+	SRS to rewrite stuff too.
+<p>
+For instance, I have:
+<pre>
+# sample SRS configuration
+[srs]
+secret = don't you wish
+maxage = 8
+hashlength = 5
+;database=/var/log/milter/srs.db
+fwdomain = bmsi.com
+sign=bmsi.com,mail.bmsi.com,gathman.org
+srs=bmsaix.bmsi.com,bmsred.bmsi.com,stl.gathman.org,bampa.gathman.org
+</pre>
+The <code>sign</code> is for local domains which are signed.
+The <code>srs</code> list is for other domains which you are relaying,
+and which need to have SRS checked/undone for bounces.

 </ol>
@@ -0,0 +1,23 @@
+<!-- -*- html -*- -->
+<h3>Subsections</h3>
+<li><a href="milter.html">Introduction</a>
+<li><a href="changes.html">Changes</a>
+<li><a href="requirements.html">Requirements</a>
+<li><a href="http://sourceforge.net/project/showfiles.php?group_id=139894">Download</a>
+<li><a href="faq.html">FAQ</a>
+<li><a href="policy.html">Policies</a>
+<li><a href="logmsgs.html">Log&nbsp;Messages</a>
+<li><a href="http://bmsi.com/mailman/listinfo/pymilter">Mailing&nbsp;List</a>
+<li><a href="credits.html">CREDITS</a>
+<li><a href="http://sourceforge.net"><img src="http://sflogo.sourceforge.net/sflogo.php?group_id=139894&amp;type=1" width="88" height="31" border="0" alt="SourceForge.net Logo" /></a>
+<h3>Links</h3>
+<li><a href="http://www.milter.org/milter_api/api.html">C&nbsp;API</a>
+<li><a href="http://www.milter.org/">Milter.Org</a>
+<li><a href="http://www.python.org/">Python.Org</a>
+<li><a href="http://www.sendmail.org/">Sendmail.Org</a>
+<li><a href="http://www.openspf.org/">SPF</a>
+<li><a href="pysrs.html">pysrs</a>
+<li><a href="http://cheeseshop.python.org/pypi/pyspf">pyspf</a>
+<li><a href="http://bmsi.com/python/pygossip.html">pygossip</a>
+<li><a href="http://bmsi.com/python/dspam.html">pydspam</a>
+<li><a href="http://bmsi.com/libdspam/dspam.html">libdspam</a>
@@ -30,8 +30,8 @@ HELO name, and it did not pass.
 <dt> INNOC: richh
 <dd> message was used to update richh's dspam dictionary

-<dt> HONEYPOT: michaelb@jsconnor.com
-<dd> message was sent to a honeypot address (michaelb@jsconnor.com), the 
+<dt> HONEYPOT: pooh@bwicorp.com
+<dd> message was sent to a honeypot address (pooh@bwicorp.com), the 
 message was added to the honeypot dspam dictionary as spam

 <dt> REJECT: numeric hello name: 63.217.19.146
@@ -57,8 +57,17 @@ MX record, we told the sender to try again later
 <dt> REJECT: Subject: Cialis - No prescription needed!
 <dd> message was rejected because its subject contained a bad expression

-<dt> DSPAM: tonyc tonyc@jsconnor.com
-<dd> message was sent to tonyc@jsconnor.com and it was identified as spam 
+<dt> REJECT: zombie PC at  192.168.3.37  sending MAIL FROM  seajdr@amritind.com
+<dd> message was rejected because the connect ip was internal, but the
+sender was not.  This is usually because a Windows PC is infected with
+malware.
+
+<dt> X-Guessed-SPF: pass
+<dd> When the SPF result is NONE, we guess a result based on the generic
+SPF policy "v=spf1 a/24 mx/24 ptr".
+
+<dt> DSPAM: tonyc tonyc@example.com
+<dd> message was sent to tonyc@example.com and it was identified as spam 
 and placed in the tonyc dspam quarantine

 <dt> REJECT: CBV: 550 calvinalstonis@ix.netcom.com...User unknown
@@ -66,6 +75,17 @@ and placed in the tonyc dspam quarantine
 <dt> REJECT: CBV: 554 delivery error: dd This user doesn't have an account 
 <dd> message was rejected because call back verification gave us a fatal
 error
+<dt> Auto-Whitelist: user@example.com
+<dd> recipient has been added to auto_whitelist.log because the message
+was sent from an internal IP and the recipient is not internal.
+<dt> WHITELIST user@example.com
+<dd> message is whitelisted because sender appears in auto_whitelist.log
+<dt> BLACKLIST user@example.com
+<dd> message is blacklisted because sender appears in blacklist.log or
+     failed a CBV test.
+<dt> TRAINSPAM: honeypot X-Dspam-Score: 0.002278
+<dd> message was used to train screener dictionary as spam
+<dt> TRAIN: honeypot X-Dspam-Score: 0.980203
+<dd> message was used to train screener dictionary as ham
 </dl>
-
-Please add more tags to this list if you know of any. Thanks.
+<br>
@@ -20,7 +20,7 @@ ALT="Viewable With Any Browser" BORDER="0"></A>
  Stuart D. Gathman</a><br>
 This web page is written by Stuart D. Gathman<br>and<br>sponsored by
 <a href="http://www.bmsi.com">Business Management Systems, Inc.</a> <br>
-Last updated Dec 29, 2005</h4>
+Last updated Mar 30, 2007</h4>

 See the <a href="faq.html">FAQ</a> | <a href="http://sourceforge.net/project/showfiles.php?group_id=139894">Download now</a> |
 <a href="http://bmsi.com/mailman/listinfo/pymilter">Subscribe to mailing list</a> |
@@ -4,8 +4,7 @@ Title: Python Milter Mail Policy

 These are the policies implemented by the <code>bms.py</code> milter
 application.  The milter and Milter modules do not implement any policies
-by themselves.  Eventually, I'll get the bms.py milter moved to its
-own package.
+by themselves.  

 <h3> Classify connection </h3>

@@ -77,161 +76,119 @@ altered accordingly.

 <h2> SPF check </h2>

-Finally, the MAIL FROM, connect IP, and HELO name are checked against
-any SPF records published via DNS for the alleged sender (MAIL FROM).
-If there is no SPF record, we check for a local substitute under the
-domain defined in the <code>[spf]delegate</code> configuration.
-Further checks depend on the result.
+The MAIL FROM, connect IP, and HELO name are checked against
+any SPF records published via DNS for the alleged sender (MAIL FROM)
+to determine the official SPF policy result.
+The offical SPF result is then logged in the Received-SPF header field,
+but certain results are subjected to further processing to create
+an effective result for policy purposes.

-<table border=1>
-<tr><th>NONE</th><td>
-If there is no SPF record (official or delegated), then we 
-initiate a "three strikes and your out" regime, which looks for
-<b>some</b> form of validated identification.
-<ol>
-<li>We try a "best guess" SPF record of "v=spf1 a/24 mx/24 ptr".  If this
-    passes, good.  
-<li> We try to validate the HELO name. First check for an SPF record.
-   Otherwise, check whether the connect IP matches any A record for
-   the HELO name, or any A record for any MX name for the HELO name,
-   or is at least in the same /24 subnet as any of the above.
-   (In other words, a HELO SPF "best guess" of "v=spf1 a/24 mx/24".)
-   If so, good.  We consider the HELO validated.  If the HELO SPF
-   check fails, we reject the email.
-</ol>
+If the official result is 'none', we try to turn it into an effective result of
+'pass' or 'fail'.  First, we check for a local substitute SPF record
+under the domain defined in the <code>[spf]delegate</code> configuration.  
+It is often useful to add local SPF records for correspondents that are
+too clueless to add their own.  If there is no local substitute, we use a "best
+guess" SPF record of "v=spf1 a/24 mx/24 ptr" for MAIL FROM or "v=spf1 a/24
+mx/24" for HELO.  In addition, a HELO that is a subdomain of MAIL FROM and
+resolves to the connect IP results in an effective result of 'pass'.
+
+If there is no local SPF record, and the effective result is still not
+'pass', we check for either a valid HELO name or a valid PTR record for
+the connect IP.  A valid HELO or PTR cannot look like a dynamic name
+as determined by the heuristic in <code>Milter.dynip</code>.
+
+If HELO has an SPF record, and the result is anything but pass, we reject
+the connection:
 <pre>
 2005Jul30 19:45:16 [93991] connect from [221.200.41.54] at ('221.200.41.54', 3581) EXTERNAL DYN
 2005Jul30 19:45:18 [93991] hello from adelphia.net
 2005Jul30 19:45:19 [93991] mail from <wendy.stubbsua@link-it.com> ()
 2005Jul30 19:45:19 [93991] REJECT: hello SPF: fail 550 access denied
 </pre>
-<ol>
-<li> If there is a validated PTR name, and it doesn't look
-   like a dynamic name, good.  We consider the connection validated.
-</ol>
-If any of the above can be validated, we continue on.
-If none of the above can be validated, and the <code>[SPF]reject_noptr</code>
-option is true, we reject the message immediately with the explanation
-that we need some form of valid identification before we accept an email.
-If <code>[SPF]reject_noptr</code> is false, we flag the message as
-needing Call Back Validation.  
-The Call Back Valildation sends a DSN to the purported sender informing
-them of the lack of identification.  If the message is legitimate, the
-sender needs to know that their email setup is broken and should be corrected.
-If the message is forged, the sender is informed of the forgery, 
-and their need to publish an SPF record or at least use a valid HELO name.
-If the purported sender does not accept the DSN,
-then the message is rejected.  The CBV status is cached to avoid
-annoying the purported sender with too many DSNs.  Currently, the DSN
-is repeated to the same sender once per month.
-<p>
-In this example, although 3com.com has no SPF record, we assume that 
-any legitimate mail from them will at least have a valid HELO or PTR.
-<pre>
-2005Jul30 23:52:03 [96777] connect from [222.252.233.200] at ('222.252.233.200', 29934) EXTERNAL DYN
-2005Jul30 23:52:03 [96777] hello from 3mail.3com.com
-2005Jul30 23:52:04 [96777] mail from <etec_nic_family@3mail.3com.com> ()
-2005Jul30 23:52:04 [96777] REJECT: no PTR, HELO or SPF
-</pre>
-</td></tr>
+Note that HELO does not have any forwarding issues like MAIL FROM, and so
+any result other than 'pass' or 'none' should be treated like 'fail'.

-<tr><th>PASS</th><td>
-A pass result normally lets the email continue on, but the domain is 
-tracked for reputation (and may be blocked), and may skip content scanning if
-it matches a whitelist.
-<pre>
-2005Jul24 17:44:26 [2104] mail from <gnucash-devel-bounces@gnucash.org> ('SIZE=4410',)
-2005Jul24 17:44:26 [2104] Received-SPF: pass (mail.bmsi.com: domain of gnucash.org
-	designates 204.107.200.65 as permitted sender)
-	client-ip=204.107.200.65; envelope-from=gnucash-devel-bounces@gnucash.org; helo=cvs.gnucash.org;
-</pre>
-</td></tr>
+Only if nothing about the SMTP envelope can be validated does the effective
+result remain 'none.  I call this the "3 strikes" rule.

-<tr><th>NEUTRAL</th><td>
-A neutral result normally lets the email continue on, but the domain is not
-tracked for reputation or matched against any whitelists. 
-Highly forged domains listed in <code>[SPF]reject_neutral</code> are
-rejected.
-<pre>
-2005Jul24 17:41:37 [2070] connect from cp500627-a.dbsch1.nb.home.nl at ('84.27.225.3', 3465) EXTERNAL
-2005Jul24 17:41:37 [2070] hello from cp500627-a.dbsch1.nb.home.nl
-2005Jul24 17:41:38 [2070] mail from <nwarjejkw@yahoo.com> ()
-2005Jul24 17:41:38 [2070] REJECT: SPF neutral for nwarjejkw@yahoo.com
-</pre>
-</td></tr>
+If the official result is 'permerror' (a syntax error in the sender's
+policy), we use the 'lax' option in pyspf to try various heuristics to guess 
+what they really meant.  For instance, the invalid mechanism "ip:1.2.3.4" is
+treated as "ip4:1.2.3.4".  The result of lax processing is then used
+as the effective result for policy purposes.

-<tr><th>SOFTFAIL</th><td>
-A softfail result normally lets the email continue on, but the domain is not
-tracked for reputation or matched against any whitelists.  Furthermore,
-the message is flagged as needing Call Back Validation,
-and the highly forged domains listed in <code>[SPF]reject_neutral</code> are
-rejected as well.  
-<p>
-At present, we also require a valid HELO or PTR to avoid rejecting 
-a softfail.  But this should probably change to only require a
-successful CBV.
-<p>
-The Call Back Valildation sends a DSN to the purported sender informing
-them of the softfail.  If the message is legitimate, the sender needs
-to know about the softfail so that their email setup can be corrected.
-If the message is forged, the sender is informed of the forgery, confirming
-that SPF is protecting their reputation and encouraging a rapid transition
-to a strict policy.  If the purported sender does not accept the DSN,
-then the message is rejected.  The CBV status is cached to avoid
-annoying the purported sender with too many DSNs.  Currently, the DSN
-is repeated to the same sender once per month.
-<pre>
-2005Jul24 15:41:33 [801] mail from <Aitp@horafeliz.com> ()
-2005Jul24 15:41:33 [801] Received-SPF: softfail (mail.bmsi.com: transitioning domain of horafeliz.com
-	does not designate 221.184.83.185 as permitted sender)
-	client-ip=221.184.83.185; envelope-from=Aitp@horafeliz.com;
-	helo=p8185-ipad30funabasi.chiba.ocn.ne.jp;
-2005Jul24 15:41:33 [801] rcpt to <david@example.com> ()
-2005Jul24 15:41:35 [801] Subject: Microsoft, Adobe, Macromedia, Corel software. Up to 80% discount.
-2005Jul24 15:41:35 [801] X-Mailer: Microsoft Outlook, Build 10.0.2605
-2005Jul24 15:41:35 [801] CBV: Aitp@horafeliz.com
-2005Jul24 15:41:38 [801] REJECT: CBV: 550 <Aitp@horafeliz.com>: User unknown
-</pre>
-</td></tr>
+With an effective SPF result in hand, we consult the sendmail access
+database to find our receiver policy for the sender.  

-<tr><th>FAIL</th><td>
-The message is rejected with a reference the SPF why page.
-<pre>
-2005Jul30 19:53:27 [94070] connect from [212.70.52.16] at ('212.70.52.16', 3192) EXTERNAL DYN
-2005Jul30 19:53:27 [94070] hello from winzip.com
-2005Jul30 19:53:27 [94070] mail from <dan@winzip.com> ()
-2005Jul30 19:53:27 [94070] REJECT: SPF fail 550 SPF fail:
-	see http://openspf.com/why.html?sender=dan@winzip.com&ip=212.70.52.16
-</pre>
+<table border=1>
+<tr><th>REJECT</th><td>
+Reject the sender with a 550 5.7.1 SMTP code.  The SMTP rejection
+includes a detailed description of the problem.
 </td></tr>
-
-<tr><th>PERMERROR</th><td>
-Permanent errors were called "unknown", and are still show that way
-in the log.  The message is rejected.  Previously, we enabled "lax" parsing
-of the SPF record, but rejecting is better because it informs the
-sender about their problem.  The next milter version will 
-look for a local substitute SPF record (as for a missing SPF record)
-before rejecting.  This will inform the sender of their problem, but
-also let the receiver install a temporary workaround.
-<pre>
-2005Jul24 18:05:37 [2312] mail from <b-mihdbcgaacaa-becibijh-000-@msg.euxiphipops.com> () 
-2005Jul24 18:05:37 [2312] REJECT: SPF unknown 550 SPF Permanent Error:
-	include mechanism missing domain: include
-</pre>
-The SPF record for msg.euxiphipops.com looked like this at the time of the
-above error:
-<pre>
-msg.euxiphipops.com	TXT "v=spf1 mx ptr a include"
-</pre>
+<tr><th>CBV</th><td>
+Do a Call Back Validation by connecting to an MX of the sender
+and checking that using the sender as the RCPT TO is not rejected.
+We quit the CBV connection before actualling sending a message.
+If the CBV is rejected, our SMTP connection is rejected with the
+same error code and message.  CBV results are cached.
 </td></tr>
-
-<tr><th>TEMPERROR</th><td>
-Temporary errors result in a 451 "Try again later" response.  The sender
-should retry the message at a later time.
-<pre>
-2005Jul24 07:33:13 [29846] mail from <quickenloans@rate.quicken.com> ('SIZE=73775', 'BODY=8BITMIME')
-2005Jul24 07:33:43 [29846] TEMPFAIL: SPF error 450 SPF Temporary Error: DNS Timeout
-</pre>
+<tr><th>DSN</th><td>
+Do a Call Back Validation by connecting to an MX of the sender
+and checking that using the sender as the RCPT TO is not rejected.
+Unlike a CBV, we continue on to data and send a detailed message
+explaining the problem.  This can be useful for reporting PermError
+or SoftFail to the sender.  Keep in mind that for any result other
+than 'pass', the sender could be forged, and your DSN could annoy the
+wrong person.  However, a SoftFail result is requesting such feedback
+for debugging and a PermError result needs to be fixed by the sender ASAP
+whether forged or not.  DSN results are cached so that senders are
+annoyed only weekly.
+</td></tr>
+<tr><th>OK</th><td>
+Accept the sender.  The message may still be rejected via reputation
+or content filtering.
 </td></tr>
-
 </table>
+
+<h3> SPF policy syntax </h3>
+
+First, the full sender is checked:
+<pre>
+SPF-Fail:abeb@adelphia.net     DSN
+</pre>
+This says to accept mail from that adelphia.net user despite the
+SPF fail, but only after annoying them with a DSN about their ISP's broken
+policy. 
+
+If there is no match on the full sender, the domain is checked:
+<pre>
+SPF-Neutral:aol.com     REJECT
+</pre>
+This says to reject mail from AOL with an SPF result of neutral.
+This means AOL users can't use their AOL address with another mail service
+to send us mail.  This is good because the other mail service is 
+likely a badly configured greeting card site or a virus.
+
+Finally, a default policy for the result is checked.  While there are program
+defaults, you should have defaults in the access database for SPF results:
+<pre>
+SPF-Neutral:            CBV
+SPF-Softfail:           DSN
+SPF-PermError:          DSN
+SPF-TempError:          REJECT
+SPF-None:               REJECT
+SPF-Fail:               REJECT
+SPF-Pass:               OK
+</pre>
+
+<h2> Reputation </h2>
+
+If the sender has not been rejected by this point, and if a GOSSiP server is
+configured, we consult GOSSiP for the reputation score of the sender and
+SPF result.  The score is a number from -100 to 100 with a confidence
+percentage from 0 to 100.  A really bad reputation (less than -50 with
+confidence greater than 3) is rejected.  Note that the reputation is tracked
+independently for each SPF result and sender combination.  So aol.com:neutral
+might have a really bad reputation, while aol.com:pass would be ok.
+Furthermore, when a sender finally publishes an SPF policy and starts
+getting SPF pass, their reputation is effectively reset.
@@ -0,0 +1,155 @@
+## To roll your own milter, create a class that extends Milter.  
+#  See the pymilter project at http://bmsi.com/python/milter.html
+#  based on Sendmail's milter API http://www.milter.org/milter_api/api.html
+#  This code is open-source on the same terms as Python.
+
+## Milter calls methods of your class at milter events.
+## Return REJECT,TEMPFAIL,ACCEPT to short circuit processing for a message.
+## You can also add/del recipients, replacebody, add/del headers, etc.
+
+import Milter
+import StringIO
+import time
+import email
+from socket import AF_INET, AF_INET6
+
+def parse_addr(t):
+  """Split email into user,domain.
+
+  >>> parse_addr('user@example.com')
+  ['user', 'example.com']
+  >>> parse_addr('"user@example.com"')
+  ['user@example.com']
+  >>> parse_addr('"user@bar"@example.com')
+  ['user@bar', 'example.com']
+  >>> parse_addr('foo')
+  ['foo']
+  """
+  if t.startswith('<') and t.endswith('>'): t = t[1:-1]
+  if t.startswith('"'):
+    if t.endswith('"'): return [t[1:-1]]
+    pos = t.find('"@')
+    if pos > 0: return [t[1:pos],t[pos+2:]]
+  return t.split('@')
+
+class myMilter(Milter.Milter):
+
+  def __init__(self):  # A new instance with each new connection.
+    self.id = Milter.uniqueID()  # Integer incremented with each call.
+
+  # each connection runs in its own thread and has its own myMilter
+  # instance.  Python code must be thread safe.  This is trivial if only stuff
+  # in myMilter instances is referenced.
+  def connect(self, IPname, family, hostaddr):
+    # (self, 'ip068.subnet71.example.com', AF_INET, ('215.183.71.68', 4720) )
+    # (self, 'ip6.mxout.example.com', AF_INET6,
+    #	('3ffe:80e8:d8::1', 4720, 1, 0) )
+    self.IP = hostaddr[0]
+    self.port = hostaddr[1]
+    if family == AF_INET6:
+      self.flow = hostaddr[2]
+      self.scope = hostaddr[3]
+    else:
+      self.flow = None
+      self.scope = None
+    self.IPname = IPname  # Name from a reverse IP lookup
+    self.H = None
+    self.fp = None
+    self.receiver = self.getsymval('j')
+    self.log("connect from %s at %s" % (IPname, hostaddr) )
+    
+    return Milter.CONTINUE
+
+
+  ##  def hello(self,hostname):
+  def hello(self, heloname):
+    # (self, 'mailout17.dallas.texas.example.com')
+    self.H = heloname
+    self.log("HELO %s" % heloname)
+    if heloname.find('.') < 0:	# illegal helo name
+      # NOTE: example only - too many real braindead clients to reject on this
+      self.setreply('550','5.7.1','Sheesh people!  Use a proper helo name!')
+      return Milter.REJECT
+      
+    return Milter.CONTINUE
+
+  ##  def envfrom(self,f,*str):
+  def envfrom(self, mailfrom, *str):
+    self.F = mailfrom
+    self.R = []  # list of recipients
+    self.fromparms = Milter.dictfromlist(str)	# ESMTP parms
+    self.user = self.getsymval('{auth_authen}')	# authenticated user
+    self.log("mail from:", mailfrom, *str)
+    self.fp = StringIO.StringIO()
+    self.canon_from = '@'.join(parse_addr(mailfrom))
+    self.fp.write('From %s %s\n' % (self.canon_from,time.ctime()))
+    return Milter.CONTINUE
+
+
+  ##  def envrcpt(self, to, *str):
+  def envrcpt(self, recipient, *str):
+    rcptinfo = to,Milter.dictfromlist(str)
+    self.R.append(rcptinfo)
+    
+    return Milter.CONTINUE
+
+
+  def header(self, name, hval):
+    self.fp.write("%s: %s\n" % (name,hval))	# add header to buffer
+    return Milter.CONTINUE
+
+
+  def eoh(self):
+    self.fp.write("\n")				# terminate headers
+    return Milter.CONTINUE
+
+
+  def body(self, chunk):
+    self.fp.write(chunk)
+    return Milter.CONTINUE
+
+
+  def eom(self):
+    self.fp.seek(0)
+    msg = email.message_from_file(self.fp)
+    self.setreply('250','2.5.1','Grokked by pymilter')
+    # many milter functions can only be called from eom()
+    # example of adding a Bcc:
+    self.addrcpt('<%s>' % 'spy@example.com')
+    return Milter.ACCEPT
+
+
+  def close(self):
+    # always called, even when abort is called.  Clean up
+    # any external resources here.
+    return Milter.CONTINUE
+
+  def abort(self):
+    # client disconnected prematurely
+    return Milter.CONTINUE
+
+  ## === Support Functions ===
+
+  def log(self,*msg):
+    print "%s [%d]" % (time.strftime('%Y%b%d %H:%M:%S'),self.id),
+    # 2005Oct13 02:34:11 [1] msg1 msg2 msg3 ...
+    for i in msg: print i,
+    print
+
+
+## ===
+    
+def main():
+  # Register to have the Milter factory create instances of your class:
+  Milter.factory = myMilter
+  flags = Milter.CHGBODY + Milter.CHGHDRS + Milter.ADDHDRS
+  flags += Milter.ADDRCPT
+  flags += Milter.DELRCPT
+  Milter.set_flags(flags)       # tell Sendmail which features we use
+  print "%s milter startup" % time.strftime('%Y%b%d %H:%M:%S')
+  sys.stdout.flush()
+  Milter.runmilter("pythonfilter",socketname,timeout)
+  print "%s bms milter shutdown" % time.strftime('%Y%b%d %H:%M:%S')
+
+if __name__ == "__main__":
+  main()
@@ -1,4 +1,6 @@
 [milter]
+# the directory with log and data files
+datadir = /var/log/milter
 # the socket used to communicate with sendmail.  Must match sendmail.cf
 socket=/var/run/milter/pythonsock
 # where to save original copies of defanged and failed messages
@@ -6,9 +8,10 @@ tempdir = /var/log/milter/save
 # how long to wait for a response from sendmail before giving up 
 ;timeout=600
 log_headers = 0
-# connection ips and hostnames are matched against this glob style list
-# to recognize internal senders.
-;internal_connect = 192.168.*.*,127.*
+# Connection ips and hostnames are matched against this glob style list
+# to recognize internal senders.  You probably need to change this.
+# The default is a good guess to try and prevent newbie frustration.
+internal_connect = 192.168.0.0/16,127.*

 # mail that is not an internal_connect and claims to be from an
 # internal domain is rejected.  Furthermore, internal mail that
@@ -23,6 +26,12 @@ log_headers = 0
 # SPF checks are bypassed for internal connections and trusted relays.
 ;trusted_relay = 1.2.3.4, 66.12.34.56

+# Relaying to these domains is allowed from internal connections only.
+# You might want to restrict aol.com, for instance, so that stupid
+# users don't forward their spam to aol for filtering and get your MTA 
+# blacklisted by aol.
+;private_relay = aol.com, yahoo.com
+
 # Reject external senders with hello names no legit external sender would use.
 # SPF will do this also, but listing your own domain and mailserver here
 # will save some DNS lookups when rejecting certain viruses.
@@ -31,17 +40,20 @@ log_headers = 0
 # Reject mail for domains mentioned unless user is mentioned here also
 ;check_user = joe@mycorp.com, mary@mycorp.com, file:bigcorp.com

+# Treat localparts in milter.cfg as case-insensitive
+case_sensitive_localpart = true
+
 # features intended to filter or block incoming mail
 [defang]

 # do virus scanning on attached messages also
-scan_rfc822 = 1
+scan_rfc822 = 0
 # do virus scanning on attached zipfiles also
 scan_zip = 0
 # Comment out scripts in HTML attachments.  Can be CPU intensive.
 scan_html = 0
 # reject messages with asian fonts because we can't read them
-block_chinese = 1
+block_chinese = 0
 # list users who hate forwarded mail
 ;block_forward = egghead@mycorp.com, busybee@mycorp.com
 # reject mail with these case insensitive strings in the subject
@@ -50,7 +62,7 @@ porn_words = penis, breast, pussy, horse cock, porn, xenical, diet pill, d1ck,
 	p-e-n-i-s, hydrocodone, vicodin, xanax, vicod1n, x@nax, diazepam,
 	v1@gra, xan@x, cialis, ci@lis, frëe, xãnax, valíum, vãlium, via-gra,
 	x@n3x, vicod3n, penís, c0d1n, phentermine, en1arge, dip1oma, v1codin,
-	valium, rolex, sexual, fuck, adv1t
+	valium, rolex, sexual, fuck, adv1t, vgaira, medz, acai berry
 # reject mail with these case sensitive strings in the subject
 spam_words = $$$, !!!, XXX, FREE, HGH
 # attachments with these extensions will be replaced with a warning
@@ -75,7 +87,7 @@ reject_spoofed = 0
 # refuses mail from user names commonly abused in that way.
 ;banned_users = postmaster, mailer-daemon, clamav

-# See http://spf.pobox.com for more info on SPF.
+# See http://www.openspf.com for more info on SPF.
 [spf]
 # namespace where SPF records can be supplied for domains without one
 # records are searched for under _spf.domain.com
@@ -125,11 +137,15 @@ blind = 1
 # discard outgoing mail without alerting sender
 # can be used in conjunction with wiretap to censor outgoing mail
 ;discard_users = canned@bigcorp.com
+# archive copies all delivered mail to a file
+;mail_archive = /var/log/mail_archive
+
 #
 # smart aliases trigger on both sender and recipient
+#   alias = sender, recipient[, destination]
 #
-;smart_alias = copycust,walter,spy1,spy2
-# multiple wiretap monitors
+[smart_alias]
+# multiple wiretap monitors.  Smart aliases are applied after wiretap.
 ;spy1 = disloyal@bigcorp.com,spy@bigcorp.com
 ;spy2 = bigmouth@bigcorp.com,spy@bigcorp.com
 # mail from client@clientcorp.com to sue@bigcorp.com is redirected to 
@@ -142,7 +158,7 @@ blind = 1
 ;walter1 = cust@othercorp.com,walter@bigcorp.com,boss@bigcorp.com,
 ;	walter@bigcorp.com
 ;bulk = soruce@telex.com,bob@jsconnor.com
-;bulk = soruce@telex.com,larry@jsconnor.com
+;bulk1 = soruce@telex.com,larry@jsconnor.com,bulk

 # See http://bmsi.com/python/dspam.html
 [dspam]
@@ -175,6 +191,11 @@ blind = 1

 # Map email addresses and aliases to dspam users
 ;dspam_users=david,goliath,spam,falsepositive
+# List dspam users which train on all delivered messages, as opposed to
+# "train on error" which trains only when a spam or falsepositive is reported.
+# Training mode will build the dictionary faster, but requires close attention
+# so as not to miss any spam or false positives.
+;dspam_train=goliath
 ;david=david@foocorp.com,david.yelnetz@foocorp.com,david@bar.foocorp.com
 ;goliath=giant@foocorp.com,goliath.philistine@foocorp.com
 # address to forward spam to.  milter will process these and not deliver
@@ -189,3 +210,20 @@ blind = 1
 # delivered.
 ;dspam_screener=david,goliath
 # The dspam CGI can also be used: logins must match dspam users
+
+# Optional pygossip interface
+#
+# GOSSiP tracks reputation of domain:qualifier pairs.  For instance,
+# the reputation of example.com:SPF is tracked separately from
+# example.com:neutral.  Currently qualifiers are
+# SPF,neutral,softfail,fail,permerror,GUESS,HELO
+[gossip]
+# Use a dedicated GOSSiP server.  If not specified, a local database
+# will be used.
+;server=host:11900
+# To include peers of a peer in reputation, set ttl=2
+;ttl=1
+# If a local database is used, also consult these GOSSiP servers about 
+# domains.  Peer reputation is also tracked as to how often they
+# agree with us, and weighted accordingly.
+;peers=host1:port,host2
@@ -8,7 +8,7 @@
 # config: /etc/mail/pymilter.cfg
 # pidfile: /var/run/milter/milter.pid

-python="python2.3"
+python="python2.4"

 pidof() {
 	set - ""
@@ -23,7 +23,7 @@ pidof() {
 # Source function library.
 . /etc/rc.d/init.d/functions

-[ -x /var/log/milter/start.sh ] || exit 0
+[ -x /usr/lib/pymilter/start.sh ] || exit 0

 RETVAL=0
 prog="milter"
@@ -32,7 +32,11 @@ start() {
 	# Start daemons.

 	echo -n "Starting $prog: "
-	daemon --check milter --user mail /var/log/milter/start.sh
+        if ! test -d /var/run/milter; then
+		mkdir -p /var/run/milter
+		chown mail:mail /var/run/milter
+	fi
+	daemon --check milter --user mail /usr/lib/pymilter/start.sh milter bms
 	RETVAL=$?
 	echo
 	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/milter
@@ -42,7 +46,7 @@ start() {
 stop() {
 	# Stop daemons.
 	echo -n "Shutting down $prog: "
-	killproc milter
+	killproc -d 9 milter
 	RETVAL=$?
 	echo
 	[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/milter
@@ -8,7 +8,7 @@
 # config: /etc/mail/pymilter.cfg
 # pidfile: /var/run/milter/milter.pid

-python="python2.3"
+python="python2.4"

 pidof() {
 	set - ""
@@ -23,7 +23,7 @@ pidof() {
 # Source function library.
 . /etc/rc.d/init.d/functions

-[ -x /var/log/milter/start.sh ] || exit 0
+[ -x /usr/lib/pymilter/start.sh ] || exit 0

 RETVAL=0
 prog="milter"
@@ -32,7 +32,11 @@ start() {
 	# Start daemons.

 	echo -n "Starting $prog: "
-	daemon --check milter --user mail /var/log/milter/start.sh
+        if ! test -d /var/run/milter; then
+		mkdir -p /var/run/milter
+		chown mail:mail /var/run/milter
+	fi
+	daemon --check milter --user mail /usr/lib/pymilter/start.sh milter bms
 	RETVAL=$?
 	echo
 	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/milter
@@ -1,19 +1,20 @@
 /* Copyright (C) 2001  James Niemira (niemira@colltech.com, urmane@urmane.org)
- * Portions Copyright (C) 2001,2002,2003,2004  Stuart Gathman (stuart@bmsi.com)
+ * Portions Copyright (C) 2001,2002,2003,2004,2005,2006,2007
+ *   Stuart Gathman (stuart@bmsi.com)
 *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version 2
- * of the License, or (at your option) any later version.
+ * This program is free software: you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation, either version 2 of the License, or (at your
+ * option) any later version.
 *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
 *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 *
 * milterContext object and thread interface contributed by
 * 	Stuart D. Gathman <stuart@bmsi.com>
@@ -34,6 +35,12 @@ $ python setup.py help
     libraries=["milter","smutil","resolv"]

 * $Log$
+ * Revision 1.10  2006/02/12 02:00:42  customdesigned
+ * Resolve FIXME for wrap_close.
+ *
+ * Revision 1.9  2005/12/23 21:46:36  customdesigned
+ * Compile on sendmail-8.12 (ifdef SMFIR_INSHEADER)
+ *
 * Revision 1.8  2005/10/20 23:23:36  customdesigned
 * Include smfi_progress is SMFIR_PROGRESS defined
 *
@@ -286,7 +293,7 @@ _find_context(PyObject *c) {
  if (c->ob_type == &milter_ContextType) {
    milter_ContextObject *self = (milter_ContextObject *)c;
    ctx = self->ctx;
-    if (smfi_getpriv(ctx) != self)
+    if (ctx != NULL && smfi_getpriv(ctx) != self)
      ctx = NULL;
  }
  if (ctx == NULL)
@@ -294,23 +301,6 @@ _find_context(PyObject *c) {
  return ctx;
 }

-/* Release the Python Context for a SMFICTX.  */
-static void
-_clear_context(SMFICTX *ctx) {
-  milter_ContextObject *self = smfi_getpriv(ctx);
-  if (self) {
-    PyThreadState *t = self->t;
-    PyEval_AcquireThread(t);
-    self->t = 0;
-    self->ctx = 0;
-    smfi_setpriv(ctx,0);
-    Py_DECREF(self);
-    PyThreadState_Clear(t);
-    PyEval_ReleaseThread(t);
-    PyThreadState_Delete(t);
-  }
-}
-
 static void
 milter_Context_dealloc(PyObject *s) {
  milter_ContextObject *self = (milter_ContextObject *)s;
@@ -544,13 +534,19 @@ milter_set_exception_policy(PyObject *self, PyObject *args) {
  return NULL;
 }

+static void
+_release_thread(PyThreadState *t) {
+  if (t != NULL)
+    PyEval_ReleaseThread(t);
+}
+
 /** Report and clear any python exception before returning to libmilter. 
  The interpreter is locked when we are called, and we unlock it.  */
 static int _report_exception(milter_ContextObject *self) {
  if (PyErr_Occurred()) {
    PyErr_Print();
    PyErr_Clear();	/* must clear since not returning to python */
-    PyEval_ReleaseThread(self->t);
+    _release_thread(self->t);
    switch (exception_policy) {
      case SMFIS_REJECT:
 	smfi_setreply(self->ctx, "554", "5.3.0", "Filter failure");
@@ -561,7 +557,7 @@ static int _report_exception(milter_ContextObject *self) {
    }
    return SMFIS_CONTINUE;
  }
-  PyEval_ReleaseThread(self->t);
+  _release_thread(self->t);
  return SMFIS_CONTINUE;
 }

@@ -580,7 +576,7 @@ _generic_wrapper(milter_ContextObject *self, PyObject *cb, PyObject *arglist) {
  retval = PyInt_AsLong(result);
  Py_DECREF(result);
  if (PyErr_Occurred()) return _report_exception(self);
-  PyEval_ReleaseThread(self->t);
+  _release_thread(self->t);
  return retval;
 }

@@ -777,17 +773,23 @@ milter_wrap_close(SMFICTX *ctx) {
  PyObject *cb = close_callback;
  milter_ContextObject *self = smfi_getpriv(ctx);
  int r = SMFIS_CONTINUE;
-  if (self != NULL && cb != NULL && self->ctx == ctx) {
-    PyObject *arglist;
-    PyEval_AcquireThread(self->t);
-    arglist = Py_BuildValue("(O)", self);
-    r = _generic_wrapper(self, cb, arglist);
+  if (self != NULL) {
+    PyThreadState *t = self->t;
+    PyEval_AcquireThread(t);
+    self->t = 0;
+    if (cb != NULL && self->ctx == ctx) {
+      PyObject *arglist = Py_BuildValue("(O)", self);
+      /* Call python close callback, but do not ReleaseThread, because
+       * self->t is NULL */
+      r = _generic_wrapper(self, cb, arglist);
+    }
+    self->ctx = 0;
+    smfi_setpriv(ctx,0);
+    Py_DECREF(self);
+    PyThreadState_Clear(t);
+    PyEval_ReleaseThread(t);
+    PyThreadState_Delete(t);
  }
-  /* FIXME: It is inefficient to have released the interp lock only to
-     acquire it again in _clear_context.  We can tell _generic_return and
-     friends not to release the lock by, for instance, setting self->t to NULL.
-     However, first we make it work. */
-  _clear_context(ctx);
  return r;
 }

@@ -1,4 +1,7 @@
+To: %(sender)s
+From: postmaster@%(receiver)s
 Subject: SPF %(result)s (POSSIBLE FORGERY)
+Auto-Submitted: auto-generated (sender verification)

 This is an automatically generated Delivery Status Notification.

@@ -27,6 +30,8 @@ If you never sent the above message, then your domain has been forged.
 Your mail admin needs to publish a strict SPF record so that I can reject
 those forgeries instead of bugging you about them.  

+See http://openspf.org for details.
+
 If you need further assistance, please do not hesitate to contact me.

 Kind regards,
@@ -1,4 +1,7 @@
+To: %(sender)s
+From: postmaster@%(receiver)s
 Subject: Critical SPF configuration error
+Auto-Submitted: auto-generated (configuration error)

 This is an automatically generated Delivery Status Notification.

@@ -11,6 +14,7 @@ Delivery to the following recipients has been delayed.
 	%(rcpt)s

 Subject: %(subject)s 
+Received-SPF: %(spf_result)s

 Your spf record has a permanent error.  The error was:

@@ -1,75 +1,81 @@
-%define name milter
-%define version 0.8.5
-%define release 1.RH7
+# This spec file contains 2 noarch packages in addition to the pymilter
+# module.  To compile all three on 32-bit Intel, use:
+# rpmbuild -ba --target=i386,noarch pymilter.spec
+
+%define __python python2.4
+%define version 0.8.10
+%define release 2%{?dist}.py24
 # what version of RH are we building for?
-%define redhat9 0
-%define redhat7 1
-%define redhat6 0
+%define redhat7 0

 # Options for Redhat version 6.x:
-# rpm -ba|--rebuild --define "rh6 1"
-%{?rh6:%define redhat7 0}
-%{?rh6:%define redhat6 1}
+# rpm -ba|--rebuild --define "rh7 1"
+%{?rh7:%define redhat7 1}

 # some systems dont have initrddir defined
 %{?_initrddir:%define _initrddir /etc/rc.d/init.d}

-%if %{redhat9}
-%define sysvinit milter.rc
-%else	# Redhat 7.x and earlier (multiple ps lines per thread)
+%if %{redhat7} 
+# Redhat 7.x and earlier (multiple ps lines per thread)
 %define sysvinit milter.rc7
+%else	
+%define sysvinit milter.rc
 %endif
 # RH9, other systems (single ps line per process)
-%ifos Linux
-%define python python2.4
+%ifos aix4.1
+%define libdir /var/log/milter
 %else
-%define python python
+%define libdir /usr/lib/pymilter
 %endif

-Summary: Python interface to sendmail milter API
-Name: %{name}
+%ifarch noarch
+Name: milter
+Group: Applications/System
+Summary:  BMS spam and reputation milter
 Version: %{version}
 Release: %{release}
-Source: %{name}-%{version}.tar.gz
+Source: pymilter-%{version}.tar.gz
 #Patch: %{name}-%{version}.patch
-Copyright: GPL
+License: GPL
 Group: Development/Libraries
 BuildRoot: %{_tmppath}/%{name}-buildroot
 Prefix: %{_prefix}
 Vendor: Stuart D. Gathman <stuart@bmsi.com>
 Packager: Stuart D. Gathman <stuart@bmsi.com>
 Url: http://www.bmsi.com/python/milter.html
-Requires: %{python} >= 2.4, sendmail >= 8.13
+Requires: %{__python} >= 2.4, pyspf >= 2.0.4, pymilter
 %ifos Linux
 Requires: chkconfig
 %endif
-BuildRequires: %{python}-devel , sendmail-devel >= 8.13

-%description
-This is a python extension module to enable python scripts to
-attach to sendmail's libmilter functionality.  Additional python
-modules provide for navigating and modifying MIME parts.
+%description -n milter
+A complex but effective spam filtering, SPF checking, and reputation tracking
+mail application.  It uses pydspam if installed for bayesian filtering.
+
+%package spf
+Group: Applications/System
+Summary:  BMS spam and reputation milter
+Requires: pyspf >= 2.0.4, pymilter
+Obsoletes: pymilter-spf
+
+%description spf
+A simple mail filter to add Received-SPF headers and reject forged mail.
+Rejection policy is configured via sendmail access file.

 %prep
-%setup
-#%patch -p0 -b .bms
-
-%build
-if %{redhat9}; then
-  LDFLAGS="-g"
-else
-  LDFLAGS="-s"
-fi
-env CFLAGS="$RPM_OPT_FLAGS" LDFLAGS="$LDFLAGS" %{python} setup.py build
+%setup -n pymilter-%{version}
+#patch -p0 -b .bms

 %install
 rm -rf $RPM_BUILD_ROOT
-%{python} setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
 mkdir -p $RPM_BUILD_ROOT/var/log/milter
 mkdir -p $RPM_BUILD_ROOT/etc/mail
 mkdir $RPM_BUILD_ROOT/var/log/milter/save
-cp bms.py *.txt $RPM_BUILD_ROOT/var/log/milter
+mkdir -p $RPM_BUILD_ROOT%{libdir}
+cp *.txt $RPM_BUILD_ROOT/var/log/milter
+cp bms.py spfmilter.py $RPM_BUILD_ROOT%{libdir}
 cp milter.cfg $RPM_BUILD_ROOT/etc/mail/pymilter.cfg
+cp spfmilter.cfg $RPM_BUILD_ROOT/etc/mail

 # logfile rotation
 mkdir -p $RPM_BUILD_ROOT/etc/logrotate.d
@@ -78,6 +84,11 @@ cat >$RPM_BUILD_ROOT/etc/logrotate.d/milter <<'EOF'
  copytruncate
  compress
 }
+/var/log/milter/banned_ips {
+  rotate 7
+  daily
+  copytruncate
+}
 EOF

 # purge saved defanged message copies
@@ -91,67 +102,65 @@ cat >$RPM_BUILD_ROOT/etc/cron.daily/milter <<'EOF'
 #!/bin/sh

 find /var/log/milter/save -mtime +7 | xargs $R rm
+# work around memory leak
+/etc/init.d/milter condrestart
 EOF
 chmod a+x $RPM_BUILD_ROOT/etc/cron.daily/milter

-%ifos aix4.1
-cat >$RPM_BUILD_ROOT/var/log/milter/start.sh <<'EOF'
-#!/bin/sh
-cd /var/log/milter
-# uncomment to enable sgmlop if installed
-#export PYTHONPATH=/usr/local/lib/python2.1/site-packages
-exec /usr/local/bin/python bms.py >>milter.log 2>&1
-EOF
-%else
-cat >$RPM_BUILD_ROOT/var/log/milter/start.sh <<'EOF'
-#!/bin/sh
-cd /var/log/milter
-exec >>milter.log 2>&1
-%{python} bms.py &
-echo $! >/var/run/milter/milter.pid
-EOF
+%ifnos aix4.1
 mkdir -p $RPM_BUILD_ROOT/etc/rc.d/init.d
 cp %{sysvinit} $RPM_BUILD_ROOT/etc/rc.d/init.d/milter
+cp spfmilter.rc $RPM_BUILD_ROOT/etc/rc.d/init.d/spfmilter
 ed $RPM_BUILD_ROOT/etc/rc.d/init.d/milter <<'EOF'
 /^python=/
 c
-python="%{python}"
+python="%{__python}"
 .
 w
 q
 EOF
-%endif
-chmod a+x $RPM_BUILD_ROOT/var/log/milter/start.sh
+ed $RPM_BUILD_ROOT/etc/rc.d/init.d/spfmilter <<'EOF'
+/^python=/
+c
+python="%{__python}"
+.
+w
+q
+EOF
+%endif	# aix4.1

-mkdir -p $RPM_BUILD_ROOT/var/run/milter
 mkdir -p $RPM_BUILD_ROOT/usr/share/sendmail-cf/hack
 cp -p rhsbl.m4 $RPM_BUILD_ROOT/usr/share/sendmail-cf/hack

 %ifos aix4.1
 %post
-mkssys -s milter -p /var/log/milter/start.sh -u 25 -S -n 15 -f 9 -G mail || :
+mkssys -s milter -p %{libdir}/start.sh -u 25 -S -n 15 -f 9 -G mail || :

 %preun
 if [ $1 = 0 ]; then
  rmssys -s milter || :
 fi
-%else
-%post
+%else # not aix4.1
+%post -n milter
 #echo "pythonsock has moved to /var/run/milter, update /etc/mail/sendmail.cf"
 /sbin/chkconfig --add milter

-%preun
+%preun -n milter
 if [ $1 = 0 ]; then
  /sbin/chkconfig --del milter
 fi
-%endif
+%post spf
+#echo "pythonsock has moved to /var/run/milter, update /etc/mail/sendmail.cf"
+/sbin/chkconfig --add spfmilter

-%clean
-rm -rf $RPM_BUILD_ROOT
+%preun spf
+if [ $1 = 0 ]; then
+  /sbin/chkconfig --del spfmilter
+fi
+%endif # aix4.1

-%files -f INSTALLED_FILES
+%files 
 %defattr(-,root,root)
-%doc README HOWTO NEWS TODO CREDITS sample.py
 /etc/logrotate.d/milter
 /etc/cron.daily/milter
 %ifos aix4.1
@@ -161,19 +170,156 @@ rm -rf $RPM_BUILD_ROOT
 %defattr(-,mail,mail)
 %endif
 %dir /var/log/milter
-%dir /var/run/milter
 %dir /var/log/milter/save
-%config /var/log/milter/start.sh
-%config /var/log/milter/bms.py
+%config %{libdir}/bms.py
 %config(noreplace) /var/log/milter/strike3.txt
 %config(noreplace) /var/log/milter/softfail.txt
+%config(noreplace) /var/log/milter/fail.txt
 %config(noreplace) /var/log/milter/neutral.txt
 %config(noreplace) /var/log/milter/quarantine.txt
 %config(noreplace) /var/log/milter/permerror.txt
+%config(noreplace) /var/log/milter/temperror.txt
 %config(noreplace) /etc/mail/pymilter.cfg
 /usr/share/sendmail-cf/hack/rhsbl.m4

+%files spf
+%defattr(-,root,root)
+%dir /var/log/milter
+%{libdir}/spfmilter.py*
+%config(noreplace) /etc/mail/spfmilter.cfg
+/etc/rc.d/init.d/spfmilter
+
+%else # not noarch
+
+%define name pymilter
+Summary: Python interface to sendmail milter API
+Name: %{name}
+Version: %{version}
+Release: %{release}
+Source: %{name}-%{version}.tar.gz
+#Patch: %{name}-%{version}.patch
+License: GPL
+Group: Development/Libraries
+BuildRoot: %{_tmppath}/%{name}-buildroot
+Prefix: %{_prefix}
+Vendor: Stuart D. Gathman <stuart@bmsi.com>
+Packager: Stuart D. Gathman <stuart@bmsi.com>
+Url: http://www.bmsi.com/python/milter.html
+Requires: %{__python} >= 2.4, sendmail >= 8.13
+BuildRequires: %{__python}-devel >= 2.4, sendmail-devel >= 8.13
+
+%description
+This is a python extension module to enable python scripts to
+attach to sendmail's libmilter functionality.  Additional python
+modules provide for navigating and modifying MIME parts, sending
+DSNs, and doing CBV.
+
+%prep
+%setup
+#patch -p0 -b .bms
+
+%build
+%if %{redhat7} 
+  LDFLAGS="-s"
+%else # Redhat builds debug packages after 7.3
+  LDFLAGS="-g"
+%endif
+env CFLAGS="$RPM_OPT_FLAGS" LDFLAGS="$LDFLAGS" %{__python} setup.py build
+
+%install
+rm -rf $RPM_BUILD_ROOT
+%{__python} setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
+mkdir -p $RPM_BUILD_ROOT/var/run/milter
+mkdir -p $RPM_BUILD_ROOT%{libdir}
+%ifos aix4.1
+cat >$RPM_BUILD_ROOT%{libdir}/start.sh <<'EOF'
+#!/bin/sh
+cd /var/log/milter
+# uncomment to enable sgmlop if installed
+#export PYTHONPATH=/usr/local/lib/python2.1/site-packages
+exec /usr/local/bin/python bms.py >>milter.log 2>&1
+EOF
+%else # not aix4.1
+cp start.sh $RPM_BUILD_ROOT%{libdir}
+ed $RPM_BUILD_ROOT%{libdir}/start.sh <<'EOF'
+/^python=/
+c
+python="%{__python}"
+.
+w
+q
+EOF
+%endif
+chmod a+x $RPM_BUILD_ROOT%{libdir}/start.sh
+%if !%{redhat7}
+#grep '.pyc$' INSTALLED_FILES | sed -e 's/c$/o/' >>INSTALLED_FILES
+%endif
+
+# start.sh is used by spfmilter and milter, and could be used by
+# other milters running on redhat
+%files -f INSTALLED_FILES
+%defattr(-,root,root)
+%doc README HOWTO ChangeLog NEWS TODO CREDITS sample.py milter-template.py
+%config %{libdir}/start.sh
+%dir %attr(0755,mail,mail) /var/run/milter
+
+%endif # noarch
+
+%clean
+rm -rf $RPM_BUILD_ROOT
+
 %changelog
+* Mon Aug 25 2008 Stuart Gathman <stuart@bmsi.com> 0.8.10-2
+- /var/run/milter directory must be owned by mail
+* Mon Aug 25 2008 Stuart Gathman <stuart@bmsi.com> 0.8.10-1
+- log rcpt for SRS rejections
+- improved parsing into email and fullname (still 2 self test failures)
+- implement no-DSN CBV, reduce full DSNs
+- check for porn words in MAIL FROM fullname
+- ban IP for too many bad MAIL FROMs or RCPT TOs
+- temperror policy in access
+- no CBV for whitelisted MAIL FROM except permerror, softfail
+- Allow explicitly whitelisted email from banned_users.
+- configure gossip TTL
+* Mon Sep 24 2007 Stuart Gathman <stuart@bmsi.com> 0.8.9-1
+- Use %ifarch hack to build milter and milter-spf packages as noarch
+- Remove spf dependency from dsn.py, add dns.py
+* Fri Jan 05 2007 Stuart Gathman <stuart@bmsi.com> 0.8.8-1
+- move AddrCache, parse_addr, iniplist to Milter package
+- move parse_header to Milter.utils
+- fix plock for missing source and can't change owner/group
+- add sample spfmilter.py milter
+- private_relay config option
+- persist delayed DSN blacklisting
+- handle gossip server restart without disabling gossip
+- split out pymilter and pymilter-spf packages
+- move milter apps to /usr/lib/pymilter
+* Sat Nov 04 2006 Stuart Gathman <stuart@bmsi.com> 0.8.7-1
+- More lame bounce heuristics
+- SPF moved to pyspf RPM
+- wiretap archive option
+- Do plain CBV if missing template
+- SMTP AUTH policy in access
+* Tue May 23 2006 Stuart Gathman <stuart@bmsi.com> 0.8.6-2
+- Support CBV timeout
+- Support fail template, headers in templates
+- Create GOSSiP record only when connection will procede to DATA.
+- More SPF lax heuristics
+- Don't require SPF pass for white/black listing mail from trusted relay.
+- Support localpart wildcard for white and black lists.
+* Thu Feb 23 2006 Stuart Gathman <stuart@bmsi.com> 0.8.6-1
+- Delay reject of unsigned RCPT for postmaster and abuse only
+- Fix dsn reporting of hard permerror
+- Resolve FIXME for wrap_close in miltermodule.c
+- Add Message-ID to DSNs
+- Use signed Message-ID in delayed reject to blacklist senders
+- Auto-train via blacklist and auto-whitelist
+- Don't check userlist for signed MFROM
+- Accept but skip DSPAM and training for whitelisted senders without SPF PASS
+- Report GC stats 
+- Support CIDR matching for IP lists
+- Support pysrs sign feature
+- Support localpart specific SPF policy in access file
 * Thu Dec 29 2005 Stuart Gathman <stuart@bmsi.com> 0.8.5-1
 - Simple trusted_forwarder implementation.
 - Fix access_file neutral policy
@@ -1,4 +1,7 @@
+To: %(sender)s
+From: postmaster@%(receiver)s
 Subject: DELIVERY STATUS (POSSIBLE SPAM)
+Auto-Submitted: auto-generated (content analysis)

 This is an automatically generated Delivery Status Notification.

@@ -3,7 +3,10 @@ import sys
 from distutils.core import setup, Extension

 # FIXME: on some versions of sendmail, smutil is renamed to sm
+# on slackware and debian, leave it out entirely.  It depends
+# on how libmilter was built by the sendmail package.
 libs = ["milter", "smutil"]
+libdirs = ["/usr/lib/libmilter"]    # needed for Debian

 # patch distutils if it can't cope with the "classifiers" or
 # "download_url" keywords
@@ -13,13 +16,13 @@ if sys.version < '2.2.3':
  DistributionMetadata.download_url = None

 # NOTE: importing Milter to obtain version fails when milter.so not built
-setup(name = "milter", version = '0.8.5',
+setup(name = "pymilter", version = '0.8.10',
 	description="Python interface to sendmail milter API",
 	long_description="""\
 This is a python extension module to enable python scripts to
 attach to sendmail's libmilter functionality.  Additional python
 modules provide for navigating and modifying MIME parts, and
-querying SPF records.
+sending DSNs or doing CBVs.
 """,
 	author="Jim Niemira",
 	author_email="urmane@urmane.org",
@@ -27,10 +30,11 @@ querying SPF records.
 	maintainer_email="stuart@bmsi.com",
 	license="GPL",
 	url="http://www.bmsi.com/python/milter.html",
-	py_modules=["mime","spf"],
+	py_modules=["mime"],
 	packages = ['Milter'],
 	ext_modules=[
 	  Extension("milter", ["miltermodule.c"],
+            library_dirs=libdirs,
 	    libraries=libs,
 	    # set MAX_ML_REPLY to 1 for sendmail < 8.13
 	    define_macros = [ ('MAX_ML_REPLY',32) ]
@@ -1,4 +1,7 @@
+To: %(sender)s
+From: postmaster@%(receiver)s
 Subject: SPF %(result)s (POSSIBLE FORGERY)
+Auto-Submitted: auto-generated (configuration error)

 This is an automatically generated Delivery Status Notification.

@@ -14,7 +17,7 @@ Subject: %(subject)s
 Received-SPF: %(spf_result)s

 Your sender policy indicated that the above email was likely forged and that
-feedback was desired.  If you are sending from a foreign ISP,
+feedback was desired for debugging.  If you are sending from a foreign ISP,
 then you may need to follow your home ISPs instructions for configuring
 your outgoing mail server.

@@ -0,0 +1,20 @@
+[milter]
+# The socket used to communicate with sendmail
+socketname = /var/run/milter/spfmiltersock
+# Name of the milter given to sendmail
+name = pyspffilter
+# Trusted relays such as secondary MXes that should not have SPF checked.
+;trusted_relay =
+# Internal networks that should not have SPF checked.
+internal_connect = 127.0.0.1,192.168.0.0/16,10.0.0.0/8
+
+# See http://www.openspf.com for more info on SPF.
+[spf]
+# Use sendmail access map or similar format for detailed spf policy.
+# SPF entries in the access map will override defaults.
+access_file = /etc/mail/access.db
+# Connections that get an SPF pass for a pretend MAIL FROM of 
+# postmaster@sometrustedforwarder.com skip SPF checks for the real MAIL FROM.
+# This is for non-SRS forwarders.  It is a simple implementation that
+# is inefficient for more than a few entries.
+;trusted_forwarder = careerbuilder.com
@@ -0,0 +1,253 @@
+# A simple SPF milter.
+# You must install pyspf for this to work.
+
+# http://www.sendmail.org/doc/sendmail-current/libmilter/docs/installation.html
+
+# Author: Stuart D. Gathman <stuart@bmsi.com>
+# Copyright 2007 Business Management Systems, Inc.
+# This code is under GPL.  See COPYING for details.
+
+import sys
+import Milter
+import spf
+import syslog
+import anydbm
+from Milter.config import MilterConfigParser
+from Milter.utils import iniplist,parse_addr
+
+syslog.openlog('spfmilter',0,syslog.LOG_MAIL)
+
+class Config(object):
+  "Hold configuration options."
+  pass
+
+def read_config(list):
+  "Return new config object."
+  cp = MilterConfigParser()
+  cp.read(list)
+  if cp.has_option('milter','datadir'):
+        os.chdir(cp.get('milter','datadir'))
+  conf = Config()
+  conf.socketname = cp.getdefault('milter','socketname', '/tmp/spfmiltersock')
+  conf.miltername = cp.getdefault('milter','name','pyspffilter')
+  conf.trusted_relay = cp.getlist('milter','trusted_relay')
+  conf.internal_connect = cp.getlist('milter','internal_connect')
+  conf.trusted_forwarder = cp.getlist('spf','trusted_relay')
+  conf.access_file = cp.getdefault('spf','access_file',None)
+  return conf
+
+class SPFPolicy(object):
+  "Get SPF policy by result from sendmail style access file."
+  def __init__(self,sender,access_file=None):
+    self.sender = sender
+    self.domain = sender.split('@')[-1].lower()
+    if access_file:
+      try: acf = anydbm.open(access_file,'r')
+      except: acf = None
+    else: acf = None
+    self.acf = acf
+
+  def getPolicy(self,pfx):
+    acf = self.acf
+    if not acf: return None
+    try:
+      return acf[pfx + self.sender]
+    except KeyError:
+      try:
+	return acf[pfx + self.domain]
+      except KeyError:
+	try:
+	  return acf[pfx]
+	except KeyError:
+	  return None
+  
+class spfMilter(Milter.Milter):
+  "Milter to check SPF.  Each connection gets its own instance."
+
+  def log(self,*msg):
+    syslog.syslog('[%d] %s' % (self.id,' '.join([str(m) for m in msg])))
+
+  def __init__(self):
+    self.mailfrom = None
+    self.id = Milter.uniqueID()
+    # we don't want config used to change during a connection
+    self.conf = config
+
+  # addheader can only be called from eom().  This accumulates added headers
+  # which can then be applied by alter_headers()
+  def add_header(self,name,val,idx=-1):
+    self.new_headers.append((name,val,idx))
+    self.log('%s: %s' % (name,val))
+
+  def connect(self,hostname,unused,hostaddr):
+    self.internal_connection = False
+    self.trusted_relay = False
+    self.hello_name = None
+    # sometimes people put extra space in sendmail config, so we strip
+    self.receiver = self.getsymval('j').strip()
+    if hostaddr and len(hostaddr) > 0:
+      ipaddr = hostaddr[0]
+      if iniplist(ipaddr,self.conf.internal_connect):
+	self.internal_connection = True
+      if iniplist(ipaddr,self.conf.trusted_relay):
+        self.trusted_relay = True
+    else: ipaddr = ''
+    self.connectip = ipaddr
+    if self.internal_connection:
+      connecttype = 'INTERNAL'
+    else:
+      connecttype = 'EXTERNAL'
+    if self.trusted_relay:
+      connecttype += ' TRUSTED'
+    self.log("connect from %s at %s %s" % (hostname,hostaddr,connecttype))
+    return Milter.CONTINUE
+
+  def hello(self,hostname):
+    self.hello_name = hostname
+    self.log("hello from %s" % hostname)
+    return Milter.CONTINUE
+
+  # multiple messages can be received on a single connection
+  # envfrom (MAIL FROM in the SMTP protocol) seems to mark the start
+  # of each message.
+  def envfrom(self,f,*str):
+    self.log("mail from",f,str)
+    if not self.hello_name:
+      self.log('REJECT: missing HELO')
+      self.setreply('550','5.7.1',"It's polite to say helo first.")
+      return Milter.REJECT
+    self.mailfrom = f
+    self.new_headers = []
+    t = parse_addr(f)
+    if len(t) == 2: t[1] = t[1].lower()
+    self.canon_from = '@'.join(t)
+    if not (self.internal_connection or self.trusted_relay) and self.connectip:
+      rc = self.check_spf()
+      if rc != Milter.CONTINUE: return rc
+    return Milter.CONTINUE
+
+  def envrcpt(self,f,*str):
+    return Milter.CONTINUE
+
+  def header(self,name,hval):
+    return Milter.CONTINUE
+
+  def eoh(self):
+    return Milter.CONTINUE
+
+  def eom(self):
+    for name,val,idx in self.new_headers:
+      try:
+	self.addheader(name,val,idx)
+      except:
+	self.addheader(name,val)	# older sendmail can't insheader
+    return Milter.CONTINUE
+
+  def close(self):
+    return Milter.CONTINUE
+
+  def check_spf(self):
+    receiver = self.receiver
+    for tf in self.conf.trusted_forwarder:
+      q = spf.query(self.connectip,'',tf,receiver=receiver,strict=False)
+      res,code,txt = q.check()
+      if res == 'pass':
+        self.log("TRUSTED_FORWARDER:",tf)
+        break
+    else:
+      q = spf.query(self.connectip,self.canon_from,self.hello_name,
+	  receiver=receiver,strict=False)
+      q.set_default_explanation(
+	'SPF fail: see http://openspf.org/why.html?sender=%s&ip=%s' % (q.s,q.i))
+      res,code,txt = q.check()
+    if res not in ('pass','temperror'):
+      if self.mailfrom != '<>':
+	# check hello name via spf unless spf pass
+	h = spf.query(self.connectip,'',self.hello_name,receiver=receiver)
+	hres,hcode,htxt = h.check()
+	if hres in ('deny','fail','neutral','softfail'):
+	  self.log('REJECT: hello SPF: %s 550 %s' % (hres,htxt))
+	  self.setreply('550','5.7.1',htxt,
+	    "The hostname given in your MTA's HELO response is not listed",
+	    "as a legitimate MTA in the SPF records for your domain.  If you",
+	    "get this bounce, the message was not in fact a forgery, and you",
+	    "should IMMEDIATELY notify your email administrator of the problem."
+	  )
+	  return Milter.REJECT
+      else:
+        hres,hcode,htxt = res,code,txt
+    else: hres = None
+
+    p = SPFPolicy(q.s,self.conf.access_file)
+
+    if res == 'fail':
+      policy = p.getPolicy('spf-fail:')
+      if not policy or policy == 'REJECT':
+	self.log('REJECT: SPF %s %i %s' % (res,code,txt))
+	self.setreply(str(code),'5.7.1',txt)
+	# A proper SPF fail error message would read:
+	# forger.biz [1.2.3.4] is not allowed to send mail with the domain
+	# "forged.org" in the sender address.  Contact <postmaster@forged.org>.
+	return Milter.REJECT
+    if res == 'softfail':
+      policy = p.getPolicy('spf-softfail:')
+      if policy and policy == 'REJECT':
+	self.log('REJECT: SPF %s %i %s' % (res,code,txt))
+	self.setreply(str(code),'5.7.1',txt)
+	# A proper SPF fail error message would read:
+	# forger.biz [1.2.3.4] is not allowed to send mail with the domain
+	# "forged.org" in the sender address.  Contact <postmaster@forged.org>.
+	return Milter.REJECT
+    elif res == 'permerror':
+      policy = p.getPolicy('spf-permerror:')
+      if not policy or policy == 'REJECT':
+	self.log('REJECT: SPF %s %i %s' % (res,code,txt))
+	# latest SPF draft recommends 5.5.2 instead of 5.7.1
+	self.setreply(str(code),'5.5.2',txt,
+	  'There is a fatal syntax error in the SPF record for %s' % q.o,
+	  'We cannot accept mail from %s until this is corrected.' % q.o
+	)
+	return Milter.REJECT
+    elif res == 'temperror':
+      policy = p.getPolicy('spf-temperror:')
+      if not policy or policy == 'REJECT':
+	self.log('TEMPFAIL: SPF %s %i %s' % (res,code,txt))
+	self.setreply(str(code),'4.3.0',txt)
+	return Milter.TEMPFAIL
+    elif res == 'neutral' or res == 'none':
+      policy = p.getPolicy('spf-neutral:')
+      if policy and policy == 'REJECT':
+        self.log('REJECT NEUTRAL:',q.s)
+	self.setreply('550','5.7.1',
+  "%s requires and SPF PASS to accept mail from %s. [http://openspf.org]"
+	  % (receiver,q.s))
+	return Milter.REJECT
+    elif res == 'pass':
+      policy = p.getPolicy('spf-pass:')
+      if policy and policy == 'REJECT':
+        self.log('REJECT PASS:',q.s)
+	self.setreply('550','5.7.1',
+		"%s has been blacklisted by %s." % (q.s,receiver))
+	return Milter.REJECT
+    self.add_header('Received-SPF',q.get_header(res,receiver),0)
+    if hres and q.h != q.o:
+      self.add_header('X-Hello-SPF',hres,0)
+    return Milter.CONTINUE
+
+if __name__ == "__main__":
+  Milter.factory = spfMilter
+  Milter.set_flags(Milter.CHGHDRS + Milter.ADDHDRS)
+  global config
+  config = read_config(['spfmilter.cfg','/etc/mail/spfmilter.cfg'])
+  miltername = config.miltername
+  socketname = config.socketname
+  print """To use this with sendmail, add the following to sendmail.cf:
+
+O InputMailFilters=%s
+X%s,        S=local:%s
+
+See the sendmail README for libmilter.
+sample spfmilter startup""" % (miltername,miltername,socketname)
+  sys.stdout.flush()
+  Milter.runmilter("pyspffilter",socketname,240)
+  print "sample spfmilter shutdown"
@@ -0,0 +1,85 @@
+#!/bin/bash
+#
+# spfmilter	This shell script takes care of starting and stopping spfmilter.
+#
+# chkconfig: 2345 80 30
+# description: a process that checks SPF for messages sent through sendmail.
+# processname: spfmilter
+# config: /etc/mail/spfmilter.cfg
+# pidfile: /var/run/milter/spfmilter.pid
+
+python="python2.4"
+
+pidof() {
+	set - ""
+	if set - `ps -e -o pid,cmd | grep "${python} spfmilter.py"` &&
+	  [ "$2" != "grep" ]; then
+	  echo $1
+	  return 0
+	fi
+	return 1
+}
+
+# Source function library.
+. /etc/rc.d/init.d/functions
+
+[ -x /usr/lib/pymilter/start.sh ] || exit 0
+
+RETVAL=0
+prog="spfmilter"
+
+start() {
+	# Start daemons.
+
+	echo -n "Starting $prog: "
+        if ! test -d /var/run/milter; then
+		mkdir -p /var/run/milter
+		chown mail:mail /var/run/milter
+	fi
+	daemon --check milter --user mail /usr/lib/pymilter/start.sh spfmilter
+	RETVAL=$?
+	echo
+	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/spfmilter
+	return $RETVAL
+}
+
+stop() {
+	# Stop daemons.
+	echo -n "Shutting down $prog: "
+	killproc milter
+	RETVAL=$?
+	echo
+	[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/spfmilter
+	return $RETVAL
+}
+
+# See how we were called.
+case "$1" in
+  start)
+	start
+	;;
+  stop)
+	stop
+	;;
+  restart|reload)
+	stop
+	start
+	RETVAL=$?
+	;;
+  condrestart)
+	if [ -f /var/lock/subsys/spfmilter ]; then
+	    stop
+	    start
+	    RETVAL=$?
+	fi
+	;;
+  status)
+	status spfmilter
+	RETVAL=$?
+	;;
+  *)
+	echo "Usage: $0 {start|stop|restart|condrestart|status}"
+	exit 1
+esac
+
+exit $RETVAL
@@ -1,99 +0,0 @@
-#!/usr/bin/python2.3
-
-# Author: Stuart D. Gathman <stuart@bmsi.com>
-# Copyright 2004 Business Management Systems, Inc.
-# This code is under the GNU General Public License.  See COPYING for details.
-
-# $Log$
-# Revision 1.1.1.1  2005/05/31 18:07:19  customdesigned
-# Release 0.6.9
-#
-# Revision 2.3  2004/04/19 22:12:11  stuart
-# Release 0.6.9
-#
-# Revision 2.2  2004/04/18 03:29:35  stuart
-# Pass most tests except -local and -rcpt-to
-#
-# Revision 2.1  2004/04/08 18:41:15  stuart
-# Reject numeric hello names
-#
-# Driver for SPF test system
-
-import spf
-import sys
-
-from optparse import OptionParser
-
-class PerlOptionParser(OptionParser):
-    def _process_args (self, largs, rargs, values):
-        """_process_args(largs : [string],
-                         rargs : [string],
-                         values : Values)
-
-        Process command-line arguments and populate 'values', consuming
-        options and arguments from 'rargs'.  If 'allow_interspersed_args' is
-        false, stop at the first non-option argument.  If true, accumulate any
-        interspersed non-option arguments in 'largs'.
-        """
-        while rargs:
-            arg = rargs[0]
-            # We handle bare "--" explicitly, and bare "-" is handled by the
-            # standard arg handler since the short arg case ensures that the
-            # len of the opt string is greater than 1.
-            if arg == "--":
-                del rargs[0]
-                return
-            elif arg[0:2] == "--":
-                # process a single long option (possibly with value(s))
-                self._process_long_opt(rargs, values)
-            elif arg[:1] == "-" and len(arg) > 1:
-                # process a single perl style long option
-		rargs[0] = '-' + arg
-                self._process_long_opt(rargs, values)
-            elif self.allow_interspersed_args:
-                largs.append(arg)
-                del rargs[0]
-            else:
-		return
-
-def format(q):
-  res,code,txt = q.check()
-  print res
-  if res in ('pass','neutral','unknown'): print
-  else: print txt
-  print 'spfquery:',q.get_header_comment(res)
-  print 'Received-SPF:',q.get_header(res,'spfquery')
-
-def main(argv):
-  parser = PerlOptionParser()
-  parser.add_option("--file",dest="file")
-  parser.add_option("--ip",dest="ip")
-  parser.add_option("--sender",dest="sender")
-  parser.add_option("--helo",dest="hello_name")
-  parser.add_option("--local",dest="local_policy")
-  parser.add_option("--rcpt-to",dest="rcpt")
-  parser.add_option("--default-explanation",dest="explanation")
-  parser.add_option("--sanitize",type="int",dest="sanitize")
-  parser.add_option("--debug",type="int",dest="debug")
-  opts,args = parser.parse_args(argv)
-  if opts.ip:
-    q = spf.query(opts.ip,opts.sender,opts.hello_name,local=opts.local_policy)
-    if opts.explanation:
-      q.set_default_explanation(opts.explanation)
-    format(q)
-  if opts.file:
-    if opts.file == '0':
-      fp = sys.stdin
-    else:
-      fp = open(opts.file,'r')
-    for ln in fp:
-      ip,sender,helo,rcpt = ln.split(None,3)
-      q = spf.query(ip,sender,helo,local=opts.local_policy)
-      if opts.explanation:
-	q.set_default_explanation(opts.explanation)
-      format(q)
-    fp.close()
-    
-if __name__ == "__main__":
-  import sys
-  main(sys.argv[1:])
@@ -0,0 +1,14 @@
+#!/bin/sh
+appname="$1"
+script="${2:-${appname}}"
+datadir=/var/log/milter
+python="python2.4"
+exec >>${datadir}/${appname}.log 2>&1
+if test -s ${datadir}/${script}.py; then
+  cd ${datadir} # use version in log dir if it exists for debugging
+else
+  cd /usr/lib/pymilter
+fi
+
+${python} ${script}.py &
+echo $! >/var/run/milter/${appname}.pid
@@ -1,4 +1,7 @@
+To: %(sender)s
+From: postmaster@%(receiver)s
 Subject: Critical mail server configuration error
+Auto-Submitted: auto-generated (configuration error)

 This is an automatically generated Delivery Status Notification.

@@ -0,0 +1,33 @@
+To: %(sender)s
+From: postmaster@%(receiver)s
+Subject: Critical DNS configuration error
+Auto-Submitted: auto-generated (configuration error)
+
+This is an automatically generated Delivery Status Notification.
+
+THIS IS A WARNING MESSAGE ONLY.
+
+YOU DO *NOT* NEED TO RESEND YOUR MESSAGE.
+
+Delivery to the following recipients has been delayed.
+
+	%(rcpt)s
+
+Subject: %(subject)s 
+Received-SPF: %(spf_result)s
+
+Your DNS server is not responding to TXT queries.  In other words,
+it is BROKEN.  You need to get somebody to fix it ASAP.  We
+are attempting to do TXT queries to see if you have an SPF record.
+
+See http://openspf.org
+
+We are sending you this message to alert you to the fact that
+you have problems with your DNS.
+
+If you need further assistance, please do not hesitate to
+contact me again.
+
+Kind regards,
+
+postmaster@%(receiver)s
@@ -2,6 +2,7 @@ import unittest
 import testbms
 import testmime
 import testsample
+import testutils
 import os

 def suite(): 
@@ -9,6 +10,7 @@ def suite():
  s.addTest(testbms.suite())
  s.addTest(testmime.suite())
  s.addTest(testsample.suite())
+  s.addTest(testutils.suite())
  return s

 if __name__ == '__main__':
@@ -44,7 +44,7 @@ class TestMilter(bms.bmsMilter):
      self._msg[field] = value
    self.headerschanged = True

-  def addheader(self,field,value):
+  def addheader(self,field,value,idx=-1):
    if not self._body:
      raise IOError,"addheader not called from eom()"
    self.log('addheader: %s=%s' % (field,value))
@@ -238,7 +238,7 @@ class BMSMilterTestCase(unittest.TestCase):
    milter = TestMilter()
    milter.connect('testSmartAlias')
    # test smart alias feature
-    key = ('foo@bar.com','baz@bat.com')
+    key = ('foo@example.com','baz@bat.com')
    bms.smart_alias[key] = ['ham@eggs.com']
    rc = milter.feedMsg('test8',key[0],key[1])
    self.assertEqual(rc,Milter.ACCEPT)
@@ -277,6 +277,25 @@ class BMSMilterTestCase(unittest.TestCase):
    fp = milter._body
    open("test/test1.tstout","w").write(fp.getvalue())

+  def testFindsrs(self):
+    if not bms.srs:
+      import SRS
+      bms.srs = SRS.new(secret='test')
+    sender = bms.srs.forward('foo@bar.com','mail.example.com')
+    sndr = bms.findsrs(StringIO.StringIO(
+"""Received: from [1.16.33.86] (helo=mail.example.com)
+	by bastion4.mail.zen.co.uk with smtp (Exim 4.50) id 1H3IBC-00013b-O9
+	for foo@bar.com; Sat, 06 Jan 2007 20:30:17 +0000
+X-Mailer: "PyMilter-0.8.5"
+	<%s> foo
+MIME-Version: 1.0
+Content-Type: text/plain
+To: foo@bar.com
+From: postmaster@mail.example.com
+""" % sender
+    ))
+    self.assertEqual(sndr,'foo@bar.com')
+
 #  def testReject(self):
 #    "Test content based spam rejection."
 #    milter = TestMilter()
@@ -0,0 +1,48 @@
+import unittest
+import doctest
+import os
+import Milter.utils
+from Milter.cache import AddrCache
+from Milter.dynip import is_dynip
+
+class AddrCacheTestCase(unittest.TestCase):
+
+  def setUp(self):
+    self.fname = 'test.dat'
+
+  def tearDown(self):
+    os.remove(self.fname)
+
+  def testAdd(self):
+    cache = AddrCache(fname=self.fname)
+    cache['foo@bar.com'] = None
+    cache.addperm('baz@bar.com')
+    cache['temp@bar.com'] = 'testing'
+    self.failUnless(cache.has_key('foo@bar.com'))
+    self.failUnless(not cache.has_key('hello@bar.com'))
+    self.failUnless('baz@bar.com' in cache)
+    self.assertEquals(cache['temp@bar.com'],'testing')
+    s = open(self.fname).readlines()
+    self.failUnless(len(s) == 2)
+    self.failUnless(s[0].startswith('foo@bar.com '))
+    self.assertEquals(s[1].strip(),'baz@bar.com')
+    # check that new result overrides old
+    cache['temp@bar.com'] = None
+    self.failUnless(not cache['temp@bar.com'])
+
+  def testDomain(self):
+    fp = open(self.fname,'w')
+    print >>fp,'spammer.com'
+    fp.close()
+    cache = AddrCache(fname=self.fname)
+    cache.load(self.fname,30)
+    self.failUnless('spammer.com' in cache)
+
+def suite(): 
+  s = unittest.makeSuite(AddrCacheTestCase,'test')
+  s.addTest(doctest.DocTestSuite(Milter.utils))
+  s.addTest(doctest.DocTestSuite(Milter.dynip))
+  return s
+
+if __name__ == '__main__':
+  unittest.TextTestRunner().run(suite())
Author	SHA1	Message	Date
cvs2svn	2fcbc27f2a	This commit was manufactured by cvs2svn to create tag 'pymilter-0_8_10'. Sprout from master 2008-08-25 22:03:24 UTC Stuart Gathman <stuart@gathman.org> 'Fix /var/run/milter owner' Cherrypick from bmsi 2005-05-31 18:23:49 UTC Stuart Gathman <stuart@gathman.org> 'Development changes since 0.7.2': rejects.py rhsbl.m4 sample.py test/amazon test/big5 test/bounce test/bounce1 test/bound test/honey test/missingboundary test/samp1 test/spam44 test/spam7 test/spam8 test/test1 test/test8 test/virus1 test/virus13 test/virus2 test/virus3 test/virus4 test/virus5 test/virus6 test/virus7 testsample.py	2008-08-25 22:03:25 +00:00
Stuart Gathman	e5bf1aee09	Fix /var/run/milter owner	2008-08-25 22:03:24 +00:00
Stuart Gathman	5df3a80f7b	Fix /var/run/milter owner	2008-08-25 22:02:39 +00:00
Stuart Gathman	df67ee9147	Report failure to remove milter socket	2008-08-25 22:00:46 +00:00
Stuart Gathman	593384d610	/var/run/milter must be owned by mail	2008-08-25 21:41:18 +00:00
Stuart Gathman	1280f1360e	Release 0.8.10	2008-08-25 20:00:51 +00:00
Stuart Gathman	3e1e528abe	Release 0.8.10	2008-08-25 19:49:02 +00:00
Stuart Gathman	04ce8f81b9	Release 0.8.10	2008-08-25 18:49:13 +00:00
Stuart Gathman	bc390e69b9	Update docs	2008-08-25 18:45:21 +00:00
Stuart Gathman	c07ed917ab	Handle missing gossip_node so self tests pass.	2008-08-25 18:32:23 +00:00
Stuart Gathman	a14d676fb6	Release 0.8.10	2008-08-25 18:18:30 +00:00
Stuart Gathman	600e3dfbfb	Update docs for 0.8.10	2008-08-25 18:14:56 +00:00
Stuart Gathman	8cfa03bbc4	Log rcpt for SRS rejections.	2008-08-18 17:47:57 +00:00
Stuart Gathman	28a0e551bd	CBV policy sends no DSN. DSN policy sends DSN.	2008-08-06 00:52:38 +00:00
Stuart Gathman	be3f463450	Send quarantine DSN to SPF PASS only.	2008-08-05 18:04:06 +00:00
Stuart Gathman	a420148b1e	Parse ESMTP params	2008-07-29 21:59:29 +00:00
Stuart Gathman	f4465ea816	Allow explicitly whitelisted email from banned_users.	2008-05-08 21:35:57 +00:00
Stuart Gathman	1845876665	Configure gossip TTL.	2008-04-10 14:59:35 +00:00
Stuart Gathman	cee6bc3bea	Release 0.8.10	2008-04-02 18:59:14 +00:00
Stuart Gathman	71403de50e	Do not CBV whitelisted addresses. We already know they are good.	2008-04-01 00:13:10 +00:00
Stuart Gathman	017784b5a7	Handle multi-hop source path in parseaddr.	2008-01-10 16:41:04 +00:00
Stuart Gathman	632e7b4248	Handle unquoted fullname when parsing email.	2008-01-09 20:15:49 +00:00
Stuart Gathman	10f4f2613e	Packaging tweaks.	2007-11-29 14:35:17 +00:00
Stuart Gathman	69369c3b2a	Support temperror policy in access.	2007-11-01 20:09:14 +00:00
Stuart Gathman	5386e08ca5	Send quarantine DSN to SPF pass (official or guessed) only. Reject blacklisted email too big for dspam.	2007-10-10 18:23:54 +00:00
Stuart Gathman	d0fe3b0b84	Check porn keywords in From header field.	2007-10-10 18:07:50 +00:00
Stuart Gathman	670e97cb79	Test on Centos5	2007-09-25 17:07:32 +00:00
Stuart Gathman	6397b7027f	Tested on RH7	2007-09-25 16:37:26 +00:00
Stuart Gathman	94ce032559	Update license.	2007-09-25 02:26:29 +00:00
Stuart Gathman	91230381cb	Test dns.py	2007-09-25 02:15:35 +00:00
Stuart Gathman	46ed3ddbcb	Allow arbitrary object, not just spf.query like, to provide data for create_msg	2007-09-25 01:24:59 +00:00
Stuart Gathman	6048fe6e8c	Remove explicit spf dependency.	2007-09-24 20:13:26 +00:00
Stuart Gathman	d225384829	Create milter and milter-spf as noarch packages.	2007-09-24 18:00:58 +00:00
Stuart Gathman	a84f6aa574	Specify library_dirs for Debian.	2007-09-24 17:44:51 +00:00
Stuart Gathman	344e8f0a0a	Report domain on reputation reject.	2007-09-13 14:51:03 +00:00
Stuart Gathman	1fa4b72c84	Delete unparseable timestamps when loading address cache. These have arisen because of failure to parse MAIL FROM properly. Will have to tighten up MAIL FROM parsing to match RFC.	2007-09-03 16:18:45 +00:00
Stuart Gathman	021ea96748	Fixes from test on EL5.	2007-07-25 19:04:44 +00:00
Stuart Gathman	a490e79564	Build on EL5	2007-07-25 17:43:34 +00:00
Stuart Gathman	33e8f7c4cc	Multi-package build fixes.	2007-07-25 17:30:30 +00:00
Stuart Gathman	6bbb6b3f02	Move milter apps to /usr/lib/pymilter	2007-07-25 17:14:59 +00:00
Stuart Gathman	6577e40bfb	Build pymilter as separate package.	2007-07-25 15:32:09 +00:00
Stuart Gathman	04eeeab2e1	Clarify docs.	2007-07-25 15:20:41 +00:00
Stuart Gathman	cdfeb2d792	Ban ips on bad mailfrom offenses as well as bad rcpts.	2007-07-02 03:06:10 +00:00
Stuart Gathman	46545cab94	Fix missed comcast dynip.	2007-06-28 20:33:25 +00:00
Stuart Gathman	9a8fdcb120	Ban IPs based on too many invalid recipients in a connection. Requires configuring check_user. Tighten HELO best_guess policy.	2007-06-23 20:53:05 +00:00
Stuart Gathman	218f5168bc	Do not process valid SRS recipients as delayed_failure.	2007-04-19 16:02:43 +00:00
Stuart Gathman	ddbb8ac3ea	Ban ips with too many bad rcpts on a connection.	2007-04-15 01:01:13 +00:00
Stuart Gathman	a2215124bb	Ban ips with too many bad rcpts on a connection.	2007-04-15 00:54:30 +00:00
Stuart Gathman	e505d2bb28	Check access_file at startup. Compress rcpt to log.	2007-04-13 17:20:09 +00:00
Stuart Gathman	9f40f265cd	Stop querying gossip server twice.	2007-04-05 17:59:07 +00:00
Stuart Gathman	20a875b84d	Don't disable gossip for temporary error.	2007-04-02 18:37:25 +00:00
Stuart Gathman	1da5ca54b5	Report bestguess and helo-spf as key-value pairs in Received-SPF instead of in their own headers.	2007-03-30 18:13:41 +00:00
Stuart Gathman	bac593f05d	Don't count DSN and unqualified MAIL FROM as internal_domain.	2007-03-29 03:06:10 +00:00
Stuart Gathman	dbba488d58	Do not CBV for internal domains.	2007-03-24 00:30:24 +00:00
Stuart Gathman	6936b599fe	Get SMTP-Auth policy from access_file.	2007-03-23 22:39:10 +00:00
Stuart Gathman	cee38f8149	Properly log From: and Sender:	2007-03-21 04:02:13 +00:00
Stuart Gathman	188e8256f3	Gossip configuration options: client or standalone with optional peers.	2007-03-18 02:32:21 +00:00
Stuart Gathman	4013365a3d	New delayed DSN pattern. Retab (expandtab).	2007-03-17 21:22:48 +00:00
Stuart Gathman	e571ccc5a5	Fix missing HELO log.	2007-03-13 21:18:28 +00:00
Stuart Gathman	f65294b470	Include Received-SPF in permerror DSN.	2007-03-13 18:45:09 +00:00
Stuart Gathman	b2d8e838a2	Fix continuing findsrs when srs.reverse fails.	2007-03-03 19:18:57 +00:00
Stuart Gathman	f136e973dc	Improve delayed failure detection.	2007-03-03 18:46:26 +00:00
Stuart Gathman	d289822f42	Handle DNS error sending DSN.	2007-03-03 18:19:40 +00:00
Stuart Gathman	806aa5a6de	Updated	2007-03-02 14:18:21 +00:00
Stuart Gathman	e84a803cc1	Handle missing HELO.	2007-02-21 22:14:41 +00:00
Stuart Gathman	20612240f3	Use re for auto-reply recognition.	2007-02-07 23:21:26 +00:00
Stuart Gathman	c9e6bb68d9	Newbie friendly default for internal_connect	2007-02-07 23:20:28 +00:00
Stuart Gathman	4d69b8fbfe	Handle null in header value.	2007-01-26 03:47:23 +00:00
Stuart Gathman	21e3c6f489	Persist blacklisting from delayed DSNs.	2007-01-25 22:47:26 +00:00
Stuart Gathman	83529320ae	Add private relay.	2007-01-23 19:46:20 +00:00
Stuart Gathman	e5685c6035	Convert tabs to spaces.	2007-01-22 02:46:01 +00:00
Stuart Gathman	4c72135b0e	Move parse_header to Milter.utils. Test case for delayed DSN parsing. Fix plock when source missing or cannot set owner/group.	2007-01-19 23:31:38 +00:00
Stuart Gathman	393aa6140a	Doc update. Parse From header for delayed failure detection. Don't check reputation of trusted host. Track IP reputation only when missing PTR.	2007-01-18 16:48:44 +00:00
Stuart Gathman	2a6a68230b	REJECT after data for blacklisted emails - so in case of mistakes, a legitimate sender will know what happened.	2007-01-16 05:17:29 +00:00
Stuart Gathman	279c831a8e	Purge old entries in auto_whitelist and send_dsn logs.	2007-01-11 19:59:40 +00:00
Stuart Gathman	c0aa632e16	Negative feedback for bad headers. Purge cache logs on startup.	2007-01-11 04:31:26 +00:00
Stuart Gathman	a875ac7834	Documentation updates.	2007-01-10 04:44:25 +00:00
Stuart Gathman	9f8cef5ee2	Get user feedback.	2007-01-08 23:20:54 +00:00
Stuart Gathman	4b0e7b22da	Tested on spidey2	2007-01-06 04:32:57 +00:00
Stuart Gathman	40fb05b0e3	Forgot import	2007-01-06 04:25:12 +00:00
Stuart Gathman	8ae7bd4217	Add config file to spfmilter	2007-01-06 04:21:30 +00:00
Stuart Gathman	139e141e1e	Make blacklist an AddrCache	2007-01-05 23:33:55 +00:00
Stuart Gathman	8932dc36db	Move parse_addr, iniplist, ip4re to Milter.utils	2007-01-05 23:12:13 +00:00
Stuart Gathman	bda654b7a0	Added sample spfmilter.py application.	2007-01-05 22:48:48 +00:00
Stuart Gathman	09b671f47b	Test AddrCache.	2007-01-05 21:26:03 +00:00
Stuart Gathman	732e7317f1	Move AddrCache to Milter package.	2007-01-05 21:25:40 +00:00
Stuart Gathman	702ec2d4ca	Link to pyspf.	2007-01-05 21:24:29 +00:00
Stuart Gathman	7bbff66000	Release 0.8.7	2007-01-04 18:04:37 +00:00
Stuart Gathman	5ad6d321bd	Do plain CBV when template missing.	2007-01-04 18:01:11 +00:00
Stuart Gathman	d01dc65f39	Use HELO identity if good when MAILFROM is bad.	2006-12-31 03:07:20 +00:00
Stuart Gathman	b703031c7e	Skip reputation/whitelist/blacklist when rejecting on SPF. Add X-Hello-SPF.	2006-12-30 18:58:53 +00:00
Stuart Gathman	1bc0a4faef	Reject on bad_reputation or blacklist and nodspam. Match valid helo like PTR for guessed SPF pass.	2006-12-28 01:54:32 +00:00
Stuart Gathman	2bea6ad76f	Add archive option to wiretap.	2006-12-19 00:59:30 +00:00
Stuart Gathman	c9f0c94b92	Reject multiple recipients to DSN. Auto-disable gossip on DB error.	2006-12-04 18:47:04 +00:00
Stuart Gathman	59bf86e747	Release 0.8.7	2006-11-22 18:32:37 +00:00
Stuart Gathman	8f5513a502	SRS domains were missing srs_reject check when SES was active.	2006-11-22 16:31:22 +00:00
Stuart Gathman	87482d5740	Replace last use of deprecated rfc822 module.	2006-11-22 01:03:28 +00:00
Stuart Gathman	b227ca6bb0	Update a use of deprecated rfc822. Recognize report-type=delivery-status	2006-11-21 18:45:49 +00:00
Stuart Gathman	dd0125b641	Another lame DSN heuristic. Block PTR cache poisoning attack.	2006-11-04 22:09:39 +00:00
Stuart Gathman	a7e98f411e	More SPF fixes and tests from pyspf.	2006-10-09 17:59:47 +00:00
Stuart Gathman	ea76acdd3d	Fix defaults.	2006-10-04 03:46:01 +00:00
Stuart Gathman	b92154934b	SPF updates from pyspf.	2006-10-04 02:15:57 +00:00
Stuart Gathman	33aeefa19f	case_sensitive_localpart option, more delayed bounce heuristics, optional smart_alias section.	2006-10-01 01:44:06 +00:00
Stuart Gathman	2fe8fa8813	Use latest pyspf verbatim. Will depend on package when pyspf-2.0 is packaged.	2006-10-01 01:42:33 +00:00
Stuart Gathman	e0f58cce1f	Merge changes from pyspf to pass test suite.	2006-09-08 22:02:57 +00:00
Stuart Gathman	157f33edb8	Permerror for multiple TXT SPF records.	2006-07-31 15:25:39 +00:00
Stuart Gathman	64bf954a17	Remove debug print	2006-07-28 01:21:33 +00:00
Stuart Gathman	357cd1b740	More fixes from pyspf	2006-07-28 01:21:02 +00:00
Stuart Gathman	3a90a35cbc	Support CBV timeout	2006-07-26 16:42:26 +00:00
Stuart Gathman	30923ab3a1	Support timeout.	2006-07-26 16:37:35 +00:00
Stuart Gathman	d38cf5885e	Handle multi-line headers in delayed dsns.	2006-06-21 22:22:00 +00:00
Stuart Gathman	8c4cca8f55	initialize perm_error	2006-06-21 21:13:07 +00:00
Stuart Gathman	a20eeda04d	More delayed reject token headers. Don't require HELO pass for CBV.	2006-06-21 21:12:04 +00:00
Stuart Gathman	d50215d0ba	Include header fields in DSN template.	2006-06-21 21:07:11 +00:00
Stuart Gathman	c5b2169509	Remove default templates. Scrub test.	2006-05-24 20:56:35 +00:00
Stuart Gathman	2e42eea306	Release 0.8.6	2006-05-21 04:04:02 +00:00
Stuart Gathman	1c78384da9	Release 0.8.6	2006-05-21 03:56:13 +00:00
Stuart Gathman	053c32e450	Fail dsn	2006-05-21 03:41:44 +00:00
Stuart Gathman	b57e365349	Default templates need headers also.	2006-05-21 03:39:59 +00:00
Stuart Gathman	99396a1eee	Fail template, move most header fields into template.	2006-05-21 03:30:06 +00:00
Stuart Gathman	528810c31a	Create GOSSiP record only when connection will procede to DATA.	2006-05-17 21:28:07 +00:00
Stuart Gathman	a9ffc3ae28	a:1.2.3.4 -> ip4:1.2.3.4 'lax' heuristic.	2006-05-12 16:15:20 +00:00
Stuart Gathman	eda8680b70	Don't require SPF pass for white/black listing mail from trusted relay. Support localpart wildcard for white and black lists.	2006-05-12 16:14:48 +00:00
Stuart Gathman	afd3e0f042	Check whitelist/blacklist even when not checking SPF (e.g. trusted relay).	2006-04-06 18:14:17 +00:00
Stuart Gathman	f42ddbfb53	Fix spec bug	2006-03-25 17:33:36 +00:00
Stuart Gathman	44d76a63d8	0.8.6 release candidate	2006-03-25 17:29:28 +00:00
Stuart Gathman	ec4f9fdd99	Import note_error from pyspf. Handle timeout on type99 lookup specially (sender actually has no SPF record and a braindead DNS server).	2006-03-21 18:48:51 +00:00
Stuart Gathman	6102d641c5	Use re to recognize failure DSNs.	2006-03-10 20:52:49 +00:00
Stuart Gathman	d69b805690	Use signed Message-ID in delayed reject to blacklist senders	2006-03-07 20:50:54 +00:00
Stuart Gathman	994bcce7dc	Properly report hard PermError (lax mode fails also) by always setting perm_error attribute with PermError exception. Improve reporting of invalid domain PermError.	2006-02-24 02:12:54 +00:00
Stuart Gathman	7f5d8b6b11	Use SRS sign domain list. Accept but do not use for training whitelisted senders without SPF pass. Immediate rejection of unsigned bounces.	2006-02-17 05:04:29 +00:00
Stuart Gathman	8d02ab1771	User specific SPF receiver policy.	2006-02-16 02:16:36 +00:00
Stuart Gathman	18759c3698	Remove spf dependency for iniplist	2006-02-12 04:15:01 +00:00
Stuart Gathman	2f533c4591	Use CIDR notation for internal connect list.	2006-02-12 02:12:08 +00:00
Stuart Gathman	04c8b2e1fc	Resolve FIXME for wrap_close.	2006-02-12 02:00:42 +00:00
Stuart Gathman	56c1cbd0fd	Don't check rcpt user list when signed MFROM.	2006-02-12 01:13:58 +00:00
Stuart Gathman	ce51034f69	Use CIDR notation for trusted_forwarder iplist	2006-02-09 20:39:43 +00:00
Stuart Gathman	285d4663c9	put back eom condition	2006-01-30 23:14:48 +00:00
Stuart Gathman	5830e13d00	New milter.log tags	2006-01-12 20:53:51 +00:00
Stuart Gathman	1b685fca76	Accelerate training via whitelist and blacklist.	2006-01-12 20:31:24 +00:00
Stuart Gathman	71e769ef0c	New FAQ	2006-01-05 03:17:10 +00:00