Documentation updates.

TODO

@@ -1,15 +1,10 @@
-DONE When bms.py can't find templates, it passes None to dsn.create_msg(),
-which uses local variable as backup, which no longer exist.  Do plain
-CBV in that case instead.
 
-Find and use X-GOSSiP: header for SPAM: and FP: submissions.  Would need to 
+Feedback from user training is ignored because UMIS has already been
-keep tags longer.
+removed from queue.  Maybe keep UMIS in queue, and add method to 
+alter last feedback for ID.
 
 Generate DSNs according to RFC 3464
 
-Parse incoming 3464 DSNs for "Action: failed" to recognize delayed
-failures.  This works regardless of Subject.
-
 Get temperror policy from access file.
 
 When training with spam, REJECT after data so that mistakenly blacklisted
@@ -18,14 +13,8 @@ senders at least get an error.
 Reporting explanation for failure should show source if sender
 provided explanation.
 
-Reports PROBATION even when rejecting message (works, but confusing in log).
-
 Bug in Auto-whitelist.  Recent Auto-whitelist doesn't override expired entry.
 
-DONE Delayed_failure detection needs to handle multi-line header fields.  
-Also, delayed_failure should be recognized when addressed to
-postmaster@helodomain 
-
 Need to use wildcards in blacklist.log: *.madcowsrecord.net
 Need to exclude emails like !*-admin@example.com in whitelist_sender.
 
@@ -42,24 +31,16 @@ Received-SPF header field should show identity that was checked.
 Check SPF for outgoing mail (including local policy for internal addresses).
 This could also solve the second part of the mail from relay problem below.
 
-Whitelisted sender from trusted relay get PROBATION.  Need to extracted
+Whitelisted senders from trusted relay get PROBATION.  Need to extracted
 SPF result from headers - and in the case of mail internal to relay
 (e.g. bmsi.com), supply 'pass' result.  
 
-FIXME: DSN for Permerror shows 'None' for error under some condition.
-
-Another metaDSN format:
-Subject: Delivery Report
-...
-Original-Envelope-ID: SRS0...@...
-
 For selected domains, check rcpts via CBV before accepting mail.  Cache
 results.  This will kick out dictonary attacks against a mail domain
 behind a gateway sooner.
 
-Allow blacklisted emails as well as domains in blacklist.log.  Use same
+Add emails blacklisted via CBV so that they are remembered across milter
-data structure as autowhitelist.log.  Add emails blacklisted via CBV
+restarts.
-so that they are remembered across milter restarts.
 
 Make all dictionaries work like honeypot.  Do not train as ham unless
 whitelisted.  Train on blacklisted messages, or spam feedback.  This
@@ -146,8 +127,6 @@ a separate process.  However, a significant amount of memory is wasted
 for each additional Python VM, and communication between milters
 is cumbersome (e.g., adding mail headers, writing external files).
 
-Backup copies for outgoing/incoming mail.
-
 Copy incoming wiretap mail, even though sendmail alias works perfectly
 for the purpose, to avoid having to change two configs for a wiretap.
 
@@ -165,3 +144,28 @@ embarrass yourself), and also removing Received headers with hidepath.
 Need a test module to feed sample messages to a milter though a live 
 sendmail and SMTP.  The mockup currently used is probably not very accurate,
 and doesn't test the threading code.
+
+DONE When bms.py can't find templates, it passes None to dsn.create_msg(),
+which uses local variable as backup, which no longer exist.  Do plain
+CBV in that case instead.
+
+DONE Find and use X-GOSSiP: header for SPAM: and FP: submissions.  Would need
+to keep tags longer.
+
+DONE Parse incoming 3464 DSNs for "Action: failed" to recognize delayed
+failures.  This works regardless of Subject.
+
+DONE Reports PROBATION even when rejecting message (works, but confusing in
+log).
+
+DONE Delayed_failure detection needs to handle multi-line header fields.  
+Also, delayed_failure should be recognized when addressed to
+postmaster@helodomain 
+
+DONE DSN for Permerror shows 'None' for error under some condition.
+
+DONE Allow blacklisted emails as well as domains in blacklist.log.  Use same
+data structure as autowhitelist.log.  
+
+DONE Backup copies for outgoing/incoming mail.
+

bms.py

@@ -1,6 +1,9 @@
 #!/usr/bin/env python
 # A simple milter that has grown quite a bit.
 # $Log$
+# Revision 1.83  2007/01/08 23:20:54  customdesigned
+# Get user feedback.
+#
 # Revision 1.82  2007/01/06 04:21:30  customdesigned
 # Add config file to spfmilter
 #
@@ -1471,7 +1474,16 @@ class bmsMilter(Milter.Milter):
       rc = self.send_dsn(q,msg,template_name)
       self.cbv_needed = None
       if rc == Milter.REJECT:
-        if gossip and self.umis:
+        # Do not feedback here, because feedback should only occur
+	# for messages that have gone to DATA.  Reputation lets us
+	# reject before DATA for persistent spam domains, saving
+	# cycles and bandwidth.
+
+	# Do feedback here, because CBV costs quite a bit more than
+	# simply rejecting before DATA.  Bad reputation will acrue to
+	# the IP or HELO, since we won't get here for validated MAILFROM.
+	# 	See Proverbs 26:4,5
+	if gossip and self.umis:
 	  gossip_node.feedback(self.umis,1)
 	self.train_spam()
 	return Milter.DISCARD

doc/faq.ht

@@ -2,8 +2,17 @@ Title: Python Milter FAQ
 
 <h1> Python Milter <a name=faq>FAQ</a> </h1>
 
+<menu>
+<li> <a href="#compiling">Compiling Python Milter</a>
+<li> <a href="#running">Running Python Milter</a>
+<li> <a href="#spf">Using SPF</a>
+<li> <a href="#srs">Using SRS</a>
+</menu>
+
 <ol>
-<h3> Compiling Python Milter </h3>
+
+<h3> <a name="compiling">Compiling Python Milter </a> </h3>
+
 <li> Q. I have installed sendmail from source, but Python milter won't
 compile.
 <p>  A. Even though libmilter is officially supported in sendmail-8.12, 
@@ -36,7 +45,7 @@ in setup.py to
             define_macros = [ ('MAX_ML_REPLY',1) ]
 </pre>
 
-<h3> Running Python Milter </h3>
+<h3> <a name="running">Running Python Milter </a></h3>
 
 <li> Q. The sample.py milter prints a message, then just sits there.
 <pre>
@@ -186,10 +195,16 @@ The <code>internal_domains</code> option is simplistic, it assumes all
 valid senders of the domains are internal.  SPF provides a much more general
 check of IP and MAIL FROM for external email.  Pymilter should soon
 have a local policy feature for more general checking of internal mail.
+<li> Q. <code>mail_archive</code> isn't working.  Or I don't understand how
+	it's suppose to work.  I have
+	<code>mail_archive = /var/mail/mail_archive</code>
+	in <code>pymilter.cfg</code> but nothing ever gets dumped into
+	<code>/var/mail/mail_archive</code>.
+<p>  A. The 'mail' user needs to have write access.  Permission failures
+	should be logged as a traceback in milter.log if it doesn't.
 
-<h3> Using SPF </h3>
+<h3> <a name="spf">Using SPF </a></h3>
 
-<a name="spf">
 <li> Q. So how do I use the SPF support?  The sample.py milter doesn't seem
         to use it.
 <p>  A. The bms.py milter supports spf.  The RedHat RPMs will set almost
@@ -209,5 +224,63 @@ everything up for you.  For other systems:
      logfiles and a simple cron script using <code>find</code> to clean
      <code>tempdir</code>.
 </ol>
+	In CVS, there is <code>spfmilter.py</code>.  Run that as a service,
+	and it does just SPF.  It uses the sendmail <code>access</code>
+	file to configure SPF responses just like <code>bms.py</code>, but
+	supports only REJECT and OK.
+<li> Q. The SPF DSN is sent at least once for domains that don't publish a SPF.
+	How do I stop this behavior?
+<p>  A. The SPF response is controlled by <code>/etc/mail/access</code>
+	(actually the file you specify with <code>access_file</code> in
+	the <code>[spf]</code> section of <code>pymilter.cfg</code>.  Responses
+	are OK, CBV, and REJECT.  CBV sends the DSN.
+<p>
+You can change the defaults.  For instance, I have:
+<pre>
+SPF-None:	REJECT
+SPF-Neutral:	CBV
+SPF-Softfail:	CBV
+SPF-Permerror:	CBV
+</pre>
+I have best_guess = 1, so SPF none is converted to PASS/NEUTRAL for policy
+lookup, and 3 strikes (no PTR, no HELO, no SPF) becomes "SPF NONE" for local
+policy purposes (the Received-SPF header always shows the official SPF
+result.)
+<p>
+You can change the default for specific domains:
+<pre>
+# these guys aren't going to pay attention to CBVs anyway...
+SPF-None:cia.gov	REJECT
+SPF-None:fbi.gov	REJECT
+SPF-Neutral:aol.com	REJECT
+SPF-Softfail:ebay.com	REJECT
+</pre>
+
+<h3> <a name="srs">Using SRS </a></h3>
+
+<li> Q. The SRS part doesn't seem to work as whenever I try to start
+	<code>/etc/init.d/pysrs</code>, I get this in
+	<code>/var/log/milter/pysrs.log</code>:
+<pre>
+ConfigParser.NoOptionError: No option 'fwdomain' in section: 'srs'
+</pre>
+<p>  A. You need to specify the forward domain - i.e. the domain you want
+	SRS to rewrite stuff too.
+<p>
+For instance, I have:
+<pre>
+# sample SRS configuration
+[srs]
+secret = don't you wish
+maxage = 8
+hashlength = 5
+;database=/var/log/milter/srs.db
+fwdomain = bmsi.com
+sign=bmsi.com,mail.bmsi.com,gathman.org
+srs=bmsaix.bmsi.com,bmsred.bmsi.com,stl.gathman.org,bampa.gathman.org
+</pre>
+The <code>sign</code> is for local domains which are signed.
+The <code>srs</code> list is for other domains which you are relaying,
+and which need to have SRS checked/undone for bounces.
 
 </ol>