From 600e3dfbfb952d6d0eda293163cea2096378aa05 Mon Sep 17 00:00:00 2001 From: Stuart Gathman Date: Mon, 25 Aug 2008 18:14:56 +0000 Subject: [PATCH] Update docs for 0.8.10 --- doc/changes.ht | 41 ++++++++ doc/credits.ht | 2 +- doc/policy.ht | 247 ++++++++++++++++++++----------------------------- 3 files changed, 144 insertions(+), 146 deletions(-) diff --git a/doc/changes.ht b/doc/changes.ht index a0eaa4a..719d09b 100644 --- a/doc/changes.ht +++ b/doc/changes.ht @@ -2,6 +2,47 @@ Title: Recent Changes

Recent Changes

+

0.8.10

+ +SRS rejections now log the recipient. +I have finally implemented plain CBV (no DSN). The CBV policy +will do a plain CBV from now on, and the DSN policy is required +if you want to send a DSN. +I started checking the MAIL FROM fullname (human readable part +of an email) for porn keywords. There is now a banned IP database. +IPs are banned for too many bad MAIL FROMs or RCPT TOs, and remain banned +for 7 days. + +

0.8.9

+ +I use the %ifarch hack to build milter and milter-spf +packages as noarch, while pymilter is built as native. + +I removed the spf dependency from dsn.py, so pymilter can be used without +installing pyspf, and added a Milter.dns module to let python milters do +general DNS lookups without loading pyspf. + +

0.8.8

+ +Programs do not belong in the /var/log directory. I moved the +milter apps to /usr/lib/pymilter. Since having the programs and +data in the same directory is convenient for debugging, it will +still use an executable present in the datadir. + +Several general utility classes and functions are now in the Milter package +for possible use by other python milters. In addition to the trivial example +milter, a simple SPF only milter is included as a realistic example. + +The spec file now build 3 RPMs: + + +

0.8.7

The spf module has been moved to the diff --git a/doc/credits.ht b/doc/credits.ht index 849ae3d..9973a4d 100644 --- a/doc/credits.ht +++ b/doc/credits.ht @@ -5,7 +5,7 @@ Title: Credits Jim Niemira wrote the original C module and some quick and dirty python to use it. -Stuart D. Gathman +Stuart D. Gathman took that kludge and added threading and context objects to it, wrote a proper OO wrapper (Milter.py) that handles attachments, did lots of testing, packaged it with distutils, and generally transformed it from a quick hack to a diff --git a/doc/policy.ht b/doc/policy.ht index c559cb1..750cf08 100644 --- a/doc/policy.ht +++ b/doc/policy.ht @@ -4,8 +4,7 @@ Title: Python Milter Mail Policy These are the policies implemented by the bms.py milter application. The milter and Milter modules do not implement any policies -by themselves. Eventually, I'll get the bms.py milter moved to its -own package. +by themselves.

Classify connection

@@ -77,161 +76,119 @@ altered accordingly.

SPF check

-Finally, the MAIL FROM, connect IP, and HELO name are checked against -any SPF records published via DNS for the alleged sender (MAIL FROM). -If there is no SPF record, we check for a local substitute under the -domain defined in the [spf]delegate configuration. -Further checks depend on the result. +The MAIL FROM, connect IP, and HELO name are checked against +any SPF records published via DNS for the alleged sender (MAIL FROM) +to determine the official SPF policy result. +The offical SPF result is then logged in the Received-SPF header field, +but certain results are subjected to further processing to create +an effective result for policy purposes. - - +Note that HELO does not have any forwarding issues like MAIL FROM, and so +any result other than 'pass' or 'none' should be treated like 'fail'. - +Only if nothing about the SMTP envelope can be validated does the effective +result remain 'none. I call this the "3 strikes" rule. - +If the official result is 'permerror' (a syntax error in the sender's +policy), we use the 'lax' option in pyspf to try various heuristics to guess +what they really meant. For instance, the invalid mechanism "ip:1.2.3.4" is +treated as "ip4:1.2.3.4". The result of lax processing is then used +as the effective result for policy purposes. - +With an effective SPF result in hand, we consult the sendmail access +database to find our receiver policy for the sender. -
NONE -If there is no SPF record (official or delegated), then we -initiate a "three strikes and your out" regime, which looks for -some form of validated identification. -
    -
  1. We try a "best guess" SPF record of "v=spf1 a/24 mx/24 ptr". If this - passes, good. -
  2. We try to validate the HELO name. First check for an SPF record. - Otherwise, check whether the connect IP matches any A record for - the HELO name, or any A record for any MX name for the HELO name, - or is at least in the same /24 subnet as any of the above. - (In other words, a HELO SPF "best guess" of "v=spf1 a/24 mx/24".) - If so, good. We consider the HELO validated. If the HELO SPF - check fails, we reject the email. -
+If the official result is 'none', we try to turn it into an effective result of +'pass' or 'fail'. First, we check for a local substitute SPF record +under the domain defined in the [spf]delegate configuration. +It is often useful to add local SPF records for correspondents that are +too clueless to add their own. If there is no local substitute, we use a "best +guess" SPF record of "v=spf1 a/24 mx/24 ptr" for MAIL FROM or "v=spf1 a/24 +mx/24" for HELO. In addition, a HELO that is a subdomain of MAIL FROM and +resolves to the connect IP results in an effective result of 'pass'. + +If there is no local SPF record, and the effective result is still not +'pass', we check for either a valid HELO name or a valid PTR record for +the connect IP. A valid HELO or PTR cannot look like a dynamic name +as determined by the heuristic in Milter.dynip. + +If HELO has an SPF record, and the result is anything but pass, we reject +the connection: 
 2005Jul30 19:45:16 [93991] connect from [221.200.41.54] at ('221.200.41.54', 3581) EXTERNAL DYN
 2005Jul30 19:45:18 [93991] hello from adelphia.net
 2005Jul30 19:45:19 [93991] mail from  ()
 2005Jul30 19:45:19 [93991] REJECT: hello SPF: fail 550 access denied
-
    -
  1. If there is a validated PTR name, and it doesn't look - like a dynamic name, good. We consider the connection validated. -
-If any of the above can be validated, we continue on. -If none of the above can be validated, and the [SPF]reject_noptr -option is true, we reject the message immediately with the explanation -that we need some form of valid identification before we accept an email. -If [SPF]reject_noptr is false, we flag the message as -needing Call Back Validation. -The Call Back Valildation sends a DSN to the purported sender informing -them of the lack of identification. If the message is legitimate, the -sender needs to know that their email setup is broken and should be corrected. -If the message is forged, the sender is informed of the forgery, -and their need to publish an SPF record or at least use a valid HELO name. -If the purported sender does not accept the DSN, -then the message is rejected. The CBV status is cached to avoid -annoying the purported sender with too many DSNs. Currently, the DSN -is repeated to the same sender once per month. -

-In this example, although 3com.com has no SPF record, we assume that -any legitimate mail from them will at least have a valid HELO or PTR. -

-2005Jul30 23:52:03 [96777] connect from [222.252.233.200] at ('222.252.233.200', 29934) EXTERNAL DYN
-2005Jul30 23:52:03 [96777] hello from 3mail.3com.com
-2005Jul30 23:52:04 [96777] mail from  ()
-2005Jul30 23:52:04 [96777] REJECT: no PTR, HELO or SPF
-
-
PASS -A pass result normally lets the email continue on, but the domain is -tracked for reputation (and may be blocked), and may skip content scanning if -it matches a whitelist. -
-2005Jul24 17:44:26 [2104] mail from  ('SIZE=4410',)
-2005Jul24 17:44:26 [2104] Received-SPF: pass (mail.bmsi.com: domain of gnucash.org
-	designates 204.107.200.65 as permitted sender)
-	client-ip=204.107.200.65; envelope-from=gnucash-devel-bounces@gnucash.org; helo=cvs.gnucash.org;
-
-
NEUTRAL -A neutral result normally lets the email continue on, but the domain is not -tracked for reputation or matched against any whitelists. -Highly forged domains listed in [SPF]reject_neutral are -rejected. -
-2005Jul24 17:41:37 [2070] connect from cp500627-a.dbsch1.nb.home.nl at ('84.27.225.3', 3465) EXTERNAL
-2005Jul24 17:41:37 [2070] hello from cp500627-a.dbsch1.nb.home.nl
-2005Jul24 17:41:38 [2070] mail from  ()
-2005Jul24 17:41:38 [2070] REJECT: SPF neutral for nwarjejkw@yahoo.com
-
-
SOFTFAIL -A softfail result normally lets the email continue on, but the domain is not -tracked for reputation or matched against any whitelists. Furthermore, -the message is flagged as needing Call Back Validation, -and the highly forged domains listed in [SPF]reject_neutral are -rejected as well. -

-At present, we also require a valid HELO or PTR to avoid rejecting -a softfail. But this should probably change to only require a -successful CBV. -

-The Call Back Valildation sends a DSN to the purported sender informing -them of the softfail. If the message is legitimate, the sender needs -to know about the softfail so that their email setup can be corrected. -If the message is forged, the sender is informed of the forgery, confirming -that SPF is protecting their reputation and encouraging a rapid transition -to a strict policy. If the purported sender does not accept the DSN, -then the message is rejected. The CBV status is cached to avoid -annoying the purported sender with too many DSNs. Currently, the DSN -is repeated to the same sender once per month. -

-2005Jul24 15:41:33 [801] mail from  ()
-2005Jul24 15:41:33 [801] Received-SPF: softfail (mail.bmsi.com: transitioning domain of horafeliz.com
-	does not designate 221.184.83.185 as permitted sender)
-	client-ip=221.184.83.185; envelope-from=Aitp@horafeliz.com;
-	helo=p8185-ipad30funabasi.chiba.ocn.ne.jp;
-2005Jul24 15:41:33 [801] rcpt to  ()
-2005Jul24 15:41:35 [801] Subject: Microsoft, Adobe, Macromedia, Corel software. Up to 80% discount.
-2005Jul24 15:41:35 [801] X-Mailer: Microsoft Outlook, Build 10.0.2605
-2005Jul24 15:41:35 [801] CBV: Aitp@horafeliz.com
-2005Jul24 15:41:38 [801] REJECT: CBV: 550 : User unknown
-
-
FAIL -The message is rejected with a reference the SPF why page. -
-2005Jul30 19:53:27 [94070] connect from [212.70.52.16] at ('212.70.52.16', 3192) EXTERNAL DYN
-2005Jul30 19:53:27 [94070] hello from winzip.com
-2005Jul30 19:53:27 [94070] mail from  ()
-2005Jul30 19:53:27 [94070] REJECT: SPF fail 550 SPF fail:
-	see http://openspf.com/why.html?sender=dan@winzip.com&ip=212.70.52.16
-
+ + - - - - + -
REJECT +Reject the sender with a 550 5.7.1 SMTP code. The SMTP rejection +includes a detailed description of the problem.
PERMERROR -Permanent errors were called "unknown", and are still show that way -in the log. The message is rejected. Previously, we enabled "lax" parsing -of the SPF record, but rejecting is better because it informs the -sender about their problem. The next milter version will -look for a local substitute SPF record (as for a missing SPF record) -before rejecting. This will inform the sender of their problem, but -also let the receiver install a temporary workaround. -
-2005Jul24 18:05:37 [2312] mail from  () 
-2005Jul24 18:05:37 [2312] REJECT: SPF unknown 550 SPF Permanent Error:
-	include mechanism missing domain: include
-
-The SPF record for msg.euxiphipops.com looked like this at the time of the -above error: -
-msg.euxiphipops.com	TXT "v=spf1 mx ptr a include"
-
+
CBV +Do a Call Back Validation by connecting to an MX of the sender +and checking that using the sender as the RCPT TO is not rejected. +We quit the CBV connection before actualling sending a message. +If the CBV is rejected, our SMTP connection is rejected with the +same error code and message. CBV results are cached.
TEMPERROR -Temporary errors result in a 451 "Try again later" response. The sender -should retry the message at a later time. -
-2005Jul24 07:33:13 [29846] mail from  ('SIZE=73775', 'BODY=8BITMIME')
-2005Jul24 07:33:43 [29846] TEMPFAIL: SPF error 450 SPF Temporary Error: DNS Timeout
-
+
DSN +Do a Call Back Validation by connecting to an MX of the sender +and checking that using the sender as the RCPT TO is not rejected. +Unlike a CBV, we continue on to data and send a detailed message +explaining the problem. This can be useful for reporting PermError +or SoftFail to the sender. Keep in mind that for any result other +than 'pass', the sender could be forged, and your DSN could annoy the +wrong person. However, a SoftFail result is requesting such feedback +for debugging and a PermError result needs to be fixed by the sender ASAP +whether forged or not. DSN results are cached so that senders are +annoyed only weekly. +
OK +Accept the sender. The message may still be rejected via reputation +or content filtering.
+ +

SPF policy syntax

+ +First, the full sender is checked: +
+SPF-Fail:abeb@adelphia.net     DSN
+
+This says to accept mail from that adelphia.net user despite the +SPF fail, but only after annoying them with a DSN about their ISP's broken +policy. + +If there is no match on the full sender, the domain is checked: +
+SPF-Neutral:aol.com     REJECT
+
+This says to reject mail from AOL with an SPF result of neutral. +This means AOL users can't use their AOL address with another mail service +to send us mail. This is good because the other mail service is +likely a badly configured greeting card site or a virus. + +Finally, a default policy for the result is checked. While there are program +defaults, you should have defaults in the access database for SPF results: +
+SPF-Neutral:            CBV
+SPF-Softfail:           DSN
+SPF-PermError:          DSN
+SPF-TempError:          REJECT
+SPF-None:               REJECT
+SPF-Fail:               REJECT
+SPF-Pass:               OK
+
+ +

Reputation

+ +If the sender has not been rejected by this point, and if a GOSSiP server is +configured, we consult GOSSiP for the reputation score of the sender and +SPF result. The score is a number from -100 to 100 with a confidence +percentage from 0 to 100. A really bad reputation (less than -50 with +confidence greater than 3) is rejected. Note that the reputation is tracked +independently for each SPF result and sender combination. So aol.com:neutral +might have a really bad reputation, while aol.com:pass would be ok. +Furthermore, when a sender finally publishes an SPF policy and starts +getting SPF pass, their reputation is effectively reset.