DandelionNStuff/pymilter
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs for 0.8.10

Browse Source
This commit is contained in:
Stuart Gathman
2008-08-25 18:14:56 +00:00
parent 8cfa03bbc4
commit 600e3dfbfb
3 changed files with 144 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/changes.ht
+41
View File
@@ -2,6 +2,47 @@ Title: Recent Changes
<h2> Recent Changes </h2> <h2> Recent Changes </h2>
<h3> 0.8.10 </h3>
SRS rejections now log the recipient.
I have finally implemented plain CBV (no DSN). The CBV policy
will do a plain CBV from now on, and the DSN policy is required
if you want to send a DSN.
I started checking the MAIL FROM fullname (human readable part
of an email) for porn keywords. There is now a banned IP database.
IPs are banned for too many bad MAIL FROMs or RCPT TOs, and remain banned
for 7 days.
<h3> 0.8.9 </h3>
I use the <code>%ifarch</code> hack to build milter and milter-spf
packages as noarch, while pymilter is built as native.
I removed the spf dependency from dsn.py, so pymilter can be used without
installing pyspf, and added a Milter.dns module to let python milters do
general DNS lookups without loading pyspf.
<h3> 0.8.8 </h3>
Programs do not belong in the /var/log directory. I moved the
milter apps to /usr/lib/pymilter. Since having the programs and
data in the same directory is convenient for debugging, it will
still use an executable present in the datadir.
Several general utility classes and functions are now in the Milter package
for possible use by other python milters. In addition to the trivial example
milter, a simple SPF only milter is included as a realistic example.
The spec file now build 3 RPMs:
<ul>
<li> pymilter is the milter module and Milter package for use by all python
milters.
<li> milter is the all-singing, all-dancing python milter application, with
supporting <code>/etc/init.d</code>, logrotate and other scripts.
<li> milter-spf is the simple SPF only milter application.
</ul>
<h3> 0.8.7 </h3> <h3> 0.8.7 </h3>
The spf module has been moved to the The spf module has been moved to the
doc/credits.ht
+1 -1
View File
@@ -5,7 +5,7 @@ Title: Credits
<a href="mailto:Jim Niemira <urmane@urmane.org>">Jim Niemira</a> <a href="mailto:Jim Niemira <urmane@urmane.org>">Jim Niemira</a>
wrote the original C module and some quick wrote the original C module and some quick
and dirty python to use it. and dirty python to use it.
<a href="mailto:Stuart Gathman <stuart@bmsi.com>">Stuart D. Gathman</a> <a href="http://gathman.org/vitae">Stuart D. Gathman</a>
took that kludge and added threading and context objects to it, wrote a proper took that kludge and added threading and context objects to it, wrote a proper
OO wrapper (Milter.py) that handles attachments, did lots of testing, packaged OO wrapper (Milter.py) that handles attachments, did lots of testing, packaged
it with distutils, and generally transformed it from a quick hack to a it with distutils, and generally transformed it from a quick hack to a
doc/policy.ht
+102 -145
View File
@@ -4,8 +4,7 @@ Title: Python Milter Mail Policy
These are the policies implemented by the <code>bms.py</code> milter These are the policies implemented by the <code>bms.py</code> milter
application. The milter and Milter modules do not implement any policies application. The milter and Milter modules do not implement any policies
by themselves. Eventually, I'll get the bms.py milter moved to its by themselves.
own package.
<h3> Classify connection </h3> <h3> Classify connection </h3>
@@ -77,161 +76,119 @@ altered accordingly.
<h2> SPF check </h2> <h2> SPF check </h2>
Finally, the MAIL FROM, connect IP, and HELO name are checked against The MAIL FROM, connect IP, and HELO name are checked against
any SPF records published via DNS for the alleged sender (MAIL FROM). any SPF records published via DNS for the alleged sender (MAIL FROM)
If there is no SPF record, we check for a local substitute under the to determine the official SPF policy result.
domain defined in the <code>[spf]delegate</code> configuration. The offical SPF result is then logged in the Received-SPF header field,
Further checks depend on the result. but certain results are subjected to further processing to create
an effective result for policy purposes.
<table border=1> If the official result is 'none', we try to turn it into an effective result of
<tr><th>NONE</th><td> 'pass' or 'fail'. First, we check for a local substitute SPF record
If there is no SPF record (official or delegated), then we under the domain defined in the <code>[spf]delegate</code> configuration.
initiate a "three strikes and your out" regime, which looks for It is often useful to add local SPF records for correspondents that are
<b>some</b> form of validated identification. too clueless to add their own. If there is no local substitute, we use a "best
<ol> guess" SPF record of "v=spf1 a/24 mx/24 ptr" for MAIL FROM or "v=spf1 a/24
<li>We try a "best guess" SPF record of "v=spf1 a/24 mx/24 ptr". If this mx/24" for HELO. In addition, a HELO that is a subdomain of MAIL FROM and
passes, good. resolves to the connect IP results in an effective result of 'pass'.
<li> We try to validate the HELO name. First check for an SPF record.
Otherwise, check whether the connect IP matches any A record for If there is no local SPF record, and the effective result is still not
the HELO name, or any A record for any MX name for the HELO name, 'pass', we check for either a valid HELO name or a valid PTR record for
or is at least in the same /24 subnet as any of the above. the connect IP. A valid HELO or PTR cannot look like a dynamic name
(In other words, a HELO SPF "best guess" of "v=spf1 a/24 mx/24".) as determined by the heuristic in <code>Milter.dynip</code>.
If so, good. We consider the HELO validated. If the HELO SPF
check fails, we reject the email. If HELO has an SPF record, and the result is anything but pass, we reject
</ol> the connection:
<pre> <pre>
2005Jul30 19:45:16 [93991] connect from [221.200.41.54] at ('221.200.41.54', 3581) EXTERNAL DYN 2005Jul30 19:45:16 [93991] connect from [221.200.41.54] at ('221.200.41.54', 3581) EXTERNAL DYN
2005Jul30 19:45:18 [93991] hello from adelphia.net 2005Jul30 19:45:18 [93991] hello from adelphia.net
2005Jul30 19:45:19 [93991] mail from <wendy.stubbsua@link-it.com> () 2005Jul30 19:45:19 [93991] mail from <wendy.stubbsua@link-it.com> ()
2005Jul30 19:45:19 [93991] REJECT: hello SPF: fail 550 access denied 2005Jul30 19:45:19 [93991] REJECT: hello SPF: fail 550 access denied
</pre> </pre>
<ol> Note that HELO does not have any forwarding issues like MAIL FROM, and so
<li> If there is a validated PTR name, and it doesn't look any result other than 'pass' or 'none' should be treated like 'fail'.
like a dynamic name, good. We consider the connection validated.
</ol>
If any of the above can be validated, we continue on.
If none of the above can be validated, and the <code>[SPF]reject_noptr</code>
option is true, we reject the message immediately with the explanation
that we need some form of valid identification before we accept an email.
If <code>[SPF]reject_noptr</code> is false, we flag the message as
needing Call Back Validation.
The Call Back Valildation sends a DSN to the purported sender informing
them of the lack of identification. If the message is legitimate, the
sender needs to know that their email setup is broken and should be corrected.
If the message is forged, the sender is informed of the forgery,
and their need to publish an SPF record or at least use a valid HELO name.
If the purported sender does not accept the DSN,
then the message is rejected. The CBV status is cached to avoid
annoying the purported sender with too many DSNs. Currently, the DSN
is repeated to the same sender once per month.
<p>
In this example, although 3com.com has no SPF record, we assume that
any legitimate mail from them will at least have a valid HELO or PTR.
<pre>
2005Jul30 23:52:03 [96777] connect from [222.252.233.200] at ('222.252.233.200', 29934) EXTERNAL DYN
2005Jul30 23:52:03 [96777] hello from 3mail.3com.com
2005Jul30 23:52:04 [96777] mail from <etec_nic_family@3mail.3com.com> ()
2005Jul30 23:52:04 [96777] REJECT: no PTR, HELO or SPF
</pre>
</td></tr>
<tr><th>PASS</th><td> Only if nothing about the SMTP envelope can be validated does the effective
A pass result normally lets the email continue on, but the domain is result remain 'none. I call this the "3 strikes" rule.
tracked for reputation (and may be blocked), and may skip content scanning if
it matches a whitelist.
<pre>
2005Jul24 17:44:26 [2104] mail from <gnucash-devel-bounces@gnucash.org> ('SIZE=4410',)
2005Jul24 17:44:26 [2104] Received-SPF: pass (mail.bmsi.com: domain of gnucash.org
designates 204.107.200.65 as permitted sender)
client-ip=204.107.200.65; envelope-from=gnucash-devel-bounces@gnucash.org; helo=cvs.gnucash.org;
</pre>
</td></tr>
<tr><th>NEUTRAL</th><td> If the official result is 'permerror' (a syntax error in the sender's
A neutral result normally lets the email continue on, but the domain is not policy), we use the 'lax' option in pyspf to try various heuristics to guess
tracked for reputation or matched against any whitelists. what they really meant. For instance, the invalid mechanism "ip:1.2.3.4" is
Highly forged domains listed in <code>[SPF]reject_neutral</code> are treated as "ip4:1.2.3.4". The result of lax processing is then used
rejected. as the effective result for policy purposes.
<pre>
2005Jul24 17:41:37 [2070] connect from cp500627-a.dbsch1.nb.home.nl at ('84.27.225.3', 3465) EXTERNAL
2005Jul24 17:41:37 [2070] hello from cp500627-a.dbsch1.nb.home.nl
2005Jul24 17:41:38 [2070] mail from <nwarjejkw@yahoo.com> ()
2005Jul24 17:41:38 [2070] REJECT: SPF neutral for nwarjejkw@yahoo.com
</pre>
</td></tr>
<tr><th>SOFTFAIL</th><td> With an effective SPF result in hand, we consult the sendmail access
A softfail result normally lets the email continue on, but the domain is not database to find our receiver policy for the sender.
tracked for reputation or matched against any whitelists. Furthermore,
the message is flagged as needing Call Back Validation,
and the highly forged domains listed in <code>[SPF]reject_neutral</code> are
rejected as well.
<p>
At present, we also require a valid HELO or PTR to avoid rejecting
a softfail. But this should probably change to only require a
successful CBV.
<p>
The Call Back Valildation sends a DSN to the purported sender informing
them of the softfail. If the message is legitimate, the sender needs
to know about the softfail so that their email setup can be corrected.
If the message is forged, the sender is informed of the forgery, confirming
that SPF is protecting their reputation and encouraging a rapid transition
to a strict policy. If the purported sender does not accept the DSN,
then the message is rejected. The CBV status is cached to avoid
annoying the purported sender with too many DSNs. Currently, the DSN
is repeated to the same sender once per month.
<pre>
2005Jul24 15:41:33 [801] mail from <Aitp@horafeliz.com> ()
2005Jul24 15:41:33 [801] Received-SPF: softfail (mail.bmsi.com: transitioning domain of horafeliz.com
does not designate 221.184.83.185 as permitted sender)
client-ip=221.184.83.185; envelope-from=Aitp@horafeliz.com;
helo=p8185-ipad30funabasi.chiba.ocn.ne.jp;
2005Jul24 15:41:33 [801] rcpt to <david@example.com> ()
2005Jul24 15:41:35 [801] Subject: Microsoft, Adobe, Macromedia, Corel software. Up to 80% discount.
2005Jul24 15:41:35 [801] X-Mailer: Microsoft Outlook, Build 10.0.2605
2005Jul24 15:41:35 [801] CBV: Aitp@horafeliz.com
2005Jul24 15:41:38 [801] REJECT: CBV: 550 <Aitp@horafeliz.com>: User unknown
</pre>
</td></tr>
<tr><th>FAIL</th><td> <table border=1>
The message is rejected with a reference the SPF why page. <tr><th>REJECT</th><td>
<pre> Reject the sender with a 550 5.7.1 SMTP code. The SMTP rejection
2005Jul30 19:53:27 [94070] connect from [212.70.52.16] at ('212.70.52.16', 3192) EXTERNAL DYN includes a detailed description of the problem.
2005Jul30 19:53:27 [94070] hello from winzip.com
2005Jul30 19:53:27 [94070] mail from <dan@winzip.com> ()
2005Jul30 19:53:27 [94070] REJECT: SPF fail 550 SPF fail:
see http://openspf.com/why.html?sender=dan@winzip.com&ip=212.70.52.16
</pre>
</td></tr> </td></tr>
<tr><th>CBV</th><td>
<tr><th>PERMERROR</th><td> Do a Call Back Validation by connecting to an MX of the sender
Permanent errors were called "unknown", and are still show that way and checking that using the sender as the RCPT TO is not rejected.
in the log. The message is rejected. Previously, we enabled "lax" parsing We quit the CBV connection before actualling sending a message.
of the SPF record, but rejecting is better because it informs the If the CBV is rejected, our SMTP connection is rejected with the
sender about their problem. The next milter version will same error code and message. CBV results are cached.
look for a local substitute SPF record (as for a missing SPF record)
before rejecting. This will inform the sender of their problem, but
also let the receiver install a temporary workaround.
<pre>
2005Jul24 18:05:37 [2312] mail from <b-mihdbcgaacaa-becibijh-000-@msg.euxiphipops.com> ()
2005Jul24 18:05:37 [2312] REJECT: SPF unknown 550 SPF Permanent Error:
include mechanism missing domain: include
</pre>
The SPF record for msg.euxiphipops.com looked like this at the time of the
above error:
<pre>
msg.euxiphipops.com TXT "v=spf1 mx ptr a include"
</pre>
</td></tr> </td></tr>
<tr><th>DSN</th><td>
<tr><th>TEMPERROR</th><td> Do a Call Back Validation by connecting to an MX of the sender
Temporary errors result in a 451 "Try again later" response. The sender and checking that using the sender as the RCPT TO is not rejected.
should retry the message at a later time. Unlike a CBV, we continue on to data and send a detailed message
<pre> explaining the problem. This can be useful for reporting PermError
2005Jul24 07:33:13 [29846] mail from <quickenloans@rate.quicken.com> ('SIZE=73775', 'BODY=8BITMIME') or SoftFail to the sender. Keep in mind that for any result other
2005Jul24 07:33:43 [29846] TEMPFAIL: SPF error 450 SPF Temporary Error: DNS Timeout than 'pass', the sender could be forged, and your DSN could annoy the
</pre> wrong person. However, a SoftFail result is requesting such feedback
for debugging and a PermError result needs to be fixed by the sender ASAP
whether forged or not. DSN results are cached so that senders are
annoyed only weekly.
</td></tr>
<tr><th>OK</th><td>
Accept the sender. The message may still be rejected via reputation
or content filtering.
</td></tr> </td></tr>
</table> </table>
<h3> SPF policy syntax </h3>
First, the full sender is checked:
<pre>
SPF-Fail:abeb@adelphia.net DSN
</pre>
This says to accept mail from that adelphia.net user despite the
SPF fail, but only after annoying them with a DSN about their ISP's broken
policy.
If there is no match on the full sender, the domain is checked:
<pre>
SPF-Neutral:aol.com REJECT
</pre>
This says to reject mail from AOL with an SPF result of neutral.
This means AOL users can't use their AOL address with another mail service
to send us mail. This is good because the other mail service is
likely a badly configured greeting card site or a virus.
Finally, a default policy for the result is checked. While there are program
defaults, you should have defaults in the access database for SPF results:
<pre>
SPF-Neutral: CBV
SPF-Softfail: DSN
SPF-PermError: DSN
SPF-TempError: REJECT
SPF-None: REJECT
SPF-Fail: REJECT
SPF-Pass: OK
</pre>
<h2> Reputation </h2>
If the sender has not been rejected by this point, and if a GOSSiP server is
configured, we consult GOSSiP for the reputation score of the sender and
SPF result. The score is a number from -100 to 100 with a confidence
percentage from 0 to 100. A really bad reputation (less than -50 with
confidence greater than 3) is rejected. Note that the reputation is tracked
independently for each SPF result and sender combination. So aol.com:neutral
might have a really bad reputation, while aol.com:pass would be ok.
Furthermore, when a sender finally publishes an SPF policy and starts
getting SPF pass, their reputation is effectively reset.