Files
pymilter/milter.html
T
Stuart Gathman 3d7003a638 Web site updates.
2005-06-14 22:02:09 +00:00

503 lines
24 KiB
HTML

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>Python Milters</title>
</head><body>
<P ALIGN="CENTER"><A HREF="http://www.anybrowser.org/campaign/">
<IMG SRC="/art/brain1.gif"
ALT="Viewable With Any Browser" BORDER="0"></A>
<img src="/art/banner_4.gif" width="468" height="60" border="0"
usemap="#banner_4" alt="Your vote?">
<map name="banner_4">
<area shape="rect" coords="330,25,426,59"
href="http://education-survey.org/" alt="I Disagree">
<area shape="rect" coords="234,28,304,57" href="http://www.honestEd.com/" alt="I Agree">
</map>
</P>
<h1 align=center>Sendmail Milters in Python</h1>
<h4 align=center>by <a href="mailto:%75%72%6D%61%6E%65%40%6E%65%75%72%61l%61%63%63%65%73%73%2E%63%6F%6D">Jim Niemira</a>
and <a href="mailto:%73%74%75%61%72%74%40%62%6D%73%69%2E%63%6F%6D">
Stuart D. Gathman</a><br>
This web page is written by Stuart D. Gathman<br>and<br>sponsored by
<a href="http://www.bmsi.com">Business Management Systems, Inc.</a> <br>
Last updated Jun 09, 2005</h4>
See the <a href="faq.html">FAQ</a> | <a href="http://sourceforge.net/project/showfiles.php?group_id=139894">Download now</a> |
<a href="/mailman/listinfo/pymilter">Subscribe to mailing list</a> |
<a href="#overview">Overview</a> |
<a href="/python/dspam.html">pydspam</a> |
<a href="/libdspam/dspam.html">libdspam</a>
<p>
<a href="//www.python.org">
<img src="python55.gif" align=left alt="A Python"></a>
<a href="//www.sendmail.org/">Sendmail</a> introduced a
<a href="http://www.milter.org/milter_api/api.html"> new API</a> beginning with version 8.10 -
libmilter. The milter module for <a href="//www.python.org">Python</a>
provides a python interface to libmilter that exploits all its features.
<p>
Sendmail 8.12 officially releases libmilter.
Version 8.12 seems to be more robust, and includes new privilege
separation features to enhance security. Even better, sendmail 8.13
supports socket maps, which makes <a href="pysrs.html">pysrs</a> much more
efficient and secure. I recommend upgrading.
<h2> Recent Changes </h2>
Python milter is being moved to
<a href="http://sourceforge.net/projects/pymilter/">pymilter Sourceforge
project</a> for development.
<p>
Release 0.8.0 is the first <a href="http://sourceforge.net/">Sourceforge</a>
release. It supports Python-2.4, and provides an option to accept mail
that gets an SPF softfail or fails the 3 strikes rule, provided the
alleged sender accepts a DSN explaining the problem. Python-2.3 is
no longer supported by the reworked mime.py module, although API changes
could be backported. There are too many incompatible changes to the
python email package.
<p>
Release 0.7.2 tightens the authentication screws with a "3 strikes and
you're out" policy. A sender must have a valid PTR, HELO, or SPF record
to send email. Specific senders can be whitelisted using the
"delegate" option in the spf configuration section by adding a
default SPF record for them. The PTR and HELO are required
by RFC anyway, so this is not an unreasonable requirement.
There is now a coherent policy for an SPF softfail result. A softfail
is accepted if there is a valid PTR or HELO, or if the domain
is listed in the "accept_softfail" option of the spf configuration section.
A neutral result is accepted by default if there is a valid PTR or
HELO, (and the SPF record was not guessed), unless the domain is listed in the
"reject_neutral" option. Common forms of PTR records for dynamic IPs are
recognized, and do not count as a valid PTR. This does not prevent anyone
from sending mail from a dynamic IP - they just need to configure a
valid HELO name or publish an SPF record.
<p>
As SPF adoption continues to rise, forged spam is not getting through. So
spammers are publishing their SPF records as predicted. The 0.7.2 RPM
now provides the <code>rhsbl</code> sendmail hack so that spammer domains
can be blacklisted. With the RPM installed, add a line like the following
to your <code>sendmail.mc</code>.
<pre>
HACK(rhsbl,`blackholes.example.com',"550 Rejected: " $&{RHS} " has been spamming our customers.")dnl
</pre>
<p>
Of course, spammers are now starting to register
throwaway domains. The next thing we need is a custom DNS server,
in Python, that
can recognize patterns. For instance, one spammer registers ded304.com,
ded305.com, ded306.com, etc. We also need the custom DNS server to
let SPF classic clients check SES (which will be part of pysrs).
The <a href="http://twistedmatrix.com/products/twisted">Twisted Python</a>
framework provides a custom DNS server - but I
would like a smaller implementation for our use.
<p>
The RPM for release 0.7.0 moves the config file and socket locations to
/etc/mail and /var/run/milter respectively. We now parse Microsoft CID records
- but only hotmail.com uses them. They seem to have applied for a patent on
the brilliant idea of examining the mail headers to see who the message is
from. We aren't doing that here, so not to worry - but I am not a lawyer, so
if you are worried, change spf.py around line 626 to return None instead of
calling CIDParser(). There is a new option to reject mail with no PTR
and no SPF.
<p>
Microsoft is pushing an anti-opensource license for their pending patent
along with their sender-ID proposal before the IETF.
It is royalty free - but requires anyone distributing a binary they've
compiled from source to sign a license agreement. The Apache Software
Foundation <a
href="http://www.apache.org/foundation/docs/sender-id-position.html"> explains
the problem with sender-ID</a>, and Debian <a
href="http://www.debian.org/News/2004/20040904">concurs</a>. Since
the <a href="http://download.microsoft.com/download/4/3/9/439b024b-09fd-44ee-8ff0-10e834004c36/senderid_FAQ.PDF">Microsoft license</a> is
<a href="http://www.circleid.com/article/732_0_1_0_C/">incompatible with free
software in general</a> and the <a
href="http://www.imc.org/ietf-mxcomp/mail-archive/msg03678.html">GPL in
particular</a>, Python milter will not be able to implement sender-ID in its
current form. This was, no doubt, Microsoft's intent all along.
<p>
Sender-ID attempts to do for RFC2822 headers what SPF does for RFC2821 headers.
Unlike SPF, it has never been tried, and is encumbered by a stupid patent. I
recommend ignoring it and continuing to implement and improve SPF until a
working and unencumbered proposal for RFC2822 headers surfaces.
<p>
<a href="http://spf.pobox.com">
<img src="SPF.gif" align=left alt="SPF logo"></a>
Release 0.6.6 adds support for <a href="http://spf.pobox.com/">SPF</a>,
a protocol to prevent forging of the envelope from address.
SPF support requires <a href="http://pydns.sourceforge.net/">pydns</a>.
The included spf.py module is an updated version of the original 1.6
version at <a href="http://www.wayforward.net/spf/">wayforward.net</a>.
The updated version tracks the draft RFC and test suite.
<p>
The FAQ addresses <a href="faq.html#spf">how to get started with SPF</a>.
<p>
Release 0.6.1 adds a full milter based dspam application.
<p>
I have selected the <a href="http://www.nuclearelephant.com/projects/dspam/">
dspam bayes filter project</a> and <a href="dspam.html">
packaged it for python</a>.
Release 0.6.0 offers a simple application of dspam I call "header triage",
which rejects messages with spammy headers.
To use header triage, you must have <a href="dspam.html">DSPAM</a> installed,
and select a dictionary that is well moderated by someone who gets
lots of spam. That dictionary can be used to block spam that is
obvious from the headers (e.g. X-Mailer and Subject) before it ties
up any more resources. I have yet to see any false positives from this
approach (check the milter log), but if there are, the sender will
get a REJECT with the message "Your message looks spammy."
<h2> Enough Already! </h2>
Nearly a dozen people have emailed me begging for a feature to copy
outgoing and/or incoming mail to a backup directory by user. Ok, it
looks like this is a most requested feature for 0.5.6. In the meantime,
here are some things to consider:
<ul>
<li> If you want to equivalent of a Bcc added to each message, this
is very easy to do in the python code for bms.py. See below.
<li> If you want to copy to a file in a directory (thus avoiding having to
set up aliases), this is slightly more involved. The bms.py milter already
copies the message to a temporary file for use in replacing the message body
when banned attachments are found. You have to open a file, and copy the
Mesage object to it in eom().
<li> Finally, you are probably aware that most email clients already
keep a copy of outgoing mail? Presumably there is a good reason for
keeping another copy on the server.
</ul>
<p>
To Bcc a message, call <code>self.add_recipient(rcpt)</code> in envfrom after
determining whether you want to copy (e.g. whether the sender is local). For
example,
<pre>
def envfrom(...
...
if len(t) == 2:
self.rejectvirus = t[1] in reject_virus_from
if t[0] in wiretap_users.get(t[1],()):
self.add_recipient(wiretap_dest)
if t[1] == 'mydomain.com':
self.add_recipient('&lt;copy-%s&gt;' % t[0])
...
</pre>
<p>
To make this a generic feature requires thinking about how the configuration
would look. Feel free to make specific suggestions about config file
entries. Be sure to handle both Bcc and file copies, and designating what
mail should be copied. How should "outgoing" be defined? Implementing it is
easy once the configuration is designed.
<h3><a name=overview>Overview</a></h3>
This package provides a robust toolkit for Python <a
href="#milter">milters</a>, and the beginnings of a general purpose mail
filtering system written in Python.
<p>
At the lowest level, the 'milter' module provides a thin wrapper around the
<a href="http://www.milter.org/milter_api/api.html">
sendmail libmilter API</a>. This API lets you register callbacks for
a number of events in the
<a href="http://www.cs.concordia.ca/~group/fig/public/email/relay/milter+ruleset-checks.html">process of sendmail receiving a message via SMTP</a>.
These events include the initial connection from a MTA,
the envelope sender and recipients, the top level mail headers, and
the message body. There are options to mangle all of these components
of the message as it passes through the milter.
<p>
At the next level, the 'Milter' module (note the case difference) provides a
Python friendly object oriented wrapper for the low level API. To use the
Milter module, an application registers a 'factory' to create an object
for each connection from a MTA to sendmail. These connection objects
must provide methods corresponding to the libmilter callback events.
<p>
Each event method returns a code to tell sendmail whether to proceed
with processing the message. This is a big advantage of milters over
other mail filtering systems. Unwanted mail can be stopped in its
tracks at the earliest possible point.
<p>
The Milter.Milter class provides default implementations for event
methods that
do nothing, and also provides wrappers for the libmilter methods to mutate
the message.
<p>
The 'spf' module provides an implementation of <a href="http://spf.pobox.com">
SPF</a> useful for detecting email forgery.
<p>
The 'mime' module provides a wrapper for the Python email package that
fixes some bugs, and simplifies modifying selected parts of a MIME message.
<p>
Finally, the bms.py application is both a sample of how to use the
Milter and spf modules, and the beginnings of a general purpose SPAM filtering,
wiretapping, SPF checking, and Win32 virus protecting milter. It can
make use of the <a href="pysrs.html">pysrs</a> package when available for
SRS/SES checking and the <a href="dspam.html">pydspam</a> package for Bayesian
content filtering. SPF checking
requires <a href="http://pydns.sourceforge.net/">
pydns</a>. Configuration documentation is currently included as comments
in the <a href="milter.cfg">sample config file</a> for the bms.py milter.
<p>
Python milter is under GPL. The authors can probably be convinced to
change this to LGPL if needed.
<h3>What is a <a name="milter">milter</a>?</h3>
Milters can run on the same machine as sendmail, or another machine. The
milter can even run with a different operating system or processor than
sendmail.
Sendmail talks to the milter via a local or internet socket.
Sendmail keeps the
milter informed of events as it processes a mail connection. At any
point, the milter can cut the conversation short by telling sendmail
to ACCEPT, REJECT, or DISCARD the message. After receiving a complete
message from sendmail, the milter can again REJECT or DISCARD it, but it
can also ACCEPT it with changes to the headers or body.
<h3> What can you do with a milter? </h3>
<menu>
<li> A milter can DISCARD or REJECT spam based based on algorithms scripted
in python rather than sendmail's cryptic "cf" language.
<li> A milter can alter or remove attachments from mail that are poisonous to
Windows.
<li> A milter can scan for viruses and clean them when detected.
<li> A milter scans outgoing as well as incoming mail.
<li> A milter can add and delete recipients to forward or secretly
copy mail.
<li> For more ideas, check the <a href="//www.milter.org">Milter Web Page</a>.
</menu>
<a href="http://www.milter.org/milter_api/api.html">
Documentation</a> for the C API is provided with sendmail. Miltermodule
provides a thin python wrapper for the C API. Milter.py provides a simple
OO wrapper on top of that.
<p>
The Python milter package includes a sample milter that replaces dangerous
attachments with a warning message, discards mail addressed to
MAILER-DAEMON, and demonstrates several SPAM abatement strategies.
The MimeMessage class to do this used to be based on the
<code>mimetools</code> and <code>multifile</code> standard python packages.
As of milter version 0.6.0, it is based on the email standard
python packages, which were derived from the
<a href="http://sourceforge.net/projects/mimelib">mimelib</a> project.
The MimeMessage class patches several bugs in the email package,
and provides some backward compatibility.
<p>
The "defang" function of the sample milter was inspired by
<a href="http://www.roaringpenguin.com/mimedefang/">MIMEDefang</a>,
a Perl milter with flexible attachment processing options. The latest
version of MIMEDefang uses an apache style process pool to avoid reloading
the Perl interpreter for each message. This makes it fast enough for
production without using Perl threading.
<p>
<a href="http://sourceforge.net/projects/mailchecker">mailchecker</a> is
a Python project to provide flexible attachment processing for mail. I
will be looking at plugging mailchecker into a milter.
<p>
<a href="http://software.libertine.org/tmda/">TMDA</a> is a Python project
to require confirmation the first time someone tries to send to your
mailbox. This would be a nice feature to have in a milter.
<p>
There is also a <a href="http://www.milter.org/">Milter community website</a>
where milter software and gory details of the API are discussed.
<h3> Is a milter written in python efficient? </h3>
The python milter process is multi-threaded and startup cost is incurred
only once. This is much more efficient than some implementations that
start a new interpreter for each connection. Testing in a production
environment did not use a significant percentage of the CPU. Furthermore,
python is easily extended in C for any step requiring expensive CPU
processing.
<p>
For example, the HTML parsing feature to remove scripts from HTML attachments
is rather CPU intensive in pure python. Using the C replacement for sgmllib
greatly speeds things up.
<h3> Goals </h3>
<menu>
<li> Implement RRS - a backdoor for non-SRS forwarders. User lists non-SRS
forwarder accounts (perhaps in <code>~/.forwarders</code>), and a util
provides a special local alias for the user to give to the forwarder.
Alias only works for mail from that forwarder. Milter gets forwarder
domain from alias and uses it to SPF check forwarder. Requires
milter to have read access to <code>~/.forwarders</code> or else
a way for user to submit entries to milter database.
<li> The bms.py milter has too many features. Create a framework where
numerous small feature modules can be plugged together in the
configuration.
<li> Create a pure python substitute for miltermodule and libmilter that
implements the <a
href="http://www.duh.org/cvsweb.cgi/~checkout~/pmilter/doc/milter-protocol.txt?rev=1">
libmilter protocol</a> in python.
<li> Find or write a faster implementation of sgmllib. The
<a href="http://www.effbot.org/zone/sgmlop-index.htm">sgmlop package</a>
is not very compatible with
<a href="http://www.python.org/doc/2.1.3/lib/module-sgmllib.html">
Python-2.1 sgmllib</a>, but it is a start, and is supported in
milter-0.4.5 or later.
<li> Implement all or most of the features of
<a href="http://www.roaringpenguin.com/mimedefang/">MIMEDefang</a>.
<li> Follow the official <a href="http://www.python.org/peps/pep-0008.html">
Python coding standards</a> more closely.
<li> Make unit test code more like other python modules.
</menu>
<h3> Confirmed Installations </h3>
Please <a href="mailto:%73%74%75%61%72%74%40%62%6D%73%69%2E%63%6F%6D">email</a>
me if you successfully install milter on a system not mentioned below.
<p>
<table>
<tr>
<th>Operating System</th> <th>Compiler</th> <th>Python</th> <th>Sendmail</th>
<th>milter</th>
<tr>
<td>Mandrake 8.0</td><td>gcc-3.0.1</td><td>2.1.1</td><td>8.12.0</td>
<td>0.3.3</td><tr>
<td>Mandrake 8.0</td><td>gcc-2.96</td><td>2.0</td><td>8.11.2</td>
<td>0.3.6</td><tr>
<td>RedHat 6.2</td><td>egcs-1.1.2</td><td>2.2.2</td><td>8.11.6</td>
<td>0.5.4</td><tr>
<td>RedHat 7.1</td><td>gcc-2.96</td><td>?</td><td>8.12.1</td>
<td>0.3.5</td><tr>
<td>RedHat 7.3</td><td>gcc-2.96</td><td>2.2.2</td><td>8.11.6</td>
<td>0.5.5</td><tr>
<td>RedHat 7.3</td><td>gcc-2.96</td><td>2.3.3</td><td>8.13.1</td>
<td>0.7.2</td><tr>
<td>RedHat 8.0</td><td>gcc-3.2</td><td>2.2.1</td><td>8.12.6</td>
<td>0.5.2</td><tr>
<td>Debian Linux</td><td>gcc-2.95.2</td><td>2.1.1</td><td>8.12.0</td>
<td>0.3.7</td><tr>
<td>Debian Linux</td><td>gcc-3.2.2</td><td>2.2.2</td><td>8.12.7</td>
<td>0.5.4</td><tr>
<td>AIX-4.1.5</td><td>gcc-2.95.2</td><td>2.1.1</td><td>8.11.5</td>
<td>0.3.3</td><tr>
<td>AIX-4.1.5</td><td>gcc-2.95.2</td><td>2.1.1</td><td>8.12.1</td>
<td>0.3.4</td><tr>
<td>AIX-4.1.5</td><td>gcc-2.95.2</td><td>2.1.3</td><td>8.12.3</td>
<td>0.4.2</td><tr>
<td>AIX-4.1.5</td><td>gcc-2.95.2</td><td>2.2.3</td><td>8.13.1</td>
<td>0.7.1</td><tr>
<td>Slackware 7.1</td><td>?</td><td>?</td><td>8.12.1</td>
<td>0.3.8</td><tr>
<td>Slackware 9.0</td><td>gcc-3.2.2</td><td>2.2.3</td><td>8.12.9</td>
<td>0.5.4</td><tr>
<td>OpenBSD</td><td>?</td><td>2.3.3?</td><td>8.13.1?</td>
<td>0.7.2</td><tr>
<td>SuSE 7.3</td><td>gcc-2.95.3</td><td>2.1.1</td><td>8.12.2</td>
<td>0.3.9</td><tr>
<td>FreeBSD</td><td>gcc-2.95.3</td><td>2.2.1</td><td>8.12.3</td>
<td>0.4.0</td><tr>
<td>FreeBSD</td><td>gcc-2.95.3</td><td>2.2.2</td><td>?</td>
<td>0.5.5</td><tr>
<td>FreeBSD 4.4</td><td>gcc-2.95.3</td><td>?</td><td>8.12.10</td>
<td>0.6.6</td><tr>
</table>
<h3> Requirements </h3>
<menu>
<li> While the miltermodule will work with python 1.5, you probably
want to use python 2.0 or better. The python code uses a number of
python 2 features.
<li> Python must be configured with thread support. This is because
sendmail's libmilter requires thread support.
<li> You must compile sendmail with libmilter enabled. In versions of
sendmail prior to 8.12 libmilter is marked FFR (For Future Release) and
is not installed by default.
Sendmail 8.12 still does not enable libmilter by default. You must
explicitly select the "MILTER" option when compiling.
<li> Python milter has been tested against sendmail-8.11 and sendmail-8.12.
<li> Python milter must be compiled for the specific version of sendmail
it will run with. (Since the result is dynamically loaded, there could
conceivably be multiple versions available and selected at startup - but
that will have to wait.) This situation may only exist for sendmail
versions prior to 8.12. The protocol seems designed for backward
compatibility - and 8.12 is the first official milter release.
<li> Mea Culpa! After reading the Python Style guide, I realize that
my Python code is not up to snuff. Apparently mixed tabs and spaces
are anathema to those using Windows editors, where tabs can be expanded using
any arbitrary algorithm. Other than that, my
intuition matched Guido's pretty well - although I like to indent by 2
rather than 4. I will arrange to have tabs expanded to spaces when
exporting new versions. Until then, beware!
</menu>
<h3> <a name="aix4"> AIX 4.1.5 Requirements </a> </h3>
To create sendmail RPMs for AIX, you can download my AIX 4.1.5 spec files
for <a href="/aix/sendmail.spec">sendmail-8.11.5</a>
or <a href="/aix/sendmail12.spec">sendmail-8.12.3</a>. If you have
not already set it up, I use a <a href="/aix/aix.spec">dummy RPM package</a>
to represent the stuff that comes with AIX. You might also want
my <a href="/aix/python.spec">python-2.1.1</a> spec file for AIX. It
does not include Tk or curses modules, sorry. If y'all trust me, you can
download rpms for AIX 4.x from my <a href="/aix">AIX RPM directory</a>.
<p>
Sendmail-8.12 renames
libsmutil.a to libsm.a. Unfortunately, libsm.a is an important AIX system
shared library. Therefore, I rename libsm.a back to libsmutil.a for
AIX. This presents a problem for setup.py.
<h3> <a name="rh72"> RedHat 7.2 Requirements </a> </h3>
If you are running Redhat 7.2, the distributed version of sendmail
now enables libmilter by default. RedHat 7.2 bundles
the development libraries with the main sendmail package, so
there is no sendmail-devel package. However, they forgot to include the
headers! So you'll have to get the SRPM and modify it. I suggest
moving the static libs to a devel package and adding the headers. If
this is too much trouble, you can get the <a href="mfapi.h">mfapi.h</a>
header for sendmail-8.6.11 from here and manually install it as
<code>/usr/include/libmilter/mfapi.h</code>.
<p>
If you do modify the SRPM, I suggest renaming libsmutil.a
to libsm.a - just like sendmail-8.12 will. If you manually install
mfapi.h or don't rename libsmutil.a, you'll
need to force <code>libs = ["milter", "smutil"]</code> in setup.py.
<p>
If you have installed python2, and want
python-milter to use python2, add <code>python=python2</code> to setup.cfg
and build with <code>python2 setup.py bdist_rpm</code>.
<h3> <a name="rh62"> Redhat 6.2 Requirements </a> </h3>
If you are running Redhat 6.2, the distributed version of sendmail
does not enable libmilter. You can download the Redhat 7.2 sendmail.spec
modified to compile on RedHat 6.2:
<a href="http://www.bmsi.com/linux/rh62/sendmail-rhmilter.spec">
sendmail-rhmilter.spec</a>. The <a
href="ftp://updates.redhat.com/7.0/en/os/SRPMS/sendmail-8.11.6-1.7.0.src.rpm">
SRPM for sendmail-8.11.6</a> is available from
<a href="http://www.redhat.com">Redhat</a> under
<a href="http://www.redhat.com/support/errata/RHSA-2001-106.html">
Errata for RH6.2</a>. But that doesn't include the latest security
patches since RH6.2 is no longer supported.
<p>
If y'all trust me, you can pick up source and binary sendmail RPMs for RH6.2
from my <a href="http://www.bmsi.com/linux/rh62">linux downloads</a> directory.
The lastest RPMs were built by taking a RH7.2 SRPMS and removing some
RPM features from the spec file that RH6.2 doesn't support, then
recompiling on RH6.2. You can check this by installing the RH7.2 SRPM,
then diffing my sendmail.spec with theirs. Then run
"rpm -bb sendmail-rhmilter.spec" when you are satisfied.
<p>
If you have installed python2, and want
python-milter to use python2, add <code>python=python2</code> to setup.cfg
and build with <code>python2 setup.py bdist_rpm</code>.
You'll need to install the sendmail-devel package to compile milter.
<hr>
<p>
<a href="http://validator.w3.org/check/referer">
<img border=0 src="/vh32.png" alt=" [ Valid HTML 3.2! ] " height=31 width=88></a>
<a href="http://www.redhat.com">
<img src="/art/powered_by.gif" width="88" height="31" alt=" [ Powered By Red Hat Linux ] " border="0"></a>
</p>
</body></html>