%ifarch hack to build milter and milter-spf
-packages as noarch, while pymilter is built as native.
-
-I removed the spf dependency from dsn.py, so pymilter can be used without
-installing pyspf, and added a Milter.dns module to let python milters do
-general DNS lookups without loading pyspf.
-
-/etc/init.d, logrotate and other scripts.
-
-As SPF adoption continues to rise, forged spam is not getting through. So
-spammers are publishing their SPF records as predicted. The 0.7.2 RPM
-now provides the rhsbl sendmail hack so that spammer domains
-can be blacklisted. With the RPM installed, add a line like the following
-to your sendmail.mc.
-
-HACK(rhsbl,`blackholes.example.com',"550 Rejected: " $&{RHS} " has been spamming our customers.")dnl
-
--Of course, spammers are now starting to register -throwaway domains. The next thing we need is a custom DNS server, -in Python, that -can recognize patterns. For instance, one spammer registers ded304.com, -ded305.com, ded306.com, etc. We also need the custom DNS server to -let SPF classic clients check SES (which will be part of pysrs). -The Twisted Python -framework provides a custom DNS server - but I -would like a smaller implementation for our use. -
-The RPM for release 0.7.0 moves the config file and socket locations to -/etc/mail and /var/run/milter respectively. We now parse Microsoft CID records -- but only hotmail.com uses them. They seem to have applied for a patent on -the brilliant idea of examining the mail headers to see who the message is -from. We aren't doing that here, so not to worry - but I am not a lawyer, so -if you are worried, change spf.py around line 626 to return None instead of -calling CIDParser(). There is a new option to reject mail with no PTR -and no SPF. -
-Microsoft is pushing an anti-opensource license for their pending patent -along with their sender-ID proposal before the IETF. -It is royalty free - but requires anyone distributing a binary they've -compiled from source to sign a license agreement. The Apache Software -Foundation explains -the problem with sender-ID, and Debian concurs. Since -the Microsoft license is -incompatible with free -software in general and the GPL in -particular, Python milter will not be able to implement sender-ID in its -current form. This was, no doubt, Microsoft's intent all along. -
-Sender-ID attempts to do for RFC2822 headers what SPF does for RFC2821 headers. -Unlike SPF, it has never been tried, and is encumbered by a stupid patent. I -recommend ignoring it and continuing to implement and improve SPF until a -working and unencumbered proposal for RFC2822 headers surfaces. - -
-
-
-Release 0.6.6 adds support for SPF,
-a protocol to prevent forging of the envelope from address.
-SPF support requires pydns.
-The included spf.py module is an updated version of the original 1.6
-version at wayforward.net.
-The updated version tracks the draft RFC and test suite.
-
-The FAQ addresses how to get started with SPF. -
-Release 0.6.1 adds a full milter based dspam application. -
-I have selected the -dspam bayes filter project and -packaged it for python. -Release 0.6.0 offers a simple application of dspam I call "header triage", -which rejects messages with spammy headers. -To use header triage, you must have DSPAM installed, -and select a dictionary that is well moderated by someone who gets -lots of spam. That dictionary can be used to block spam that is -obvious from the headers (e.g. X-Mailer and Subject) before it ties -up any more resources. I have yet to see any false positives from this -approach (check the milter log), but if there are, the sender will -get a REJECT with the message "Your message looks spammy." - diff --git a/doc/credits.ht b/doc/credits.ht deleted file mode 100644 index 9973a4d..0000000 --- a/doc/credits.ht +++ /dev/null @@ -1,55 +0,0 @@ -Title: Credits - -
A. The milter source includes a number of deactivated viruses in -the test directory. All but the first and last lines of the base64 -encoded virus data has been removed. I suppose I should randomize -the first and last lines as well, since pymilter just deletes executables, -and doesn't look for signatures. -
A. Even though libmilter is officially supported in sendmail-8.12, -you need to build and install it in separate steps. Take a look -at the RPM spec file for sendmail-8.12. -The %prep section shows you how to create -a site.config.m4 that enables MILTER. The %build section shows you how -to build libmilter in a separate invocation of make. The %install section -shows you how to install libmilter with a separate invocation of make. -
- -
A. RedHat forgot to include the header in the RPM. See the -RedHat 7.2 requirements. -
-
-ImportError: /usr/lib/python2.4/site-packages/milter.so: undefined symbol: smfi_setmlreply --
A. Your libmilter.a is from sendmail-8.12 or earlier. You need - sendmail-8.13 or later to support setmlreply. You can disable - setmlreply by changing setup.py. Change: -
- define_macros = [ ('MAX_ML_REPLY',32) ]
-
-in setup.py to
-
- define_macros = [ ('MAX_ML_REPLY',1) ]
-
-
--To use this with sendmail, add the following to sendmail.cf: - -O InputMailFilters=pythonfilter -Xpythonfilter, S=local:inet:1030@localhost - -See the sendmail README for libmilter. -sample milter startup --
A. You need to tell sendmail to connect to your milter. The -sample milter tells you what to add to your sendmail.cf to tell -sendmail to use the milter. You can also add an INPUT_MAIL_FILTER -macro to your sendmail.mc file and rebuild sendmail.cf - see the sendmail -README for milters. -
- -
A. Sendmail only milters SMTP mail. Local mail is not miltered. -You can pipe a raw message through sendmail to test your milter: -
-$ cat rawtextmsg | sendmail myname@my.full.domain --Now check your milter log. -
- -
-File "mime.py", line 370, in ? - from sgmllib import declstringlit, declname - ImportError: cannot import name declstringlit --
A. declstringlit is not provided by sgmllib in all versions
-of python. For instance, python-2.2 does not have it. Upgrade to
-milter-0.4.5 or later to remove this dependency.
-
- -
milter.error: cannot add recipient?
---
A. You must tell libmilter how you might mutate the message with
-set_flags() before calling runmilter(). For
-instance, Milter.set_flags(Milter.ADDRCPT). You must add together
-all of ADDHDRS, CHGBODY, ADDRCPT, DELRCPT, CHGHDRS that apply.
-
NOTE - recent versions default flags to enabling all features. You
-must now call set_flags() if you wish to disable features for
-efficiency.
-
- -
A. Libmilter expects "rcpt to" shortly after getting "mail from".
-"Shortly" is defined by the timeout parameter you passed to
-Milter.runmilter()
- or milter.settimeout(). If the timeout is 10 seconds,
-and looking up the first recipient in DNS takes more than
-10 seconds, libmilter will give up and break the connection.
-Milter.runmilter() defaulted to 10 seconds in 0.3.4. In 0.3.5
-it will keep the libmilter default of 2 hours.
-
- -
A. sample.py is a sample. It is supposed to be easily modified -for your specific needs. We will of course continue to move generic -code out of the sample as the project evolves. Think of sample.py as -an active config file. -
-If you are running bms.py, then the block_chinese option in
-/etc/mail/pymilter.cfg controls this feature.
-
- -
A. Sendmail has a problem with unix sockets on old versions of OpenBSD.
-OpenBSD users report that this problem has been fixed, so upgrading
-OpenBSD will fix this. Otherwise, you can
-use an internet domain socket instead. For example, in
-sendmail.cf use
-
-Xpythonfilter, S=inet:1234@localhost --and change sample.py accordingly. -
- -
A. Configure sendmail to use virtusertable, and send all unknown -addresses to /dev/null. For example, -
-@mycorp.com dev-null -dan@mycorp.com dan -sally@mycorp.com sally --
-dev-null: /dev/null --Now your milter will get to the eom callback, and can change the -envelope recipient at will. Thanks to Dredd at -milter.org for this solution. -
- -
A. Check the sendmail log for errors. If sendmail is getting -milter timeouts, then your milter is taking too long and sendmail gave -up waiting. You can adjust the timeouts in your sendmail config. Here -is a milter declaration for sendmail.cf with all timeouts specified: -
-Xpythonfilter, S=local:/var/log/milter/pythonsock, F=T, T=C:5m;S:20s;R:60s;E:5m --
A. By default, when the milter fails with an untrapped exception, a
-TEMPFAIL result (451) is returned to the sender. The sender will then retry
-every hour or so for several days. Hopefully, someone will notice the
-traceback, and workaround or fix the problem. Beginning with milter-0.8.2,
-you can call milter.set_exception_policy(milter.CONTINUE)
-to cause an untrapped exception to continue processing with the
-next callback or milter instead. For
-completeness, you can also set the exception policy to
-milter.REJECT.
-
-
A. The internal_domains configuration specifies which
-MAIL FROM domains are used by internal connections. If an internal
-PC tries to use some other domain, it is assumed to be a "Zombie".
-
-Here is a sample log line: -
-2005Jun22 12:01:04 [12430] REJECT: zombie PC at 192.168.100.171 sending MAIL FROM debby@fedex.com --No, fedex.com does not use pymilter, and there is no one named debby at my -client. But the idiot using the PC at 192.168.100.171 has downloaded and -installed some stupid weatherbar/hotbar/aquariumscreensaver that is actually a -spam bot. -
-The internal_domains option is simplistic, it assumes all
-valid senders of the domains are internal. SPF provides a much more general
-check of IP and MAIL FROM for external email. Pymilter should soon
-have a local policy feature for more general checking of internal mail.
-
mail_archive isn't working. Or I don't understand how
- it's suppose to work. I have
- mail_archive = /var/mail/mail_archive
- in pymilter.cfg but nothing ever gets dumped into
- /var/mail/mail_archive.
-A. The 'mail' user needs to have write access. Permission failures - should be logged as a traceback in milter.log if it doesn't. - -
A. The bms.py milter supports spf. The RedHat RPMs will set almost -everything up for you. For other systems: -
tempdir. The RedHat RPM uses logrotate for
- logfiles and a simple cron script using find to clean
- tempdir.
-spfmilter.py. Run that as a service,
- and it does just SPF. It uses the sendmail access
- file to configure SPF responses just like bms.py, but
- supports only REJECT and OK.
- A. The SPF response is controlled by /etc/mail/access
- (actually the file you specify with access_file in
- the [spf] section of pymilter.cfg).
- Responses are OK, CBV, and REJECT. CBV sends the DSN.
-
-You can change the defaults. For instance, I have: -
-SPF-None: REJECT -SPF-Neutral: CBV -SPF-Softfail: CBV -SPF-Permerror: CBV --I have best_guess = 1, so SPF none is converted to PASS/NEUTRAL for policy -lookup, and 3 strikes (no PTR, no HELO, no SPF) becomes "SPF NONE" for local -policy purposes (the Received-SPF header always shows the official SPF -result.) -
-You can change the default for specific domains: -
-# these guys aren't going to pay attention to CBVs anyway... -SPF-None:cia.gov REJECT -SPF-None:fbi.gov REJECT -SPF-Neutral:aol.com REJECT -SPF-Softfail:ebay.com REJECT -- -
/etc/init.d/pysrs, I get this in
- /var/log/milter/pysrs.log:
--ConfigParser.NoOptionError: No option 'fwdomain' in section: 'srs' --
A. You need to specify the forward domain - i.e. the domain you want - SRS to rewrite stuff too. -
-For instance, I have: -
-# sample SRS configuration -[srs] -secret = don't you wish -maxage = 8 -hashlength = 5 -;database=/var/log/milter/srs.db -fwdomain = bmsi.com -sign=bmsi.com,mail.bmsi.com,gathman.org -srs=bmsaix.bmsi.com,bmsred.bmsi.com,stl.gathman.org,bampa.gathman.org --The
sign is for local domains which are signed.
-The srs list is for other domains which you are relaying,
-and which need to have SRS checked/undone for bounces.
-
-
-
-
-
-
-
-
-Mascot by Christian Hafner
- |
-
-Sendmail Milters in Python-by Jim Niemira
- and
- Stuart D. Gathman
-
-See the FAQ | Download now |
-Subscribe to mailing list |
-Overview |
-pydspam |
-libdspam
- |
-At the lowest level, the 'milter' module provides a thin wrapper around the - -sendmail libmilter API. This API lets you register callbacks for -a number of events in the -process of sendmail receiving a message via SMTP. -These events include the initial connection from a MTA, -the envelope sender and recipients, the top level mail headers, and -the message body. There are options to mangle all of these components -of the message as it passes through the milter. -
-At the next level, the 'Milter' module (note the case difference) provides a -Python friendly object oriented wrapper for the low level API. To use the -Milter module, an application registers a 'factory' to create an object -for each connection from a MTA to sendmail. These connection objects -must provide methods corresponding to the libmilter callback events. -
-Each event method returns a code to tell sendmail whether to proceed -with processing the message. This is a big advantage of milters over -other mail filtering systems. Unwanted mail can be stopped in its -tracks at the earliest possible point. -
-The Milter.Milter class provides default implementations for event -methods that -do nothing, and also provides wrappers for the libmilter methods to mutate -the message. -
-The 'spf' module provides an implementation of -SPF useful for detecting email forgery. -
-The 'mime' module provides a wrapper for the Python email package that -fixes some bugs, and simplifies modifying selected parts of a MIME message. -
-Finally, the bms.py application is both a sample of how to use the -Milter and spf modules, and the beginnings of a general purpose SPAM filtering, -wiretapping, SPF checking, and Win32 virus protecting milter. It can -make use of the pysrs package when available for -SRS/SES checking and the pydspam package for Bayesian -content filtering. SPF checking -requires -pydns. Configuration documentation is currently included as comments -in the sample config file for the bms.py milter. -See also the HOWTO and -Milter Log Message Tags. -
-Python milter is under GPL. The authors can probably be convinced to -change this to LGPL if needed. - -
-The Python milter package includes a sample milter that replaces dangerous
-attachments with a warning message, discards mail addressed to
-MAILER-DAEMON, and demonstrates several SPAM abatement strategies.
-The MimeMessage class to do this used to be based on the
-mimetools and multifile standard python packages.
-As of milter version 0.6.0, it is based on the email standard
-python packages, which were derived from the
-mimelib project.
-The MimeMessage class patches several bugs in the email package,
-and provides some backward compatibility.
-
-
-The "defang" function of the sample milter was inspired by -MIMEDefang, -a Perl milter with flexible attachment processing options. The latest -version of MIMEDefang uses an apache style process pool to avoid reloading -the Perl interpreter for each message. This makes it fast enough for -production without using Perl threading. -
-mailchecker is -a Python project to provide flexible attachment processing for mail. I -will be looking at plugging mailchecker into a milter. -
-TMDA is a Python project -to require confirmation the first time someone tries to send to your -mailbox. This would be a nice feature to have in a milter. -
-There is also a Milter community website -where milter software and gory details of the API are discussed. - -
-For example, the HTML parsing feature to remove scripts from HTML attachments -is rather CPU intensive in pure python. Using the C replacement for sgmllib -greatly speeds things up. - -
-To Bcc a message, call self.add_recipient(rcpt) in envfrom after
-determining whether you want to copy (e.g. whether the sender is local). For
-example,
-
- def envfrom(...
- ...
- if len(t) == 2:
- self.rejectvirus = t[1] in reject_virus_from
- if t[0] in wiretap_users.get(t[1],()):
- self.add_recipient(wiretap_dest)
- if t[1] == 'mydomain.com':
- self.add_recipient('<copy-%s>' % t[0])
- ...
-
--To make this a generic feature requires thinking about how the configuration -would look. Feel free to make specific suggestions about config file -entries. Be sure to handle both Bcc and file copies, and designating what -mail should be copied. How should "outgoing" be defined? Implementing it is -easy once the configuration is designed. - - -
![[ Valid HTML 3.2! ]](http://bmsi.com/vh32.png){kind=small}
![[ Powered By Red Hat Linux ]](http://bmsi.com/art/powered_by.gif){kind=small}
bms.py milter
-application. The milter and Milter modules do not implement any policies
-by themselves.
-
-internal_connect configuration.
-IP addresses with no PTR, and PTR names that look like
-the kind assigned to dynamic IPs (as determined by a heuristic
-algorithm) are flagged as DYNAMIC. IPs that match the
-trusted_relay configuration are flagged as TRUSTED.
--Examples from the log file (not the SMTP error message returned): -
-2005Jul29 13:56:53 [71207] connect from p50863492.dip0.t-ipconnect.de at ('80.134.52.146', 1858) EXTERNAL DYN
-2005Jul29 18:10:15 [74511] connect from foopub at ('1.2.3.4', 46513) EXTERNAL TRUSTED
-2005Jul29 14:41:00 [71805] connect from foobar at ('192.168.0.1', 41205) INTERNAL
-2005Jul29 14:41:15 [71806] connect from cncln.online.ln.cn at ('218.25.240.137', 35992) EXTERNAL
-
--Certain obviously evil PTR names are blocked at this point: -"localhost" (when IP is not 127.*) and ".". -
-2005Jul29 14:49:50 [71918] connect from localhost at ('221.132.0.6', 50507) EXTERNAL
-2005Jul29 14:49:50 [71918] REJECT: PTR is localhost
-
-
-hello_blacklist configuration
-are immediately rejected. The hello_blacklist typically contains
-the current MTAs own HELO name or email domains.
-Clients that attempt to skip HELO are immediately rejected.
--2005Jul29 18:10:15 [74512] hello from example.com -2005Jul29 18:10:15 [74512] REJECT: spam from self: example.com -2005Jul29 18:17:09 [74581] hello from 80.191.244.69 -2005Jul29 18:17:09 [74581] REJECT: numeric hello name: 80.191.244.69 -- -
-The MAIL FROM address is saved for possible use by the smart-alias
-feature. First, the internal_domains is used for
-a simple screening if defined. If the MAIL FROM for an INTERNAL connection
-is NOT in internal_domains, then it is rejected (the
-PC is most likely infected and attempting to send out spam).
-If the MAIL FROM for an EXTERNAL connection IS in
-internal_domains, then the message is immediately rejected.
-This is quick and effective for most small company MTAs. For more
-complex mail networks, it is too simplistic, and should not be defined.
-SPF will handle the complex cases.
-
-
-If the official result is 'none', we try to turn it into an effective result of
-'pass' or 'fail'. First, we check for a local substitute SPF record
-under the domain defined in the [spf]delegate configuration.
-It is often useful to add local SPF records for correspondents that are
-too clueless to add their own. If there is no local substitute, we use a "best
-guess" SPF record of "v=spf1 a/24 mx/24 ptr" for MAIL FROM or "v=spf1 a/24
-mx/24" for HELO. In addition, a HELO that is a subdomain of MAIL FROM and
-resolves to the connect IP results in an effective result of 'pass'.
-
-If there is no local SPF record, and the effective result is still not
-'pass', we check for either a valid HELO name or a valid PTR record for
-the connect IP. A valid HELO or PTR cannot look like a dynamic name
-as determined by the heuristic in Milter.dynip.
-
-If HELO has an SPF record, and the result is anything but pass, we reject -the connection: -
-2005Jul30 19:45:16 [93991] connect from [221.200.41.54] at ('221.200.41.54', 3581) EXTERNAL DYN
-2005Jul30 19:45:18 [93991] hello from adelphia.net
-2005Jul30 19:45:19 [93991] mail from ()
-2005Jul30 19:45:19 [93991] REJECT: hello SPF: fail 550 access denied
-
-Note that HELO does not have any forwarding issues like MAIL FROM, and so
-any result other than 'pass' or 'none' should be treated like 'fail'.
--Only if nothing about the SMTP envelope can be validated does the effective -result remain 'none. I call this the "3 strikes" rule. -
-If the official result is 'permerror' (a syntax error in the sender's -policy), we use the 'lax' option in pyspf to try various heuristics to guess -what they really meant. For instance, the invalid mechanism "ip:1.2.3.4" is -treated as "ip4:1.2.3.4". The result of lax processing is then used -as the effective result for policy purposes. -
-With an effective SPF result in hand, we consult the sendmail access -database to find our receiver policy for the sender. - -
| REJECT | -Reject the sender with a 550 5.7.1 SMTP code. The SMTP rejection -includes a detailed description of the problem. - |
|---|---|
| CBV | -Do a Call Back Validation by connecting to an MX of the sender -and checking that using the sender as the RCPT TO is not rejected. -We quit the CBV connection before actualling sending a message. -If the CBV is rejected, our SMTP connection is rejected with the -same error code and message. CBV results are cached. - |
| DSN | -Do a Call Back Validation by connecting to an MX of the sender -and checking that using the sender as the RCPT TO is not rejected. -Unlike a CBV, we continue on to data and send a detailed message -explaining the problem. This can be useful for reporting PermError -or SoftFail to the sender. Keep in mind that for any result other -than 'pass', the sender could be forged, and your DSN could annoy the -wrong person. However, a SoftFail result is requesting such feedback -for debugging and a PermError result needs to be fixed by the sender ASAP -whether forged or not. DSN results are cached so that senders are -annoyed only weekly. - |
| OK | -Accept the sender. The message may still be rejected via reputation -or content filtering. - |
-SPF-Fail:abeb@adelphia.net DSN --This says to accept mail from that adelphia.net user despite the -SPF fail, but only after annoying them with a DSN about their ISP's broken -policy. -
-If there is no match on the full sender, the domain is checked: -
-SPF-Neutral:aol.com REJECT --This says to reject mail from AOL with an SPF result of neutral. -This means AOL users can't use their AOL address with another mail service -to send us mail. This is good because the other mail service is -likely a badly configured greeting card site or a virus. -
-Finally, a default policy for the result is checked. While there are program -defaults, you should have defaults in the access database for SPF results: -
-SPF-Neutral: CBV -SPF-Softfail: DSN -SPF-PermError: DSN -SPF-TempError: REJECT -SPF-None: REJECT -SPF-Fail: REJECT -SPF-Pass: OK -- -
${datadir}/auto_whitelist.log or
-${datadir}/blacklist.log respectively. In addition,
-recipients of internal senders (except for automatic replies like vacation
-messages and return receipts) are automatically whitelisted for 60 days, and
-senders that fail CBV or DSN checks are automatically blacklisted for 30 days.
-Whitelisted and blacklisted senders are used to automatically train the
-bayesian content filter before being delivered or rejected, respectively.
--Real Soon Now users will be able to maintain their own whitelist and -blacklist that applies only when they are the recipient. - -
-Normally, you don't want email messages to silently disappear into -a black hole, so you should set the reject_spam option. However, -if you don't want your correspondent's email to get rejected, you can -check your quarantine frequently instead. - -
-Sendmail-8.12 renames -libsmutil.a to libsm.a. Unfortunately, libsm.a is an important AIX system -shared library. Therefore, I rename libsm.a back to libsmutil.a for -AIX. This presents a problem for setup.py. - -
/usr/include/libmilter/mfapi.h.
-
-If you do modify the SRPM, I suggest renaming libsmutil.a
-to libsm.a - just like sendmail-8.12 will. If you manually install
-mfapi.h or don't rename libsmutil.a, you'll
-need to force libs = ["milter", "smutil"] in setup.py.
-
-If you have installed python2, and want
-python-milter to use python2, add python=python2 to setup.cfg
-and build with python2 setup.py bdist_rpm.
-
-
-If y'all trust me, you can pick up source and binary sendmail RPMs for RH6.2 -from my linux downloads directory. -The lastest RPMs were built by taking a RH7.2 SRPMS and removing some -RPM features from the spec file that RH6.2 doesn't support, then -recompiling on RH6.2. You can check this by installing the RH7.2 SRPM, -then diffing my sendmail.spec with theirs. Then run -"rpm -bb sendmail-rhmilter.spec" when you are satisfied. -
-If you have installed python2, and want
-python-milter to use python2, add python=python2 to setup.cfg
-and build with python2 setup.py bdist_rpm.
-You'll need to install the sendmail-devel package to compile milter.