Expand test suite to cover RSA as well as ed25519

This test makes use of DNSOverride and the new verifying milter to
ensure that signatures can be verified properly.

It doesn't test the actual interaction with the public DNS, but
getting that kind of test to work on arbitrary platforms might be more
trouble than it's worth.

I note that the DNSOverride only works as long as testkey.dns is a
single line, which is fine for ed25519, but maybe not for RSA.

.gitignore

@@ -1,3 +1,4 @@
 dist
 dkimpy_milter.egg-info
 *.pyc
+*~

CHANGES

@@ -1,3 +1,51 @@
+1.0.1 2019-02-11
+ * Reorder milter start and dropping privileges so permissions on Unix socket
+   are correct (LP: 1797720)
+ - Make domain checks case insensitive for determining if signing should be
+   done (LP: #1815311)
+ - Add additional Sendmail configuration information to README from OpenDKIM
+   update based on input from Дилян Палаузов (LP: #1801619)
+ - Add information on Ed25519 key creation to README (LP: #1815313)
+
+1.0.0 2018-05-11
+ - Minor documentation updates
+ - Deleted reference to obsolete syslog target in unit file
+
+0.9.7 2018-03-19
+ - Made sysv init executable
+ - Add missing documentation key to system/dkimpy-milter.service
+ - Put version directly in setup.py and do not import dkimpy_milter to ease
+   install via pip
+ - Minor sysv init improvments
+
+0.9.6 2018-03-13
+ - Fixed typo in package installation section of README
+ - Added more to README about first run with systemd
+ - Fixed typo in path for fallback location of the config file if one is not
+   provided
+ - Added protection for malformed From addresses.  If the From does not at
+   least have an '@' in the address, then the signing domain is not extracted
+   and the message will not be signed
+
+0.9.5.1 2018-03-10
+ - Add conf file location to systemd unit file
+ - Fix setup.py install locations so they are installed correctly
+
+0.9.5 2018-03-10
+ - Beta 1 (updated Alpha -> Beta warning in README and trove classifiers)
+ - Added support for MacroList option
+ - Added support for MacroListVerify option
+ - Added example in README to show use of MacroList* to separate inbound and
+   outbound mail streams
+ - Added support for SyslogSuccess option (both signing and verifying)
+ - Rationalized logging to be much less verbose unless SyslogSuccess or
+   debugLevel are set - default is generally start/stop/errors only
+ - Fixed install_requires so either dnspython (preferred if neither is
+   installed) or PyDNS satisfies the install requirements
+ - Updated Authentication Results result comment not to mention key size for
+   ed25519 signatures, since it's irrelevant
+ - Enhanced signature verification logging to provide more useful information
+
 0.9.4 2018-03-09
  - Create PID directory if it is missing
  - Fix crash when verifying if domain for signing was not set

README

@@ -1,11 +1,17 @@
-This is a DKIM signing and verification milter.  In theory it works with both
+OVERVIEW
-Postfix and Sendmail, but the author has zero experience with Sendmail, so
+========
-reports of success/failure with Sendmail and patches are welcom.
+
+This is a DKIM signing and verification milter.  It has been tested with both
+Postfix and Sendmail.
 
 The configuration file is designed to be compatible with OpenDKIM, but only
 a subset of OpenDKIM options are supported.  If an unsupported option is
 specified, an error will be raised.
 
+
+INSTALLATION
+===========
+
 This package includes a default configuration file and man pages.  For those
 to be installed when installing using setup.py, the following incantation is
 required because setuptools developers decided not being able to do this by
@@ -13,11 +19,11 @@ default is a feature:
 
 python setup.py install --single-version-externally-managed --record=/dev/null
 
-For users of Debian Stable (Debian 9, Codename Squueze), all dependencies are
+For users of Debian Stable (Debian 9, Codename Squeeze), all dependencies are
 available in either the main or backports repositories:
 
-[sudo] apt install python-milter python-nacl pthon-ipaddress python-dnspython
+[sudo] apt install python-milter python-nacl python-ipaddress python-dnspython
-[sudo] apt install -t squeeze-backports python-authres python-dkim
+[sudo] apt install -t stretch-backports python-authres python-dkim
 
 The preferred method of installation is from PyPi using pip (if distribution
 packages are not available):
@@ -25,12 +31,57 @@ packages are not available):
 [sudo] pip install dkimpy_milter
 
 Using pip will cause required packages to be installed via easy_install if they
-have not been previously installed.
+have not been previously installed.  Because pymilter and PyNaCl are compiled
+Python extensions, the system will need appropriate development packages and
+an C compiler.  Alternately, install these dependencies from dsitribution/OS
+packages and then pip install dkimpy_milter.
 
 The milter will work with either pydns (DNS) or dnspython (dns), preferring
 dnspython is both are available.  The dkimpy DKIM module also works with
 either.
 
+
+SETUP
+====
+
+SIGNING KEYS
+============
+
+In order to create DKIM signatures, a private key must be available.  Signing
+keys should be protected (owned by root:root with permissions 600 in a
+directory that is not world readable).  Different keys are required for RSA
+and (if used) Ed25519.
+
+RSA
+===
+
+Both public and private keys for RSA have standard formats and there are many
+tools available to create them.  Keys must (RFC 8302) have a minimum size of
+1024 bits and should have a size of at least 2048 bits.  The dknewkey script
+that is provided with dkimpy is one such tool:
+
+dknewkey exampleprivkey
+
+will produce both the private key file (.key suffix) and a file with the DKIM
+public key record to be published DNS (.dns suffix).  RSA is the default key
+type.  2048 bits is the default key size.
+
+ED25519
+=======
+
+There is no standardized non-binary representation for Ed25519 private keys,
+so in order to generate Ed25519 keys for dkimpy-milter, dkimpy specific tools
+must be used to be compatible.  The same dknewkey script support Ed25519:
+
+dknewkey --ktype ed25519 anothernewkey
+
+will provide both the private key file (.key suffix) and a file with the DKIM
+public key record to be published DNS (.dns suffix).  Ed25519 keys do not have
+variable bit lengths.
+
+MTA INTEGRATION
+==============
+
 Both a systemd unit file and a sysv init file are provided.  Both make
 assumptions about defaults being used, e.g. if a non-standard pidfile name is
 used, they will need to be updated.  The sysv init file is Debian specific and
@@ -48,10 +99,19 @@ As an example, using the default dkimpy-user on Debian, the command would be:
 Since /var/run or /run is sometimes on a tempfs, if the PID file directory is
 missing, the milter will create it on startup.
 
+To start dkimpy-milter with systemd for the first time, you will need to take
+the following steps:
+
+[sudo] systemctl daemon-reload
+[sudo] systemctl enable dkimpy-milter
+[sudo] systemctl start dkimpy-milter
+[sudo] systemctl status dkimpy-milter (to verify it started correctly)
+
 As with all milters, dkimpy-milter needs to be integrated with your MTA of
 choice (Sendmail or Postfix).
 
-For Sendmail:
+SENDMAIL
+========
 
 Configuration is very similar to opendkim, but needs some adjustment for
 dkimpy-milter. Here's an example configuration line to include in your
@@ -67,12 +127,64 @@ Milter support should be present by default in most versions of sendmail
 these days, but if not included in your Sendmail build, see:
 http://www.elandsys.com/resources/sendmail/milter.html
 
-For Postfix:
+ISSUES USING SENDMAIL TO SIGN AND VERIFY
+========================================
+
+When using the sendmail MTA in both signing and verifying mode, there are
+a few issues of which to be aware that might cause operational problems
+and deserve consideration.
+
+(a) When the MTA will be used for relaying emails, e.g. delivering to other
+    hosts using the aliases mechanism, it is important not to break
+    signatures inserted by the original sender.  This is particularly sensitive
+    particular when the sending domain has published a "reject" DMARC policy.
+
+    By default, sendmail quotes to address header fields when there are no
+    quotes and the display part of the address contains a period or an
+    apostrophe.  However, opendkim only sees the raw, unmodified form of
+    the header field, and so the content that gets verified and what gets
+    signed will not be the same, guaranteeing the attached signature is not
+    valid.
+
+    To direct sendmail not to modify the headers, add this to your sendmail.mc:
+
+    	conf(`confMUST_QUOTE_CHARS', `')
+
+(b) As stated in sendmail's KNOWNBUGS file, sendmail truncates header field
+    values longer than 256 characters, which could mean truncating the domain
+    of a long From: header field value and invalidating the signature.
+    You may wish to consider increasing MAXNAME in sendmail/conf.h to mitigate
+    changing the messages and invalidating their signatures.  This change
+    requires recompiling sendmail.
+
+(c) Similar to (a) above, sendmail may wrap very long single-line recipient
+    fields for presentation purposes; for example:
+
+    To: very long name <a@example.org>,anotherloo...ong name b <b@example.org>
+
+    ...might be rewritten as:
+
+    To: very long name <a@example.org>,
+    	anotherloo...ong name b <b@example.org>
+
+    This rewrite is also done after opendkim has seen the message, meaning
+    the signature opendkim attaches to the message does not match the
+    content it signed.  There is not a known configuration change to
+    mitigate this mutation.
+
+    The only known mechanism for dealing with this is to have distinct
+    instances of opendkim do the verifying (inbound) and signing (outbound)
+    so that the version that arrives at the signing instance is already
+    in the rewritten form, guaranteeing the input and output are the same
+    and thus the signature matches the payload.
+
+POSTFIX
+=======
 
 Integration of dkimpy-milter into Postfix is like any milter (See Postfix's
-README_FILES/MILTER_README).  Here's an example master.cf excerpt the talks to
+README_FILES/MILTER_README).  Here's an example master.cf excerpt that talks
-two dkimpy-milter instances, one configured for signing and one configured for
+to two dkimpy-milter instances, one configured for signing and one configured
-verification:
+for verification:
 
 smtp       inet  n       -       -       -      -       smtpd
     ...
@@ -86,10 +198,46 @@ submission inet  n       -       -       -      -       smtpd
 
 These need to match the Socket value for each dkimpy-milter instance.
 
-The python DKIM library, dkimpy, requires the entire message being signed or
+Care is required to segregate outbound mail to be signed and inbound mail to
-verified to be in memory, so dkimpy-milter does not write messages out to a temp
+be verified.  The above example uses two instances of dkimpy-milter to do
-file.  This may impact performance on low-memory systems.
+this.  There are many possible ways.  Here is another example using milter
+macros to keep the mail streams segregated:
 
-WARNING: This is an alpha grade release to support interoperability testing with
+Postfix master.cf:
-Ed25519 signatures and basic functionality.  It is known to be incomplete and
+
-not suitable for general use.
+smtp       inet  n       -       -       -       -       smtpd
+    ...
+    -o smtpd_milters=inet:localhost:8891
+    -o milter_macro_daemon_name=VERIFYING
+    ...
+
+submission inet n       -       -       -       -       smtpd
+    -o syslog_name=postfix/submission
+    -o smtpd_tls_security_level=encrypt
+    -o smtpd_sasl_auth_enable=yes
+    ...
+    -o milter_macro_daemon_name=ORIGINATING
+    -o smtpd_milters=inet:localhost:8891
+    ...
+
+Dkimpy-milter.conf:
+
+...
+Mode			sv
+MacroList		dameon_name|ORIGINATING
+MacroListVerify		daemon_name|VERIFYING
+...
+
+
+NOTES
+=====
+
+The python DKIM library, dkimpy, requires the entire message being signed or
+verified to be in memory, so dkimpy-milter does not write messages out to a
+temp file.  This may impact performance on low-memory systems.
+
+DKIM with Ed25519 signatures are described in RFC 8463.  Version 1.0.0 and
+later support Ed25519 signing and verification.  RFC 8301 removed rsa-sha1
+from DKIM.  dkimpy-milter does not sign with rsa-sha1, but still considers
+rsa-sha1 signatures as valid for verification because they are still in
+common use and are not known to be cryptographically broken.

TODO

@@ -34,16 +34,21 @@ DiagnosticDirectory     implemented     verified
 InternalHosts   implemented     verified
 
 0.9.5 (Beta)
-
+MacroList	implemented	verified
-
+MacroListVerify	implemented	verified
-SyslogSuccess
+SyslogSuccess   implemented     verified
 
 1.0.0
-Convert dkim-milter-python config
+No additional features
-No additional features planned
 
-Plannedataset type support:
+1.0.1
-db:/.db
+Bug fix only, improved documentation
+
+1.1.0 (planned)
+Port to Python 3
+Subdomain support
+
+Planned dataset type support (if needed):
 mdb:
 
 Considered for near-term feature release
@@ -51,10 +56,10 @@ Considered for near-term feature release
 AlwaysAddARHeader
 ChangeRootDirectory
 ClockDrift (requires dkimpy change)
-DNSTimeout (requires dkmpy change)
+DNSTimeout (requires dkimpy change)
-MacroList
 MilterDebug
 MinimumKeyBits
+OversignHeaders (may require dkimpy changes)
 PeerList
 SignatureAlgorithm
 
@@ -85,7 +90,6 @@ On-InternalError
 On-KeyNotFound
 On-NoSignature
 On-SignatureError
-OversignHeaders
 RemoveARAll
 RemoveARFrom
 RemoveOldSignatures 

dkimpy_milter/__init__.py

@@ -25,15 +25,12 @@ import sys
 import syslog
 import Milter
 import dkim
-from dkim.dnsplug import get_txt
-from dkim.util import parse_tag_value
 import authres
 import os
 import tempfile
 import StringIO
 import re
-from Milter.config import MilterConfigParser
+from Milter.utils import parse_addr, parseaddr
-from Milter.utils import iniplist,parse_addr,parseaddr
 import dkimpy_milter.config as config
 from dkimpy_milter.util import drop_privileges
 from dkimpy_milter.util import setExceptHook
@@ -42,230 +39,293 @@ from dkimpy_milter.util import read_keyfile
 from dkimpy_milter.util import own_socketfile
 from dkimpy_milter.util import fold
 
-__version__ = "0.9.4"
+__version__ = "1.0.1"
 FWS = re.compile(r'\r?\n[ \t]+')
 
+
 class dkimMilter(Milter.Base):
-  "Milter to check and sign DKIM.  Each connection gets its own instance."
+    "Milter to check and sign DKIM.  Each connection gets its own instance."
 
-  def __init__(self):
+    def __init__(self):
-    self.mailfrom = None
+        self.mailfrom = None
-    self.id = Milter.uniqueID()
+        self.id = Milter.uniqueID()
-    # we don't want config used to change during a connection
+        # we don't want config used to change during a connection
-    self.conf = milterconfig
+        self.conf = milterconfig
-    self.privatersa = privateRSA
+        self.privatersa = privateRSA
-    self.privateed25519 = privateEd25519
+        self.privateed25519 = privateEd25519
-    self.fp = None
+        self.fp = None
 
-  @Milter.noreply
+    @Milter.noreply
-  def connect(self,hostname,unused,hostaddr):
+    def connect(self, hostname, unused, hostaddr):
-    self.internal_connection = False
+        self.internal_connection = False
-    self.hello_name = None
+        self.external_connection = False
-    # sometimes people put extra space in sendmail config, so we strip
+        self.hello_name = None
-    self.receiver = self.getsymval('j').strip()
+        # sometimes people put extra space in sendmail config, so we strip
-    try:
+        self.receiver = self.getsymval('j')
-        self.AuthservID = milterconfig['AuthservID']
+        if self.receiver is not None:
-    except:
+            self.receiver = self.receiver.strip()
-        self.AuthservID = self.receiver
+        try:
-    if hostaddr and len(hostaddr) > 0:
+            self.AuthservID = milterconfig['AuthservID']
-        ipaddr = hostaddr[0]
+        except:
-        if milterconfig['InternalHostsObj']:
+            self.AuthservID = self.receiver
-            if milterconfig['InternalHostsObj'].match(ipaddr):
+        if hostaddr and len(hostaddr) > 0:
-                self.internal_connection = True
+            ipaddr = hostaddr[0]
-    else: ipaddr = ''
+            if milterconfig['IntHosts']:
-    self.connectip = ipaddr
+                if milterconfig['IntHosts'].match(ipaddr):
-    if self.internal_connection:
+                    self.internal_connection = True
-        connecttype = 'INTERNAL'
+        else:
-    else:
+            ipaddr = ''
-        connecttype = 'EXTERNAL'
+        self.connectip = ipaddr
-    if milterconfig.get('Syslog'):
+        if milterconfig.get('MacroList') and not self.internal_connection:
-        syslog.syslog("connect from {0} at {1} {2}".format(hostname,hostaddr,connecttype))
+            macrolist = milterconfig.get('MacroList')
-    return Milter.CONTINUE
+            for macro in macrolist:
+                macroname = macro.split('|')[0]
+                macroname = '{' + macroname + '}'
+                macroresult = self.getsymval(macroname)
+                if ((len(macro.split('|')) == 1 and macroresult) or macroresult
+                        in macro.split('|')[1:]):
+                    self.internal_connection = True
+        if milterconfig.get('MacroListVerify'):
+            macrolist = milterconfig.get('MacroListVerify')
+            for macro in macrolist:
+                macroname = macro.split('|')[0]
+                macroname = '{' + macroname + '}'
+                macroresult = self.getsymval(macroname)
+                if ((len(macro.split('|')) == 1 and macroresult) or macroresult
+                        in macro.split('|')[1:]):
+                    self.external_connection = True
+        if self.internal_connection:
+            connecttype = 'INTERNAL'
+        else:
+            connecttype = 'EXTERNAL'
+        if milterconfig.get('Syslog') and milterconfig.get('debugLevel') >= 1:
+            syslog.syslog("connect from {0} at {1} {2}"
+                          .format(hostname, hostaddr, connecttype))
+        return Milter.CONTINUE
 
-  # multiple messages can be received on a single connection
+    # multiple messages can be received on a single connection
-  # envfrom (MAIL FROM in the SMTP protocol) seems to mark the start
+    # envfrom (MAIL FROM in the SMTP protocol) seems to mark the start
-  # of each message.
+    # of each message.
-  @Milter.noreply
+    @Milter.noreply
-  def envfrom(self,f,*str):
+    def envfrom(self, f, *str):
-    if milterconfig.get('Syslog'):  
+        if milterconfig.get('Syslog') and milterconfig.get('debugLevel') >= 2:
-        syslog.syslog("mail from: {0} {1}".format(f,str))
+            syslog.syslog("mail from: {0} {1}".format(f, str))
-    self.fp = StringIO.StringIO()
+        self.fp = StringIO.StringIO()
-    self.mailfrom = f
+        self.mailfrom = f
-    t = parse_addr(f)
+        t = parse_addr(f)
-    if len(t) == 2: t[1] = t[1].lower()
+        if len(t) == 2:
-    self.canon_from = '@'.join(t)
+            t[1] = t[1].lower()
-    self.user = self.getsymval('{auth_authen}')
+        self.canon_from = '@'.join(t)
-    self.has_dkim = 0
+        self.has_dkim = 0
-    self.author = None
+        self.author = None
-    self.arheaders = []
+        self.arheaders = []
-    self.arresults = []
+        self.arresults = []
-    '''if self.user:
+        return Milter.CONTINUE
-      # Very simple SMTP AUTH policy by default:
-      #   any successful authentication is considered INTERNAL
-      self.internal_connection = True
-      auth_type = self.getsymval('{auth_type}')
-      ssl_bits =  self.getsymval('{cipher_bits}')
-      if milterconfig.get('Syslog'):
-          syslog.syslog(
-              "SMTP AUTH:",self.user,"sslbits =",ssl_bits, auth_type,
-              "ssf =",self.getsymval('{auth_ssf}'), "INTERNAL"
-      )
-      # Detailed authorization policy is configured in the access file below.
-      self.arresults.append(
-        authres.SMTPAUTHAuthenticationResult(result = 'pass',
-      	result_comment = auth_type+' sslbits='+ssl_bits, smtp_auth = self.user)
-      )'''
-    return Milter.CONTINUE
 
-  @Milter.noreply
+    @Milter.noreply
-  def header(self,name,val):
+    def header(self, name, val):
-    lname = name.lower()
+        lname = name.lower()
-    if lname == 'dkim-signature':
+        if lname == 'dkim-signature':
-        if milterconfig.get('Syslog'):
+            if (milterconfig.get('Syslog') and
-          syslog.syslog("{0}: {1}".format(name,val))
+                    milterconfig.get('debugLevel') >= 1):
-        self.has_dkim += 1
+                syslog.syslog("{0}: {1}".format(name, val))
-    if lname == 'from':
+            self.has_dkim += 1
-        fname,self.author = parseaddr(val)
+        if lname == 'from':
-        self.fdomain = self.author.split('@')[1]
+            fname, self.author = parseaddr(val)
-        if milterconfig.get('Syslog'):
+            try:
-            syslog.syslog("{0}: {1}".format(name,val))
+                self.fdomain = self.author.split('@')[1].lower()
-    elif lname == 'authentication-results':
+            except IndexError as er:
-        self.arheaders.append(val)
+                self.fdomain = ''  # self.author was not a proper email address
-    if self.fp:
+            if (milterconfig.get('Syslog') and
-        self.fp.write("%s: %s\n" % (name,val))
+                    milterconfig.get('debugLevel') >= 1):
-    return Milter.CONTINUE
+                syslog.syslog("{0}: {1}".format(name, val))
+        elif lname == 'authentication-results':
+            self.arheaders.append(val)
+        if self.fp:
+            self.fp.write("%s: %s\n" % (name, val))
+        return Milter.CONTINUE
 
-  @Milter.noreply
+    @Milter.noreply
-  def eoh(self):
+    def eoh(self):
-    if self.fp:
+        if self.fp:
-      self.fp.write("\n")                         # terminate headers
+            self.fp.write("\n")   # terminate headers
-    self.bodysize = 0
+        self.bodysize = 0
-    return Milter.CONTINUE
+        return Milter.CONTINUE
 
-  @Milter.noreply
+    @Milter.noreply
-  def body(self,chunk):         # copy body to temp file
+    def body(self, chunk):        # copy body to temp file
-    if self.fp:
+        if self.fp:
-      self.fp.write(chunk)      # IOError causes TEMPFAIL in milter
+            self.fp.write(chunk)  # IOError causes TEMPFAIL in milter
-      self.bodysize += len(chunk)
+            self.bodysize += len(chunk)
-    return Milter.CONTINUE
+        return Milter.CONTINUE
 
-  def eom(self):
+    def eom(self):
-    if not self.fp:
+        if not self.fp:
-      return Milter.ACCEPT      # no message collected - so no eom processing
+            return Milter.ACCEPT  # no message collected - so no eom processing
-    # Remove existing Authentication-Results headers for our authserv_id
+        # Remove existing Authentication-Results headers for our authserv_id
-    for i,val in enumerate(self.arheaders,1):
+        for i, val in enumerate(self.arheaders, 1):
-      # FIXME: don't delete A-R headers from trusted MTAs
+            # FIXME: don't delete A-R headers from trusted MTAs
-      try:
+            try:
-          ar = authres.AuthenticationResultsHeader.parse_value(FWS.sub('',val))
+                ar = (authres.AuthenticationResultsHeader
-          if ar.authserv_id == self.AuthservID:
+                      .parse_value(FWS.sub('', val)))
-              self.chgheader('authentication-results',i,'')
+                if ar.authserv_id == self.AuthservID:
-              if milterconfig.get('Syslog'):
+                    self.chgheader('authentication-results', i, '')
-                  syslog.syslog('REMOVE: {0}'.format(val))
+                    if (milterconfig.get('Syslog') and
-      except:
+                            milterconfig.get('debugLevel') >= 1):
-          # Don't error out on unparseable AR header fiels
+                        syslog.syslog('REMOVE: {0}'.format(val))
-          pass
+            except:
-    # Check or sign DKIM
+                # Don't error out on unparseable AR header fiels
-    self.fp.seek(0)
+                pass
-    if milterconfig.get('Domain'):
+        # Check or sign DKIM
-        domain = milterconfig.get('Domain')
+        self.fp.seek(0)
-    else:
+        if milterconfig.get('Domain'):
-        domain = ''
+            domain = milterconfig.get('Domain')
-    if (self.fdomain in domain) and (not milterconfig.get('Mode') == 'v'):
+        else:
-      txt = self.fp.read()
+            domain = ''
-      self.sign_dkim(txt)
+        if ((self.fdomain in domain) and not milterconfig.get('Mode') == 'v'
-      result = None
+                and not self.external_connection):
-    if (self.has_dkim) and (not self.internal_connection) and (milterconfig.get('Mode') == 'v' or milterconfig.get('Mode') == 'sv'):
+            txt = self.fp.read()
-      txt = self.fp.read()
+            self.sign_dkim(txt)
-      self.check_dkim(txt)
+        if ((self.has_dkim) and (not self.internal_connection) and
-    else:
+            (milterconfig.get('Mode') == 'v' or
-      result = 'none'
+             milterconfig.get('Mode') == 'sv')):
-    if self.arresults:
+            txt = self.fp.read()
-        h = authres.AuthenticationResultsHeader(authserv_id = self.AuthservID, 
+            self.check_dkim(txt)
-            results=self.arresults)
+        if self.arresults:
-        h = fold(str(h))
+            h = authres.AuthenticationResultsHeader(authserv_id=
-        if milterconfig.get('Syslog'):
+                                                    self.AuthservID,
-            syslog.syslog(str(h))
+                                                    results=self.arresults)
-        name,val = str(h).split(': ',1)
+            h = fold(str(h))
-        self.addheader(name,val,0)
+            if (milterconfig.get('Syslog') and
-    return Milter.CONTINUE
+                    milterconfig.get('debugLevel') >= 2):
+                syslog.syslog(str(h))
+            name, val = str(h).split(': ', 1)
+            self.addheader(name, val, 0)
+        return Milter.CONTINUE
 
-  def sign_dkim(self,txt):
+    def sign_dkim(self, txt):
-      canon = milterconfig.get('Canonicalization')
+        canon = milterconfig.get('Canonicalization')
-      canonicalize = []
+        canonicalize = []
-      if len(canon.split('/')) == 2:
+        if len(canon.split('/')) == 2:
-          canonicalize.append(canon.split('/')[0])
+            canonicalize.append(canon.split('/')[0])
-          canonicalize.append(canon.split('/')[1])
+            canonicalize.append(canon.split('/')[1])
-      else:
+        else:
-          canonicalize.append(canon)
+            canonicalize.append(canon)
-          canonicalize.append(canon)
+            canonicalize.append(canon)
-      syslog.syslog('canonicalize: {0}'.format(canonicalize))
+            if (milterconfig.get('Syslog') and
-      try:
+                    milterconfig.get('debugLevel') >= 1):
-        if privateRSA:
+                syslog.syslog('canonicalize: {0}'.format(canonicalize))
-            d = dkim.DKIM(txt)
+        try:
-            h = d.sign(milterconfig.get('Selector'), self.fdomain, privateRSA,
+            if privateRSA:
-                    canonicalize=(canonicalize[0], canonicalize[1]))
+                d = dkim.DKIM(txt)
-            name,val = h.split(': ',1)
+                h = d.sign(milterconfig.get('Selector'), self.fdomain,
-            self.addheader(name,val.strip().replace('\r\n','\n'),0)
+                           privateRSA, canonicalize=(canonicalize[0],
-        if privateEd25519:
+                                                     canonicalize[1]))
-            d = dkim.DKIM(txt)
+                name, val = h.split(': ', 1)
-            h = d.sign(milterconfig.get('SelectorEd25519'), self.fdomain, privateEd25519,
+                self.addheader(name, val.strip().replace('\r\n', '\n'), 0)
-                    canonicalize=(canonicalize[0], canonicalize[1]), signature_algorithm='ed25519-sha256')
+                if (milterconfig.get('Syslog') and
-            name,val = h.split(': ',1)
+                    (milterconfig.get('SyslogSuccess')
-            self.addheader(name,val.strip().replace('\r\n','\n'),0)
+                     or milterconfig.get('debugLevel') >= 1)):
-      except dkim.DKIMException as x:
+                    syslog.syslog('{0}: {1} DKIM-Signature field added (s={2} '
-          if milterconfig.get('Syslog'):
+                                  'd={3})'.format(self.getsymval('i'),
-              syslog.syslog('DKIM: {0}'.format(x))
+                                                  d.signature_fields.get(b'a'),
-      except Exception as x:
+                                                  d.signature_fields.get(b's'),
-          if milterconfig.get('Syslog'):
+                                                  d.domain.lower()))
-              syslog.syslog("sign_dkim: {0}".format(x))
+            if privateEd25519:
-          raise
+                d = dkim.DKIM(txt)
-      
+                h = d.sign(milterconfig.get('SelectorEd25519'), self.fdomain,
-  def check_dkim(self,txt):
+                           privateEd25519, canonicalize=(canonicalize[0],
-      res = False
+                                                         canonicalize[1]),
-      conf = self.conf
+                           signature_algorithm='ed25519-sha256')
-      for y in range(self.has_dkim): # Verify _ALL_ the signatures
+                name, val = h.split(': ', 1)
-          d = dkim.DKIM(txt)
+                self.addheader(name, val.strip().replace('\r\n', '\n'), 0)
-          try:
+                if (milterconfig.get('Syslog') and
-            res = d.verify(idx=y)
+                    (milterconfig.get('SyslogSuccess')
-            if res:
+                     or milterconfig.get('debugLevel') >= 1)):
-              self.dkim_comment = 'Good {0} bit {1} signature.'.format(d.keysize, d.signature_fields.get(b'a'))
+                    syslog.syslog('{0}: {1} DKIM-Signature field added (s={2} '
-            else:
+                                  'd={3})'.format(self.getsymval('i'),
-              self.dkim_comment = 'Bad {0} bit {1} signature.'.format(d.keysize, d.signature_fields.get(b'a'))
+                                                  d.signature_fields.get(b'a'),
-          except dkim.DKIMException as x:
+                                                  d.signature_fields.get(b's'),
-            self.dkim_comment = str(x)
+                                                  d.domain.lower()))
+        except dkim.DKIMException as x:
             if milterconfig.get('Syslog'):
                 syslog.syslog('DKIM: {0}'.format(x))
-          except Exception as x:
+        except Exception as x:
-            self.dkim_comment = str(x)
             if milterconfig.get('Syslog'):
-                syslog.syslog("check_dkim: {0}".format(x))
+                syslog.syslog("sign_dkim: {0}".format(x))
-          self.header_i = d.signature_fields.get(b'i')
+            raise
-          self.header_d = d.signature_fields.get(b'd')
+
-          self.header_a = d.signature_fields.get(b'a')
+    def check_dkim(self, txt):
-          if res:
+        res = False
-              if milterconfig.get('Syslog'):
+        for y in range(self.has_dkim):  # Verify _ALL_ the signatures
-                  syslog.syslog('DKIM: Pass ({0})'.format(d.domain))
+            d = dkim.DKIM(txt)
-              self.dkim_domain = d.domain
+            try:
-          else:
+                dnsoverride = milterconfig.get('DNSOverride')
-              if milterconfig.get['DiagnosticDirectory']:
+                if isinstance(dnsoverride, str):
-                  fd,fname = tempfile.mkstemp(".dkim")
+                    syslog.syslog("DNSOverride: {0}".format(dnsoverride))
-                  with os.fdopen(fd,"w+b") as fp:
+                    res = d.verify(idx=y, dnsfunc=lambda _x: dnsoverride)
-                      fp.write(txt)
+                else:
-                  if milterconfig.get('Syslog'):
+                    res = d.verify(idx=y)
-                      syslog.syslog('DKIM: Fail (saved as {0})'.format(fname))
+                if res:
-              else:
+                    if d.signature_fields.get(b'a') == 'ed25519-sha256':
-                  syslog.syslog('DKIM: Fail ({0})'.format(d.domain))
+                        self.dkim_comment = ('Good {0} signature'
-          if res:
+                                             .format(d.signature_fields
-            result = 'pass'
+                                                     .get(b'a')))
-          else:
+                    else:
-            result = 'fail'
+                        self.dkim_comment = ('Good {0} bit {1} signature'
-          res = False
+                                             .format(d.keysize,
-          self.arresults.append(
+                                                     d.signature_fields
-            authres.DKIMAuthenticationResult(result=result,
+                                                     .get(b'a')))
-              header_i = self.header_i, header_d = self.header_d, header_a = self.header_a,
+                else:
-              result_comment = self.dkim_comment)
+                    self.dkim_comment = ('Bad {0} bit {1} signature.'
-          )
+                                         .format(d.keysize,
-      return
+                                                 d.signature_fields.get(b'a')))
+            except dkim.DKIMException as x:
+                self.dkim_comment = str(x)
+                if milterconfig.get('Syslog'):
+                    syslog.syslog('DKIM: {0}'.format(x))
+            except Exception as x:
+                self.dkim_comment = str(x)
+                if milterconfig.get('Syslog'):
+                    syslog.syslog("check_dkim: {0}".format(x))
+            self.header_i = d.signature_fields.get(b'i')
+            self.header_d = d.signature_fields.get(b'd')
+            self.header_a = d.signature_fields.get(b'a')
+            if res:
+                if (milterconfig.get('Syslog') and
+                        (milterconfig.get('SyslogSuccess') or
+                         milterconfig.get('debugLevel') >= 1)):
+                    syslog.syslog('{0}: {1} DKIM signature verified (s={2} '
+                                  'd={3})'.format(self.getsymval('i'),
+                                                  d.signature_fields.get(b'a'),
+                                                  d.signature_fields.get(b's'),
+                                                  d.domain.lower()))
+                self.dkim_domain = d.domain.lower()
+            else:
+                if milterconfig.get('DiagnosticDirectory'):
+                    fd, fname = tempfile.mkstemp(".dkim")
+                    with os.fdopen(fd, "w+b") as fp:
+                        fp.write(txt)
+                    if milterconfig.get('Syslog'):
+                        syslog.syslog('DKIM: Fail (saved as {0})'
+                                      .format(fname))
+                else:
+                    syslog.syslog('DKIM: Fail ({0})'.format(d.domain.lower()))
+            if res:
+                result = 'pass'
+            else:
+                result = 'fail'
+            res = False
+            self.arresults.append(
+                authres.DKIMAuthenticationResult(result=result,
+                                                 header_i=self.header_i,
+                                                 header_d=self.header_d,
+                                                 header_a=self.header_a,
+                                                 result_comment=
+                                                 self.dkim_comment)
+            )
+        return
+
 
 def main():
     # Ugh, but there's no easy way around this.
@@ -274,15 +334,16 @@ def main():
     global privateEd25519
     privateRSA = False
     privateEd25519 = False
-    configFile = '/etc/dkimpy-milter.conf'
+    configFile = '/usr/local/etc/dkimpy-milter.conf'
     if len(sys.argv) > 1:
-        if sys.argv[1] in ( '-?', '--help', '-h' ):
+        if sys.argv[1] in ('-?', '--help', '-h'):
             print('usage: dkimpy-milter [<configfilename>]')
             sys.exit(1)
         configFile = sys.argv[1]
-    milterconfig = config._processConfigFile(filename = configFile)
+    milterconfig = config._processConfigFile(filename=configFile)
     if milterconfig.get('Syslog'):
-        facility = eval("syslog.LOG_{0}".format(milterconfig.get('SyslogFacility').upper()))
+        facility = eval("syslog.LOG_{0}"
+                        .format(milterconfig.get('SyslogFacility').upper()))
         syslog.openlog(os.path.basename(sys.argv[0]), syslog.LOG_PID, facility)
         setExceptHook()
     pid = write_pid(milterconfig)
@@ -294,12 +355,13 @@ def main():
     Milter.set_flags(Milter.CHGHDRS + Milter.ADDHDRS)
     miltername = 'dkimpy-filter'
     socketname = milterconfig.get('Socket')
-    if milterconfig.get('Syslog'):
-        syslog.syslog('dkimpy-milter started:{0} user:{1}'.format(pid,milterconfig.get('UserID')))
-    sys.stdout.flush()
-    Milter.runmilter(miltername,socketname,240)
     own_socketfile(milterconfig)
     drop_privileges(milterconfig)
+    sys.stdout.flush()
+    Milter.runmilter(miltername, socketname, 240)
+    if milterconfig.get('Syslog'):
+        syslog.syslog('dkimpy-milter started:{0} user:{1}'
+                      .format(pid, milterconfig.get('UserID')))
 
 if __name__ == "__main__":
     main()

dkimpy_milter/__main__.py

@@ -0,0 +1,6 @@
+#!/usr/bin/python2
+
+from dkimpy_milter import main
+
+if __name__ == "__main__":
+    main()

dkimpy_milter/config.py

@@ -27,8 +27,6 @@
 import syslog
 import os
 import sys
-import re
-import urllib
 import stat
 import dkim
 import socket
@@ -37,26 +35,29 @@ from dnsplug import Session
 
 #  default values
 defaultConfigData = {
-        'Syslog' : 'yes',
+    'Syslog': 'yes',
-        'SyslogFacility' : 'mail',
+    'SyslogFacility': 'mail',
-        'UMask' : 007,
+    'UMask': 007,
-        'Mode' : 'sv',
+    'Mode': 'sv',
-        'Socket' : 'local:/var/run/dkimpy-milter/dkimpy-milter.sock',
+    'Socket': 'local:/var/run/dkimpy-milter/dkimpy-milter.sock',
-        'PidFile'  : '/var/run/dkimpy-milter/dkimpy-milter.pid',
+    'PidFile': '/var/run/dkimpy-milter/dkimpy-milter.pid',
-        'UserID' : 'dkimpy-milter',
+    'UserID': 'dkimpy-milter',
-        'Canonicalization' : 'relaxed/simple',
+    'Canonicalization': 'relaxed/simple',
-        'InternalHosts' : '127.0.0.1',
+    'InternalHosts': '127.0.0.1',
-        'InternalHostsObj' : False,
+    'IntHosts': False,
-        'DiagnosticDirectory' : ''
+    'DiagnosticDirectory': '',
-        }
+    'MacroList': '',
+    'MacroListVerify': '',
+    'DNSOverride': None,
+    'debugLevel': 0  # Undocumented config item for developer use
+    }
 
 
-#################################
 class ConfigException(Exception):
     '''Exception raised when there's a configuration file error.'''
     pass
 
-#################################
+
 class HostsDataset(object):
     '''Hold a group of host related dataset objects'''
 
@@ -85,34 +86,41 @@ class HostsDataset(object):
                 self.negative = True
             try:
                 self.item = ipaddress.ip_address(unicode(self.item, "utf-8"))
-                if isinstance(self.item, ipaddress.IPv4Address): self.isipv4 = True
+                if isinstance(self.item, ipaddress.IPv4Address):
-                elif isinstance(self.item, ipaddress.IPv6Address): self.isipv6 = True
+                    self.isipv4 = True
+                elif isinstance(self.item, ipaddress.IPv6Address):
+                    self.isipv6 = True
             except ValueError as e:
                 try:
-                    self.item = ipaddress.ip_network(unicode(self.item, "utf-8"), strict=False)
+                    self.item = ipaddress.ip_network(unicode
-                    if isinstance(self.item, ipaddress.IPv4Network): self.isipv4cidr = True
+                                                     (self.item, "utf-8"),
-                    elif isinstance(self.item, ipaddress.IPv6Network): self.isipv6cidr = True
+                                                     strict=False)
+                    if isinstance(self.item, ipaddress.IPv4Network):
+                        self.isipv4cidr = True
+                    elif isinstance(self.item, ipaddress.IPv6Network):
+                        self.isipv6cidr = True
                 except ValueError as e2:
                     if self.item[0] == '.' and len(self.item.split('.')) > 2:
                         self.isdomain = True
-                    elif len(self.item.split('.')) > 1: # It has a '.' in it
+                    elif len(self.item.split('.')) > 1:  # It has a '.' in it
                         self.ishostname = True
                     else:
-                        raise ConfigException('Unknown dataset item: {0}'.format(item))
+                        raise ConfigException('Unknown dataset item: {0}'
+                                              .format(item))
 
     def match(self, connectip):
         '''Check if the connect IP is part of the dataset'''
         source = ipaddress.ip_address(unicode(connectip, "utf-8"))
         for item in self.dataset:
             if item.isdomain or item.ishostname:
-                result = self.matchname(source) # Match host/domain names first
+                result = self.matchname(source)   # Match host/domains first
                 if result:
                     return(result)
-            elif item.isipv4 or item.isipv4cidr:
+            elif item.isipv4 or item.isipv4cidr:  # Then IPv4/6 addresses or
-                if isinstance(source, ipaddress.IPv4Address): # Then IPv4/6 addresses
+                if isinstance(source, ipaddress.IPv4Address):  # networks
-                    return(self.match4(source))               # or networks depending
+                    return(self.match4(source))   # depending on the item type
-            elif item.isipv6 or item.isipv6cidr:              # on the item type and
+            elif item.isipv6 or item.isipv6cidr:  # and connect type
-                if isinstance(source, ipaddress.IPv6Address): # connection type
+                if isinstance(source, ipaddress.IPv6Address):
                     return(self.match6(source))
 
     def matchname(self, source):
@@ -126,7 +134,7 @@ class HostsDataset(object):
         for item in self.dataset:
             if item.isdomain:
                 for ptr in ptrlist:
-                    # Strip the leading '.' off the domain name so exact match works.
+                    # Strip the leading '.' off the domain name for exact match
                     if item.item[1:] == ptr[-len(item.item)+1:]:
                         matchdomain = True
                         negativedomain = item.negative
@@ -211,21 +219,16 @@ class HostsDataset(object):
             match = False
         return(match)
 
-    def dump(self):
-        for item in self.dataset:
-            print 'name: {0} ip4: {1} cidr4: {2} ip6: {3} cidr6: {4} host: {5} domain: {6} negat: {7} type: {8}'.format(item.item,
-                    item.isipv4, item.isipv4cidr, item.isipv6, item.isipv6cidr, item.ishostname, item.isdomain,
-                    item.negative, type(item.item))
 
-####################################################################
+def _processConfigFile(filename=None, configdata=None, useSyslog=1,
-def _processConfigFile(filename = None, configdata = None, useSyslog = 1,
+                       useStderr=0):
-        useStderr = 0):
     '''Load the specified config file, exit and log errors if it fails,
     otherwise return a config dictionary.'''
 
     import config
-    if configdata == None: configdata = config.defaultConfigData
+    if configdata is None:
-    if filename != None:
+        configdata = config.defaultConfigData
+    if filename is not None:
         try:
             _readConfigFile(filename, configdata)
         except Exception, e:
@@ -237,7 +240,7 @@ def _processConfigFile(filename = None, configdata = None, useSyslog = 1,
             sys.exit(1)
     return(configdata)
 
-####################
+
 def _find_boolean(item):
     if type(item) == int:
         item = str(item)
@@ -248,14 +251,15 @@ def _find_boolean(item):
     else:
         raise dkim.ParameterError()
     return item
-####################
+
-def _calculate_authserv_id(as_id):
+
+def _make_authserv_id(as_id):
     """Determine AuthservID if needed"""
     if as_id == 'HOSTNAME':
         as_id = socket.gethostname()
     return as_id
 
-####################
+
 def _dataset_to_list(dataset):
     """Convert a dataset (as defined in dkimpymilter.8) and return a python
        list of values."""
@@ -291,78 +295,79 @@ def _dataset_to_list(dataset):
         else:
             return [dataset.strip().strip(',')]
         if dataset[-3:] == '.db' or dataset[:3] == 'db:':
-            # This is a Sleepycat (Oracle) DB  dataset
+            #  This is a Sleepycat (Oracle) DB  dataset, which we dont support
-            import whichdb # Will need rewriting someday for python3
+            raise dkim.ParameterError('Unsupported dataset db datase: {0}'
-            if dataset[-3:] == '.db':
+                                      .format(type(dataset)))
-                dbname = dataset
-            elif dataset[:3] == 'db:':
-                dbname = dataset[3:]
-            else:
-                raise dkim.ParameterError('Unimplmented dataset type: {0}'.format(type(dataset)))
-            if whichdb.whichdb(dbname) != 'dbhash':
-                raise dkim.ParameterError('Unimplmented dataset type: {0}'.format(type(dataset)))
-            #TODO replace this with code to use db maps
-            raise dkim.ParameterError('Unsupported dataset db dataset not yet used: {0}'.format(type(dataset)))
 
-    raise dkim.ParameterError('Unimplmented dataset type: {0}'.format(type(dataset)))
+    raise dkim.ParameterError('Unimplmented dataset type: {0}'
+                              .format(type(dataset)))
 
-###############################################################
+
-commentRx = re.compile(r'^(.*)#.*$')
+def _readConfigFile(path, configData=None, configGlobal={}):
-def _readConfigFile(path, configData = None, configGlobal = {}):
     '''Reads a configuration file from the specified path, merging it
     with the configuration data specified in configData.  Returns a
     dictionary of name/value pairs based on configData and the values
     read from path.'''
 
     debugLevel = configGlobal.get('debugLevel', 0)
-    if debugLevel >= 5: syslog.syslog('readConfigFile: Loading "%s"' % path)
+    if debugLevel >= 5:
-    if configData == None: configData = {}
+        syslog.syslog('readConfigFile: Loading "%s"' % path)
+    if configData is None:
+        configData = {}
     nameConversion = {
-        'AuthservID' : 'str',
+        'AuthservID': 'str',
-        'Syslog' : 'bool',
+        'Syslog': 'bool',
-        'SyslogFacility' : 'str',
+        'SyslogFacility': 'str',
-        'SyslogSuccess' : 'bool',
+        'SyslogSuccess': 'bool',
-        'UMask' : 'int',
+        'UMask': 'int',
-        'Mode' : 'str',
+        'Mode': 'str',
-        'Socket' : 'str',
+        'Socket': 'str',
-        'PidFile'  : 'str',
+        'PidFile': 'str',
-        'UserID' : 'str',
+        'UserID': 'str',
-        'Domain' : 'dataset',
+        'Domain': 'dataset',
-        'KeyFile' : 'str',
+        'KeyFile': 'str',
-        'KeyFileEd25519' : 'str',
+        'KeyFileEd25519': 'str',
-        'Selector' : 'str',
+        'Selector': 'str',
         'SelectorEd25519': 'str',
-        'Canonicalization' : 'str',
+        'Canonicalization': 'str',
-        'InternalHosts' : 'dataset',
+        'InternalHosts': 'dataset',
-        'InternalHostsObj': 'bool',
+        'IntHosts': 'bool',
-        'DiagnosticDirectory' : 'str'
+        'DiagnosticDirectory': 'str',
-            }
+        'MacroList': 'dataset',
+        'MacroListVerify': 'dataset',
+        'DNSOverride': 'str',
+        'debugLevel': 'int'
+        }
 
     #  check to see if it's a file
     try:
         mode = os.stat(path)[0]
     except OSError, e:
-        syslog.syslog(syslog.LOG_ERR,'ERROR stating "%s": %s' % ( path, e.strerror ))
+        syslog.syslog(syslog.LOG_ERR, 'ERROR stating "%s": %s'
+                      % (path, e.strerror))
         return(configData)
     if not stat.S_ISREG(mode):
-        syslog.syslog(syslog.LOG_ERR,'ERROR: is not a file: "%s", mode=%s' % ( path, oct(mode) ))
+        syslog.syslog(syslog.LOG_ERR, 'ERROR: is not a file: "%s", mode=%s'
+                      % (path, oct(mode)))
         return(configData)
 
     #  load file
     fp = open(path, 'r')
     while 1:
         line = fp.readline()
-        if not line: break
+        if not line:
+            break
 
         #  parse line
         line = line.split('#', 1)[0].strip()
-        if not line: continue
+        if not line:
+            continue
         data = line.split()
         if len(data) != 2:
             if len(data) == 1:
                 if debugLevel >= 1:
-                    syslog.syslog('Configuration item "%s" not defined in file "%s"'
+                    syslog.syslog('Config item "%s" not defined in file "%s"'
-                        % ( line, path ))
+                                  % (line, path))
         if len(data) == 1:
             name = data
             value = ''
@@ -374,27 +379,33 @@ def _readConfigFile(path, configData = None, configGlobal = {}):
 
         #  check validity of name
         conversion = nameConversion.get(name)
-        if conversion == None:
+        if conversion is None:
-            syslog.syslog('ERROR: Unknown name "%s" in file "%s"' % ( name, path ))
+            syslog.syslog('ERROR: Unknown name "%s" in file "%s"'
+                          % (name, path))
             continue
 
-        if debugLevel >= 5: syslog.syslog('readConfigFile: Found entry "%s=%s"'
+        if debugLevel >= 5:
-                % ( name, value ))
+            syslog.syslog('readConfigFile: Found entry "%s=%s"'
+                          % (name, value))
         if conversion == 'bool':
             configData[name] = _find_boolean(value)
         elif conversion == 'str':
-            configData[name] = str(value)
+            if isinstance(value, list):
+                configData[name] = line.split(None, 1)[1]
+            else:
+                configData[name] = str(value)
         elif conversion == 'int':
             configData[name] = int(value)
         elif conversion == 'dataset':
             configData[name] = _dataset_to_list(value)
         else:
-            syslog.syslog(str('name: ' + name + ' value: ' + value + ' conversion: ' + conversion))
+            syslog.syslog(str('name: ' + name + ' value: ' + value +
+                              ' conversion: ' + conversion))
             configData[name] = conversion(value)
     fp.close()
     try:
-        configData['AuthservID'] = _calculate_authserv_id(configData['AuthservID'])
+        configData['AuthservID'] = _make_authserv_id(configData.get('AuthservID', 'HOSTNAME'))
-        configData['InternalHostsObj'] = HostsDataset(configData['InternalHosts'])
+        configData['IntHosts'] = HostsDataset(configData['InternalHosts'])
     except:
         pass
 

dkimpy_milter/util.py

@@ -16,6 +16,7 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
+
 def fold(header):
     """Fold a header line into multiple crlf-separated lines at column 72.
     Borrowed from dkimpy and updated to only add \n instead of \r\n because
@@ -46,9 +47,9 @@ def fold(header):
             j = i + 1
         pre += header[:j] + b"\n   "
         header = header[j:]
-        namelen = 0
     return pre + header
 
+
 def user_group(userid):
     """Return user and group from UserID"""
     import grp
@@ -64,13 +65,14 @@ def user_group(userid):
     running_gid = grp.getgrnam(gidname).gr_gid
     return running_uid, running_gid
 
+
 def drop_privileges(milterconfig):
     import os
     import syslog
 
     if os.getuid() != 0:
         if milterconfig.get('Syslog'):
-            syslog.syslog('drop_privileges: Not running as root. Cannot drop permissions.')
+            syslog.syslog('drop_privileges: Not root. No action taken.')
         return
 
     # Get user and group
@@ -86,9 +88,9 @@ def drop_privileges(milterconfig):
     # Set umask
     old_umask = os.umask(milterconfig.get('UMask'))
 
-#################
+
 class ExceptHook:
-    def __init__(self, useSyslog = 1, useStderr = 0):
+    def __init__(self, useSyslog=1, useStderr=0):
         self.useSyslog = useSyslog
         self.useStderr = useStderr
 
@@ -104,12 +106,11 @@ class ExceptHook:
                 sys.stderr.write(line)
 
 
-####################
 def setExceptHook():
     import sys
-    sys.excepthook = ExceptHook(useSyslog = 1, useStderr = 1)
+    sys.excepthook = ExceptHook(useSyslog=1, useStderr=1)
+
 
-####################
 def write_pid(milterconfig):
     """Write PID in pidfile.  Will not overwrite an existing file."""
     import os
@@ -126,10 +127,11 @@ def write_pid(milterconfig):
                 os.chown(piddir, user, group)
                 f = open(milterconfig.get('PidFile'), 'w')
                 if milterconfig.get('Syslog'):
-                    syslog.syslog('Missing pid dir created: {0}'.format(piddir))
+                    syslog.syslog('PID dir created: {0}'.format(piddir))
             else:
                 if milterconfig.get('Syslog'):
-                    syslog.syslog('Unable to write pidfle {0}.  IOError: {1}'.format(milterconfig.get('PidFile'), e))
+                    syslog.syslog('Unable to write pidfle {0}.  IOError: {1}'
+                                  .format(milterconfig.get('PidFile'), e))
                 raise
         f.write(pid)
         f.close()
@@ -137,20 +139,31 @@ def write_pid(milterconfig):
         os.chown(milterconfig.get('PidFile'), user, group)
     else:
         if milterconfig.get('Syslog'):
-            syslog.syslog('Unable to write pidfle {0}.  File exists.'.format(milterconfig.get('PidFile')))
+            syslog.syslog('Unable to write pidfle {0}.  File exists.'
-        raise RuntimeError('Unable to write pidfle {0}.  File exists.'.format(milterconfig.get('PidFile')))
+                          .format(milterconfig.get('PidFile')))
+        raise RuntimeError('Unable to write pidfle {0}.  File exists.'
+                           .format(milterconfig.get('PidFile')))
     return pid
 
+
 def own_socketfile(milterconfig):
     """If socket is Unix socket, chown to UserID before dropping privileges"""
     import os
     user, group = user_group(milterconfig.get('UserID'))
-    if milterconfig.get('Socket')[:1] == '/':
+    offset = None
-        os.chown(milterconfig.get('Socket')[1:], user, group)
+    sockname = milterconfig.get('Socket')
-    if milterconfig.get('Socket')[:6] == "local:":
+    if sockname[:1] == '/':
-        os.chown(milterconfig.get('Socket')[6:], user, group)
+        offset = 0
+    elif sockname[:6] == "local:":
+        offset = 6
+    elif sockname[:5] == "unix:":
+        offset = 5
+
+    if offset is not None:
+        if os.path.exists(sockname[offset:]):
+            os.chown(sockname[offset:], user, group)
+
 
-####################
 def read_keyfile(milterconfig, keytype):
     """Read private key from file."""
     import syslog
@@ -163,7 +176,8 @@ def read_keyfile(milterconfig, keytype):
         keylist = f.readlines()
     except IOError as e:
         if milterconfig.get('Syslog'):
-            syslog.syslog('Unable to read keyfile {0}.  IOError: {1}'.format(keyfile, e))
+            syslog.syslog('Unable to read keyfile {0}.  IOError: {1}'
+                          .format(keyfile, e))
         raise
     f.close()
     key = ''

man/dkimpy-milter.8

@@ -127,7 +127,6 @@
 .rm #[ #] #H #V #F C
 .\" ========================================================================
 .\"
-.IX Title "dkimpy-milter 8"
 .TH dkimpyy-milter 8
 .SH NAME
 .B dkimpy
@@ -269,7 +268,7 @@ proposal, and Cisco's
 .B Internet Identified Mail
 (IIM) proposal.
 .SH VERSION
-This man page covers version 0.9.4 of
+This man page covers version 1.1.0 of
 .I dkimpy-milter.
 .SH COPYRIGHT
 Copyright (c) 2005-2008, Sendmail, Inc. and its suppliers.  All rights
@@ -278,7 +277,7 @@ reserved.
 Copyright (c) 2009-2013, 2015, The Trusted Domain Project.
 All rights reserved.
 
-Copyright (c) 2018 Scott Kitterman <scott@kitterman.com>
+Copyright (c) 2018, 2019 Scott Kitterman <scott@kitterman.com>
 .SH SEE ALSO
 .I dkimpy-milter.conf(5), sendmail(8)
 .P
@@ -292,4 +291,6 @@ RFC6376 - DomainKeys Identified Mail
 .P
 RFC7601 - Message Header Field for Indicating Message Authentication Status
 .P
-draft-ietf-dcrup-dkim-crypto - A new cryptographic signature method for DKIM
+RFC8301 - Cryptographic Algorithm and Key Usage Update to DomainKeys Identified Mail (DKIM)
+.P
+RFC8463 - A New Cryptographic Signature Method for DomainKeys Identified Mail (DKIM)

man/dkimpy-milter.conf.5

@@ -127,16 +127,13 @@
 .rm #[ #] #H #V #F C
 .\" ========================================================================
 .\"
-.IX Title "dkimpy-milter.conf 5"
 .TH dkimpy-milter.conf 5 "2018-02-12"
 .SH "NAME"
 dkimpy-milter \- Python milter for DKIM signing and validation
 .SH "VERSION"
-.IX Header "VERSION"
 0\.9\.2
 
 .SH "DESCRIPTION"
-.IX Header "DESCRIPTION"
 .I dkimpy-milter(8)
 implements the
 .B DKIM
@@ -160,18 +157,15 @@ The provided setup.py installs this configuration file in /etc or
 Command line invocation of parameters as is done by OpenDKIM is not supported.
 
 .SH "USAGE"
-.IX Header "USAGE"
 Usage:
   dkimpy-milter [/etc/dkimpy-milter.conf]
 
 .SH "OTHER DOCUMENTATION"
-.IX Header "OTHER DOCUMENTATION"
 This documentation assumes you have read Postfix's README_FILES/MILTER_README
 (or Sendmail equivalent) and are generally familiar with Domain Keys Identified
 Mail (DKIM).  See RFC 6376 for details.
 
 .SH "SYNOPSIS"
-.IX Header "SYNOPSIS"
 
 dkimpy-milter operates with a default installed configuration file and 
 set of default configuration options that are used if the configuration file
@@ -181,14 +175,12 @@ files can be used directly.  Not all OpenDKIM options are supported.  If an
 unsupported option from OpenDKIM is specified, an error will be raised.
 
 .SH "DESCRIPTION"
-.IX Header "DESCRIPTION"
 
 Configuration options are described here and in the configuration file 
 provided with the package.  The provided setup.py installs this configuration 
 file in /etc or /usr/local/etc.
 
 .SH "OPTIONS"
-.IX Header "OPTIONS"
 
 .TP
 .I AuthservID (string)
@@ -255,13 +247,50 @@ all messages.  Ignored if a
 is defined. [KeyTable NOT IMPLEMENTED]
 
 .TP
-.I KeyFileEd25119 (string)
+.I KeyFileEd25519 (string)
 Gives the location of a Ed25519 private key to be used for Ed25519 signing
 all messages.  File is the Base64 encoded output of RFC 8032 Ed25519 private Key
 generation (as used in dkimpy).  Ignored if a 
 .I KeyTableEd25519
 is defined.  [KeyTableEd25519 NOT IMPLEMENTED]
 
+.TP
+.I MacroList (dataset)
+Defines a set of MTA-provided
+.I macros
+that should be checked to see if the sender has been determined to be a
+local user and therefore whether or not the message should be signed.  If
+a
+.I value
+is specified matching a macro name in the data set, the value of the macro
+must match a value specified (matching is case-sensitive), otherwise the
+macro must be defined but may contain any value.  The set is empty by
+default, meaning macros are not considered when making the sign-verify
+decision.  The general format of the value is
+.I value1[|value2[|...]];
+if one or more value is defined then the macro must be set to one of the
+listed values, otherwise the macro must be set but can contain any
+value.
+
+In order for the macro and its value to be available to the filter for
+checking, the MTA must send it during the protocol exchange.  This is either
+accomplished via manual configuration of the MTA to send the desired macros
+or, for MTA/filter combinations that support the feature, the filter can
+request those macros that are of interest.  The latter is a feature negotiated
+at the time the filter receives a connection from the MTA and its availability
+depends upon the version of milter used to compile the filter and the version
+of the MTA making the connection.
+
+.TP
+.I MacroListVerify (dataset)
+Defines a set of MTA-provided
+.I macros
+that should be checked to see if the sender has been determined to be an
+external source and therefore whether or not the message should be signed.
+Entries in this data set follow the same form as those of the
+.I MacroList
+option above.  [this option is not inhereted from OpenDKIM]  
+
 .TP
 .I Mode (string)
 Selects operating modes.  The string is a concatenation of characters that
@@ -282,6 +311,13 @@ be set:
 (b) KeyTable, SigningTable, no Domain, no KeyFile, no Selector;
 [fooTable options NOT IMPLEMENTED]
 
+.TP 
+.I DNSOverride (string)
+Provide a text string that a verifying milter should use instead of
+consulting the DNS on each message.  This is useful primarily for
+testing purposes in environments where it is awkward to modify the
+system DNS resolution.  It should not be used in production.
+
 .TP
 .I PeerList (dataset)
 Identifies a set of "peers" that identifies clients whose connections
@@ -404,7 +440,6 @@ unless an alternate
 is specified.
 
 .SH "AUTHORS"
-.IX Header "AUTHORS"
 \ddkimpy-milter\fR was written by Scott Kitterman <scott@kitterman.com>.
 It is based on dkimpy-milter.py  Copyright (c) 2001-2013 Business Management Systems, Inc.
 Copyright (c) 2013-2015 Stuart D. Gathman

setup.py

@@ -18,20 +18,26 @@
 
 from setuptools import setup
 import os
-import dkimpy_milter
 
 description = "Domain Keys Identified Mail (DKIM) signing/verifying milter for Postfix/Sendmail."
 
+kw = {}  # Work-around for lack of 'or' requires in setuptools.
+try:
+    import dns
+    kw['install_requires'] = ['dkimpy>=0.7', 'pymilter', 'authres>=1.1.0', 'PyNaCl', 'ipaddress', 'dnspython']
+except ImportError:  # If PyDNS is not installed, prefer dnspython
+    kw['install_requires'] = ['dkimpy>=0.7', 'pymilter', 'authres>=1.1.0', 'PyNaCl', 'ipaddress', 'PyDNS']
+
 setup(
     name='dkimpy-milter',
-    version=dkimpy_milter.__version__,
+    version='1.0.1',
     author='Scott Kitterman',
     author_email='scott@kitterman.com',
     url='https://launchpad.net/dkimpy-milter',
     description=description,
     download_url = "https://pypi.python.org/pypi/dkimpy-milter",
     classifiers= [
-        'Development Status :: 3 - Alpha',
+        'Development Status :: 5 - Production/Stable',
         'Environment :: No Input/Output (Daemon)',
         'Intended Audience :: System Administrators',
         'License :: OSI Approved :: GNU General Public License (GPL)',
@@ -52,9 +58,9 @@ setup(
     data_files=[(os.path.join('share', 'man', 'man5'),
         ['man/dkimpy-milter.conf.5']), (os.path.join('share', 'man', 'man8'),
         ['man/dkimpy-milter.8']), ('etc', ['etc/dkimpy-milter.conf']),
-        (os.path.join('/lib', 'systemd', 'system'),
+        (os.path.join('lib', 'systemd', 'system'),
-        ['system/dkimpy-milter.service']),(os.path.join('/etc', 'init.d'),
+        ['system/dkimpy-milter.service']),(os.path.join('etc', 'init.d'),
         ['system/dkimpy-milter'])],
-    install_requires = ['dkimpy>=0.7', 'pymilter', 'authres>=1.1.0', 'PyNaCl', 'ipaddress', 'dns'],
     zip_safe = False,
+    **kw
 )

system/dkimpy-milter

@@ -13,8 +13,6 @@
 # Provides:          dkim-milter dkim-milter-python dkimpy-milter
 # Required-Start:    $remote_fs $syslog $network $time
 # Required-Stop:     $remote_fs $syslog $network
-# Should-Start:
-# Should-Stop:
 # Default-Start:     2 3 4 5
 # Default-Stop:      0 1 6
 # Short-Description: dkimpy-milter
@@ -31,7 +29,7 @@ NAME=dkimpy-milter
 DESC="Python DKIM Milter"
 USER=dkimpy-milter
 GROUP=dkimpy-milter
-SOCKET=$RUNDIR/dkimpy-milter.pid
+SOCKET=$RUNDIR/dkimpy-milter.sock
 
 test -x $DAEMON || exit 0
 
@@ -70,8 +68,8 @@ case "$1" in
 		fi
 	fi
 
-	start-stop-daemon --start --chuid $USER --background --quiet --pidfile \
+        start-stop-daemon --start --quiet --pidfile $RUNDIR/$NAME.pid --startas \
-		$RUNDIR/$NAME.pid --exec $DAEMON $sysconfdir/$NAME.conf
+                $DAEMON $sysconfdir/$NAME.conf --name $NAME --test > /dev/null \
 	echo "$NAME."
 	;;
   stop)

system/dkimpy-milter.service

@@ -1,11 +1,12 @@
 [Unit]
 Description=DKIMpy Milter
-After=syslog.target network.target
+Documentation=man:dkimpy-milter(8) man:dkimpy-milter.conf(5) 
+After=network.target
 
 [Service]
 Type=simple
 PIDFile=/var/run/dkimpy-milter/dkimpy-milter.pid
-ExecStart=/usr/local/bin/dkimpy-milter 
+ExecStart=/usr/local/bin/dkimpy-milter /usr/local/etc/dkimpy-milter.conf 
 
 [Install]
 WantedBy=multi-user.target

tests/00_minimal.miltertest

@@ -0,0 +1,12 @@
+-- -*- lua -*-
+for _, keytype in ipairs({"ed25519", "rsa"}) do
+   for _, func in ipairs({"signing", "verify"}) do
+      mt.echo("testing "..keytype.." "..func)
+      conn = mt.connect("unix:"..keytype.."."..func..".sock")
+      if conn == nil then
+         error("mt.connect() failed "..keytype.." "..func)
+      end
+      mt.disconnect(conn)
+      mt.echo(keytype.." "..func.." complete")
+   end
+end

tests/01_connect.miltertest

@@ -0,0 +1,40 @@
+-- -*- lua -*-
+for _, keytype in ipairs({"ed25519", "rsa"}) do
+   for _, func in ipairs({"signing", "verify"}) do
+      mt.echo("testing "..keytype.." "..func)
+      conn = mt.connect("unix:"..keytype.."."..func..".sock")
+      if conn == nil then
+         error("mt.connect() failed "..keytype.." "..func)
+      end
+      if mt.conninfo(conn, "localhost", "127.0.0.1") ~= nil then
+         error("mt.conninfo() failed "..keytype.." "..func)
+      end
+      if mt.getreply(conn) ~= SMFIR_CONTINUE then
+         error("mt.conninfo() unexpected reply "..keytype.." "..func)
+      end
+
+      if mt.test_action(conn, SMFIF_ADDHDRS) then
+         print("could add headers "..keytype.." "..func)
+      else
+         error("mt.test_action() says could not add headers "..keytype.." "..func)
+      end
+
+      if mt.test_action(conn, SMFIF_CHGHDRS) then
+         print("could change headers "..keytype.." "..func)
+      else
+         error("mt.test_action() says could not change headers "..keytype.." "..func)
+      end
+
+-- -- FIXME: this part of the test fails, as apparently the
+-- -- dkimpy-milter claims the right to change the body of a message,
+-- -- even though it shouldn't.  How can we fix the negotiation?
+--      if mt.test_action(conn, SMFIF_CHGBODY) then
+--         error("mt.test_action() says could change body "..keytype.." "..func)
+--      else
+--         print("could not change body "..keytype.." "..func)
+--      end
+
+      mt.disconnect(conn)
+      mt.echo(keytype.." "..func.." test complete")
+   end
+end

tests/02_sign_message.miltertest

@@ -0,0 +1,100 @@
+-- -*- lua -*-
+
+msg = {
+   ['headers'] = {
+      ['From'] = 'Alice <alice@example.net>',
+      ['Message-Id'] = '<dkimpy-milter-test-02@example.net>',
+      ['To'] = 'Bob <bob@example.biz>',
+      ['Date'] = 'Mon, 18 Feb 2019 08:32:50 -0500',
+      ['Subject'] = 'Signing test',
+      ['Content-Type'] = 'text/plain',
+   },
+   ['body'] = "This is a test!\r\n",
+}
+
+-- returns miltertest connection object
+function connect_and_send (sockname, headers, body)
+   conn = mt.connect(sockname)
+   if conn == nil then
+      error "mt.connect() failed"
+   end
+   if mt.conninfo(conn, "localhost", "127.0.0.1") ~= nil then
+      error "mt.conninfo() failed"
+   end
+   if mt.getreply(conn) ~= SMFIR_CONTINUE then
+      error "mt.conninfo() unexpected reply"
+   end
+
+   -- mt.macro(conn, SMFIC_MAIL, "i", "simple-message")
+   if mt.mailfrom(conn, "<alice@example.net>") ~= nil then
+      error "mt.mailfrom() failed"
+   end
+   if mt.getreply(conn) ~= SMFIR_CONTINUE then
+      error "mt.mailfrom() unexpected reply"
+   end
+   -- mt.rcptto() is called implicitly
+
+   -- send headers
+   for key,value in pairs(headers) do
+      if mt.header(conn, key, value) ~= nil then
+         error("mt.header(" .. key .. ") failed")
+      end
+      if mt.getreply(conn) ~= SMFIR_CONTINUE then
+         error("mt.header(" .. key .. ") unexpected reply")
+      end
+   end
+   -- send EOH
+   if mt.eoh(conn) ~= nil then
+      error "mt.eoh() failed"
+   end
+   if mt.getreply(conn) ~= SMFIR_CONTINUE then
+      error "mt.eoh() unexpected reply"
+   end
+
+   -- send body
+   if mt.bodystring(conn, body) ~= nil then
+      error "mt.bodystring() failed"
+   end
+   if mt.getreply(conn) ~= SMFIR_CONTINUE then
+      error "mt.bodystring() unexpected reply"
+   end
+   -- end of message; let the filter react
+   if mt.eom(conn) ~= nil then
+      error "mt.eom() failed"
+   end
+   reply = mt.getreply(conn)
+   if reply ~= SMFIR_CONTINUE then
+      error ("mt.eom() unexpected reply: " .. reply)
+   end
+   return conn
+end
+
+for _, keytype in ipairs({"ed25519", "rsa"}) do
+   mt.echo("testing "..keytype)
+   signing = connect_and_send("unix:"..keytype..".signing.sock", msg.headers, msg.body)
+   -- verify that a test header field got added
+   if not mt.eom_check(signing, MT_HDRINSERT) then
+      error "no header added by signer"
+   end
+
+   signature = mt.getheader(signing, "DKIM-Signature", 0)
+
+   mt.disconnect(signing)
+
+   mt.echo("DKIM-Signature: " .. signature)
+
+   msg.headers['DKIM-Signature'] = signature
+
+   verify = connect_and_send("unix:"..keytype..".verify.sock", msg.headers, msg.body)
+
+   if not mt.eom_check(verify, MT_HDRINSERT) then
+      error "no header added in verify"
+   end
+
+   authres = mt.getheader(verify, "Authentication-Results", 0)
+   mt.echo("Authentication-Results: "..authres)
+
+   mt.disconnect(verify)
+
+   mt.echo(keytype.." complete")
+end

tests/dkimpy-milter

@@ -0,0 +1,2 @@
+#!/bin/sh
+python2 -m dkimpy_milter "$@"

tests/runtests

@@ -0,0 +1,84 @@
+#!/bin/bash
+
+set -e
+WORKDIR=$(mktemp -d)
+TESTDIR=$(realpath "$(dirname "$0")")
+DKIMPY_MILTER=${DKIMPY_MILTER:-"$TESTDIR/dkimpy-milter"}
+KEY_TYPES=(ed25519 rsa)
+
+cd "$WORKDIR"
+
+printf "Testing %s from directory %s\n" "$DKIMPY_MILTER" "$WORKDIR"
+
+for keytype in "${KEY_TYPES[@]}"; do
+    dknewkey --ktype "$keytype" "testkey.$keytype"
+    if [ "$keytype" = ed25519 ]; then
+        keyfile=KeyFileEd25519
+        selector=SelectorEd25519
+    else
+        keyfile=KeyFile
+        selector=Selector
+    fi
+    cat > "$keytype.signing.conf" <<EOF
+Domain example.net
+$keyfile testkey.$keytype.key
+$selector testkey
+Socket unix:$keytype.signing.sock
+PidFile $keytype.signing.pid
+Mode s
+UserID $(id --name --user):$(id --name --group)
+EOF
+
+    cat > "$keytype.verify.conf" <<EOF
+Socket unix:$keytype.verify.sock
+PidFile $keytype.verify.pid
+Mode v
+DNSOverride $(cat testkey.$keytype.dns)
+UserID $(id --name --user):$(id --name --group)
+EOF
+done
+
+cleanup() {
+    echo cleaning up jobs:
+    jobs
+    for keytype in "${KEY_TYPES[@]}"; do
+        for func in signing verify; do
+            if [ -s "$keytype.$func.pid" ] && kill -0 "$(cat "$keytype.$func.pid")"; then
+                kill "$(cat $keytype.$func.pid)"
+            fi
+        done
+    done
+    wait
+    for keytype in "${KEY_TYPES[@]}"; do
+        for func in signing verify; do
+            errdata="$keytype.$func.stderr"
+            if [ -s "$errdata" ]; then
+                printf -- "-> %s:\n" "$errdata"
+                cat "$errdata"
+                printf -- "-> end %s\n" "$errdata"
+            fi
+        done
+    done
+    rm -rf "$WORKDIR"
+}
+
+for keytype in "${KEY_TYPES[@]}"; do
+    for func in signing verify; do
+        PYTHONPATH="$(dirname "$TESTDIR")" "$DKIMPY_MILTER" "$keytype.$func.conf" 2>"$keytype.$func.stderr" &
+    done
+done
+trap cleanup EXIT
+
+# ugly ugly (how are we supposed to know that the milters are all ready?):
+sleep 2
+
+# uses miltertest from opendkim:
+for x in ${TESTS:-"$TESTDIR"/*.miltertest}; do
+    if ! [ -e "$x" ]; then
+        if [ -e "$TESTDIR/$x" ]; then
+            x="$TESTDIR/$x"
+        fi
+    fi
+    printf -- "-> running %s...\n" "$x"
+    miltertest -s "$x"
+done