Fix setup.py install locations so they are installed correctly and drop unneeded README changes.

locations

CHANGES

@@ -1,11 +1,48 @@
+0.9.5.1 2018-03-10
+ - Add conf file location to systemd unit file
+ - Fix setup.py install locations so they are installed correctly
 
-0.9.1 2018-02-17
+0.9.5 2018-03-10
- - DKIM signing and verification using both RSA and Ed25519
+ - Beta 1 (updated Alpha -> Beta warning in README and trove classifiers)
- - The following configuration options are supported (same definition as
+ - Added support for MacroList option
-   OpenDKIM): Domain, KeyFile, KeyFileEd25519, Mode, PidFile, Selector,
+ - Added support for MacroListVerify option
-   Socket, Syslog, UMask, and UserID (see dkimpy-milter.conf.5)
+ - Added example in README to show use of MacroList* to separate inbound and
- - This is an Alpha grade release and while the implemented features work, it
+   outbound mail streams
-   is nowhere near being a complete package
+ - Added support for SyslogSuccess option (both signing and verifying)
+ - Rationalized logging to be much less verbose unless SyslogSuccess or
+   debugLevel are set - default is generally start/stop/errors only
+ - Fixed install_requires so either dnspython (preferred if neither is
+   installed) or PyDNS satisfies the install requirements
+ - Updated Authentication Results result comment not to mention key size for
+   ed25519 signatures, since it's irrelevant
+ - Enhanced signature verification logging to provide more useful information
+
+0.9.4 2018-03-09
+ - Create PID directory if it is missing
+ - Fix crash when verifying if domain for signing was not set
+ - Fix header folding to use \n only to align with milter protocol
+   requirements
+ - Added information about creating a dedicated user and PID file directory
+   creation to README
+ - Fixed a bug where dkim fail might be reported as pass when verifying
+   multiple signatures and a previous signature had passed
+ - Make RSA signatures in dkimpy-milter optional, so dkimpy-milter can be
+   added after an existing DKIM signing application to add an Ed25519
+   signature (Thanks to A. Schulze for the patch)
+ - Added support for AuthservID option
+ - Added support for InternalHosts option (ipaddress and either dns (dnspython)
+   or pydns (DNS) modules are now required)
+ - Added support for DiagnosticDirectory and updated dkimpy-milter specifics in
+   dkimpy-milter.conf.5
+
+0.9.3 2018-03-02
+ - Fixup csl dataset processing for single item lists
+ - file: dataset support
+ - Bump minimum authres version to 1.1.0 due to known issues with 1.0.2
+ - Ignore errors parsing broken authres header fields
+ - Fold added authres header fields
+ - Fix pidfile permissions
+ - Fix socket setup sequence so Unix sockets work
 
 0.9.2 2018-02-19
  - Improved package requirements definition
@@ -16,3 +53,12 @@
  - Initial dataset support: csl
  - Only sign if mail from from a domain in Domain and only if Mode is not
    verfication only
+   
+0.9.1 2018-02-17
+ - DKIM signing and verification using both RSA and Ed25519
+ - The following configuration options are supported (same definition as
+   OpenDKIM): Domain, KeyFile, KeyFileEd25519, Mode, PidFile, Selector,
+   Socket, Syslog, UMask, and UserID (see dkimpy-milter.conf.5)
+ - This is an Alpha grade release and while the implemented features work, it
+   is nowhere near being a complete package
+

README

@@ -13,16 +13,115 @@ default is a feature:
 
 python setup.py install --single-version-externally-managed --record=/dev/null
 
+For users of Debian Stable (Debian 9, Codename Squueze), all dependencies are
+available in either the main or backports repositories:
+
+[sudo] apt install python-milter python-nacl pthon-ipaddress python-dnspython
+[sudo] apt install -t squeeze-backports python-authres python-dkim
+
+The preferred method of installation is from PyPi using pip (if distribution
+packages are not available):
+
+[sudo] pip install dkimpy_milter
+
+Using pip will cause required packages to be installed via easy_install if they
+have not been previously installed.
+
+The milter will work with either pydns (DNS) or dnspython (dns), preferring
+dnspython is both are available.  The dkimpy DKIM module also works with
+either.
+
 Both a systemd unit file and a sysv init file are provided.  Both make
 assumptions about defaults being used, e.g. if a non-standard pidfile name is
 used, they will need to be updated.  The sysv init file is Debian specific and
 untested, since the developers are not using sysv init.  Feedback/patches
 welcome.
 
+The dkimpy-milter drops priviledges after setup to the user/group specified in
+UserID.  During initial setup, this system user needs to be manually created.
+As an example, using the default dkimpy-user on Debian, the command would be:
+
+[sudo] adduser --system --no-create-home --quiet --disabled-password \
+               --disabled-login --shell /bin/false --group \
+               --home /var/run/dkimpy-milter dkimpy-milter
+
+Since /var/run or /run is sometimes on a tempfs, if the PID file directory is
+missing, the milter will create it on startup.
+
+As with all milters, dkimpy-milter needs to be integrated with your MTA of
+choice (Sendmail or Postfix).
+
+For Sendmail:
+
+Configuration is very similar to opendkim, but needs some adjustment for
+dkimpy-milter. Here's an example configuration line to include in your
+sendmail.mc:
+
+INPUT_MAIL_FILTER(`dkimpy-milter', `S=local:/var/run/dkimpy-milter/dkimpy-milter.sock')dnl
+
+Changing the sendmail.mc file requires a Make (to compile it into sendmail.cf)
+and a restart of sendmail.  Note that S= needs to match the value of Socket in
+the dkimpy-milter configuration file.
+
+Milter support should be present by default in most versions of sendmail
+these days, but if not included in your Sendmail build, see:
+http://www.elandsys.com/resources/sendmail/milter.html
+
+For Postfix:
+
+Integration of dkimpy-milter into Postfix is like any milter (See Postfix's
+README_FILES/MILTER_README).  Here's an example master.cf excerpt the talks to
+two dkimpy-milter instances, one configured for signing and one configured for
+verification:
+
+smtp       inet  n       -       -       -      -       smtpd
+    ...
+    -o smtpd_milters=inet:localhost:8892
+    ...
+
+submission inet  n       -       -       -      -       smtpd
+    ...
+    -o smtpd_milters=inet:localhost:8891
+    ...
+
+These need to match the Socket value for each dkimpy-milter instance.
+
+Care is required to segregate outbound mail to be signed and inbound mail to
+be verified.  The above example uses two instances of dkimpy-milter to do
+this.  There are many possible ways.  Here is another example using milter
+macros to keep the mail streams segregated:
+
+Postfix master.cf:
+
+smtp       inet  n       -       -       -       -       smtpd
+    ...
+    -o smtpd_milters=inet:localhost:8891
+    -o milter_macro_daemon_name=VERIFYING
+    ...
+
+submission inet n       -       -       -       -       smtpd
+    -o syslog_name=postfix/submission
+    -o smtpd_tls_security_level=encrypt
+    -o smtpd_sasl_auth_enable=yes
+    ...
+    -o milter_macro_daemon_name=ORIGINATING
+    -o smtpd_milters=inet:localhost:8891
+    ...
+
+Dkimpy-milter.conf:
+
+...
+Mode			sv
+MacroList		dameon_name|ORIGINATING
+MacroListVerify		daemon_name|VERIFYING
+...
+
+
 The python DKIM library, dkimpy, requires the entire message being signed or
 verified to be in memory, so dkimpy-milter does not write messages out to a temp
 file.  This may impact performance on low-memory systems.
 
-WARNING: This is an alpha grade release to support interoperability testing with
+This is an beta grade release to support interoperability testing with Ed25519
-Ed25519 signatures and basic functionality.  It is known to be incomplete and
+signatures sufficient functionality for basic use.  The documented
-not suitable for general use.
+functionality has been implemented and at least partially tested.  It is free
+of known major defects, but is not fully tested in a variety of environments.

TODO

@@ -25,23 +25,24 @@ Sign based on Domain   implemented      verified
 Canonicalization       implemented      verified
 SyslogFacility         implemented      verified
 
-0.9.5 (Beta)
+0.9.3 (Alpha)
-AuthservID
+File dataset           implemented      verified
-Diagnostics
-DiagnosticDirectory
-InternalHosts
 
-SyslogSuccess
+0.9.4 (Alpha)
+AuthservID      implemented     verified
+DiagnosticDirectory     implemented     verified
+InternalHosts   implemented     verified
+
+0.9.5 (Beta)
+MacroList	implemented	verified
+MacroListVerify	implemented	verified
+SyslogSuccess   implemented     verified
 
 1.0.0
-Convert dkim-milter-python config
 No additional features planned
 
-Plannedataset type support:
+Plannedataset type support (if needed):
-file://
-refile:
 db:/.db
-csl:
 mdb:
 
 Considered for near-term feature release
@@ -50,7 +51,6 @@ AlwaysAddARHeader
 ChangeRootDirectory
 ClockDrift (requires dkimpy change)
 DNSTimeout (requires dkmpy change)
-MacroList
 MilterDebug
 MinimumKeyBits
 PeerList
@@ -59,6 +59,7 @@ SignatureAlgorithm
 Later
 
 BaseDirectory
+Diagnostics (requires dkimpy changes)
 DontSignMailTo
 ExemptDomains
 ExternalIgnoreList

dkimpy_milter/__init__.py

@@ -25,24 +25,24 @@ import sys
 import syslog
 import Milter
 import dkim
-from dkim.dnsplug import get_txt
-from dkim.util import parse_tag_value
 import authres
 import os
 import tempfile
 import StringIO
 import re
-from Milter.config import MilterConfigParser
+from Milter.utils import parse_addr, parseaddr
-from Milter.utils import iniplist,parse_addr,parseaddr
 import dkimpy_milter.config as config
 from dkimpy_milter.util import drop_privileges
 from dkimpy_milter.util import setExceptHook
 from dkimpy_milter.util import write_pid
 from dkimpy_milter.util import read_keyfile
+from dkimpy_milter.util import own_socketfile
+from dkimpy_milter.util import fold
 
-__version__ = "0.9.2"
+__version__ = "0.9.5.1"
 FWS = re.compile(r'\r?\n[ \t]+')
 
+
 class dkimMilter(Milter.Base):
     "Milter to check and sign DKIM.  Each connection gets its own instance."
 
@@ -56,76 +56,88 @@ class dkimMilter(Milter.Base):
         self.fp = None
 
     @Milter.noreply
-  def connect(self,hostname,unused,hostaddr):
+    def connect(self, hostname, unused, hostaddr):
         self.internal_connection = False
+        self.external_connection = False
         self.hello_name = None
         # sometimes people put extra space in sendmail config, so we strip
         self.receiver = self.getsymval('j').strip()
+        try:
+            self.AuthservID = milterconfig['AuthservID']
+        except:
+            self.AuthservID = self.receiver
         if hostaddr and len(hostaddr) > 0:
             ipaddr = hostaddr[0]
-        """if iniplist(ipaddr,self.conf.internal_connect): FIXME
+            if milterconfig['IntHosts']:
-            self.internal_connection = True"""
+                if milterconfig['IntHosts'].match(ipaddr):
-    else: ipaddr = ''
+                    self.internal_connection = True
+        else:
+            ipaddr = ''
         self.connectip = ipaddr
+        if milterconfig.get('MacroList') and not self.internal_connection:
+            macrolist = milterconfig.get('MacroList')
+            for macro in macrolist:
+                macroname = macro.split('|')[0]
+                macroname = '{' + macroname + '}'
+                macroresult = self.getsymval(macroname)
+                if ((len(macro.split('|')) == 1 and macroresult) or macroresult
+                        in macro.split('|')[1:]):
+                    self.internal_connection = True
+        if milterconfig.get('MacroListVerify'):
+            macrolist = milterconfig.get('MacroListVerify')
+            for macro in macrolist:
+                macroname = macro.split('|')[0]
+                macroname = '{' + macroname + '}'
+                macroresult = self.getsymval(macroname)
+                if ((len(macro.split('|')) == 1 and macroresult) or macroresult
+                        in macro.split('|')[1:]):
+                    self.external_connection = True
         if self.internal_connection:
             connecttype = 'INTERNAL'
         else:
             connecttype = 'EXTERNAL'
-    if milterconfig.get('Syslog'):
+        if milterconfig.get('Syslog') and milterconfig.get('debugLevel') >= 1:
-        syslog.syslog("connect from {0} at {1} {2}".format(hostname,hostaddr,connecttype))
+            syslog.syslog("connect from {0} at {1} {2}"
+                          .format(hostname, hostaddr, connecttype))
         return Milter.CONTINUE
 
     # multiple messages can be received on a single connection
     # envfrom (MAIL FROM in the SMTP protocol) seems to mark the start
     # of each message.
     @Milter.noreply
-  def envfrom(self,f,*str):
+    def envfrom(self, f, *str):
-    if milterconfig.get('Syslog'):  
+        if milterconfig.get('Syslog') and milterconfig.get('debugLevel') >= 2:
-        syslog.syslog("mail from: {0} {1}".format(f,str))
+            syslog.syslog("mail from: {0} {1}".format(f, str))
         self.fp = StringIO.StringIO()
         self.mailfrom = f
         t = parse_addr(f)
-    if len(t) == 2: t[1] = t[1].lower()
+        if len(t) == 2:
+            t[1] = t[1].lower()
         self.canon_from = '@'.join(t)
-    self.user = self.getsymval('{auth_authen}')
         self.has_dkim = 0
         self.author = None
         self.arheaders = []
         self.arresults = []
-    '''if self.user:
-      # Very simple SMTP AUTH policy by default:
-      #   any successful authentication is considered INTERNAL
-      self.internal_connection = True
-      auth_type = self.getsymval('{auth_type}')
-      ssl_bits =  self.getsymval('{cipher_bits}')
-      if milterconfig.get('Syslog'):
-          syslog.syslog(
-              "SMTP AUTH:",self.user,"sslbits =",ssl_bits, auth_type,
-              "ssf =",self.getsymval('{auth_ssf}'), "INTERNAL"
-      )
-      # Detailed authorization policy is configured in the access file below.
-      self.arresults.append(
-        authres.SMTPAUTHAuthenticationResult(result = 'pass',
-      	result_comment = auth_type+' sslbits='+ssl_bits, smtp_auth = self.user)
-      )'''
         return Milter.CONTINUE
 
     @Milter.noreply
-  def header(self,name,val):
+    def header(self, name, val):
         lname = name.lower()
         if lname == 'dkim-signature':
-        if milterconfig.get('Syslog'):
+            if (milterconfig.get('Syslog') and
-          syslog.syslog("{0}: {1}".format(name,val))
+                    milterconfig.get('debugLevel') >= 1):
+                syslog.syslog("{0}: {1}".format(name, val))
             self.has_dkim += 1
         if lname == 'from':
-        fname,self.author = parseaddr(val)
+            fname, self.author = parseaddr(val)
             self.fdomain = self.author.split('@')[1]
-        if milterconfig.get('Syslog'):
+            if (milterconfig.get('Syslog') and
-            syslog.syslog("{0}: {1}".format(name,val))
+                    milterconfig.get('debugLevel') >= 1):
+                syslog.syslog("{0}: {1}".format(name, val))
         elif lname == 'authentication-results':
             self.arheaders.append(val)
         if self.fp:
-        self.fp.write("%s: %s\n" % (name,val))
+            self.fp.write("%s: %s\n" % (name, val))
         return Milter.CONTINUE
 
     @Milter.noreply
@@ -136,7 +148,7 @@ class dkimMilter(Milter.Base):
         return Milter.CONTINUE
 
     @Milter.noreply
-  def body(self,chunk):         # copy body to temp file
+    def body(self, chunk):        # copy body to temp file
         if self.fp:
             self.fp.write(chunk)  # IOError causes TEMPFAIL in milter
             self.bodysize += len(chunk)
@@ -146,34 +158,47 @@ class dkimMilter(Milter.Base):
         if not self.fp:
             return Milter.ACCEPT  # no message collected - so no eom processing
         # Remove existing Authentication-Results headers for our authserv_id
-    for i,val in enumerate(self.arheaders,1):
+        for i, val in enumerate(self.arheaders, 1):
             # FIXME: don't delete A-R headers from trusted MTAs
-      ar = authres.AuthenticationResultsHeader.parse_value(FWS.sub('',val))
+            try:
-      if ar.authserv_id == self.receiver:
+                ar = (authres.AuthenticationResultsHeader
-        self.chgheader('authentication-results',i,'')
+                      .parse_value(FWS.sub('', val)))
-        if milterconfig.get('Syslog'):
+                if ar.authserv_id == self.AuthservID:
+                    self.chgheader('authentication-results', i, '')
+                    if (milterconfig.get('Syslog') and
+                            milterconfig.get('debugLevel') >= 1):
                         syslog.syslog('REMOVE: {0}'.format(val))
+            except:
+                # Don't error out on unparseable AR header fiels
+                pass
         # Check or sign DKIM
         self.fp.seek(0)
-    if (self.fdomain in milterconfig.get('Domain')) and (not milterconfig.get('Mode') == 'v'):
+        if milterconfig.get('Domain'):
+            domain = milterconfig.get('Domain')
+        else:
+            domain = ''
+        if ((self.fdomain in domain) and not milterconfig.get('Mode') == 'v'
+                and not self.external_connection):
             txt = self.fp.read()
             self.sign_dkim(txt)
-      result = None
+        if ((self.has_dkim) and (not self.internal_connection) and
-    if (self.has_dkim) and (not self.internal_connection) and (milterconfig.get('Mode') == 'v' or milterconfig.get('Mode') == 'sv'):
+            (milterconfig.get('Mode') == 'v' or
+             milterconfig.get('Mode') == 'sv')):
             txt = self.fp.read()
             self.check_dkim(txt)
-    else:
-      result = 'none'
         if self.arresults:
-        h = authres.AuthenticationResultsHeader(authserv_id = self.receiver, 
+            h = authres.AuthenticationResultsHeader(authserv_id=
+                                                    self.AuthservID,
                                                     results=self.arresults)
-        if milterconfig.get('Syslog'):
+            h = fold(str(h))
+            if (milterconfig.get('Syslog') and
+                    milterconfig.get('debugLevel') >= 2):
                 syslog.syslog(str(h))
-        name,val = str(h).split(': ',1)
+            name, val = str(h).split(': ', 1)
-        self.addheader(name,val,0)
+            self.addheader(name, val, 0)
         return Milter.CONTINUE
 
-  def sign_dkim(self,txt):
+    def sign_dkim(self, txt):
         canon = milterconfig.get('Canonicalization')
         canonicalize = []
         if len(canon.split('/')) == 2:
@@ -182,19 +207,41 @@ class dkimMilter(Milter.Base):
         else:
             canonicalize.append(canon)
             canonicalize.append(canon)
+            if (milterconfig.get('Syslog') and
+                    milterconfig.get('debugLevel') >= 1):
                 syslog.syslog('canonicalize: {0}'.format(canonicalize))
         try:
+            if privateRSA:
                 d = dkim.DKIM(txt)
-        h = d.sign(milterconfig.get('Selector'), self.fdomain, privateRSA,
+                h = d.sign(milterconfig.get('Selector'), self.fdomain,
-                canonicalize=(canonicalize[0], canonicalize[1]))
+                           privateRSA, canonicalize=(canonicalize[0],
-        name,val = h.split(': ',1)
+                                                     canonicalize[1]))
-        self.addheader(name,val.strip().replace('\r\n','\n'),0)
+                name, val = h.split(': ', 1)
+                self.addheader(name, val.strip().replace('\r\n', '\n'), 0)
+                if (milterconfig.get('Syslog') and
+                    (milterconfig.get('SyslogSuccess')
+                     or milterconfig.get('debugLevel') >= 1)):
+                    syslog.syslog('{0}: {1} DKIM-Signature field added (s={2} '
+                                  'd={3})'.format(self.getsymval('i'),
+                                                  d.signature_fields.get(b'a'),
+                                                  d.signature_fields.get(b's'),
+                                                  d.domain))
             if privateEd25519:
                 d = dkim.DKIM(txt)
-            h = d.sign(milterconfig.get('SelectorEd25519'), self.fdomain, privateEd25519,
+                h = d.sign(milterconfig.get('SelectorEd25519'), self.fdomain,
-                    canonicalize=(canonicalize[0], canonicalize[1]), signature_algorithm='ed25519-sha256')
+                           privateEd25519, canonicalize=(canonicalize[0],
-            name,val = h.split(': ',1)
+                                                         canonicalize[1]),
-            self.addheader(name,val.strip().replace('\r\n','\n'),0)
+                           signature_algorithm='ed25519-sha256')
+                name, val = h.split(': ', 1)
+                self.addheader(name, val.strip().replace('\r\n', '\n'), 0)
+                if (milterconfig.get('Syslog') and
+                    (milterconfig.get('SyslogSuccess')
+                     or milterconfig.get('debugLevel') >= 1)):
+                    syslog.syslog('{0}: {1} DKIM-Signature field added (s={2} '
+                                  'd={3})'.format(self.getsymval('i'),
+                                                  d.signature_fields.get(b'a'),
+                                                  d.signature_fields.get(b's'),
+                                                  d.domain))
         except dkim.DKIMException as x:
             if milterconfig.get('Syslog'):
                 syslog.syslog('DKIM: {0}'.format(x))
@@ -203,17 +250,26 @@ class dkimMilter(Milter.Base):
                 syslog.syslog("sign_dkim: {0}".format(x))
             raise
 
-  def check_dkim(self,txt):
+    def check_dkim(self, txt):
         res = False
-      conf = self.conf
         for y in range(self.has_dkim):  # Verify _ALL_ the signatures
             d = dkim.DKIM(txt)
             try:
                 res = d.verify(idx=y)
                 if res:
-              self.dkim_comment = 'Good {0} bit {1} signature.'.format(d.keysize, d.signature_fields.get(b'a'))
+                    if d.signature_fields.get(b'a') == 'ed25519-sha256':
+                        self.dkim_comment = ('Good {0} signature'
+                                             .format(d.signature_fields
+                                                     .get(b'a')))
                     else:
-              self.dkim_comment = 'Bad {0} bit {1} signature.'.format(d.keysize, d.signature_fields.get(b'a'))
+                        self.dkim_comment = ('Good {0} bit {1} signature'
+                                             .format(d.keysize,
+                                                     d.signature_fields
+                                                     .get(b'a')))
+                else:
+                    self.dkim_comment = ('Bad {0} bit {1} signature.'
+                                         .format(d.keysize,
+                                                 d.signature_fields.get(b'a')))
             except dkim.DKIMException as x:
                 self.dkim_comment = str(x)
                 if milterconfig.get('Syslog'):
@@ -226,26 +282,41 @@ class dkimMilter(Milter.Base):
             self.header_d = d.signature_fields.get(b'd')
             self.header_a = d.signature_fields.get(b'a')
             if res:
-              if milterconfig.get('Syslog'):
+                if (milterconfig.get('Syslog') and
-                  syslog.syslog('DKIM: Pass ({0})'.format(d.domain))
+                        (milterconfig.get('SyslogSuccess') or
+                         milterconfig.get('debugLevel') >= 1)):
+                    syslog.syslog('{0}: {1} DKIM signature verified (s={2} '
+                                  'd={3})'.format(self.getsymval('i'),
+                                                  d.signature_fields.get(b'a'),
+                                                  d.signature_fields.get(b's'),
+                                                  d.domain))
                 self.dkim_domain = d.domain
             else:
-            fd,fname = tempfile.mkstemp(".dkim")
+                if milterconfig.get('DiagnosticDirectory'):
-            with os.fdopen(fd,"w+b") as fp:
+                    fd, fname = tempfile.mkstemp(".dkim")
+                    with os.fdopen(fd, "w+b") as fp:
                         fp.write(txt)
                     if milterconfig.get('Syslog'):
-                syslog.syslog('DKIM: Fail (saved as {0})'.format(fname))
+                        syslog.syslog('DKIM: Fail (saved as {0})'
+                                      .format(fname))
+                else:
+                    syslog.syslog('DKIM: Fail ({0})'.format(d.domain))
             if res:
                 result = 'pass'
             else:
                 result = 'fail'
+            res = False
             self.arresults.append(
                 authres.DKIMAuthenticationResult(result=result,
-              header_i = self.header_i, header_d = self.header_d, header_a = self.header_a,
+                                                 header_i=self.header_i,
-              result_comment = self.dkim_comment)
+                                                 header_d=self.header_d,
+                                                 header_a=self.header_a,
+                                                 result_comment=
+                                                 self.dkim_comment)
             )
         return
 
+
 def main():
     # Ugh, but there's no easy way around this.
     global milterconfig
@@ -253,15 +324,16 @@ def main():
     global privateEd25519
     privateRSA = False
     privateEd25519 = False
-    configFile = '/etc/dkimpy-milter.conf'
+    configFile = '/usr/loca/etc/dkimpy-milter.conf'
     if len(sys.argv) > 1:
-        if sys.argv[1] in ( '-?', '--help', '-h' ):
+        if sys.argv[1] in ('-?', '--help', '-h'):
             print('usage: dkimpy-milter [<configfilename>]')
             sys.exit(1)
         configFile = sys.argv[1]
-    milterconfig = config._processConfigFile(filename = configFile)
+    milterconfig = config._processConfigFile(filename=configFile)
     if milterconfig.get('Syslog'):
-        facility = eval("syslog.LOG_{0}".format(milterconfig.get('SyslogFacility').upper()))
+        facility = eval("syslog.LOG_{0}"
+                        .format(milterconfig.get('SyslogFacility').upper()))
         syslog.openlog(os.path.basename(sys.argv[0]), syslog.LOG_PID, facility)
         setExceptHook()
     pid = write_pid(milterconfig)
@@ -269,15 +341,17 @@ def main():
         privateRSA = read_keyfile(milterconfig, 'RSA')
     if milterconfig.get('KeyFileEd25519'):
         privateEd25519 = read_keyfile(milterconfig, 'Ed25519')
-    drop_privileges(milterconfig)
-    if milterconfig.get('Syslog'):
-        syslog.syslog('dkimpy-milter started:{0} user:{1}'.format(pid,milterconfig.get('UserID')))
     Milter.factory = dkimMilter
     Milter.set_flags(Milter.CHGHDRS + Milter.ADDHDRS)
     miltername = 'dkimpy-filter'
     socketname = milterconfig.get('Socket')
+    if milterconfig.get('Syslog'):
+        syslog.syslog('dkimpy-milter started:{0} user:{1}'
+                      .format(pid, milterconfig.get('UserID')))
     sys.stdout.flush()
-    Milter.runmilter(miltername,socketname,240)
+    Milter.runmilter(miltername, socketname, 240)
+    own_socketfile(milterconfig)
+    drop_privileges(milterconfig)
 
 if __name__ == "__main__":
     main()

dkimpy_milter/config.py

@@ -27,39 +27,207 @@
 import syslog
 import os
 import sys
-import re
-import urllib
 import stat
 import dkim
-
+import socket
+import ipaddress
+from dnsplug import Session
 
 #  default values
 defaultConfigData = {
-        'Syslog' : 'yes',
+    'Syslog': 'yes',
-        'SyslogFacility' : 'mail',
+    'SyslogFacility': 'mail',
-        'UMask' : 007,
+    'UMask': 007,
-        'Mode' : 'sv',
+    'Mode': 'sv',
-        'Socket' : 'local:/var/run/dkimpy-milter/dkimpy-milter.sock',
+    'Socket': 'local:/var/run/dkimpy-milter/dkimpy-milter.sock',
-        'PidFile'  : '/var/run/dkimpy-milter/dkimpy-milter.pid',
+    'PidFile': '/var/run/dkimpy-milter/dkimpy-milter.pid',
-        'UserID' : 'dkimpy-milter',
+    'UserID': 'dkimpy-milter',
-        'Canonicalization' : 'relaxed/simple'
+    'Canonicalization': 'relaxed/simple',
+    'InternalHosts': '127.0.0.1',
+    'IntHosts': False,
+    'DiagnosticDirectory': '',
+    'MacroList': '',
+    'MacroListVerify': '',
+    'debugLevel': 0  # Undocumented config item for developer use
     }
 
 
-#################################
 class ConfigException(Exception):
     '''Exception raised when there's a configuration file error.'''
     pass
 
-####################################################################
+
-def _processConfigFile(filename = None, configdata = None, useSyslog = 1,
+class HostsDataset(object):
-        useStderr = 0):
+    '''Hold a group of host related dataset objects'''
+
+    def __init__(self, dataset):
+        self.dataset = []
+        # Self.dataset will end up being a list of DataSetItem(s).
+        for item in dataset:
+            item = item.rstrip(']')
+            item = item.lstrip('[')
+            self.dataset.append(self.DatasetItem(item))
+
+    class DatasetItem(object):
+        '''Individual dataset item'''
+
+        def __init__(self, item):
+            self.item = item
+            self.isipv4 = False
+            self.isipv4cidr = False
+            self.isipv6 = False
+            self.isipv6cidr = False
+            self.ishostname = False
+            self.isdomain = False
+            self.negative = False
+            if self.item[0] == '!':
+                self.item = item[1:]
+                self.negative = True
+            try:
+                self.item = ipaddress.ip_address(unicode(self.item, "utf-8"))
+                if isinstance(self.item, ipaddress.IPv4Address):
+                    self.isipv4 = True
+                elif isinstance(self.item, ipaddress.IPv6Address):
+                    self.isipv6 = True
+            except ValueError as e:
+                try:
+                    self.item = ipaddress.ip_network(unicode
+                                                     (self.item, "utf-8"),
+                                                     strict=False)
+                    if isinstance(self.item, ipaddress.IPv4Network):
+                        self.isipv4cidr = True
+                    elif isinstance(self.item, ipaddress.IPv6Network):
+                        self.isipv6cidr = True
+                except ValueError as e2:
+                    if self.item[0] == '.' and len(self.item.split('.')) > 2:
+                        self.isdomain = True
+                    elif len(self.item.split('.')) > 1:  # It has a '.' in it
+                        self.ishostname = True
+                    else:
+                        raise ConfigException('Unknown dataset item: {0}'
+                                              .format(item))
+
+    def match(self, connectip):
+        '''Check if the connect IP is part of the dataset'''
+        source = ipaddress.ip_address(unicode(connectip, "utf-8"))
+        for item in self.dataset:
+            if item.isdomain or item.ishostname:
+                result = self.matchname(source)   # Match host/domains first
+                if result:
+                    return(result)
+            elif item.isipv4 or item.isipv4cidr:  # Then IPv4/6 addresses or
+                if isinstance(source, ipaddress.IPv4Address):  # networks
+                    return(self.match4(source))   # depending on the item type
+            elif item.isipv6 or item.isipv6cidr:  # and connect type
+                if isinstance(source, ipaddress.IPv6Address):
+                    return(self.match6(source))
+
+    def matchname(self, source):
+        '''Does source IP address relate to a domain/hostname in the dataset'''
+        match = False
+        matchone = False
+        negativeone = False
+        matchdomain = False
+        negativedomain = False
+        ptrlist = self.getptr(source)
+        for item in self.dataset:
+            if item.isdomain:
+                for ptr in ptrlist:
+                    # Strip the leading '.' off the domain name for exact match
+                    if item.item[1:] == ptr[-len(item.item)+1:]:
+                        matchdomain = True
+                        negativedomain = item.negative
+            elif item.ishostname:
+                for ptr in ptrlist:
+                    if item.item == ptr:
+                        matchone = True
+                        negativeone = item.negative
+        if matchdomain and not negativedomain:
+            match = True
+        if matchone and not negativeone:
+            return True
+        if matchone and negativeone:
+            match = False
+        return(match)
+
+    def getptr(self, source):
+        '''Get validated PTR name of IP address'''
+        results = []
+        s = Session()
+        ptrnames = s.dns(source.reverse_pointer, 'PTR')
+        for name in ptrnames:
+            if isinstance(source, ipaddress.IPv4Address):
+                ips = s.dns(name, 'A')
+                for ip in ips:
+                    ip = ipaddress.IPv4Address(unicode(ip, 'UTF-8'))
+                    if ip == source:
+                        results.append(name)
+            if isinstance(source, ipaddress.IPv6Address):
+                ips = s.dns(name, 'AAAA')
+                for ip in ips:
+                    ip = ipaddress.IPv6Address(unicode(ip, 'UTF-8'))
+                if ip == source:
+                    results.append(name)
+        return results
+
+    def match4(self, source):
+        '''Is the source IP related to a IPv4 address/network in the dataset'''
+        match = False
+        matchone = False
+        negativeone = False
+        matchcidr = False
+        negativecidr = False
+        for item in self.dataset:
+            if item.isipv4:
+                if source == item.item:
+                    matchone = True
+                    negativeone = item.negative
+            elif item.isipv4cidr:
+                if source in item.item:
+                    matchcidr = True
+                    negativecidr = item.negative
+        if matchcidr and not negativecidr:
+            match = True
+        if matchone and not negativeone:
+            return True
+        if matchone and negativeone:
+            match = False
+        return(match)
+
+    def match6(self, source):
+        '''Is the source IP realted to a IPv6 address/network in the dataset'''
+        match = False
+        matchone = False
+        negativeone = False
+        matchcidr = False
+        negativecidr = False
+        for item in self.dataset:
+            if item.isipv6:
+                if source == item.item:
+                    matchone = True
+                    negativeone = item.negative
+            elif item.isipv6cidr:
+                if source in item.item:
+                    matchcidr = True
+                    negativecidr = item.negative
+        if matchcidr and not negativecidr:
+            match = True
+        if matchone and not negativeone:
+            return True
+        if matchone and negativeone:
+            match = False
+        return(match)
+
+
+def _processConfigFile(filename=None, configdata=None, useSyslog=1,
+                       useStderr=0):
     '''Load the specified config file, exit and log errors if it fails,
     otherwise return a config dictionary.'''
 
     import config
-    if configdata == None: configdata = config.defaultConfigData
+    if configdata is None:
-    if filename != None:
+        configdata = config.defaultConfigData
+    if filename is not None:
         try:
             _readConfigFile(filename, configdata)
         except Exception, e:
@@ -71,7 +239,7 @@ def _processConfigFile(filename = None, configdata = None, useSyslog = 1,
             sys.exit(1)
     return(configdata)
 
-####################
+
 def _find_boolean(item):
     if type(item) == int:
         item = str(item)
@@ -84,74 +252,132 @@ def _find_boolean(item):
     return item
 
 
+def _make_authserv_id(as_id):
+    """Determine AuthservID if needed"""
+    if as_id == 'HOSTNAME':
+        as_id = socket.gethostname()
+    return as_id
+
+
 def _dataset_to_list(dataset):
     """Convert a dataset (as defined in dkimpymilter.8) and return a python
        list of values."""
-    if not isinstance(dataset, basestring):
+    if not isinstance(dataset, str):
-        # If it was a csl, it's already a list, we only need to remove the name
+        # If it was a csl with more than one value, it's already a list, we
-        # from the first value
+        # only need to remove the name from the first value.
         if dataset[0][:4] == 'csl:':
             dataset[0] = dataset[0][4:]
         for item in dataset:
             dataset[dataset.index(item)] = item.strip().strip(',')
         return dataset
+    elif isinstance(dataset, str):
+        if dataset[0] == '/' or dataset[:5] == 'file:':
+            # This is a flat file dataset
+            ds = []
+            if dataset[0] == '/':
+                dsname = dataset
+            if dataset[:5] == 'file:':
+                dsname = dataset[5:]
+            dsf = open(dsname, 'r')
+            for line in dsf.readlines():
+                if line[0] != '#':
+                    if len(line.split(':')) == 1:
+                        ds.append(line.strip())
                     else:
-        raise dkim.ParameterError('Unimplmented dataset type')
+                        for element in line.split(':'):
+                            ds.append(element.strip().strip(':'))
+            dsf.close()
+            return ds
+        # If it's a str and csl, it has one value and we return a list
+        if dataset[:4] == 'csl:':
+            return [dataset[4:].strip().strip(',')]
+        else:
+            return [dataset.strip().strip(',')]
+        if dataset[-3:] == '.db' or dataset[:3] == 'db:':
+            #  This is a Sleepycat (Oracle) DB  dataset
+            import whichdb  # Will need rewriting someday for python3
+            if dataset[-3:] == '.db':
+                dbname = dataset
+            elif dataset[:3] == 'db:':
+                dbname = dataset[3:]
+            else:
+                raise dkim.ParameterError('Unimplmented dataset type: {0}'
+                                          .format(type(dataset)))
+            if whichdb.whichdb(dbname) != 'dbhash':
+                raise dkim.ParameterError('Unimplmented dataset type: {0}'
+                                          .format(type(dataset)))
+            #TODO replace this with code to use db maps
+            raise dkim.ParameterError('Unsupported dataset db datase: {0}'
+                                      .format(type(dataset)))
 
-###############################################################
+    raise dkim.ParameterError('Unimplmented dataset type: {0}'
-commentRx = re.compile(r'^(.*)#.*$')
+                              .format(type(dataset)))
-def _readConfigFile(path, configData = None, configGlobal = {}):
+
+
+def _readConfigFile(path, configData=None, configGlobal={}):
     '''Reads a configuration file from the specified path, merging it
     with the configuration data specified in configData.  Returns a
     dictionary of name/value pairs based on configData and the values
     read from path.'''
 
     debugLevel = configGlobal.get('debugLevel', 0)
-    if debugLevel >= 5: syslog.syslog('readConfigFile: Loading "%s"' % path)
+    if debugLevel >= 5:
-    if configData == None: configData = {}
+        syslog.syslog('readConfigFile: Loading "%s"' % path)
+    if configData is None:
+        configData = {}
     nameConversion = {
-        'AuthservID' : 'str',
+        'AuthservID': 'str',
-        'Syslog' : 'bool',
+        'Syslog': 'bool',
-        'SyslogFacility' : 'str',
+        'SyslogFacility': 'str',
-        'SyslogSuccess' : 'bool',
+        'SyslogSuccess': 'bool',
-        'UMask' : 'int',
+        'UMask': 'int',
-        'Mode' : 'str',
+        'Mode': 'str',
-        'Socket' : 'str',
+        'Socket': 'str',
-        'PidFile'  : 'str',
+        'PidFile': 'str',
-        'UserID' : 'str',
+        'UserID': 'str',
-        'Domain' : 'dataset',
+        'Domain': 'dataset',
-        'KeyFile' : 'str',
+        'KeyFile': 'str',
-        'KeyFileEd25519' : 'str',
+        'KeyFileEd25519': 'str',
-        'Selector' : 'str',
+        'Selector': 'str',
         'SelectorEd25519': 'str',
-        'Canonicalization' : 'str',
+        'Canonicalization': 'str',
+        'InternalHosts': 'dataset',
+        'IntHosts': 'bool',
+        'DiagnosticDirectory': 'str',
+        'MacroList': 'dataset',
+        'MacroListVerify': 'dataset',
+        'debugLevel': 'int'
         }
 
     #  check to see if it's a file
     try:
         mode = os.stat(path)[0]
     except OSError, e:
-        syslog.syslog(syslog.LOG_ERR,'ERROR stating "%s": %s' % ( path, e.strerror ))
+        syslog.syslog(syslog.LOG_ERR, 'ERROR stating "%s": %s'
+                      % (path, e.strerror))
         return(configData)
     if not stat.S_ISREG(mode):
-        syslog.syslog(syslog.LOG_ERR,'ERROR: is not a file: "%s", mode=%s' % ( path, oct(mode) ))
+        syslog.syslog(syslog.LOG_ERR, 'ERROR: is not a file: "%s", mode=%s'
+                      % (path, oct(mode)))
         return(configData)
 
     #  load file
     fp = open(path, 'r')
     while 1:
         line = fp.readline()
-        if not line: break
+        if not line:
+            break
 
         #  parse line
         line = line.split('#', 1)[0].strip()
-        if not line: continue
+        if not line:
+            continue
         data = line.split()
         if len(data) != 2:
             if len(data) == 1:
                 if debugLevel >= 1:
-                    syslog.syslog('Configuration item "%s" not defined in file "%s"'
+                    syslog.syslog('Config item "%s" not defined in file "%s"'
-                        % ( line, path ))
+                                  % (line, path))
         if len(data) == 1:
             name = data
             value = ''
@@ -163,12 +389,14 @@ def _readConfigFile(path, configData = None, configGlobal = {}):
 
         #  check validity of name
         conversion = nameConversion.get(name)
-        if conversion == None:
+        if conversion is None:
-            syslog.syslog('ERROR: Unknown name "%s" in file "%s"' % ( name, path ))
+            syslog.syslog('ERROR: Unknown name "%s" in file "%s"'
+                          % (name, path))
             continue
 
-        if debugLevel >= 5: syslog.syslog('readConfigFile: Found entry "%s=%s"'
+        if debugLevel >= 5:
-                % ( name, value ))
+            syslog.syslog('readConfigFile: Found entry "%s=%s"'
+                          % (name, value))
         if conversion == 'bool':
             configData[name] = _find_boolean(value)
         elif conversion == 'str':
@@ -178,8 +406,14 @@ def _readConfigFile(path, configData = None, configGlobal = {}):
         elif conversion == 'dataset':
             configData[name] = _dataset_to_list(value)
         else:
-            syslog.syslog(str('name: ' + name + ' value: ' + value + ' conversion: ' + conversion))
+            syslog.syslog(str('name: ' + name + ' value: ' + value +
+                              ' conversion: ' + conversion))
             configData[name] = conversion(value)
     fp.close()
+    try:
+        configData['AuthservID'] = _make_authserv_id(configData['AuthservID'])
+        configData['IntHosts'] = HostsDataset(configData['InternalHosts'])
+    except:
+        pass
 
     return(configData)

dkimpy_milter/dnsplug.py

@@ -0,0 +1,168 @@
+## @package dnsplug
+# Provide a higher level interface to pydns or dnspython (or other provider).
+# NOT RELEASED: this is a proposed API and implementation.
+# Goals - work with both pydns and dnspython (and possibly other libraries)
+# at a simplied level.
+# TODO:
+# 1. map exceptions to common dnsplug.DNSError exception (with
+#    original exception saved as a member).
+# 2. include dict based implementation (handy for test suites)
+# 3. move implementations to subpackages to enable autoselect on first call.
+
+## Maximum number of CNAME records to follow
+MAX_CNAME = 10
+
+## Lookup DNS records by label and RR type.
+# The response can include records of other types that the DNS
+# server thinks we might need.  FIXME: empty result
+# could mean NXDOMAIN or NOANSWER.
+# @param name the DNS label to lookup
+# @param qtype the name of the DNS RR type to lookup
+# @param tcpfallback if False, raise exception instead of TCP fallback
+# @return a list of ((name,type),data) tuples
+def DNSLookup(name, qtype, tcpfallback=True, timeout=30):
+    raise NotImplementedError('No supported dns library found')
+
+class Session(object):
+  """A Session object has a simple cache with no TTL that is valid
+   for a single "session", for example an SMTP conversation."""
+  def __init__(self):
+    self.cache = {}
+
+  ## Additional DNS RRs we can safely cache.
+  # We have to be careful which additional DNS RRs we cache.  For
+  # instance, PTR records are controlled by the connecting IP, and they
+  # could poison our local cache with bogus A and MX records.  
+  # Each entry is a tuple of (query_type,rr_type).  So for instance,
+  # the entry ('MX','A') says it is safe (for milter purposes) to cache
+  # any 'A' RRs found in an 'MX' query.
+  SAFE2CACHE = frozenset((
+    ('MX','MX'), ('MX','A'),
+    ('CNAME','CNAME'), ('CNAME','A'),
+    ('A','A'),
+    ('AAAA','AAAA'),
+    ('PTR','PTR'),
+    ('NS','NS'), ('NS','A'),
+    ('TXT','TXT'),
+    ('SPF','SPF')
+  ))
+
+  ## Cached DNS lookup.
+  # @param name the DNS label to query
+  # @param qtype the query type, e.g. 'A'
+  # @param cnames tracks CNAMES already followed in recursive calls
+  def dns(self, name, qtype, cnames=None):
+    """DNS query.
+
+    If the result is in cache, return that.  Otherwise pull the
+    result from DNS, and cache ALL answers, so additional info
+    is available for further queries later.
+
+    CNAMEs are followed.
+
+    If there is no data, [] is returned.
+
+    pre: qtype in ['A', 'AAAA', 'MX', 'PTR', 'TXT', 'SPF']
+    post: isinstance(__return__, types.ListType)
+    """
+    result = self.cache.get( (name, qtype) )
+    cname = None
+
+    if not result:
+        safe2cache = Session.SAFE2CACHE
+        for k, v in DNSLookup(name, qtype):
+            if k == (name, 'CNAME'):
+                cname = v
+            if (qtype,k[1]) in safe2cache:
+                self.cache.setdefault(k, []).append(v)
+        result = self.cache.get( (name, qtype), [])
+    if not result and cname:
+        if not cnames:
+            cnames = {}
+        elif len(cnames) >= MAX_CNAME:
+            #return result    # if too many == NX_DOMAIN
+            raise DNSError('Length of CNAME chain exceeds %d' % MAX_CNAME)
+        cnames[name] = cname
+        if cname in cnames:
+            raise DNSError, 'CNAME loop'
+        result = self.dns(cname, qtype, cnames=cnames)
+    return result
+
+def DNSLookup_pydns(name, qtype, tcpfallback=True, timeout=30):
+    try:
+	# FIXME: To be thread safe, we create a fresh DnsRequest with
+	# each call.  It would be more efficient to reuse
+	# a req object stored in a Session.
+        req = DNS.DnsRequest(name, qtype=qtype, timeout=timeout)
+        resp = req.req()
+        #resp.show()
+        # key k: ('wayforward.net', 'A'), value v
+        # FIXME: pydns returns AAAA RR as 16 byte binary string, but
+        # A RR as dotted quad.  For consistency, this driver should
+        # return both as binary string.
+        #
+        if resp.header['tc'] == True:
+          if not tcpfallback:
+              raise DNS.DNSError, 'DNS: Truncated UDP Reply, SPF records should fit in a UDP packet'
+          try:
+              req = DNS.DnsRequest(name, qtype=qtype, protocol='tcp',
+                        timeout=timeout)
+              resp = req.req()
+          except DNS.DNSError, x:
+              raise DNS.DNSError, 'TCP Fallback error: ' + str(x)
+        return [((a['name'], a['typename']), a['data']) for a in resp.answers]
+    except IOError, x:
+        raise DNS.DNSError, 'DNS: ' + str(x)
+
+def DNSLookup_dnspython(name,qtype,tcpfallback=True,timeout=30):
+  retVal = []
+  try:
+    # FIXME: how to disable TCP fallback in dnspython if not tcpfallback?
+    answers = dns.resolver.query(name, qtype)
+    for rdata in answers:
+      if qtype == 'A' or qtype == 'AAAA':
+        retVal.append(((name, qtype), rdata.address))
+      elif qtype == 'MX':
+        retVal.append(((name, qtype), (rdata.preference, rdata.exchange)))
+      elif qtype == 'PTR':
+        retVal.append(((name, qtype), rdata.target.to_text(True)))
+      elif qtype == 'TXT' or qtype == 'SPF':
+        retVal.append(((name, qtype), rdata.strings))
+  except dns.resolver.NoAnswer:
+    pass
+  except dns.resolver.NXDOMAIN:
+    pass
+  return retVal
+
+try:
+    # prefer dnspython (the more complete library)
+    import dns
+    import dns.resolver  # http://www.dnspython.org
+    import dns.exception
+
+    if not hasattr(dns.rdatatype,'SPF'):
+      # patch in type99 support
+      dns.rdatatype.SPF = 99
+      dns.rdatatype._by_text['SPF'] = dns.rdatatype.SPF
+
+    DNSLookup = DNSLookup_dnspython
+except:
+    import DNS    # http://pydns.sourceforge.net
+
+    if not hasattr(DNS.Type, 'SPF'):
+        # patch in type99 support
+        DNS.Type.SPF = 99
+        DNS.Type.typemap[99] = 'SPF'
+        DNS.Lib.RRunpacker.getSPFdata = DNS.Lib.RRunpacker.getTXTdata
+
+    # Fails on Mac OS X? Add domain to /etc/resolv.conf
+    DNS.DiscoverNameServers()
+
+    DNSLookup = DNSLookup_pydns
+
+if __name__ == '__main__':
+  import sys
+  s = Session()
+  for n,t in zip(*[iter(sys.argv[1:])]*2):
+    print n,t
+    print s.dns(n,t)

dkimpy_milter/util.py

@@ -16,43 +16,81 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-def drop_privileges(milterconfig):
+
-    import os
+def fold(header):
+    """Fold a header line into multiple crlf-separated lines at column 72.
+    Borrowed from dkimpy and updated to only add \n instead of \r\n because
+    that's what the milter protocol wants.
+
+    >>> text(fold(b'foo'))
+    'foo'
+    >>> text(fold(b'foo  '+b'foo'*24).splitlines()[0])
+    'foo  '
+    >>> text(fold(b'foo'*25).splitlines()[-1])
+    ' foo'
+    >>> len(fold(b'foo'*25).splitlines()[0])
+    72
+    """
+    i = header.rfind(b"\r\n ")
+    if i == -1:
+        pre = b""
+    else:
+        i += 3
+        pre = header[:i]
+        header = header[i:]
+    maxleng = 72
+    while len(header) > maxleng:
+        i = header[:maxleng].rfind(b" ")
+        if i == -1:
+            j = maxleng
+        else:
+            j = i + 1
+        pre += header[:j] + b"\n   "
+        header = header[j:]
+    return pre + header
+
+
+def user_group(userid):
+    """Return user and group from UserID"""
     import grp
     import pwd
-    import syslog
 
-    if os.getuid() != 0:
+    userlist = userid.split(':')
-        if milterconfig.get('Syslog'):
-            syslog.syslog('drop_privileges: Not running as root. Cannot drop permissions.')
-        return
-
-    # Figure out if user and group are specified
-    userstr = milterconfig.get('UserID')
-    userlist = userstr.split(':')
     if len(userlist) == 1:
         gidname = userlist[0]
     else:
         gidname = userlist[1]
-    uidname = userlist[0]
-
     # Get the uid/gid from the name
-    running_uid = pwd.getpwnam(uidname).pw_uid
+    running_uid = pwd.getpwnam(userlist[0]).pw_uid
     running_gid = grp.getgrnam(gidname).gr_gid
+    return running_uid, running_gid
+
+
+def drop_privileges(milterconfig):
+    import os
+    import syslog
+
+    if os.getuid() != 0:
+        if milterconfig.get('Syslog'):
+            syslog.syslog('drop_privileges: Not root. No action taken.')
+        return
+
+    # Get user and group
+    uid, gid = user_group(milterconfig.get('UserID'))
 
     # Remove group privileges
     os.setgroups([])
 
     # Try setting the new uid/gid
-    os.setgid(running_gid)
+    os.setgid(gid)
-    os.setuid(running_uid)
+    os.setuid(uid)
 
     # Set umask
     old_umask = os.umask(milterconfig.get('UMask'))
 
-#################
+
 class ExceptHook:
-    def __init__(self, useSyslog = 1, useStderr = 0):
+    def __init__(self, useSyslog=1, useStderr=0):
         self.useSyslog = useSyslog
         self.useStderr = useStderr
 
@@ -68,12 +106,11 @@ class ExceptHook:
                 sys.stderr.write(line)
 
 
-####################
 def setExceptHook():
     import sys
-    sys.excepthook = ExceptHook(useSyslog = 1, useStderr = 1)
+    sys.excepthook = ExceptHook(useSyslog=1, useStderr=1)
+
 
-####################
 def write_pid(milterconfig):
     """Write PID in pidfile.  Will not overwrite an existing file."""
     import os
@@ -83,18 +120,42 @@ def write_pid(milterconfig):
         try:
             f = open(milterconfig.get('PidFile'), 'w')
         except IOError as e:
+            if str(e)[:35] == '[Errno 2] No such file or directory':
+                piddir = milterconfig.get('PidFile').rsplit('/', 1)[0]
+                os.mkdir(piddir)
+                user, group = user_group(milterconfig.get('UserID'))
+                os.chown(piddir, user, group)
+                f = open(milterconfig.get('PidFile'), 'w')
                 if milterconfig.get('Syslog'):
-                syslog.syslog('Unable to write pidfle {0}.  IOError: {1}'.format(milterconfig.get('PidFile'), e))
+                    syslog.syslog('PID dir created: {0}'.format(piddir))
+            else:
+                if milterconfig.get('Syslog'):
+                    syslog.syslog('Unable to write pidfle {0}.  IOError: {1}'
+                                  .format(milterconfig.get('PidFile'), e))
                 raise
         f.write(pid)
         f.close()
+        user, group = user_group(milterconfig.get('UserID'))
+        os.chown(milterconfig.get('PidFile'), user, group)
     else:
         if milterconfig.get('Syslog'):
-            syslog.syslog('Unable to write pidfle {0}.  File exists.'.format(milterconfig.get('PidFile')))
+            syslog.syslog('Unable to write pidfle {0}.  File exists.'
-        raise RuntimeError('Unable to write pidfle {0}.  File exists.'.format(milterconfig.get('PidFile')))
+                          .format(milterconfig.get('PidFile')))
+        raise RuntimeError('Unable to write pidfle {0}.  File exists.'
+                           .format(milterconfig.get('PidFile')))
     return pid
 
-####################
+
+def own_socketfile(milterconfig):
+    """If socket is Unix socket, chown to UserID before dropping privileges"""
+    import os
+    user, group = user_group(milterconfig.get('UserID'))
+    if milterconfig.get('Socket')[:1] == '/':
+        os.chown(milterconfig.get('Socket')[1:], user, group)
+    if milterconfig.get('Socket')[:6] == "local:":
+        os.chown(milterconfig.get('Socket')[6:], user, group)
+
+
 def read_keyfile(milterconfig, keytype):
     """Read private key from file."""
     import syslog
@@ -107,7 +168,8 @@ def read_keyfile(milterconfig, keytype):
         keylist = f.readlines()
     except IOError as e:
         if milterconfig.get('Syslog'):
-            syslog.syslog('Unable to read keyfile {0}.  IOError: {1}'.format(keyfile, e))
+            syslog.syslog('Unable to read keyfile {0}.  IOError: {1}'
+                          .format(keyfile, e))
         raise
     f.close()
     key = ''

etc/dkimpy-milter.conf

@@ -11,9 +11,9 @@ UMask			007
 
 # Sign for example.com with key in /etc/dkimkeys/dkim.key using
 # selector '2007' (e.g. 2007._domainkey.example.com)
-Domain			example.com
+#Domain			example.com
-KeyFile			/etc/mail/dkim.key
+#KeyFile			/etc/mail/dkim.key
-Selector		default
+#Selector		default
 
 # Commonly-used options; the commented-out versions show the defaults.
 #Canonicalization	relaxed/simple

man/dkimpy-milter.8

@@ -156,7 +156,7 @@ values, or to a file that contains them, or a database containing the data.
 
 Some data sets require that the value contain more than one entry.  How this
 is done depends on which data set type is used.  Not all these datasets are
-currently used by dkimp-milter.  See
+currently used by dkimpy-milter.  See
 .B dkimpy-milter.conf(5)
 for details about specific options and which dataset types they use.
 
@@ -169,17 +169,7 @@ one per line.  If a line contains whitespace-separated values, then the
 line is presumed to define a key and its corresponding value.  Blank lines
 are ignored, and the hash ("#") character denotes the start of a comment.
 If a value contains multiple entries, the entries should be separated by
-colons. [Not implemented yet]
+colons.
-.TP
-.I b)
-If the string begins with "refile:", then the remainder of the string is
-presumed to specify a file that contains a set of patterns, one per line,
-and their associated values.  The pattern is taken as the start of the line
-to the first whitespace, and the portion after that whitespace is taken as
-the value to be used when that pattern is matched.  Patterns are simple
-wildcard patterns, matching all text except that the asterisk ("*") character
-is considered a wildcard.  If a value contains multiple entries, the entries
-should be separated by colons. [Not implemented yet]
 .TP
 .I c)
 If the string begins with "db:" and the program was compiled with
@@ -196,11 +186,11 @@ is compiled in).  [Not implemented yet]
 .TP
 .I i)
 If the string contains none of these prefixes but starts with a slash ("/")
-character, it is presumed to be a flat file as described above.  [Not implemented yet]
+character, it is presumed to be a flat file as described above.
 .TP
 .I j)
 If the string begins with "csl:", the string is treated as a comma-separated
-list as described in m) below.  [Not implemented yet]
+list as described in m) below.
 .TP
 .I l)
 If the string begins with "mdb:", it refers to a directory that contains
@@ -279,7 +269,7 @@ proposal, and Cisco's
 .B Internet Identified Mail
 (IIM) proposal.
 .SH VERSION
-This man page covers version 0.9.2 of
+This man page covers version 0.9.4 of
 .I dkimpy-milter.
 .SH COPYRIGHT
 Copyright (c) 2005-2008, Sendmail, Inc. and its suppliers.  All rights

man/dkimpy-milter.conf.5

@@ -215,18 +215,12 @@ The value may include two different canonicalizations separated by a
 slash ("/") character, in which case the first will be applied to the
 header and the second to the body.
 
-.TP
-.I Diagnostics (Boolean)
-Requests the inclusion of "z=" tags in signatures, which encode the
-original header field set for use by verifiers when diagnosing verification
-failures.  Not recommended for normal operation. [dkimpy-milter specific: also
-increases the verbosity of Syslog logging if enabled.]
-
 .TP
 .I DiagnosticDirectory (string)
 Directory into which to write diagnostic reports when message verification
-fails on a message bearing a "z=" tag.  If not set (the default), these files
+fails.  If not set (the default), these files are not generated.  [Unlike
-are not generated.
+OpenDKIM, this applies to all messages, not just  on messages bearing a "z=" tag
+because dkimpy does not yet support "z=".]
 
 .TP
 .I Domain (dataset)
@@ -261,13 +255,50 @@ all messages.  Ignored if a
 is defined. [KeyTable NOT IMPLEMENTED]
 
 .TP
-.I KeyFileEd25119 (string)
+.I KeyFileEd25519 (string)
 Gives the location of a Ed25519 private key to be used for Ed25519 signing
 all messages.  File is the Base64 encoded output of RFC 8032 Ed25519 private Key
 generation (as used in dkimpy).  Ignored if a 
 .I KeyTableEd25519
 is defined.  [KeyTableEd25519 NOT IMPLEMENTED]
 
+.TP
+.I MacroList (dataset)
+Defines a set of MTA-provided
+.I macros
+that should be checked to see if the sender has been determined to be a
+local user and therefore whether or not the message should be signed.  If
+a
+.I value
+is specified matching a macro name in the data set, the value of the macro
+must match a value specified (matching is case-sensitive), otherwise the
+macro must be defined but may contain any value.  The set is empty by
+default, meaning macros are not considered when making the sign-verify
+decision.  The general format of the value is
+.I value1[|value2[|...]];
+if one or more value is defined then the macro must be set to one of the
+listed values, otherwise the macro must be set but can contain any
+value.
+
+In order for the macro and its value to be available to the filter for
+checking, the MTA must send it during the protocol exchange.  This is either
+accomplished via manual configuration of the MTA to send the desired macros
+or, for MTA/filter combinations that support the feature, the filter can
+request those macros that are of interest.  The latter is a feature negotiated
+at the time the filter receives a connection from the MTA and its availability
+depends upon the version of milter used to compile the filter and the version
+of the MTA making the connection.
+
+.TP
+.I MacroListVerify (dataset)
+Defines a set of MTA-provided
+.I macros
+that should be checked to see if the sender has been determined to be an
+external source and therefore whether or not the message should be signed.
+Entries in this data set follow the same form as those of the
+.I MacroList
+option above.  [this option is not inhereted from OpenDKIM]  
+
 .TP
 .I Mode (string)
 Selects operating modes.  The string is a concatenation of characters that
@@ -374,7 +405,7 @@ Log via calls to
 using the named facility.  The facility names are the same as the ones
 allowed in
 .I syslog.conf(5).
-The default is "mail".  [Hardcoded to default for now]
+The default is "mail".
 
 .TP
 .I SyslogSuccess (Boolean)

setup.py

@@ -22,6 +22,13 @@ import dkimpy_milter
 
 description = "Domain Keys Identified Mail (DKIM) signing/verifying milter for Postfix/Sendmail."
 
+kw = {}  # Work-around for lack of 'or' requires in setuptools.
+try:
+    import DNS
+    kw['install_requires'] = ['dkimpy>=0.7', 'pymilter', 'authres>=1.1.0', 'PyNaCl', 'ipaddress', 'PyDNS']
+except ImportError:  # If PyDNS is not installed, prefer dnspython
+    kw['install_requires'] = ['dkimpy>=0.7', 'pymilter', 'authres>=1.1.0', 'PyNaCl', 'ipaddress', 'dnspython']
+
 setup(
     name='dkimpy-milter',
     version=dkimpy_milter.__version__,
@@ -31,7 +38,7 @@ setup(
     description=description,
     download_url = "https://pypi.python.org/pypi/dkimpy-milter",
     classifiers= [
-        'Development Status :: 3 - Alpha',
+        'Development Status :: 4 - Beta',
         'Environment :: No Input/Output (Daemon)',
         'Intended Audience :: System Administrators',
         'License :: OSI Approved :: GNU General Public License (GPL)',
@@ -52,9 +59,9 @@ setup(
     data_files=[(os.path.join('share', 'man', 'man5'),
         ['man/dkimpy-milter.conf.5']), (os.path.join('share', 'man', 'man8'),
         ['man/dkimpy-milter.8']), ('etc', ['etc/dkimpy-milter.conf']),
-        (os.path.join('/lib', 'systemd', 'system'),
+        (os.path.join('lib', 'systemd', 'system'),
-        ['system/dkimpy-milter.service']),(os.path.join('/etc', 'init.d'),
+        ['system/dkimpy-milter.service']),(os.path.join('etc', 'init.d'),
         ['system/dkimpy-milter'])],
-    install_requires = ['dkimpy>=0.7', 'pymilter', 'authres>=1.0.2', 'PyNaCl'],
     zip_safe = False,
+    **kw
 )

system/dkimpy-milter.service

@@ -5,7 +5,7 @@ After=syslog.target network.target
 [Service]
 Type=simple
 PIDFile=/var/run/dkimpy-milter/dkimpy-milter.pid
-ExecStart=/usr/local/bin/dkimpy-milter 
+ExecStart=/usr/local/bin/dkimpy-milter /usr/local/etc/dkimpy-milter.conf 
 
 [Install]
 WantedBy=multi-user.target