CHANGES and setup.py updates for 1.1.1 release prep

to fail

CHANGES

@@ -1,10 +1,19 @@
-1.1.0 UNRELEASED
+1.1.1 2019-09-06
- - Port to python3
+ - Fix startup logging so it provides information at a useful time
+ - Fix verify processing so missing (optional) i= tag doesn't cause the milter
+   to fail (LP: #1842250)
+ - Fix message extraction so that signing in the same pass through the milter
+   as verifying works correctly
+
+1.1.0 2019-04-12
+ - Add SubDomains option to enable signing for sub-domains (LP: #1811535)
+ - Port to python3 (LP: #1815502)
  - Add test suite using opendkim miltertest
  - When Socket is absolute path, do not strip leading /
  - Handle unix: socket prefix the same as local:
  - Set up correct AuthservID defaults
  - config: Reassemble strings sensibly
+ - Consistently prefer dnspython to Py3DNS (LP: #1815558)
 
 1.0.1 2019-02-11
  - Reorder milter start and dropping privileges so permissions on Unix socket

README

@@ -22,8 +22,8 @@ python3 setup.py install --single-version-externally-managed --record=/dev/null
 For users of Debian Stable (Debian 9, Codename Squeeze), all dependencies are
 available in either the main or backports repositories:
 
-[sudo] apt install python-milter python-nacl python-ipaddress python-dnspython
+[sudo] apt install python3-milter python3-nacl python3-dnspython
-[sudo] apt install -t stretch-backports python-authres python-dkim
+[sudo] apt install -t stretch-backports python3-authres python3-dkim
 
 The preferred method of installation is from PyPi using pip (if distribution
 packages are not available):
@@ -33,10 +33,10 @@ packages are not available):
 Using pip will cause required packages to be installed via easy_install if they
 have not been previously installed.  Because pymilter and PyNaCl are compiled
 Python extensions, the system will need appropriate development packages and
-an C compiler.  Alternately, install these dependencies from dsitribution/OS
+an C compiler.  Alternately, install these dependencies from distribution/OS
 packages and then pip install dkimpy_milter.
 
-The milter will work with either pydns (DNS) or dnspython (dns), preferring
+The milter will work with either py3dns (DNS) or dnspython (dns), preferring
 dnspython is both are available.  The dkimpy DKIM module also works with
 either.
 
@@ -94,7 +94,7 @@ As an example, using the default dkimpy-user on Debian, the command would be:
 
 [sudo] adduser --system --no-create-home --quiet --disabled-password \
                --disabled-login --shell /bin/false --group \
-               --home /var/run/dkimpy-milter dkimpy-milter
+               --home /run/dkimpy-milter dkimpy-milter
 
 Since /var/run or /run is sometimes on a tempfs, if the PID file directory is
 missing, the milter will create it on startup.
@@ -117,7 +117,7 @@ Configuration is very similar to opendkim, but needs some adjustment for
 dkimpy-milter. Here's an example configuration line to include in your
 sendmail.mc:
 
-INPUT_MAIL_FILTER(`dkimpy-milter', `S=local:/var/run/dkimpy-milter/dkimpy-milter.sock')dnl
+INPUT_MAIL_FILTER(`dkimpy-milter', `S=local:/run/dkimpy-milter/dkimpy-milter.sock')dnl
 
 Changing the sendmail.mc file requires a Make (to compile it into sendmail.cf)
 and a restart of sendmail.  Note that S= needs to match the value of Socket in

TODO

@@ -44,15 +44,20 @@ No additional features
 1.0.1
 Bug fix only, improved documentation
 
-1.1.0 (planned)
+1.1.0
-Port to Python 3
+Port to Python 3  implemented	verified
-Subdomain support
+Subdomain support implemented	verified
+Test suite	  implemented	verified
 
 Planned dataset type support (if needed):
 mdb:
 
 Considered for near-term feature release
 
+KeyTable
+KeytableEd25519
+SigningTable
+SigningTableEd25519
 AlwaysAddARHeader
 ChangeRootDirectory
 ClockDrift (requires dkimpy change)
@@ -73,8 +78,6 @@ ExternalIgnoreList
 FixCRLF
 KeepAuthResults
 KeepTemporaryFiles
-KeyTable
-KeytableEd25519
 LogResults
 LogWhy
 MaximumHeaders
@@ -98,7 +101,6 @@ RequireSafeKeys
 SignatureAlgorithm
 SignatureTTL
 SignHeaders
-SigningTable
 SoftwareHeader
 StrictHeaders
 SubDomains

dkimpy_milter/__init__.py

@@ -177,20 +177,21 @@ class dkimMilter(Milter.Base):
             except:
                 # Don't error out on unparseable AR header fiels
                 pass
-        # Check or sign DKIM
+        # Check and/or sign DKIM
         self.fp.seek(0)
+        txt = self.fp.read()
         if milterconfig.get('Domain'):
             domain = milterconfig.get('Domain')
         else:
             domain = ''
+        if milterconfig.get('SubDomains'):
+            self.fdomain = _get_parent_domain(self.fdomain, domain)
         if ((self.fdomain in domain) and not milterconfig.get('Mode') == 'v'
                 and not self.external_connection):
-            txt = self.fp.read()
             self.sign_dkim(txt)
         if ((self.has_dkim) and (not self.internal_connection) and
             (milterconfig.get('Mode') == 'v' or
              milterconfig.get('Mode') == 'sv')):
-            txt = self.fp.read()
             self.check_dkim(txt)
         if self.arresults:
             h = authres.AuthenticationResultsHeader(authserv_id=
@@ -228,11 +229,11 @@ class dkimMilter(Milter.Base):
                 if (milterconfig.get('Syslog') and
                     (milterconfig.get('SyslogSuccess')
                      or milterconfig.get('debugLevel') >= 1)):
-                    syslog.syslog('{0}: {1} DKIM-Signature field added (s={2} '
+                    syslog.syslog('{0}: {1} DKIM signature added (s={2} '
                                   'd={3})'.format(self.getsymval('i'),
-                                                  d.signature_fields.get(b'a'),
+                                                  d.signature_fields.get(b'a').decode(),
-                                                  d.signature_fields.get(b's'),
+                                                  d.signature_fields.get(b's').decode(),
-                                                  d.domain.lower()))
+                                                  d.domain.decode().lower()))
             if privateEd25519:
                 d = dkim.DKIM(txt)
                 h = d.sign(codecs.encode(milterconfig.get('SelectorEd25519'), 'ascii'), codecs.encode(self.fdomain, 'ascii'),
@@ -244,11 +245,11 @@ class dkimMilter(Milter.Base):
                 if (milterconfig.get('Syslog') and
                     (milterconfig.get('SyslogSuccess')
                      or milterconfig.get('debugLevel') >= 1)):
-                    syslog.syslog('{0}: {1} DKIM-Signature field added (s={2} '
+                    syslog.syslog('{0}: {1} DKIM signature added (s={2} '
                                   'd={3})'.format(self.getsymval('i'),
-                                                  d.signature_fields.get(b'a'),
+                                                  d.signature_fields.get(b'a').decode(),
-                                                  d.signature_fields.get(b's'),
+                                                  d.signature_fields.get(b's').decode(),
-                                                  d.domain.lower()))
+                                                  d.domain.decode().lower()))
         except dkim.DKIMException as x:
             if milterconfig.get('Syslog'):
                 syslog.syslog('DKIM: {0}'.format(x))
@@ -287,7 +288,11 @@ class dkimMilter(Milter.Base):
                 self.dkim_comment = str(x)
                 if milterconfig.get('Syslog'):
                     syslog.syslog("check_dkim: {0}".format(x))
+            try:
+                # i= is optional and dkimpy is fine if it's not provided
                 self.header_i = codecs.decode(d.signature_fields.get(b'i'), 'ascii')
+            except TypeError as x:
+                self.header_i = None
             self.header_d = codecs.decode(d.signature_fields.get(b'd'), 'ascii')
             self.header_a = codecs.decode(d.signature_fields.get(b'a'), 'ascii')
             if res:
@@ -296,9 +301,9 @@ class dkimMilter(Milter.Base):
                          milterconfig.get('debugLevel') >= 1)):
                     syslog.syslog('{0}: {1} DKIM signature verified (s={2} '
                                   'd={3})'.format(self.getsymval('i'),
-                                                  d.signature_fields.get(b'a'),
+                                                  d.signature_fields.get(b'a').decode(),
-                                                  d.signature_fields.get(b's'),
+                                                  d.signature_fields.get(b's').decode(),
-                                                  d.domain.lower()))
+                                                  d.domain.decode().lower()))
                 self.dkim_domain = d.domain.lower()
             else:
                 if milterconfig.get('DiagnosticDirectory'):
@@ -325,6 +330,16 @@ class dkimMilter(Milter.Base):
             )
         return
 
+# get parent domain to be signed for if fdomain is a subdomain
+def _get_parent_domain(fdomain, domains):
+    for domain in domains:
+        rhs = '.'+domain
+        # compare right hand side of fdomain against .domain
+        if fdomain[-len(rhs):] == rhs:
+            # return parent domain on match
+            return domain
+    # or return the fdomain itself
+    return fdomain
 
 def main():
     # Ugh, but there's no easy way around this.
@@ -368,10 +383,10 @@ def main():
     own_socketfile(milterconfig, socketname)
     drop_privileges(milterconfig)
     sys.stdout.flush()
-    Milter.runmilter(miltername, socketname, 240)
     if milterconfig.get('Syslog'):
-        syslog.syslog('dkimpy-milter started:{0} user:{1}'
+        syslog.syslog('dkimpy-milter starting:{0} user:{1}'
                       .format(pid, milterconfig.get('UserID')))
+    Milter.runmilter(miltername, socketname, 240)
 
 if __name__ == "__main__":
     main()

dkimpy_milter/config.py

@@ -49,6 +49,7 @@ defaultConfigData = {
     'MacroList': '',
     'MacroListVerify': '',
     'DNSOverride': None,
+    'SubDomains': False,
     'debugLevel': 0  # Undocumented config item for developer use
     }
 
@@ -309,7 +310,9 @@ def _readConfigFile(path, configData=None, configGlobal={}):
     dictionary of name/value pairs based on configData and the values
     read from path.'''
 
-    debugLevel = configGlobal.get('debugLevel', 0)
+    # No config file data is available yet, so to debug _readConfigFile, set
+    # the value here.
+    debugLevel = 0
     if debugLevel >= 5:
         syslog.syslog('readConfigFile: Loading "%s"' % path)
     if configData is None:
@@ -325,6 +328,7 @@ def _readConfigFile(path, configData=None, configGlobal={}):
         'PidFile': 'str',
         'UserID': 'str',
         'Domain': 'dataset',
+        'SubDomains': 'bool',
         'KeyFile': 'str',
         'KeyFileEd25519': 'str',
         'Selector': 'str',
@@ -378,9 +382,15 @@ def _readConfigFile(path, configData=None, configGlobal={}):
             value = data[1:]
 
         #  check validity of name
+        try:
             conversion = nameConversion.get(name)
+        except TypeError:
+            name = name[0]
+            syslog.syslog('Config item "%s" does not provide a value in file "%s"'
+                          % (name, path))
+            conversion = None
         if conversion is None:
-            syslog.syslog('ERROR: Unknown name "%s" in file "%s"'
+            syslog.syslog('ERROR: Unknown name or name missing value "%s" in file "%s"'
                           % (name, path))
             continue
 

dkimpy_milter/util.py

@@ -190,3 +190,26 @@ def read_keyfile(milterconfig, keytype):
     for line in keylist:
         key += line
     return key
+
+def read_keytable(milterconfig, tabletype):
+    """Read keytables into in memory configuration data so all keys are read
+    before priviledges are dropped."""
+    import syslog
+    if tabletype == "RSA":
+        tablefile = milterconfig.get('KeyTable')
+    if tabletype == "Ed25519":
+        tablefile = milterconfig.get('KeyTableEd25519')
+    if milterconfig.get(tablefile):
+        keytabledata = []
+        try:
+            f = open(milterconfig.get(tablefile))
+            for row in f:
+                keytabledata.append(row) 
+            f.close()
+        except IOError as e:
+            if milterconfig.get('Syslog'):
+                syslog.syslog('Unable to read keytable {0}.  IOError: {1}'
+                              .format(tablefile, e))
+            raise
+        
+    return keytabledata

etc/dkimpy-milter.conf

@@ -38,7 +38,7 @@ Socket			inet:8892@localhost
 ###  Name of the file where the filter should write its pid before beginning
 ###  normal operations.
 #
-PidFile			/var/run/dkimpy-milter/dkimpy-milter.pid
+PidFile			/run/dkimpy-milter/dkimpy-milter.pid
 
 ##  Userid userid
 ###      default dkimpy-milter

man/dkimpy-milter.conf.5

@@ -131,7 +131,7 @@
 .SH "NAME"
 dkimpy-milter \- Python milter for DKIM signing and validation
 .SH "VERSION"
-0\.9\.2
+1\.1\.0
 
 .SH "DESCRIPTION"
 .I dkimpy-milter(8)
@@ -221,12 +221,16 @@ domains will be verified rather than being signed.
 
 This parameter is not required if a
 .I SigningTable
+or
+.I SigningTableEd25519
 is in use; in that case, the list of signed domains is implied by the
-lines in that file. [SigningTable NOT IMPLEMENTED]
+lines in that file.
 
 This parameter is ignored if a
 .I KeyTable
-is defined. [KeyTable NOT IMPLEMENTED]
+or
+.I KeyTableD25119
+is defined.
 
 .TP
 .I InternalHosts (dataset)
@@ -244,7 +248,7 @@ address explicitly. [PeerList NOT IMPLEMENTED]
 Gives the location of a PEM-formatted private key to be used for RSA signing
 all messages.  Ignored if a
 .I KeyTable
-is defined. [KeyTable NOT IMPLEMENTED]
+is defined.
 
 .TP
 .I KeyFileEd25519 (string)
@@ -252,7 +256,17 @@ Gives the location of a Ed25519 private key to be used for Ed25519 signing
 all messages.  File is the Base64 encoded output of RFC 8032 Ed25519 private Key
 generation (as used in dkimpy).  Ignored if a 
 .I KeyTableEd25519
-is defined.  [KeyTableEd25519 NOT IMPLEMENTED]
+is defined.
+
+.TP
+.I KeyTable (dataset)
+Gives the location of a file mapping key names to RSA signing keys. If present, overrides any KeyFile setting in the configuration file. The data set named here maps each key name to three values: (a) the name of the domain to use in the signature’s "d=" value; (b) the name of the selector to use in the signature’s "s=" value; and (c) the  path to a file containing a private key. If the first value consists solely of a percent sign ("%") character, it will be replaced by the apparent domain of the sender when generating a signature. The third value must start with a slash ("/") character, or "./" or "../" to indicate it refers to a file from which the private key should be read.  The SigningTable (see below) is used to select records from this table to be used to add signatures based on the message sender.  NOTE: direct specification of keys in the table as is done by OpenDKIM is not supported.
+
+.TP
+.I KeyTableEd25519 (dataset)
+Gives the location of a file mapping key names to Ed25519 signing keys. If present, overrides any KeyFile setting in the configuration file. The data set named here maps each key name to three values: (a) the name of the domain to use in the signature’s "d=" value; (b) the name of the selector to use in the signature’s "s=" value; and (c) the  path to a file containing a private key. If the first value consists solely of a percent sign ("%") character, it will be replaced by the apparent domain of the sender when generating a signature. The third value must start with a slash ("/") character, or "./" or "../" to indicate it refers to a file from which the private key should be read.  The SigningTable (see below) is used to select records from this table to be used to add signatures based on the message sender.  NOTE: direct specification of keys in the table as is done by OpenDKIM is not support
+ed.
+
 
 .TP
 .I MacroList (dataset)
@@ -309,7 +323,6 @@ When signing mode is enabled, one of the following combinations must also
 be set:
 (a) Domain, KeyFile, Selector, no KeyTable, no SigningTable;
 (b) KeyTable, SigningTable, no Domain, no KeyFile, no Selector;
-[fooTable options NOT IMPLEMENTED]
 
 .TP 
 .I DNSOverride (string)
@@ -352,7 +365,7 @@ parameter below for more information.
 
 This parameter is ignored if a
 .I KeyTable
-is defined. [KeyTable NOT IMPLEMENTED]
+is defined.
 
 .TP
 .I SelectorEd25519 (string)
@@ -366,7 +379,33 @@ parameter below for more information.
 
 This parameter is ignored if a
 .I KeyTableEd25519
-is defined. [KeyTable NOT IMPLEMENTED]
+is defined.
+
+.TP
+.I SigningTable (dataset)
+
+Defines a table used to select one or more signatures to apply to a message based on the address found in the From: header field. Keys in this table vary depending on the type of table used; values in this data set should include one field that contains a name found in the KeyTable (see above) that identifies which key should be used in generating the signature, and an optional second field naming the signer of the message that will be included in the "i=" tag in the generated signature. Note that the "i=" value will not be included in the signature if it conflicts with the signing domain (the "d=" value).
+
+If the first field contains only a "%" character, it will be replaced by the domain found in the From: header field. Similarly, within the optional second field, any "%" character will be replaced by the domain found in the From: header field.
+
+If this table specifies a regular expression file ("refile"), then the keys are wildcard patterns that are matched against the address found in the From: header field. Entries are checked in the order in which they appear in the file. ["refile support not implemented"].
+
+For all other database types, the full user@host is checked first, then simply host, then user@.domain (with all superdomains checked in sequence, so "foo.example.com" would first check "user@foo.example.com", then "user@.example.com", then "user@.com"), then .domain, then user@*, and finally *.
+
+In any case, only the first match is applied.
+
+.TP
+.I SigningTableEd25519 (dataset)
+
+Defines a table used to select one or more signatures to apply to a message based on the address found in the From: header field. Keys in this table vary depending on the type of table used; values in this data set should include one field that contains a name found in the KeyTable (see above) that identifies which key should be used in generating the signature, and an optional second field naming the signer of the message that will be included in the "i=" tag in the generated signature. Note that the "i=" value will not be included in the signature if it conflicts with the signing domain (the "d=" value).
+
+If the first field contains only a "%" character, it will be replaced by the domain found in the From: header field. Similarly, within the optional second field, any "%" character will be replaced by the domain found in the From: header field.
+
+If this table specifies a regular expression file ("refile"), then the keys are wildcard patterns that are matched against the address found in the From: header field. Entries are checked in the order in which they appear in the file. ["refile support not implemented"].
+
+For all other database types, the full user@host is checked first, then simply host, then user@.domain (with all superdomains checked in sequence, so "foo.example.com" would first check "user@foo.example.com", then "user@.example.com", then "user@.com"), then .domain, then user@*, and finally *.
+
+In any case, only the first match is applied.
 
 .TP
 .I Socket (string)
@@ -391,6 +430,12 @@ is not given as either a hostname or an IP address, the socket will be
 listening on all interfaces.  A literal IP address must be enclosed in
 square brackets.  This option is mandatory in the configuration file.
 
+.TP
+.I SubDomains (Boolean)
+Sign subdomains of those listed by the
+.I Domain
+parameter as well as the actual domains.
+
 .TP
 .I Syslog (Boolean)
 Log via calls to

setup.py

@@ -26,11 +26,11 @@ try:
     import dns
     kw['install_requires'] = ['dkimpy>=0.7', 'pymilter', 'authres>=1.1.0', 'PyNaCl', 'dnspython']
 except ImportError:  # If PyDNS is not installed, prefer dnspython
-    kw['install_requires'] = ['dkimpy>=0.7', 'pymilter', 'authres>=1.1.0', 'PyNaCl', 'PyDNS']
+    kw['install_requires'] = ['dkimpy>=0.7', 'pymilter', 'authres>=1.1.0', 'PyNaCl', 'Py3DNS']
 
 setup(
     name='dkimpy-milter',
-    version='1.1.0',
+    version='1.1.1',
     author='Scott Kitterman',
     author_email='scott@kitterman.com',
     url='https://launchpad.net/dkimpy-milter',

system/dkimpy-milter

@@ -22,7 +22,7 @@ prefix="/usr/local"
 exec_prefix=${prefix}
 sysconfdir="/etc/dkimpy-milter"
 bindir="${exec_prefix}/bin/"
-RUNDIR="/var/run/dkimpy-milter"
+RUNDIR="/run/dkimpy-milter"
 DAEMON=${bindir}/dkimpy-milter
 PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:
 NAME=dkimpy-milter

system/dkimpy-milter.service

@@ -5,7 +5,7 @@ After=network.target
 
 [Service]
 Type=simple
-PIDFile=/var/run/dkimpy-milter/dkimpy-milter.pid
+PIDFile=/run/dkimpy-milter/dkimpy-milter.pid
 ExecStart=/usr/local/bin/dkimpy-milter /usr/local/etc/dkimpy-milter.conf 
 
 [Install]