Logging fixups: Don't traceback for non-UTF-8 data in mail headers and don't put byte string markers in logs (some remain, but are from dkimpy and should be fixed there), related to LP: #1980821

Clarify that key names from SigningTable apply to RSA and ed25519 keys.
The way to sign with both algorithms is to provide keys in both
KeyTables, with the same name.

CHANGES

@@ -1,3 +1,19 @@
+1.2.3
+ - Logging fixups: Don't traceback for non-UTF-8 data in mail headers and
+   don't put byte string markers in logs (some remain, but are from dkimpy
+   and should be fixed there), related to LP: #1980821.
+
+1.2.2 2020-08-09
+ - Improve README.md formating for markdown display on pypi
+ - Improve documentation in dkimpy-milter.conf (5) and README.md for signing
+   for multiple domains (Thanks to Stefano Rivera)
+ - Minimal fix for dnspython 2.0.0 compatibility (still works with 1.16.0)
+
+1.2.1 2020-01-04
+ - Fix expand option not to fail if files are missing since socket activation
+   service files are not shipped in the sdist
+ - Correct dkimpy-milter.conf file install location to match expand locations
+
 1.2.0 2020-01-03
  - Add support for SigningTable, KeyTable, and KeyTableEd25519 (LP: #1797397)
  - Add support for specifying MinimumKeyBits for RSA signatures

README.md

@@ -52,22 +52,23 @@ The package includes a custom setup command called expand.  It allows various
 file locations in init scripts, man pages, and config files to be over-ridden
 at install time.
 
-expand: Expand @@ variables in input files, simlar to make macros.
-user_options:
- --sysconfigdir=, e: Specify system configuration directory.
- --sbindir=, s: Specify system binary directory [not used].
- --bindir=, b: Specify binary directory.
- --rundir=,r: Specify run state directory.
+
+    expand: Expand @@ variables in input files, simlar to make macros.
+    user_options:
+      --sysconfigdir=, e: Specify system configuration directory.
+      --sbindir=, s: Specify system binary directory [not used].
+      --bindir=, b: Specify binary directory.
+      --rundir=,r: Specify run state directory.
 
 As an example, to change the run directory to /var/run, one would do:
 
-python3 setup.py expand --rundir=/var/run
-[sudo] python3 setup.py install --single-version-externally-managed \
+    python3 setup.py expand --rundir=/var/run
+    [sudo] python3 setup.py install --single-version-externally-managed \
                                 --record=/dev/null
 
 or in a single step (the order matters):
 
-[sudo] python3 setup.py expand --rundir=/var/run install \
+    [sudo] python3 setup.py expand --rundir=/var/run install \
                                --single-version-externally-managed \
                                --record=/dev/null
 
@@ -87,7 +88,7 @@ tools available to create them.  Keys must (RFC 8302) have a minimum size of
 1024 bits and should have a size of at least 2048 bits.  The dknewkey script
 that is provided with dkimpy is one such tool:
 
-dknewkey exampleprivkey
+    dknewkey exampleprivkey
 
 will produce both the private key file (.key suffix) and a file with the DKIM
 public key record to be published DNS (.dns suffix).  RSA is the default key
@@ -99,7 +100,7 @@ There is no standardized non-binary representation for Ed25519 private keys,
 so in order to generate Ed25519 keys for dkimpy-milter, dkimpy specific tools
 must be used to be compatible.  The same dknewkey script support Ed25519:
 
-dknewkey --ktype ed25519 anothernewkey
+    dknewkey --ktype ed25519 anothernewkey
 
 will provide both the private key file (.key suffix) and a file with the DKIM
 public key record to be published DNS (.dns suffix).  Ed25519 keys do not have
@@ -135,9 +136,9 @@ for the above might look like this:
 	comkey	example.com:bar:/usr/local/etc/dkim/keys/excom
 	netkey	example.net:baz:/usr/local/etc/dkim/keys/exnet
 
-If also signing with ed25519, specify a KeyTableEd25519 pointing to the keys
-needed for ed25519.  Both KeyTable and KeyTableEd25519 are evaluated if there
-is a SigningTable (see below).
+If also signing with ed25519, specify a KeyTableEd25519, with the same
+names, pointing to the keys needed for ed25519.  Both KeyTable and
+KeyTableEd25519 are evaluated if there is a SigningTable (see below).
 
 Per the documentation, multi-field data sets that are made of flat files have
 the fields separated by colons, but the key and value(s) are separated by
@@ -185,7 +186,7 @@ The dkimpy-milter drops priviledges after setup to the user/group specified in
 UserID.  During initial setup, this system user needs to be manually created.
 As an example, using the default dkimpy-user on Debian, the command would be:
 
-[sudo] adduser --system --no-create-home --quiet --disabled-password \
+    [sudo] adduser --system --no-create-home --quiet --disabled-password \
                --disabled-login --shell /bin/false --group \
                --home /run/dkimpy-milter dkimpy-milter
 
@@ -195,10 +196,10 @@ missing, the milter will create it on startup.
 To start dkimpy-milter with systemd for the first time, you will need to take
 the following steps:
 
-[sudo] systemctl daemon-reload
-[sudo] systemctl enable dkimpy-milter
-[sudo] systemctl start dkimpy-milter
-[sudo] systemctl status dkimpy-milter (to verify it started correctly)
+    [sudo] systemctl daemon-reload
+    [sudo] systemctl enable dkimpy-milter
+    [sudo] systemctl start dkimpy-milter
+    [sudo] systemctl status dkimpy-milter (to verify it started correctly)
 
 As with all milters, dkimpy-milter needs to be integrated with your MTA of
 choice (Sendmail or Postfix).  When integrating with your MTA, the risk of
@@ -214,7 +215,7 @@ Configuration is very similar to opendkim, but needs some adjustment for
 dkimpy-milter. Here's an example configuration line to include in your
 sendmail.mc:
 
-INPUT_MAIL_FILTER(`dkimpy-milter', `S=local:/run/dkimpy-milter/dkimpy-milter.sock')dnl
+    INPUT_MAIL_FILTER(`dkimpy-milter', `S=local:/run/dkimpy-milter/dkimpy-milter.sock')dnl
 
 Changing the sendmail.mc file requires a Make (to compile it into sendmail.cf)
 and a restart of sendmail.  Note that S= needs to match the value of Socket in
@@ -237,7 +238,7 @@ and deserve consideration.
 
     By default, sendmail quotes to address header fields when there are no
     quotes and the display part of the address contains a period or an
-    apostrophe.  However, opendkim only sees the raw, unmodified form of
+    apostrophe.  However, dkimpy-milter only sees the raw, unmodified form of
     the header field, and so the content that gets verified and what gets
     signed will not be the same, guaranteeing the attached signature is not
     valid.
@@ -263,16 +264,16 @@ and deserve consideration.
     To: very long name <a@example.org>,
     	anotherloo...ong name b <b@example.org>
 
-    This rewrite is also done after opendkim has seen the message, meaning
-    the signature opendkim attaches to the message does not match the
-    content it signed.  There is not a known configuration change to
+    This rewrite is also done after dkimpy-milter has seen the message,
+    meaning the signature dkimpy-milter attaches to the message does not match
+    the content it signed.  There is not a known configuration change to
     mitigate this mutation.
 
     The only known mechanism for dealing with this is to have distinct
-    instances of opendkim do the verifying (inbound) and signing (outbound)
-    so that the version that arrives at the signing instance is already
-    in the rewritten form, guaranteeing the input and output are the same
-    and thus the signature matches the payload.
+    instances of dkimpy-milter do the verifying (inbound) and signing
+    (outbound) so that the version that arrives at the signing instance is
+    already in the rewritten form, guaranteeing the input and output are the
+    same and thus the signature matches the payload.
 
 ### POSTFIX
 
@@ -281,15 +282,15 @@ README_FILES/MILTER_README).  Here's an example master.cf excerpt that talks
 to two dkimpy-milter instances, one configured for signing and one configured
 for verification:
 
-smtp       inet  n       -       -       -      -       smtpd
-    ...
-    -o smtpd_milters=inet:localhost:8892
-    ...
+    smtp       inet  n       -       -       -      -       smtpd
+        ...
+        -o smtpd_milters=inet:localhost:8892
+        ...
 
-submission inet  n       -       -       -      -       smtpd
-    ...
-    -o smtpd_milters=inet:localhost:8891
-    ...
+    submission inet  n       -       -       -      -       smtpd
+        ...
+        -o smtpd_milters=inet:localhost:8891
+        ...
 
 These need to match the Socket value for each dkimpy-milter instance.
 
@@ -300,28 +301,28 @@ macros to keep the mail streams segregated:
 
 Postfix master.cf:
 
-smtp       inet  n       -       -       -       -       smtpd
-    ...
-    -o smtpd_milters=inet:localhost:8891
-    -o milter_macro_daemon_name=VERIFYING
-    ...
+    smtp       inet  n       -       -       -       -       smtpd
+        ...
+        -o smtpd_milters=inet:localhost:8891
+        -o milter_macro_daemon_name=VERIFYING
+        ...
 
-submission inet n       -       -       -       -       smtpd
-    -o syslog_name=postfix/submission
-    -o smtpd_tls_security_level=encrypt
-    -o smtpd_sasl_auth_enable=yes
-    ...
-    -o milter_macro_daemon_name=ORIGINATING
-    -o smtpd_milters=inet:localhost:8891
-    ...
+    submission inet n       -       -       -       -       smtpd
+        -o syslog_name=postfix/submission
+        -o smtpd_tls_security_level=encrypt
+        -o smtpd_sasl_auth_enable=yes
+        ...
+        -o milter_macro_daemon_name=ORIGINATING
+        -o smtpd_milters=inet:localhost:8891
+        ...
 
 Dkimpy-milter.conf:
 
-...
-Mode			sv
-MacroList		dameon_name|ORIGINATING
-MacroListVerify		daemon_name|VERIFYING
-...
+    ...
+    Mode			sv
+    MacroList		dameon_name|ORIGINATING
+    MacroListVerify		daemon_name|VERIFYING
+    ...
 
 
 # NOTES

dkimpy_milter/__init__.py

@@ -107,9 +107,10 @@ class dkimMilter(Milter.Base):
     # envfrom (MAIL FROM in the SMTP protocol) seems to mark the start
     # of each message.
     @Milter.noreply
-    def envfrom(self, f, *str):
+    def envfrom(self, f, *stri):
         if self.conf.get('Syslog') and self.conf.get('debugLevel') >= 2:
-            syslog.syslog("mail from: {0} {1}".format(f, str))
+            f = str(bytes(f, encoding='utf-8', errors='replace'))[2:-1]
+            syslog.syslog("mail from: {0} {1}".format(f, stri))
         self.fp = io.BytesIO()
         self.mailfrom = f
         t = parse_addr(f)
@@ -133,7 +134,8 @@ class dkimMilter(Milter.Base):
         if lname == 'dkim-signature':
             if (self.conf.get('Syslog') and
                     self.conf.get('debugLevel') >= 1):
-                syslog.syslog("{0}: {1}".format(name, val))
+                val2 = str(bytes(val, encoding='utf-8', errors='replace'))[2:-1]
+                syslog.syslog("{0}: {1}".format(name, val2))
             self.has_dkim += 1
         if lname == 'from':
             fname, self.author = parseaddr(val)
@@ -143,7 +145,8 @@ class dkimMilter(Milter.Base):
                 pass # self.author was not a proper email address
             if (self.conf.get('Syslog') and
                     self.conf.get('debugLevel') >= 1):
-                syslog.syslog("{0}: {1}".format(name, val))
+                val2 = str(bytes(val, encoding='utf-8', errors='replace'))[2:-1]
+                syslog.syslog("{0}: {1}".format(name, val2))
         elif lname == 'authentication-results':
             self.arheaders.append(val)
         if self.fp:
@@ -181,7 +184,8 @@ class dkimMilter(Milter.Base):
                     self.chgheader('authentication-results', i, '')
                     if (self.conf.get('Syslog') and
                             self.conf.get('debugLevel') >= 1):
-                        syslog.syslog('REMOVE: {0}'.format(val))
+                        val2 = str(bytes(val, encoding='utf-8', errors='replace'))[2:-1]
+                        syslog.syslog('REMOVE: {0}'.format(val2))
             except:
                 # Don't error out on unparseable AR header fiels
                 pass
@@ -363,8 +367,12 @@ class dkimMilter(Milter.Base):
             try:
                 dnsoverride = self.conf.get('DNSOverride')
                 if isinstance(dnsoverride, str):
+                    timeout = 5
+                    domain = self.fdomain
+                    def dnsfunc(domain, timeout=timeout, dnsoverride=dnsoverride):
+                        return dnsoverride
                     syslog.syslog("DNSOverride: {0}".format(dnsoverride))
-                    res = d.verify(idx=y, dnsfunc=lambda _x: dnsoverride)
+                    res = d.verify(idx=y, dnsfunc=dnsfunc)
                 else:
                     res = d.verify(idx=y)
                 algo = codecs.decode(d.signature_fields.get(b'a'), 'ascii')
@@ -424,7 +432,8 @@ class dkimMilter(Milter.Base):
                     if self.conf.get('Syslog'):
                         if d.domain:
                             syslog.syslog('DKIM: Fail ({0})'
-                                          .format(d.domain.lower()))
+                                .format(str(d.domain.lower(), 'UTF-8',
+                                errors='replace')))
                         else:
                             syslog.syslog('DKIM: Fail, unextractable domain')
             if res:
@@ -434,13 +443,13 @@ class dkimMilter(Milter.Base):
             res = False
             if self.header_d:
                 self.arresults.append(
-                    authres.DKIMAuthenticationResult(result=result,
+                    authres.DKIMAuthenticationResult(result=result[2:-1],
                                                  header_i=self.header_i,
                                                  header_d=self.header_d,
                                                  header_a=self.header_a,
                                                  result_comment=
                                                  self.dkim_comment)
-            )
+                )
             self.header_a = None
         return
 

dkimpy_milter/dnsplug.py

@@ -127,7 +127,7 @@ def DNSLookup_dnspython(name,qtype,tcpfallback=True,timeout=5):
       elif qtype == 'PTR':
         retVal.append(((name, qtype), rdata.target.to_text(True)))
       elif qtype == 'TXT' or qtype == 'SPF':
-        retVal.append(((name, qtype), rdata.strings))
+        retVal.append(((name, qtype), list(rdata.strings)))
   except dns.resolver.NoAnswer:
     pass
   except dns.resolver.NXDOMAIN:

man/dkimpy-milter.conf.5.in

@@ -428,7 +428,7 @@ of this field.
 .TP
 .I SigningTable (dataset)
 
-Defines a table used to select one or more signing identities to apply to a message based on the address found in the From: header field. Keys in this table vary depending on the type of table used; values in this data set should include one field that contains a name found in the KeyTable (see above) that identifies which key should be used in generating the signature, and an optional second field naming the signer of the message that will be included in the "i=" tag in the generated signature. Note that the "i=" value will not be included in the signature if it conflicts with the signing domain (the "d=" value).
+Defines a table used to select a signing identity to apply to a message based on the address found in the From: header field. Keys in this table vary depending on the type of table used; values in this data set should include one field that contains a name found in the KeyTable (see above) that identifies which key should be used in generating the signature, and an optional second field naming the signer of the message that will be included in the "i=" tag in the generated signature. Note that the "i=" value will not be included in the signature if it conflicts with the signing domain (the "d=" value).
 
 If the first field contains only a "%" character, it will be replaced by the domain found in the From: header field. Similarly, within the optional second field, any "%" character will be replaced by the domain found in the From: header field.
 

setup.py

@@ -53,29 +53,32 @@ class FileMacroExpand(distutils.cmd.Command):
                  'system/dkimpy-milter.service', 'system/dkimpy-milter', \
                  'system/dkimpy-milter.openrc', \
                  'system/socket-activation/dkimpy-milter.service', \
-                 'system/socket-activation/dkimpy-milter.socket',]
+                 'system/socket-activation/dkimpy-milter.socket', ]
         for infile in files:
             outfile = ''
-            filein = open(infile + '.in')
-            for line in filein:
-                for function in ["@SYSCONFDIR@", "@CONFDIR@", "@SBINDIR@", "@BINDIR@", "@RUNSTATEDIR@"]:
-                    splitline = line.split(function)
-                    if len(splitline) > 1:
-                        if function == "@SYSCONFDIR@":
-                            line = splitline[0] + self.sysconfigdir + splitline[1]
-                        elif function == "@CONFDIR@":
-                            line = splitline[0] + self.configdir + splitline[1]
-                        elif function == "@SBINDIR@":
-                            line = splitline[0] + self.sbindir + splitline[1]
-                        elif function == "@BINDIR@":
-                            line = splitline[0] + self.bindir + splitline[1]
-                        elif function == "@RUNSTATEDIR@":
-                            line = splitline[0] + self.rundir + splitline[1]
-                outfile += line
-            out = open(infile, 'w')
-            for line in outfile:
-                out.write(line)
-            out.close()
+            try:
+                filein = open(infile + '.in')
+                for line in filein:
+                    for function in ["@SYSCONFDIR@", "@CONFDIR@", "@SBINDIR@", "@BINDIR@", "@RUNSTATEDIR@"]:
+                        splitline = line.split(function)
+                        if len(splitline) > 1:
+                            if function == "@SYSCONFDIR@":
+                                line = splitline[0] + self.sysconfigdir + splitline[1]
+                            elif function == "@CONFDIR@":
+                                line = splitline[0] + self.configdir + splitline[1]
+                            elif function == "@SBINDIR@":
+                                line = splitline[0] + self.sbindir + splitline[1]
+                            elif function == "@BINDIR@":
+                                line = splitline[0] + self.bindir + splitline[1]
+                            elif function == "@RUNSTATEDIR@":
+                                line = splitline[0] + self.rundir + splitline[1]
+                    outfile += line
+                out = open(infile, 'w')
+                for line in outfile:
+                    out.write(line)
+                out.close()
+            except FileNotFoundError as x:
+                pass
 
 kw = {}  # Work-around for lack of 'or' requires in setuptools.
 try:
@@ -86,7 +89,7 @@ except ImportError:  # If PyDNS is not installed, prefer dnspython
 
 setup(
     name='dkimpy-milter',
-    version='1.2.0',
+    version='1.2.1',
     author='Scott Kitterman',
     author_email='scott@kitterman.com',
     url='https://launchpad.net/dkimpy-milter',
@@ -115,8 +118,8 @@ setup(
     include_package_data=True,
     data_files=[(os.path.join('share', 'man', 'man5'),
         ['man/dkimpy-milter.conf.5']), (os.path.join('share', 'man', 'man8'),
-        ['man/dkimpy-milter.8']), ('etc', ['etc/dkimpy-milter.conf']),
-        (os.path.join('lib', 'systemd', 'system'),
+        ['man/dkimpy-milter.8']), (os.path.join('etc', 'dkimpy-milter'),
+        ['etc/dkimpy-milter.conf']), (os.path.join('lib', 'systemd', 'system'),
         ['system/dkimpy-milter.service']),(os.path.join('etc', 'init.d'),
         ['system/dkimpy-milter']), (os.path.join('etc', 'init.d'),
         ['system/dkimpy-milter.openrc'])],