Upgrading from Dovecot 2.3 to 2.4 - side by side examples

I recently went through the process of upgrading to Debian 13 (Trixie), and by far the hardest part of it was updating my Dovecot configuration. The version of Dovecot included in Trixie (2.4) is not backwards compatible - you'll have to rewrite most of your configuration. For the most part the rewrites are straightforward, but there were two parts that I found particularly tricky - and one of them confounded me so much that I could only resolve it after getting advice from Wietse Venema himself.

In this blog post I'll go over my experience, giving side-by-side examples of what my config looks like before and after the upgrade, and talk a bit about how to configure Dovecot & Postfix to prevent backscatter while maintaining aliasing functionality. But let me give a disclaimer first: I'm merely an email enthusiast - my day job has nothing to do with anything described here, and I only self host because I enjoy the freedom that it affords me. So exercise caution on anything that I recommend here.

Also my email setup is mostly based on the ISPmail guide written by Christoph Haas, with Dovecot set up to work with Postfix and SQL. It has diverged a bit over time, but the Dovecot 2.3 configurations described here are (AFAIK) identical to what his guide describes.

The Configuration

I'll go over the iteration file-by-file as it is structured in the Debian distribution of Dovecot, in the directory /etc/dovecot/conf.d. While upgrading I simply preferred the maintainer's version for all files over mine and then ported my changes.

10-auth.conf

The changes in this file itself are fairly trivial, what was once

disable_plaintext_auth = yes

is now:

auth_allow_cleartext = no

I assume the reason for this change must be to allow better grouping of configuration settings by name. I also have auth_mechanisms set to plain here, and have !include auth-sql.conf.ext uncommented to use SQL for authentication.

The syntax for the SQL settings have changed greatly - the auth-sql.conf.ext file previously pointed to the SQL configuration file /etc/dovecot/dovecot-sql.conf.ext, which I had configured as such:

driver = mysql
connect = \
  host=127.0.0.1 \
  dbname=mailserver \
  user=mailserver \
  password=hunter2
user_query = SELECT email as user, \
  concat('*:bytes=', quota) AS quota_rule, \
  '/var/vmail/%d/%n' AS home, \
  5000 AS uid, 5000 AS gid \
  FROM virtual_users WHERE email='%u'
password_query = SELECT password FROM virtual_users WHERE email='%u'
iterate_query = SELECT email AS user FROM virtual_users

Now the file auth-sql.conf.ext directly contains this information as follows:

sql_driver = mysql

mysql 127.0.0.1 {
  user = mailserver
  password = hunter2
  dbname = mailserver
}

passdb sql {
  query = SELECT password, email AS user FROM virtual_users WHERE email='%{user}'
}

userdb sql {
  query = SELECT email as user, \
    concat(quota, 'B') AS quota_storage_size, \
    '/var/vmail/%{user | domain}/%{user | username}' AS home, \
    5000 AS uid, 5000 AS gid \
    FROM virtual_users WHERE email='%{user}'
  # For using doveadm -A:
  iterate_query = SELECT email AS user FROM virtual_users
}

10-mail.conf

The configurations I had in this file were:

mail_location = maildir:~/Maildir
mail_plugins = quota

These are now:

mail_driver = maildir
mail_home = /var/vmail/%{user|domain}/%{user|username}
mail_path = %{home}/Maildir

mail_plugins {
  quota = yes
}

It's likely that mail_home may not be necessary as the userdb query we saw above points to the home, but I included it nevertheless and didn't bother to test this.

10-master.conf

My configurations in this file did not require change, in both 2.3 and 2.4 versions I have the following:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

10-ssl.conf

Another minor change, what was once:

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.monospace.games/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.monospace.games/privkey.pem

Is now:

ssl = required
ssl_server_cert_file = /etc/letsencrypt/live/mail.monospace.games/fullchain.pem
ssl_server_key_file = /etc/letsencrypt/live/mail.monospace.games/privkey.pem

15-mailboxes.conf

The default configuration shipped with 2.4 is sufficient in this file, I think I had to alter it in the previous version to define the Junk and Trash mailboxes but I'm not certain.

20-imap.conf

Minor change again:

protocol imap {
  mail_plugins = $mail_plugins quota imap_sieve
}

changed to:

protocol imap {
  mail_plugins {
    imap_quota = yes
    imap_sieve = yes
  }
}

20-lmtp.conf

This file is a bit different and relates to both of the tricky parts I mentioned in the intro. In 2.3 I simply had the following:

protocol lmtp {
  mail_plugins = $mail_plugins sieve
}

In 2.4, this file is now:

lmtp_rcpt_check_quota = yes

protocol lmtp {
  mail_plugins {
    quota = yes
    sieve = yes
  }
  postmaster_address = postmaster@monospace.games

  # auth_username_format = %{user | username}
}

Tricky part #1 - the auth_username_format variable is uncommented in the debian distribution by default. As explained in the full file this makes Dovecot strip the domain name before delivery and is not suitable for our SQL userdb. Make sure to comment it out! I overlooked this initially and had to spend some time examining SQL queries in my dovecot logs to figure out what was going on before I realized what was happening.

Tricky part #2 - the lmtp_rcpt_check_quota setting allows Dovecot to check quotas if it is used by Postfix to verify the recipient before enqueuing mail. This allows you to retain advanced alias functionality while doing quota checks without risking backscatter - more on this towards the end of the article.

90-sieve.conf

This file is much more readable in the new version. In the old version my configurations were:

plugin {
  sieve_after = /etc/dovecot/sieve-after

  # From elsewhere to Junk folder
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_before = file:/etc/dovecot/sieve/learn-spam.sieve

  # From Junk folder to elsewhere
  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/etc/dovecot/sieve/learn-ham.sieve

  sieve_pipe_bin_dir = /etc/dovecot/sieve
  sieve_global_extensions = +vnd.dovecot.pipe
  sieve_plugins = sieve_imapsieve sieve_extprograms
}

Now this file looks like:

sieve_script after {
  path = /etc/dovecot/sieve-after
}

sieve_pipe_bin_dir = /etc/dovecot/sieve
sieve_global_extensions {
  vnd.dovecot.pipe = yes
}
sieve_plugins {
  sieve_imapsieve = yes
  sieve_extprograms = yes
}

imapsieve_from Junk {
  sieve_script ham {
    type = before
    cause = copy
    path = /etc/dovecot/sieve/learn-ham.sieve
  }
}
mailbox Junk {
  sieve_script spam {
    type = before
    cause = copy
    path = /etc/dovecot/sieve/learn-spam.sieve
  }
}

90-quota.conf

In 2.3, this file looked like:

plugin {
  quota = maildir:User quota

  quota_status_success = DUNNO
  quota_status_nouser = DUNNO
  quota_status_overquota = "452 4.2.2 Mailbox is full and cannot receive any more emails"
}

service quota-status {
  executable = /usr/lib/dovecot/quota-status -p postfix
  unix_listener /var/spool/postfix/private/quota-status {
    user = postfix
  }
}

plugin {
   quota_warning = storage=95%% quota-warning 95 %u
   quota_warning2 = storage=80%% quota-warning 80 %u
}
service quota-warning {
   executable = script /usr/local/bin/quota-warning.sh
   unix_listener quota-warning {
     user = vmail
     group = vmail
     mode = 0660
   }
}

The 2.4 equivalent is:

quota "User quota" {
  warning warn-95 {
    quota_storage_percentage = 95
    execute quota-warning {
      args = 95 %{user}
    }
  }
  warning warn-80 {
    quota_storage_percentage = 80
    execute quota-warning {
      args = 80 %{user}
    }
  }
}

quota_status_success = DUNNO
quota_status_nouser = DUNNO
quota_status_overquota = "452 4.2.2 Mailbox is full and cannot receive any more emails"

service quota-status {
  executable = quota-status -p postfix
  unix_listener /var/spool/postfix/private/quota-status {
    user = postfix
  }
  client_limit = 1
}

service quota-warning {
   executable = script /usr/local/bin/quota-warning.sh
   unix_listener quota-warning {
     user = vmail
     group = vmail
     mode = 0660
   }
}

The Ninth Configuration - Aliases and Quotas

The quota configuration I described in the 90-quota.conf section provides a quota-status service intended to be used by Postfix before it enqueues mail for delivery (by including check_policy_service unix:private/quota-status in the postfix configuration variable smtpd_recipient_restrictions). This is intended to prevent backscatter - if you don't do this but use quotas your mail server will generate and send bounce emails after the mail is accepted, and since sender addresses can be forged you'll be sending the bounce emails to whoever the sender pretended to be.

But there is one issue with this approach, quoting Wietse Venema: "Implementing 100% correct alias expansion in the SMTP server is not feasible." As a result postfix can not resolve aliases before querying the quota service, and the quota service must say OK to unknown users for aliases to work at all (hence the quota_status_nouser = DUNNO above). But this is still not ideal - it means aliases that resolve to over-quota users can generate backscatter.

It seems one solution for this is to create dummy user accounts for aliases that have the same quotas and home directories as the users that they resolve to, but this is a fairly crude solution and does not work for more complex use cases such as catchall aliases or aliases pointing to aliases.

I banged my head at this issue for a painful amount of time and ended up deciding to ask about it on the postfix-users mailing list. To my surprise I received a very quick response from Wietse where he described how to use the reject_unverified_recipient restriction of postfix, which you can read here.

It took me some time to implement this solution, partly because I was initially not aware of the lmtp_rcpt_check_quota Dovecot setting, and partly because in my tests I failed to account for the caching behavior of postfix. Postfix caches the result of recipient verifications for performance purposes - and the low-quota user I had set up for testing was initially cached as OK to deliver to, likely because I had not turned the lmtp_rcpt_check_quota setting on yet. Only after carefully going through my logs did I realize that the cache database lived in /var/lib/postfix/verify_cache.db, and after deleting this I could finally verify that reject_unverified_recipient was working as intended.

For the record I made sure to thank Mr Venema after his response, but in my excitement I responded to him directly rather than to the list, so the following exchange ended up being private.

Thanks for reading, and I hope this has been useful!

Miscellaneous Notes on Upgrading to Trixie

Do not upgrade from an emacs shell buffer. The prompt asking how to handle altered config files for the dovecot package does not handle the failure to bring up a TUI gracefully. This is not a universal problem - e.g. nginx can prompt in an emacs shell buffer just fine.

The output of the file command on encrypted files has changed from GPG to PGP.

Upgrading Python breaks virtual environments. I didn't know this and had to YOLO update my backend code.

In emacs, emms-player-mpv stopped working. I ended up just switching to emms-player-vlc.