Cyrus IMAP 2.5.0 Release Notes — Cyrus IMAP 3.12.2 documentation

# Cyrus IMAP 2.5.0 Release Notes

Important

Make sure to read the Upgrading to Cyrus IMAP 2.5.0 notes (all of them).

- New Features

- Configuration Option Changes and Enhancements

- Development: Switch to autoconf and libtool

- Development: Switch to Phabricator

- Development: Sphinx for Documentation

Download from GitHub:

> - https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.0/cyrus-imapd-2.5.0.tar.gz

> - https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.0/cyrus-imapd-2.5.0.tar.gz.sig

## Upgrading to Cyrus IMAP 2.5.0

It is strongly recommended to shut down the Cyrus IMAP services before performing the upgrade, as the newer binaries will end up writing to `mailboxes.db` in a way that is not compatible with the older binaries (that would otherwise still be running).

You can start the server immediately after upgrading, however.

### Underscores in `cmd` Names in cyrus.conf(5)

Underscores (the `_` character) are no longer allowed in the `START`, `SERVICES` and `EVENTS` sections of cyrus.conf(5), as they interfere with configuration options in imapd.conf(5) being prefixed by service names and an underscore (`_`) character.

### Database Format Upgrade for Mailboxes

Upgrading to Cyrus IMAP 2.5.0 recommends the upgrade of database formats, for which the reconstruct utility has been modified allowing administrators to run:

    # /usr/lib/cyrus-imapd/reconstruct -V max [options]

This command can be run at the administrator's convenience, and while running the database format upgrade process is not strictly required, it is certainly strongly recommended.

Releases prior to 2.5.0, namely the 2.4 series, upgraded `cyrus.index` database formats in-line (i.e. as the mailbox in question as being used). This may have caused an I/O storm in some situations, which 2.5.0 aims to address.

The minor version of the `cyrus.index` file in 2.5.0 is 13. While mailboxes with an older format will continue to work "forever", newer features such as annotation quotas will fail, as there is no space in the database formats prior to version 13 to store the new feature's information.

Until you upgrade the database format, you may experience slightly inconsistent search results, due to the `cyrus.cache` format in Cyrus IMAP versions prior to 2.4.0 running words together and not obeying word boundaries (as well as versions in the 2.4 and 2.5 series).

If you were running Cyrus 2.3.x with `expunge_mode: delayed`, then the contents of the `cyrus.expunge` file will be removed the first time the mailbox is opened, and the mailbox will act as if `expunge_mode` was set to `immediate` -- until it is upgraded.

This is because version 2.3 mailboxes used the `cyrus.expunge` file, and it's not worth the complexity to try to recreate that file.

This upgrade path supports upgrades from Cyrus IMAP version 2.2.12 and later versions.

Important

The above command can run while the Cyrus IMAP services are in operation, and does not interfere with the service's operations.

Note

Additional options can be specified to reconstruct(8) to more granularly specify which mailboxes' `cyrus.index` is to be upgraded.

### Quota Fixes and Enhancements

After all mailboxes are upgraded, you should run `quota -f` to make all message and folder counts are correct. The new quota types that Cyrus IMAP 2.5.0 supports require the counts to need to be accurate.

### Cyrus IMAP Murder Topologies

Environments that run a Cyrus IMAP Murder topology will want to upgrade their backends before they upgrade their frontends.

### Greater Memory Footprint

For those upgrading from 2.3.X; newer releases of Cyrus IMAP will use significantly more memory per selected mailbox. This is not an error or bug; it's a feature. The newer code is holding more data and metadata in memory for purposes of faster access to more of the mailbox. This is not a memory leak.

## New Features

### Index Namelock Release

Long-running (idling) connections may have previously intervened with mailbox deletions and general cleanup work, resulting in one connection blocking another from, for example, deleting and recreating a mailbox:

    1C1: SELECT "Foo"
    2C2: DELETE "Foo"
    3C2: CREATE "Foo"

The `CREATE` command on line 3 (by client 2, while client 1 still has the `Foo` mailbox selected) would have failed before, but does now succeed.

### Extended Quota Types

New ways to restrict resource usage:

- Number of Folders,

- Number of Messages,

- Number of Annotations

### CalDAV and CardDAV Support

CalDAV and CardDAV support no longer live out-of-tree in the same GIT repository, but are now mainstream and included in Cyrus IMAP 2.5.0.

This has been a major effort by Carnegie Mellon University and FastMail, and continues to be, enriching the experience of Cyrus IMAP users globally.

If upgrading to Cyrus IMAP 2.5 from one of the 2.4.17-caldav-beta releases, you MUST run the `dav_reconstruct` utility for each of your CalDAV users.

### Support for RFC 5464: IMAP METADATA

Cyrus IMAP now fully supports RFC 5464, The IMAP METADATA Extension.

This also means the support for the ANNOTATEMORE draft for IMAP will ultimately be dropped.

Aside from the trusted folder metadata, this also introduces message annotations. Users will need to be given the `n` right to allow them to set message annotations.

### Event Notifications

Various events occuring in Cyrus IMAP, among which mailbox, message and access events, can now be pushed out through a side-channel and notify client applications or provide other infrastructure with detailed information.

### Mailbox Distribution Enhancements: Backend and Partition Selection

Thanks to the work of Julien Coloos and colleagues, a new mode is available for server and partition selection upon mailbox creation.

Prior to Cyrus IMAP 2.5.0, the server and/or partition on which to create a new mailbox was selected by detecting the largest amount of absolute free disk space on all servers and partitions. The mailbox distribution feature allows for more intelligent and flexible routines to be used in the selection. Please see our Administrator Guide for more details.

### New Database Format for `mailboxes.db`

The database format for mailboxes.db has been upgraded, adding;

- A new mailbox type for deleted mailboxes.

  In versions of Cyrus IMAP prior to 2.5.0, mailboxes that were deleted may have become unavailable for actual cleanup expecting another session on the same mailbox to clean up the directories and files. See also Index Namelock Release.

- A key-value storage format is used, allowing for faster and better parsing of `mailboxes.db`, more granular updates to runtime environments, and more sustainable future upgrades.

### New Database Format `twoskip`

A new database format has been added, called `twoskip` [1].

This new database format is reputedly better, faster, safer and 64-bit capable, as opposed to our former favorite `skiplist`.

`twoskip` can be used for the following databases:

- `annotation_db`

- `duplicate_db`

- `mboxkey_db`

- `mboxlist_db`

- `ptscache_db`

- `quota_db`

- `seenstate_db`

- `subscription_db`

- `statuscache_db`

- `tls_sessions_db`

- `user_deny_db`

### Miscellaneous

#### Allowing Undefined Annotations

Cyrus IMAP 2.5.0 allows administrators to configure that undefined annotations should be allowed, using a new `annotation_allow_undefined` setting in imapd.conf(5).

#### Catchall Mailbox for LMTP

Thanks to the work by Carsten Hoeger and Ralf Haferkamp, this new feature enables administrators to configure a target mailbox for mail delivered through LMTP to targetted mailboxes that do not exist.

For example, a mail that LMTP would deliver to `user/bovik`, which for the sake of argument does not exist in this example, setting `lmtp_catchall_mailbox` to `admin` will instead deliver the mail to `user/admin`.

Note

Mailbox name, not Email Address

Note that lmtp_catchall_mailbox must be a user mailbox name, not an email address. Also note that the user/ namespace indicator as well as the hierarchy separator are to be omitted.

##### Does this impact lmtp_fuzzy_mailbox_match?

Environments that have `lmtp_fuzzy_mailbox_match` enabled, in order to have LMTP seek from the targetted, non-existent mailbox sub-folder (example: `user/bovik/spam/probably`) all the way to the toplevel mailbox folder (i.e. `user/bovik`) until it finds a mailbox (sub-)folder that does exist (example: `user/bovik/spam`), are not impacted by this setting.

##### Can the lmtp_catchall_mailbox include the path to a sub-folder of a target mailbox?

UNCONFIRMED

##### Can the lmtp_catchall_mailbox be a shared folder?

UNCONFIRMED

#### Callout for SETMETADATA

A callout program can be called when annotations are set, configured through `annotation_callout`.

#### Host & User Login Restrictions

Logins can now be restricted on a per host (source IP address) or per user basis, using the settings `maxlogins_per_host` and `maxlogins_per_user`.

## Configuration Option Changes and Enhancements

Important

While it is not mandatory to update your configuration file with these new settings, not doing so may have undesired side-effects, including but not limited to deprecation warnings in log messages.

### Option Name Changes for `autocreate`

The options related to automatic creation of user mailboxes and sub-folders (aka. autocreate) have been changed to hold a prefix of `autocreate_`.

The following autocreate options are now available:

autocreate_inbox_folders (was: `autocreateinboxfolders`)

> `autocreate_inbox_folders` controls which folders to create in addition to the INBOX folder.

> Separate the folder names by `|`.

autocreate_post (was: `createonpost`)

> Controls whether or not to create a folder when a message is first posted to it (by LTMP).

autocreate_quota (was: `autocreatequota`)

> When creating a user mailbox, set the quota for that mailbox to the value of this configuration option.

autocreate_quota_messages (not available)

> When creating a user mailbox, set the message quota (maximum number of messages allowed in the folder hierarchy) to the value of this configuration option.

autocreate_sieve_folders (was: `autosievefolders`)

> Limit the folders that can be created automatically by a Sieve script performing a "fileinto" action, to the folders listed in this configuration option.

> Separate the folder names by `|`.

autocreate_sieve_script (unchanged)

> When creating a user mailbox, associate the Sieve script configured here.

autocreate_sieve_script_compile (was: `generate_compiled_sieve_script`)

> Whether or not to compile the Sieve script configured by `autocreate_sieve_script`.

autocreate_sieve_script_compiled (was: `autocreate_sieve_compiled_script`)

> When creating a user mailbox, associate the already compiled Sieve script configured here.

autocreate_subscribe_folders (was: `autosubscribeinboxfolders`)

> List the folder names to which the user for which a mailbox is being created should be subscribed.

> Note

> All folders listed here are considered to reside in the personal namespace.

> Separate the folder names by `|`.

autocreate_subscribe_sharedfolders (was: `autosubscribesharedfolders`)

> List the folder names of shared folders to which the user for which a mailbox is being automatically created should be subscribed.

> Separate the folder names by `|`.

autocreate_subscribe_sharedfolders_all (was: `autosubscribe_all_sharedfolders`)

> Rather than subscribe the user for which a mailbox is being automatically created to some shared folders, simply subscribe the user to all shared folders.

autocreate_users (unchanged)

> Limit the users for which mailboxes may be created to the list configured here.

### Default Change: `delete_mode`

The default for the imapd.conf(5) configuration option `delete_mode` has changed from `immediate` to `delayed`.

This causes mail folders that are deleted by a client to not immediately dissappear from the filesystem. Instead, they are renamed to a deleted namespace that is visible only to administrators.

A separate job `cyr_expire -D $x` is to be included in the master service configuration file cyrus.conf(5), specifically in the EVENTS section. `$x` is a number of days to keep already deleted folders.

Example section of :cyrusman:`cyrus.conf(5)`

>     EVENTS {
>         deleteprune cmd="cyr_expire -D 69" at=0430
>     }

In the aforementioned example, folders are purged from the filesystem only after 2 times 31 plus 7 days, corresponding with 2 cycles of a monthly (full, virtual) backup of which one might fail.

### Default Change: `expunge_mode`

The default for the imapd.conf(5) configuration option `expunge_mode` has changed from `default` to `delayed`.

This causes the mail message files associated with messages that are flagged as Deleted in a folder that is subsequently expunged, or individual messages that are expunged, to not be removed from the filesystem directly.

A separate job `cyr_expire -X $x` is to be included in the master service configuration file cyrus.conf(5), specifically in the EVENTS section. `$x` is a number of days to keep the message files on the filesystem.

Example section of :cyrusman:`cyrus.conf(5)`

>     EVENTS {
>         expungeprune cmd="cyr_expire -X 69" at=0430
>     }

In the aforementioned example, message files are purged from the filesystem only after 2 times 31 plus 7 days, corresponding with 2 cycles of a monthly (full, virtual) backup of which one might fail.

### Option Name Changes for `ldap_tls_*`

Configuration option names for LDAP SSL/TLS configuration in imapd.conf(5) have been changed:

ldap_ca_dir (was: `ldap_tls_cacert_dir`)

ldap_ca_file (was: `ldap_tls_cacert_file`)

ldap_client_cert (was: `ldap_tls_cert`)

ldap_verify_peer (was: `ldap_tls_check_peer`)

ldap_ciphers (was: `ldap_tls_ciphers`)

ldap_client_key (was: `ldap_tls_key`)

### Option Name Changes for `tls_*`

Configuration option names for SSL/TLS configuration in imapd.conf(5) have been changed to better reflect how they are used, as enhancements would otherwise create great confusion.

tls_client_ca_dir (was: `tls_ca_path`)

tls_client_ca_file (was: `tls_ca_file`)

> The former `tls_ca_*` configuration options specified one or more SSL Certificate Authority certificates against which SSL certificates offered by clients could be verified.

> In a Cyrus IMAP Murder topology however, Cyrus IMAP servers themselves become clients of other Cyrus IMAP servers, but may not have been issued certificates under the same verification chain.

With the (too) generic names for `tls_ca_*` configuration options out of the way, Cyrus IMAP 2.5.0 adds the following configuration options:

tls_server_cert (was: `tls_cert_file`)

tls_server_key (was: `tls_key_file`)

> Server SSL certificate and key to use for connections from clients.

tls_sessions_db (was: `tlscache_db`)

tls_sessions_db_path (was: `tlscache_db_path`)

### New Options for `tls_*`

tls_client_cert (<none>)

tls_client_key (<none>)

> Client SSL certificate and key to use when cyrus-imapd behaves as a client (to other cyrus-imapd server (instances)).

tls_client_ca_file (<none>)

tls_client_ca_dir (<none>)

> Certificate Authority file or directory used to verify client SSL certificates.

tls_client_certs (`optional`)

> Disable (`off`), allow (`optional`) or require (`require`) clients authenticate with an SSL certificate.

tls_server_ca_file (<none>)

tls_server_ca_dir (<none>)

> Certificate Authority file or directory used to verify SSL certificates offered by other servers.

tls_compression (`0`)

> Enable TLS compression. Disabled by default.

tls_eccurve (`prime256v1`)

> Select the elliptic curve used for ECDHE. See openssl ecparams -list_curves for supported values on your platform.

tls_prefer_server_ciphers (`0`)

> Prefer the cipher order configured on the server-side.

tls_versions (`ssl2 ssl3 tls1_0 tls1_1 tls1_2`)

> Disable SSL/TLS protocols not in this list.

## Development: Switch to `autoconf` and `libtool`

With the release of Cyrus IMAP 2.5.0, the Cyrus IMAP project has switched to using autoconf and libtool.

## Development: Switch to Phabricator

An instance of Phabricator is going to be replacing our old Bugzilla.

We believe this better facilitates our processes, and will make it easier to contribute code and collaborate.

Please see https://git.cyrus.foundation/.

Note

The Phabricator installation suffered a disk failure. We now use GitHub for code and bug management. https://github.com/cyrusimap/cyrus-imapd

## Development: Sphinx for Documentation

While a work in progress still, you're looking at the new and improved documentation effort for the Cyrus project as a whole.

This documentation is written in reStructuredText, and rendered by Sphinx.

The GIT repository for the documentation is at

Footnotes

[1]

http://opera.brong.fastmail.fm.user.fm/talks/twoskip/twoskip-yapc12.pdf