v0.0.7

version: v0.0.7
date: 2023-09-24

Update instructions:

Due to a bug with "expunging" (deleting) messages that were
junk-filter-trained, messages that were removed from disk could be resurrected,
causing errors when mox would later try to open such messages again. Before
upgrading, you should first check and resolve this problem:

• Find missing files by running: mox-v0.0.6 fixmsgsize
• Create empty replacements for the missing files (use "touch data/accounts/...").
• Update the message metadata in the database for the newly created files:
  mox-v0.0.6 fixmsgsize
  (this will fix the previously missing files, but should no longer find new
  missing files)
• Fix per-mailbox message/unread counts for each affected account, run:
  mox-v0.0.6 recalculatemailboxcounts
  ("account" is the name as it appears in the data/accounts/ directory,
  not necessarily an email address)
• The affected accounts will see the recreated empty messages, which
  can be deleted.

Now with the storage consistency resolved, you can start with the upgrade. Mox
v0.0.7 adds message threading, and all messages will be read and assigned a
message thread. This is done in the background, in two steps, and may take a
while. The first step adds Message-ID and a "thread base subject" to each
message in the database. The second step reads through all messages and uses
their References/In-Reply-To/Subject message headers to match threads. In
testing, upgrading took approximately 1 minute per 100k messages, but it will
depend on the hardware. Accounts are available for reading and delivery during
the upgrade, but the webmail may tell you that threading is not yet available.

You should do a dry-run of the upgrade first:

• Make a temporary backup with the old mox version:
  mox-v0.0.6 backup data/tmp/testupgrade
• Verify that all is well with the old version:
  mox-v0.0.6 verifydata data/tmp/testupgrade
  NOTE: If you still get an error about a message with the wrong size, see
  https://github.com/mjl-/mox/issues/71 for a fix.
• Trigger the threading upgrade using the new version:
  mox-v0.0.7 openaccounts data/tmp/testupgrade
  (it prints nothing until done; the live upgrade prints progress)
• Verify the new state:
  mox-v0.0.7 verifydata data/tmp/testupgrade

With a successful dry-run, the upgrade should go smoothly. Make a new backup
with mox-v0.0.6 backup data/tmp/backup (the previous backup used for the
dry-run has been modified, so couldn't be used to restore!), replace the binary
and restart.

After the upgrade, any accounts affected by the "missing file" problem should
get their "uid validity" increased, so IMAP clients will resynchronize. Run
"mox-v0.0.7 bumpuidvalidity " for each affected account. The change is
made directly in the database file, so no IMAP/webmail sessions should be
active for the account. If any sessions are active, the database file is locked
and "bumpuidvalidity" will print a timeout error. Briefly shutting down mox is
an option, the bumpuidvalidity command operates quickly.

Apologies for the inconvenience, mox aims to make administrating a mail server
easier than this.

If you run into any problems, please create an issue.

New features:

• Keep track of message threading, and add a threaded view to the webmail. Newly
  delivered messages are matched against threads, as are imported messages. The
  message threading is currently only exposed through the webmail client, not yet
  through IMAP. The webmail has new keyboard shortcuts to navigate between
  threads, collapse/expand them (with state remembered), or mute them (so new
  deliveries to the thread are marked as read).
• Add option to accept DMARC and TLS reports for other domains. You would add
  the other domain in mox, and specify an alternative domain (of your regular
  dmarc/tls reporting address) for the recipient of the reports. The webadmin
  DNS-check page will show the required DNS records for delegated reporting.
  Incoming reports will be delivered to the configured reporting address, and
  processed by mox.
• Easier account setup on Apple devices without device management profiles
  (.mobileconfig files). Download a profile from the account web interface, at an
  email address. Or open the QR-code to easily get the profile on a mobile device.
  For issue #65 by x8x.
• Transparent gzip compression when webserving files for selected content-types
  that are likely compressible. Both for static files (with a cache of compressed
  files, max 512MB stored in $datadir/tmp/httpstaticcompresscache), and for
  forwarded requests (gzip level "fast"). Compression can be enabled per handler.

Improvements:

• In SMTP server, for submission (with authenticated clients), don't fail on a
  bad domain/IP address in the EHLO command. With submission the domain/IP is
  irrelevant, and clients often fill in something that isn't strictly correct. No
  need to prevent those users from submitting email (except in pedantic mode).
  For issue #55 reported by gimpf.
• At top of config files, mention the config file format, and hints to prevent
  likely mistakes (given how sconf is different from what admins may be used to),
  including a pointer to the sconf documentation page. For issue #56 reported by
  kikoreis.
• Recognize when quickstart is probably run behind a NAT, possibly a container,
  and set the "NATIPs" field of the "public" listener in mox.conf accordingly.
  This triggers when you set up mox for a public domain, but only have
  private/loopback IPs on the machine. For issue #59 reported by pmarini.
• When moving a message out of Rejects mailbox, mark it as unread. For issue #63
  by x8x.
• The "mox setaccountpassword" subcommand now takes an account name as
  parameter instead of an email address. The email address could be confusing
  in the face of wildcard addresses and a typo/non-explicitly-created address:
  The account holding the wildcard address would get a new password. For issue
  #68 by x8x.
• Make Mac OS X Mail use the special-use mailbox attributes so it finds the
  correct "Sent" and "Trash" mailboxes (instead of creating its own "Sent
  Messages" and "Deleted Messages" mailboxes). The IMAP server now
  unconditionally sends the special-use flags, even if the mail client doesn't
  request it. For issue #66 by x8x.

Bug fixes:

• Fix expunging of messages marked junk/nonjunk. The messages would be marked
  as expunged, then the junkfilter would retrain and clear the expunged field
  again. This would cause the message to be resurrected while the on-disk message
  file was already removed. Trying to read such messages would fail. The update
  instructions should help fix the problem.
• Fix "mox sendmail" when submitting over a TLS connection by setting the remote
  host name to verify the certificate of. Due to a logic bug the name wasn't set
  and the connection would fail due to the missing setting.
• Don't generate duplicate suggested SPF record if hostname is equal to domain
  name, e.g. [email protected]. Fixes issue #46 reported by x8x.
• Fix showing attachments of type text/plain in webmail, they weren't shown at
  all because they were skipped when parsing the message during webmail message
  processing.
• Fix parsing the List-Post header in messages, for use in webmail when replying
  to a mailing list message.
• When moving a message to the mailbox that has the special-use "junk" flag,
  mark the message as junk and retrain. This should have already worked just like
  the "AutomaticJunkFlags" config option, and the default account config already
  handles marking messages as junk based on that option, but the special-use flag
  should be recognized independently and now also takes precedence.
• Set the correct special-use mailbox flag "\Drafts" instead of "\Draft" on the
  draft mailbox. Mail clients may have not found the correct drafts mailbox
  before. For issue #66 by x8x.
• Fix "mox bumpuidvalidity" to not create a (mostly harmless) uidvalidity
  inconsistency that "mox verifydata" will warn about. For issue #61 by x8x.
• And quite a few smaller tweaks/improvements/fixes.

Special thanks for contributions and/or feedback from: x8x, gimpf, kikoreis,
pmarini, fairking, gedw99, hmfaysal (and those I missed).

Feedback, requests, bug reports, contributions (start small!) are all welcome.

Good news: Mox is now being funded for a year of continued development through
the NGI0 Entrust Fund, a fund established by NLnet with financial support from
the European Commission's Next Generation Internet programme,
https://nlnet.nl/project/Mox/!