Notmuch of a mail setup Part 1- mbsync, msmtp and systemd

Overview

Some months back, I discovered the Notmuch mail reader. It’s a pretty sweet piece of software based on Xapian to index and tag your mail. As of today, the email setup described here has displaced my Google Apps hosted Gmail for my personal domain. My goal here is to document how I’ve setup my mail workflow.

As it says in the name, Notmuch is not much of a mail client. It doesn’t get any mail, send any mail, or even display any mail. All it does is to provide an interface to a Xapian index for your mail. We need to provide other software that will help us handle the other pieces.

On one of my computers I’m using Arch Linux. I’ve tried just about every Linux distribution in the past fourteen years and have found Arch to be a pretty good sweet spot for a workstation. The distro has stabilized in recent years and a system upgrade seems to be less likely to completely bork your computer. I prefer debian stable on the server since it’s so well tested. The instructions here might be a bit Arch-Centric. I’m going to discuss sending and receiving mail before discussing how to configure Notmuch. Sending and receiving mail are critical for just about every GNU/Linux mail client and most of this should be generally applicable.

Getting the mail

My mail lives on an IMAP server somewhere in cloud-land. I’ve had an account on this machine for several years, but I’ve recently forwarded all of the mail for my domains to this server. There are several good IMAP clients these days that will synchronize the mail from your local computer with your mail on the IMAP server.

Among the ones that I’ve tried in the past are:

My mail setup requires the use of STARTTLS, so Getmail and OfflineIMAP are out of the question. They both support STARTTLS, but they’re using a Python library that doesn’t perform any verification of the certificate. This leaves me open to a MITM attack where someone upstream can present a fake certificate and proxy my traffic. I ended up using MBSync to fetch my mail from the server.

The documentation for MBSync is pretty terse, but there’s a nice page at the ArchLinux wiki.. I’ve included a configuration file that works for my needs.

 =~/.mbsyncrc= ##

# Address to connect to
Host mail.servername.info
Port 143
User BostonEnginerd
PassCmd "gnome-keyring-query get servername\\_mail"
SSLType starttls
SSLVersions TLSv1.2
SystemCertificates no # The following line should work. If get
certificate errors, uncomment the two following lines and read the
"Troubleshooting" section.


CertificateFile ~/.cert/mail.servername.pem


IMAPStore servername-remote
Account servername

MaildirStore servername-local
# The trailing "/" is important
Path ~/.mail/servername/
Inbox ~/.mail/servernamw/Inbox

Channel servername
Master :servername-remote:
Slave :servername-local:
Patterns *
# Automatically create missing mailboxes, both locally and on the server
Create Both
# Save the synchronization state files in the relevant directory
SyncState *

You will need to get the ssl certificate from your server. If it’s a “real” SSL certificate signed by a CA, including the CA certificate should be sufficient. If it’s self-signed, it’s easiest to get the certificate from the server and save it to a file. The commandline openssl command can do that for you:

openssl s_client -starttls imap -connect some.imap.server:port -showcerts

I just copied and pasted from the ——-BEGIN CERTIFICATE—– to the END CERTIFICATE line into a file and referred to that in the configuration.

I’m using the gnome-keyring-daemon to store the mail password so it’s not sitting on my hard disk in plaintext. There’s a program in the AUR called gnome-keyring-query which allows you to interact with the keyring on the command line. The get command gets the requested key. There is a set command as well for storing a password. One of these days, I might try to setup something that uses a certificate to authenticate the client.

Sending the Mail

The second most important thing that a mail client can do is send mail. I’m using the SMTP server on my host, again using STARTTLS. To send the mail, I decided to use a program called msmtp which implements a small send-only smtp client. The ArchLinux wiki has a nice tutorial on using this also.

After installing msmtp, you need to configure it. A working config file is below:

`~/.msmtprc`

# Set default values for all following accounts.
defaults
tls on
tls_starttls on
tls_fingerprint AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:
logfile ~/.msmtp.log

# A freemail service
account myserver
host my.server.tld
port 9000

from bostonenginerd@myserver
auth on
user myname
passwordeval gnome-keyring-query get myserver.smtp

# A second mail address at the same freemail service
account myotheraccountname : myserver
from myname@myotherdomain

# Set a default account
account default : myserver

I get the TLS fingerprint from the server using the openssl s_client. Again, I’m using the gnome-keyring to store the passwords encrypted on the disk.

Once msmtp is working you can send mail when your computer is online.

A nice feature would be to queue messages when the network connection is down and send them when you come back online. Fortunately recent versions of msmtp come with a queue management script. On Arch, there are two included. I’m using the one called msmtpqueue which includes three scripts - msmtp-enqueue.sh, msmtp-listqueue.sh and msmtp-runqueue.sh. These scripts pretty much do what they say. Copy them to somewhere in your path.

Using the scripts requires that you have a directory called ~/.msmtpqueue. The email will be stored in this directory when you use the msmtp-enqueue.sh command in your mail client. To send all the mail, you can use the msmtp-runqueue.sh script when you’re connected to the internet. In the past, I’ve had problems sending too many messages too quickly. If you encounter this, you can put a sleep command in the msmtp-runqueue.sh script.

Sending and Receiving Automatically

Now that we’re able to receive and send mail from the command line, it would be good to automate the process. I put together a hacked up script that checks if we’re connected to the Internet, sends all of the mail sitting in the queue and pulls in new messages. nmcli interacts with the NetworkManager daemon to query the state of the internet connection. Possible answers are none, portal, limited and full. We only want to go through the trouble of trying to send messages when we have full connectivity.

`~/scripts/checkmail.sh`

#!/bin/sh

STATE=`nmcli networking connectivity`

if [ $STATE = 'full' ]
then
    /usr/local/bin/msmtp-runqueue.sh
    mbsync -qq accountname
    exit 0
fi
echo "No internet connection."
exit 0

Now that we have this together, it would be awfully nice if my computer would run this script periodically. There are two ways to do this now. One way is to use cron. This is the standard way to accomplish something like this.

I thought that I would take the opportunity to play with what seems to be the new init system for major Linux distributions – systemd. I haven’t really been following all the drama surrounding this project, but it’s the new init system for Arch, Debian and Fedora. I’ve noticed that computers with systemd tend to boot substantially faster than the initV based systems. I was inspired to do this based on a blog post that Joey Hess wrote to document building himself an alarm clock with systemd.

I wanted to check for new mail and send old mail every fifteen minutes. To do this requires that I create two files checkmail.service and checkmail.timer in the ~/.config/systemd/user directory.

`checkmail.service`

[Unit]
Description=check mail
RefuseManualStart=no
RefuseManualStop=yes

[Service]
Type=oneshot
ExecStart=/home/bostonenginerd/scripts/checkmail.sh

This file defines what the service does. Type=oneshot indicates that this is not a daemon and will just run and exit. Setting the RefuseManualStop=yes variable will make systemctl unable to stop the service. In this case, that’s fine because the script will teriminate when it’s done running. Now we need to setup the schedule for running the job.

`checkmail.timer`

[Unit]
Description=Check Mail every fifteen minutes
RefuseManualStart=no
RefuseManualStop=no

[Timer]
Persistent=false
OnBootSec= 5min
OnUnitActiveSec= 15min
Unit=checkmail.service

[Install]
WantedBy=default.target

In this file, we’re setting RefuseManualStop=no because this will be a long running service which we will can terminate at some point. In the [Timer] section, we set OnBootSec to five minutes. This means that the checkmail.service job will not run until five minutes after the computer boots up. I put this in to wait some time for a network connection to come up before trying to get the mail. The OnUnitActiveSec variable control the time delay in between repeats of the job. Here it’s set to 15 minutes, but could be set for any interval.

To start the timer, you should type systemctl --user start checkmail.timer. Using the command systemctl --user status checkmail.timer will allow you to see if the timer was successfully started. To start the timer automatically, issue the systemctl --user start checkmail.timer command.

Another neat command is systemctl --user list-timers. This will list all of the timers that you have active and how long until they’re triggered again. Neat stuff. Systemd is pretty clever.

Ideally we could simplify our script and add a systemd dependency to the service that requires that the internet be active before running the script. I’d love to hear any ideas on how to do that, or how to do this more efficiently.

End Part 1

I’ll finish up the Notmuch specific portions of my email setup at a later date.