Take Control of your Email with Mutt, OfflineIMAP and Notmuch

mutt-1I’ve been an avid user of the Mutt email client for quite some time, preferring it to GUI clients or web for its speed, hotkey-everything approach, and integration with vim that compliment my daily workflow.  Still, I found search functionality lacking compared to Thunderbird or other GUI clients.  Mutt has built-in, powerful regex search but I sought something more intuitive – I could never remember the arcane syntax for it to be timely but I didn’t want to bother with a cumbersome GUI for searching and indexing.  Here is how I combined Mutt, OfflineIMAP and Notmuch into a powerful email/search system to take back my inbox.

A few years ago a co-worker introduced me to a blog post that explained how a marriage of Mutt and the exhaustive indexing and searching functionality of notmuch could give me what I was lacking.  I’ve found tremendous gains in productivity, speed and efficiency with this setup.  I get a ridiculous amount of email through open source project mailing lists, team and company mailing lists and other sources and it’s basically the only way I can keep up.

It’s not for everyone, but if you find that using keyboard hotkeys over kludgy mouse clicks is preferable but you’ve always avoided a console email client like Mutt, Alpine or Elm because of lacking search/indexing functionality then you may find it useful.  If you find yourself using Thunderbird or another GUI client just for the search functionality this might be useful.

Another benefit is that using a console email and a console IRC client (I like weechat) I can do 100% of my work in a screen session, basically being able to work without a laptop in a pinch and only a terminal.  While that may not mean much to most people it makes me feel more like Caine from Kung-Fu.

Please note that you must be running a recent Linux distribution due to the libraries required.  This should also work on any operating system where mutt runs natively (FreeBSD, OSX, etc).

The Goldberg Machine of Awesome: List of Components.

1) Mutt (with sidepane patch)
The Mutt MUA (mail user agent) was created by Pilgrims on the voyage of the Mayflower to the Americas, though the concept of e-mail, internet and correspondence outside of pidgeons and quill pens upon parchment had not yet been invented.
One can use the Mutt MUA without the sidepane patch, but I find it extremely useful .. especially if you have lots of IMAP folders to manage like me.  Mutt is very good at handling messages with multiple threads and mailing lists, it is also extremely fast to use – it is keyboard/console only with more customization and configuration options than you’d ever want.

Below you’ll see the sidepane and a common business email compose window

mutt-sidepane1mutt-compose

2) OfflineIMAP
OfflineIMAP is like a grocery store delivery service – it keeps your fridge stocked with food without you having do anything.  OfflineIMAP fetches mail on set intervals from IMAP and keeps it on your local filesystem.  This has two major benefits:

  • offline email composition and reading, syncs remotely when its run
  • sets up an easy way for local indexing and search

It’s been noted that you can replace OfflineIMAP with mbsync/isync or any similiar IMAP sync program if this works better for you.  This is a modular setup so you can pick other components to use if you like (some people prefer emacs to vim for example).  I’ve never had issues with OfflineIMAP but you’ve got other choices if you wish.

3) Notmuch
Notmuch is an email indexing/tagging application that basically gives you your own Google Search appliance for your email.  It can find pretty much anything except for Jimmy Hoffa.  With the “mutt-notmuch” script you can plug this functionality into Mutt via a configurable hotkey  (I use F8).  Below you can see a common search.

mutt-search

4) Mutt-Notmuch Script
(Perl)
Useful things in life like electricity, plutonium and velcro are created with Perl, mutt-notmuch is no different.  This useful script was introduced to me by a useful blog post that basically echoes the exact type of usage I’m explaining here.  You define mutt-notmuch execution as a hotkey inside your .muttrc and call it to search mail after the initial setup.

Putting it all together – How to set things up

Note: I’ve switched to neomutt, updated below.  neomutt is a well-maintained fork of mutt that contains lots of newer patches and functionality that’s not in upstream mutt yet, and it’s also extremely well-documented.  Highly recommended!

1a) Install mutt or neomutt (sidepane patch optional but encouraged)
If you opt to use the sidepane patch you’ll need to compile your own mutt from source unless you use Karel Zak’s mutt-kz fork neomutt which includes everything.  If you use an RPM-based Linux distribution it’s easier to just use any number of Fedora COPR Repos.  I am focusing on Fedora 23+ here.

dnf copr enable flatcap/neomutt
dnf install neomutt

1b) Configure Mutt:  .muttrc
Mutt is a very flexible, configurable MUA – perhaps the most configurable in existence.  Like the magical .vimrc, the .muttrc file can turn into a work of art as there are near-unlimited levels of customization and complexity one can harness.  If you are not already a mutt user and have one I’d recommend borrowing heavily from someone else.  The default one is fine for your distribution is fine too and you can build on it as needed later.

I originally borrowed mine from ashcrow and modified it heavily – you can find my sanitized .muttrc here for reference.  The default .muttrc is well commented and you can start with that but there are just a few variables you’ll need for bare-bones operation with notmuch and offlineimap.

There are two important variables below, setting your mailbox format to “maildir” and the necessary settings for including the mutt-notmuch script and binding to a hotkey. Your paths may vary.   I also have example mailcap (associate opening attachments with external programs) and mutt-gpg config files here.

— snip  .muttrc —

### save and sync my sent messages
set record=~/Maildir/Sent
set spoolfile=~/Maildir/INBOX
# set header cache
set header_cache=~/Localmail/hcache
# needed for maildir format
set mbox_type=Maildir
set folder=~/Maildir/
# IMAP and INBOX (this should map to any existing IMAP folders)
mailboxes =INBOX \
=Sent \
=Drafts \
=Junk \
=Trash \
=Lastfolder
### BEGIN NOTMUCH-MUTT SETTINGS
    macro index  \
          "unset wait_key~/.mutt/mutt-notmuch.pl --prompt search~/.cache/mutt_results" \
          "search mail (using notmuch)"
    macro index  \
          "unset wait_key~/.mutt/mutt-notmuch.pl thread~/.cache/mutt_resultsset wait_key" \
          "search and reconstruct owning thread (using notmuch)"

— snip .muttrc —

2a) Install OfflineIMAP
Most distributions like Fedora carry OfflineIMAP, make sure you install it.

dnf install offlineimap

2b) Configure OfflineIMAP: .offlineimaprc
Next, you need an offlineimaprc file.  I place this in ~/.offlineimaprc and it looks like this:

[general]
accounts = ExampleOrg

[Account ExampleOrg]
localrepository = ExampleLocal
remoterepository = ExampleRemote
status_backend = sqlite
postsynchook = notmuch new

[Repository ExampleRemote]
type = IMAP
remotehost = mail.example.com
remoteuser = you
# probably a bad idea
remotepass = ***********
ssl = yes

[Repository ExampleLocal]
type = Maildir
localfolders = ~/Maildir
restoreatime = no

Note: There are two methods to do this, GSSAPI and SSL/password.  I would not recommend the SSL/password approach if you use a system.  The only benefit of using the SSL/password in the config is so that you can continually sync mail without a kerberos ticket (though you can also specify a longer duration for ticket expiration).  This goes without saying to use proper u-g-o permissions on those types of files.

2c) Sync OfflineIMAP
Once your OfflineIMAP configuration is in place it’s time to run the first sync.  This may take a substantial amount of time depending on how many mail messages/folders you have.  With 1.3million+ emails my sync took around 12hours.

time offlineimap

3a)  Install notmuch
The next step is straightforward, simply install notmuch.

 dnf install notmuch

This will pull in all types of Perl dependencies but that’s ok.

I believe the following packages are pulled in as a result of “perl-Email-Sender perl-MailTools”

dnf install perl-Email-Sender perl-MailTools perl-Mail-Box

3b) Configure notmuch

notmuch setup

Follow the prompts, pointing it to your Maildir you set (and synced with OfflineIMAP)

3c) Index your maildir with notmuch

notmuch new

4) Download mutt-notmuch
Pull down mutt-notmuch from here.  The man page can be found here or in HTML format here.
Place this where you refer to it in your .muttrc above.

5a) Option #1: Set up cron to sync OfflineIMAP

  • notmuch is called as a post hook from your offlineimaprc if you used my example above

NOTE: some people have issues with offlineimap hanging, simply prepend “killall offlineimap ; ” to the cron line below as a work-around but a cleaner way is to use a script so you don’t generate crond email and things only sync when you are online.

# sync IMAP store every 8min for offline use
*/8 * * * * offlineimap -q -u quiet

5a) Option #2: Setup a script to only sync while online (or when mailserver is available)

I use the following script in my crontab in lieu of the sync cron above to run on my laptop

#!/bin/bash
# sync offineimap when you can do a DNS lookup against your mailserver
# run this from cron, I use every 12min (*/12 * * * *)

imapactive=`ps -ef | grep offlineimap | grep -v grep | wc -l`
mailsync="/usr/bin/offlineimap -u quiet -q"
username=`whoami`
mailhost=`cat /home/$username/.offlineimaprc | grep remotehost | awk '{print $3}'`
online=`host $mailhost | grep "has address" | wc -l`

# kill offlineimap if active, in some rare cases it may hang
case $imapactive in
'1')
   killall offlineimap && sleep 5
;;
esac

# if you can do a DNS lookup, sync your mail
case $online in
'1')
   $mailsync
;;
esac

6) Tips and Hotkeys
If you’re not an avid mutt user, you might want to check out the help menu by typing ? then use vim search (default mutt editor) to find your hotkeys as defined by your .muttrc or defaults

Using Multiple Accounts
Mutt has some ways to intelligently use multiple accounts, and try to pick the best correct one when replying to email.  Rather than explain it here, please read the Mutt MultiAccount Examples here.

Common Sidepane and Misc Hotkeys
Here are some common hotkeys I use, configurable in your .muttrc.

  • b” shows or hides the sidepane bar
  • control + n” moves down one sidepane folder
  • control + p moves up one sidepane folder
  • control + o” opens a selected sidepane folder
  • .r” marks all messages read in an IMAP folder
  • “.i” sorts message view to all unread/flagged
  • “.a” reverts message view back to all
  • control + b” inside message opens URLView
  • v” to view attachments inside a message
  • esc + r” marks a thread read
  • esc + s” saves (copies) messages to a different folder
  • g” replies-all when in message view
  • r” replies to individual sender in message view
  • L” replies to email list in message view
  • d” marks message for delete in message list
  • D” marks messages for delete on a partial match
  • U” undeletes messages based on a partial match
  • “a” creates email alias for address based via any email in mailbox

Notmuch Syntax
Notmuch search syntax is similar to Google search.  You should read the notmuch documentation for full usage.  Here are some examples, and you can combine any manner of them together for added granularity.

  • from:will@example.com AND to:john@example.com OR to:jane@example.com
  • date:”2015-09-01″..”2015-09-02″
  • budget AND FY14 OR FY15 AND date:”2015-09-01″..”2015-09-02″

Victory!  Have control over your mail!
At this point you should have offlineIMAP running in cron and keeping your local Maildir up to date, notmuch will be indexing the crap out of it and mutt will be used locally.
For an idea of my usage, I keep around 1.8Million emails (14GB) locally, discarding automated stuff and have no problem keeping up with it all.

victorywaitsonyourfingers_sm

About Will Foster

hobo devop/sysadmin, all-around nice guy.
This entry was posted in open source, sysadmin and tagged , , , , , , . Bookmark the permalink.

6 Responses to Take Control of your Email with Mutt, OfflineIMAP and Notmuch

  1. Nice!

    You can definitely enhance this approach to be more Shao Lin Monk nomad by packaging up everything you use into a few docker containers (auto-built via public github repositories) and a small cloud instance that houses your credentials and a shell-in-a-box ssh shell with a two-factor Google authenticator for login, and voila, now you have you whole setup in an always available cloud instance with all software configured in easy to spin containers.

    Liked by 1 person

    • Will Foster says:

      Hey Kayvan, this is a pretty neat idea. My only issue with this for personal usage is the ~15G+ of local mail housed somewhere (encrypted, and constantly updated over OpenVPN). For someone doing this with a more disposable, lightweight setup this might work really well – I like the transient nature of it but 2FA would be a must as you mentioned.

      Like

  2. Amanda says:


    I’m pretty thrilled to have this working nicely but I’ve been doing a ton of tagging and I finally realized that (duh) my notmuch tags are local. I’m wondering if you’ve encountered any clever ways to sync the tags across machines?

    Or to carry the information? Pack it into a header and then unpack it again? I’m really only dealing with two computers here.

    Liked by 1 person

    • Will Foster says:

      Hey Amanda, I’m glad that you’ve gotten a setup you like – looks sweet! For multiple machines I either sync them independently against the same IMAP server or if I’m setting up something new I’ll simply keep them synced with rsync. You’ve got to be careful with overwriting newer offlineimap sqlite files with older ones however, you can inadvertently push delete flags to your messages (happened to me once – was my own fault).

      Sorry, I don’t have any really clever ways of managing distributing tags inside of notmuch across computers, it might help to ask on the project github. Two small improvements I’d make is to use some logic in your sync to not try if the IMAP server is unavailable, see:

      https://github.com/sadsfae/misc-scripts/blob/master/shell/mailsync.sh

      This might help to eat less cycles on a laptop if you’re firing off the sync in cron.

      Like

  3. PJ says:

    Interesting, thanks. I use X1 (x1.com) on Windows 7 to index mail (and everything else). I’m looking to switch to a Linux alternative and notmuch wasn’t listed on the AlternativeTo site.

    Like

Have a Squat, Leave a Reply ..

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s