In a previous guide I documented my mutt + offlineimap + notmuch setup.  If you’re using Gmail with IMAP enabled you can still utilize the superpower of this trio but you’ll need to do some things differently.  I’ll show you how to use offlineimap with the Google Gmail OAuth2 API and configure it.

UPDATE: 2022-08-24:  October 3rd 2022 Google will turn off their API support for cited security reasons.  Although I find their reasoning somewhat flimsy in this regard this is what they’re doing.

I have reverted to using Google App passwords for OfflineIMAP for now.

You can (until there is another API way to do this) go to your Google App Passwords settings and create an application password (requires turning 2FA on) and then in your offlineIMAP configuration use the following settings:

remotehost = imap.gmail.com
remoteuser = jhoffa@example.com
remotepass = XXXYYYXXXAAABB

I am going to keep the old blog post guide intact below for posterity and if alternate API access becomes available in the future I’ll update it accordingly.

Why API?
You can simply enable 2-factor authentication and create an app password for offlineimap, that’s fine too.  I like the API method because it’s more secure (you require two unique items + a refresh token).

Setting up the Gmail API
Offlineimap has some steps listed to get your offlineimap setup and running on their Github but we’ll walk through them in detail.

Make sure the IMAP API is enabled (toggled in the above link) then you’ll just need three things for API access for Offlineimap which we’ll cover:

1. Client ID
2. Client Secret
3. Refresh Token

Create a New OAuth2 Application

You first need to head here and register a new OAuth2 application.  You may also be prompted for some 2-factor information here or a general authorization screen which I didn’t capture but should be self-explanatory.

• Create a new project (e.g. offlineimap, name doesn’t matter)

• In API  and Auth:  select credentials
• Click Enable
• Click Create Credentials

• Create Client ID (Application type – Other)

• Client ID and Secret

Above you should have now have an enabled client ID and secret available, copy these down because you’ll need them later.

Generate Oauth2 Refresh Token
The last thing we’ll need is an OAuth2 refresh token, to retrieve that you’ll need to use the Google OAuth tools repository.

git clone https://github.com/google/gmail-oauth2-tools

Next you’ll need to generate your oath2 token and refresh token.  Make sure to change the YOUR_* strings below with your client ID and your client secret.

Note:  This should be run with python2, thanks to folks in comments who discovered this.  At the time of writing this guide I was using Python2.

cd gmail-oauth2-tools

python2 python/oauth2.py --generate_oauth2_token \
--client_id=YOUR_CLIENT_ID --client_secret=YOUR_CLIENT_SECRET

You should see the refresh token string as the output for the above command, copy this as well since we’ll need all three to configure offlineimap.

Configure Offlineimap for API Access
We’re almost done!  Now you need the following lines in your offlineimap configuration file.  I’ve also made a sanitized copy for your reference.

[Repository ExampleCompanyRemote]
auth_mechanisms = XOAUTH2
oauth2_client_id = YOUR_CLIENT_ID_HERE
oauth2_client_secret = YOUR_CLIENT_SECRET_HERE
oauth2_request_url = https://accounts.google.com/o/oauth2/token
oauth2_refresh_token = YOUR_REFRESH_TOKEN_HERE

Fill in the above values you have received and your refresh token, remove any other authentication options.

Filter Gmail Specific IMAP Prefixes
Gmail wants to apply a [Gmail]/ prefix to all your existing IMAP folders (they call them labels, more on that later).  Let’s remove this from our mail so our beautiful mutt sidepane isn’t tarnished.

Add the following to your local repository configuration in .offlineimaprc

[Repository ExampleCompanyLocal]
## Remove GMAIL prefix on Google-specific IMAP folders that are pulled down.
nametrans = lambda f: '[Gmail]/' + f if f in ['Drafts', 'Starred', 'Important', 'Spam', 'Trash', 'All Mail', 'Sent Mail'] else f

You’ll need a similiar line for your remote repository configuration in .offlineimaprc

[Repository ExampleCompanyRemote]
## remove Gmail prefix on IMAP folders
nametrans = lambda f: f.replace('[Gmail]/', '') if f.startswith('[Gmail]/') else f

Awesome, we’re almost done.

Omit Duplicated Mail Labels for IMAP
Google thinks they can re-invent email smarter and better than everyone else, it tries to do this with the labelling system by treating all your mail as one big pile (All Mail) and then using labels (which can suffice as IMAP folders on the client side) to simply tag messages to show up in user defined pseudo-folders.  This is good enough for mutt for the most part.

Gmail sort of does IMAP, it is standards-compliant enough to let you pull mail but it also does some irritating (from an IMAP client perspective) message de-duplication.  You need to omit a few labels from being visible from IMAP so you don’t pull down your messages two or three times (one time for All Mail, the other times for Starred and Important).

Go to the gmail interface –> Settings –> Labels and uncheck show in IMAP for ‘All Mail’, ‘Starred’ and ‘Important’.

Turn Off Smartypants Predictive Analysis and Filtering
While Google is scanning all of your email for marketing purposes it tries to also do predictive analysis by default.  I turn all of this off because it can mess add unnecessary flags to your emails which don’t jive well in mutt.

I also set filters to not be overridden so you can control exactly how things are filtered as this setting is related to the first one.  This can be set in Settings -> Inbox

Sync Offlineimap
At this point you’re ready to sync offlineimap for the first time!

while sleep 50; do time offlineimap; done

Note that I am building in extra fault tolerance into the first sync command because of some issues highlighted below.  The above command will take a while to do a first full, initial sync of your mailbox and emails if it’s quite large like mine.  I believe it took me around 1-2 weeks to sync around 2million emails and attachments with countless crashes (see below).

Subsequent sync commands will not need this hack.

Once you’ve gotten a full sync you can proceed with step 3a from my original guide to setting up mutt + offlineimap + notmuch.

Drafts and Sent Mail
On the mutt side you can utilize the default mutt configuration settings to ensure your Sent Mail and Drafts are synced by ways (my config file has good examples)

# sync sent mail
set record="~/Maildir/Sent Mail"
# sync drafts
set postponed="~/Maildir/Drafts"

Syncing Mail – Gmail and Reliability
I was able to crash gmail countless times just by syncing my mail, it always came back relatively quickly but I incurred quite a bit of 500’ish errors on both offlineimap sync, mailbox unavailable and the web interface.

I am guessing my mailbox is run in a Kubernetes container because I could detect slowdowns accessing it from more than one location while syncing and when it was unavailable via IMAP it was also unavailable via the web interface.

Bulk actions also tend to make it angry like applying labels across a few hundred thousand emails at a time.  That’s ok, refresh and wait a few seconds and drive on citizen.

It always came back quickly but this is why we put some some .. ahem .. resiliency into our initial IMAP sync command, especially if you have a lot of mail.

Gmail – The Good and the Bad
On principle I don’t care for most closed source technology nor services that are not 100% RFC standards compliant.  Gmail itself also doesn’t seem as resilient as other enterprise IMAP solutions I’ve used, but it recovers quickly.

The label system also has some quirks and it’s not as flexible at adding multi-condition filtering for people who have meticulous IMAP organization or subscribe to a lot of Open Source mailing lists.  Generally you can work around this with label ordering or multiple filters per list.

One of the benefits of Gmail is that if your company uses it for hosted email you no longer require VPN access to retrieve your mail from clients.  Another benefit is that the sync just seems faster.  Being located in Ireland I’m always using Google servers very close to me which may not be the case if your company collocates their datacenter in one geographical continent for email services.

Overall I’m happy with this solution and it works well for me, with a little bit of modification I can continue to use my preferred setup.

Thanks to John Eckersberg who gave me the idea to use the Gmail API and for being the best around.

Extending Desktop Notifications with mail-notifications
Victor Kareh has suggested the usage of the mail-notification program on Linux to extend mail notifications further via inotify.  From his experience he writes

mail-notification is almost too easy, just add a new email
mailbox on Autodetect, and hard-code the location to your INBOX
directory. e.g. file:///home/vkareh/Mail/redhat.com/INBOX

Then you’ll have a notification tray icon showing you the number of
unread emails across any mailboxes. Optionally can show popup
notifications with the sender and subject. This works well in MATE and
probably XFCE and other “old-school” desktops, not quite sure if it
works in GNOME (since I think they’ve dropped support for the tray?).