Using Offlineimap with the Gmail IMAP API

In a previous guide I documented my mutt + offlineimap + notmuch setup.  If you’re using Gmail with IMAP enabled you can still utilize the superpower of this trio but you’ll need to do some things differently.  I’ll show you how to use offlineimap with the Google Gmail OAuth2 API and configure it.

 

Why API?
You can simply enable 2-factor authentication and create an app password for offlineimap, that’s fine too.  I like the API method because it’s more secure (you require two unique items + a refresh token).

Setting up the Gmail API
Offlineimap has some steps listed to get your offlineimap setup and running on their Github but we’ll walk through them in detail.

Make sure the IMAP API is enabled (toggled in the above link) then you’ll just need three things for API access for Offlineimap which we’ll cover:

  1. Client ID
  2. Client Secret
  3. Refresh Token

Create a New OAuth2 Application

You first need to head here and register a new OAuth2 application.  You may also be prompted for some 2-factor information here or a general authorization screen which I didn’t capture but should be self-explanatory.

  • Create a new project (e.g. offlineimap, name doesn’t matter)

  • In API  and Auth:  select credentials
  • Click Enable
  • Click Create Credentials

  • Create Client ID (Application type – Other)

 

  • Client ID and Secret

Above you should have now have an enabled client ID and secret available, copy these down because you’ll need them later.

Generate Oauth2 Refresh Token
The last thing we’ll need is an OAuth2 refresh token, to retrieve that you’ll need to use the Google OAuth tools repository.

git clone https://github.com/google/gmail-oauth2-tools

Next you’ll need to generate your oath2 token and refresh token.  Make sure to change the YOUR_* strings below with your client ID and your client secret.

cd gmail-oauth2-tools
python python/oauth2.py --generate_oauth2_token \
--client_id=YOUR_CLIENT_ID --client_secret=YOUR_CLIENT_SECRET

You should see the refresh token string as the output for the above command, copy this as well since we’ll need all three to configure offlineimap.

Configure Offlineimap for API Access
We’re almost done!  Now you need the following lines in your offlineimap configuration file.  I’ve also made a sanitized copy for your reference.

[Repository ExampleCompanyRemote]
auth_mechanisms = XOAUTH2
oauth2_client_id = YOUR_CLIENT_ID_HERE
oauth2_client_secret = YOUR_CLIENT_SECRET_HERE
oauth2_request_url = https://accounts.google.com/o/oauth2/token
oauth2_refresh_token = YOUR_REFRESH_TOKEN_HERE

Fill in the above values you have received and your refresh token, remove any other authentication options.

Filter Gmail Specific IMAP Prefixes
Gmail wants to apply a [Gmail]/ prefix to all your existing IMAP folders (they call them labels, more on that later).  Let’s remove this from our mail so our beautiful mutt sidepane isn’t tarnished.

Add the following to your local repository configuration in .offlineimaprc

[Repository ExampleCompanyLocal]
## Remove GMAIL prefix on Google-specific IMAP folders that are pulled down.
nametrans = lambda f: '[Gmail]/' + f if f in ['Drafts', 'Starred', 'Important', 'Spam', 'Trash', 'All Mail', 'Sent Mail'] else f

You’ll need a similiar line for your remote repository configuration in .offlineimaprc

[Repository ExampleCompanyRemote]
## remove Gmail prefix on IMAP folders
nametrans = lambda f: f.replace('[Gmail]/', '') if f.startswith('[Gmail]/') else f

Awesome, we’re almost done.

Omit Duplicated Mail Labels for IMAP
Google thinks they can re-invent email smarter and better than everyone else, it tries to do this with the labelling system by treating all your mail as one big pile (All Mail) and then using labels (which can suffice as IMAP folders on the client side) to simply tag messages to show up in user defined pseudo-folders.  This is good enough for mutt for the most part.

Gmail sort of does IMAP, it is standards-compliant enough to let you pull mail but it also does some irritating (from an IMAP client perspective) message de-duplication.  You need to omit a few labels from being visible from IMAP so you don’t pull down your messages two or three times (one time for All Mail, the other times for Starred and Important).

Go to the gmail interface –> Settings –> Labels and uncheck show in IMAP for ‘All Mail’, ‘Starred’ and ‘Important’.

Sync Offlineimap
At this point you’re ready to sync offlineimap for the first time!

while sleep 50; do time offlineimap; done

Note that I am building in extra fault tolerance into the first sync command because of some issues highlighted below.  The above command will take a while to do a first full, initial sync of your mailbox and emails if it’s quite large like mine.  I believe it took me around 1-2 weeks to sync around 2million emails and attachments with countless crashes (see below).

Subsequent sync commands will not need this hack.

Once you’ve gotten a full sync you can proceed with step 3a from my original guide to setting up mutt + offlineimap + notmuch.

Drafts and Sent Mail
On the mutt side you can utilize the default mutt configuration settings to ensure your Sent Mail and Drafts are synced by ways (my config file has good examples)

# sync sent mail
set record="~/Maildir/Sent Mail"
# sync drafts
set postponed="~/Maildir/Drafts"

Syncing Mail – Gmail and Reliability
I was able to crash gmail countless times just by syncing my mail, it always came back relatively quickly but I incurred quite a bit of 500’ish errors on both offlineimap sync, mailbox unavailable and the web interface.

I am guessing my mailbox is run in a Kubernetes container because I could detect slowdowns accessing it from more than one location while syncing and when it was unavailable via IMAP it was also unavailable via the web interface.

Bulk actions also tend to make it angry like applying labels across a few hundred thousand emails at a time.  That’s ok, refresh and wait a few seconds and drive on citizen.

It always came back quickly but this is why we put some some .. ahem .. resiliency into our initial IMAP sync command, especially if you have a lot of mail.

Gmail – The Good and the Bad
On principle I don’t care for most closed source technology nor services that are not 100% RFC standards compliant.  Gmail itself also doesn’t seem as resilient as other enterprise IMAP solutions I’ve used, but it recovers quickly.

The label system also has some quirks and it’s not as flexible at adding multi-condition filtering for people who have meticulous IMAP organization or subscribe to a lot of Open Source mailing lists.  Generally you can work around this with label ordering or multiple filters per list.

One of the benefits of Gmail is that if your company uses it for hosted email you no longer require VPN access to retrieve your mail from clients.  Another benefit is that the sync just seems faster.  Being located in Ireland I’m always using Google servers very close to me which may not be the case if your company collocates their datacenter in one geographical continent for email services.

Overall I’m happy with this solution and it works well for me, with a little bit of modification I can continue to use my preferred setup.

Thanks to John Eckersberg who gave me the idea to use the Gmail API and for being the best around.

About Will Foster

hobo devop/sysadmin, all-around nice guy.
This entry was posted in open source, sysadmin and tagged , , , , , , . Bookmark the permalink.

4 Responses to Using Offlineimap with the Gmail IMAP API

  1. Doug says:

    Thank you for this information. I’m a new-ish user- can you (briefly) explain why this API route is more secure? Isn’t the client secret still sitting in a plain text config file? I don’t understand why this would be preferable to storing your password in a gpg encrypted file. Thanks again.

    Like

    • Will Foster says:

      Thank you for this information. I’m a new-ish user- can you (briefly) explain why this API route is more secure? Isn’t the client secret still sitting in a plain text config file? I don’t understand why this would be preferable to storing your password in a gpg encrypted file. Thanks again.

      Hey Doug, that’s a good question. Ultimately if someone has control of your config file they’ll be able to usurp the same level of access so you’re right there, however I do see the API method being a bit more secure because there’s a refresh token, a forced length private secret, and an unique client id whereas with a simple application password is just a generated singular item. In this regard you’d need the sum of all the parts (API) to steal access whereas someone only needs to find the application password by itself to do the same. As you pointed out having access to the configuration file here invalidates the security of any approach.

      Like

  2. anataua says:

    Hello,
    I just followed all the steps, I was able to get refresh oauth2_refresh_token, I didi averithing well, but still I get
    error: [ALERT] Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)

    is there a log or other stuff I can use to diagnose this issue? I am desperately trying to sync gmail account by more the 10 hours :(

    Like

    • Will Foster says:

      Hey Anataua,

      Are you using 2-factor authentication? Mine is setup this way. I’d say that if you’re in a hurry you can just generate an application password and use that, it will do the same thing.

      Like

Have a Squat, Leave a Reply ..

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s