How to use Google Workspace OAuth2 with mbsync (from isync) and msmtp on NetBSD

Google will remove old OAuth out-of-band (OOB) support on 2022-10-04. And desktop clients for Google Workspace should use Loopback IP address flow instead. So I must change my script to support Loopback IP address flow. My previous setup using OOB is described in How to use Google G suite OAuth2 with mbsync (from isync) and msmtp on NetBSD post.

I have searched the web and found How to continue using msmtp OAuth 2.0 for Gmail in mutt after oob deprecation? on StackExchange. The answer is to use getmail-gmail-xoauth-tokens script from getmail6 to get refresh token and access tokens. The answer says "- which are what we've been looking for, but that 1h expiry is prohibitively awkward...". It is absolutely unacceptable for me. I am away from GUI or web browsers in most time. I have not read getmail-gmail-xoauth-tokens script. However I feel that it may be the answer's misunderstanding or misuse. Anyway I do not have enough time to investigate the potentially hopeless script.

As a result, I found that the access tokens will expire in 3600 seconds and the refresh token has much longer life time like before. I have create two scripts to confirm this result. And I can reuse the scripts for my mbsync from isync and msmtp setups.

Get refresh token and save it

At first, you must download your client secret file as client_secret_*.json. Just rename it as client_secret.json

Google provides Google Auth Python Library and related libraries for Python. For further changes, I should use common methods to get tokens. I am a pkgsrc user and I have installed required libraries as follows:

# cd /usr/pkgsrc/www/py-google-api-python-client
# make install
# cd /usr/pkgsrc/security/py-google-auth-oauthlib
# make install

To get refresh token, you can use google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file() with the client_secret.json file. My source code (google-oauth2-loopback-ip.py) is as follows:

#!/usr/pkg/bin/python3.10

# Get refresh token and save it as ~/.refresh_token.pickle.
# Saved information will be used to refresh access token hourly.

# Follow:
# https://googleapis.github.io/google-api-python-client/docs/oauth-installed.html

import os
import pickle
from google_auth_oauthlib.flow import InstalledAppFlow

# Constants
CLIENT_SECRETS_FILE = 'client_secret.json'
CREDENTIAL_CACHE_FILE = '.refresh_token.pickle'

## Configurations
### To identify scopes and services, See:
### https://developers.google.com/identity/protocols/oauth2/scopes#gmail
SCOPES=['https://mail.google.com/']
API_SERVICE_NAME = 'mail'
API_VERSION = 'v1'

## Get credentials
def get_credentials():
    ## Create client object
    flow = InstalledAppFlow.from_client_secrets_file(
        CLIENT_SECRETS_FILE,
        scopes=SCOPES)

    ## Open with local web browser, input credential manually
    flow.run_local_server(host='localhost',
        port=8091,
        authorization_prompt_message='Please visit this URI: {url}',
        success_message='The auth flow is complete; you may close this web browser window.',
        open_browser=True)

    return flow.credentials


if __name__ == '__main__':
    credentials = get_credentials()

    # Get home directory path and create a path for cache file
    homeDir = os.environ['HOME']
    if not homeDir:
        homeDir = '.'
    credPath = os.path.join(homeDir, CREDENTIAL_CACHE_FILE)

    # Save credentials as file
    with open(credPath, 'wb') as tokenCache:
        pickle.dump(credentials, tokenCache)

    # https://google-auth.readthedocs.io/en/stable/reference/google.oauth2.credentials.html#google.oauth2.credentials.Credentials
    print('Refresh Token:', credentials.refresh_token)

You will get your refresh token as Refresh Token: YOURREFRESHTOKENSTRING. This script get an access token simultaniously (See your ~/.refresh_token.pickle). However I will not use the access token in this script at all.

Get access token using the refresh token

The refresh token has longer life time than 3600 seconds (life time of the access token). And the refresh token is used to get refreshed access tokens without login via web browsers.

To get a refreshed access token, you should load saved credentials in ~/.refresh_token.pickle and invoke refresh() method. My source code (google-oauth2-refresh-access_token.py) is as follows:

#!/usr/pkg/bin/python3.10

# Get new access token using saved refresh token.

# Follow:
# https://googleapis.github.io/google-api-python-client/docs/oauth-installed.html

import os
import pickle
from google.auth.transport.requests import Request

# Constants
CREDENTIAL_CACHE_FILE = '.refresh_token.pickle'

## Configurations
### To identify scopes and services, See:
### https://developers.google.com/identity/protocols/oauth2/scopes#gmail
SCOPES=['https://mail.google.com/']
API_SERVICE_NAME = 'mail'
API_VERSION = 'v1'

## Refresh credentials and get new ones
def get_refreshed_credentials():
    # Get home directory path and create a path for cache file
    homeDir = os.environ['HOME']
    if not homeDir:
        homeDir = '.'
    credPath = os.path.join(homeDir, CREDENTIAL_CACHE_FILE)

    if os.path.exists(credPath):
        with open(credPath, 'rb') as tokenCache:
            credentials = pickle.load(tokenCache)

        if credentials:
            # Refresh access token with refresh token
            credentials.refresh(Request())

    return credentials

if __name__ == '__main__':
    credentials = get_refreshed_credentials()

    # https://google-auth.readthedocs.io/en/stable/reference/google.oauth2.credentials.html#google.oauth2.credentials.Credentials
    print('Access Token:', credentials.token)
    #print('Expiry:', credentials.expiry) # in UTC

You will get your access token as Access Token: YOURREFRESHEDACCESSTOKENSTRING.

For mbsync and msmtp

This part is almost identical to my previous post. I will include my current script and configuration files.

$ cat /opt/bin/get_teteraorg_token.sh
#!/bin/sh

python3.10 /opt/bin/google-oauth2-refresh-access_token.py | awk -F" " '{if(NR==1)print $3}'

$ cat ~/.mbsync
(snip)
IMAPAccount gmail
Host imap.gmail.com
User username@tetera.org
AuthMechs XOAUTH2
PassCmd "/opt/bin/get_teteraorg_token.sh"
SSLType IMAPS
CertificateFile /etc/openssl/certs/ca-certificates.crt
(snip)

$ cat ~/.msmtprc
(snip)
account teteraorg
tls on
tls_certcheck off
tls_starttls off
host smtp.gmail.com
port 465
protocol smtp
auth xoauth2
from username@tetera.org
user username@tetera.org
passwordeval "/opt/bin/get_teteraorg_token.sh"
(snip)