lieer + git send-email creates "broken" threads.

[granquet]

Hello.

Just trying out lieer 1.3 and I'm wondering if anyone is using 'gmi send' with git send-email and has encountered issues with threads?
email clients such as mutt or alot see the first mail as "standalone" and the following messages correctly grouped by thread.

The In-Reply-To and References in the subsequent mails are correct and point to the first message id.

Though, lieer complains that it cannot find the parent message when sending the subsequent mails:

warning: could not find parent message, sent message will not be associated in the same thread

I tried wrapping gmi send with gmi sync && notmuch new to force an update of the database and maybe help in finding the parent message but to no avail.

I'll keep digging and update here if I have something worth sharing.

Example output:

granquet@seychelles ~/projects/conf
 % git send-email -4
/tmp/R9XUsEApXW/0001-vim-fix-pathogen-and-bundles.patch
/tmp/R9XUsEApXW/0002-Add-install-script.patch
/tmp/R9XUsEApXW/0003-vim-Make-vim-autoload-symlink-relative.patch
/tmp/R9XUsEApXW/0004-tmux-bind-C-a-a-to-send-C-a.patch
To whom should the emails be sent (if anyone)? [email protected]
Message-ID to be used as In-Reply-To for the first email (if any)?
(mbox) Adding cc: Guillaume Ranquet <[email protected]> from line 'From: Guillaume Ranquet <[email protected]>'
(body) Adding cc: Guillaume Ranquet <[email protected]> from line 'Signed-off-by: Guillaume Ranquet <[email protected]>'
path: ~/.mail/[email protected]
sending message, from: Guillaume Ranquet <[email protected]>..
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  4.18it/s]
receiving metadata: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.63it/s]
message sent successfully: 1795be5d07c16444
OK. Log says:
Sendmail: /usr/bin/gmi send -C ~/.mail/[email protected] -i [email protected] [email protected]
From: Guillaume Ranquet <[email protected]>
To: [email protected]
Cc: Guillaume Ranquet <[email protected]>
Subject: [PATCH 1/4] vim: fix pathogen and bundles
Date: Tue, 11 May 2021 16:47:20 +0200
Message-Id: <[email protected]>
X-Mailer: git-send-email 2.26.3
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

Result: OK
(mbox) Adding cc: Guillaume Ranquet <[email protected]> from line 'From: Guillaume Ranquet <[email protected]>'
(body) Adding cc: Guillaume Ranquet <[email protected]> from line 'Signed-off-by: Guillaume Ranquet <[email protected]>'
path: ~/.mail/[email protected]
sending message, from: Guillaume Ranquet <[email protected]>..
looking for original message: [email protected]
warning: could not find parent message, sent message will not be associated in the same thread
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  4.43it/s]
receiving metadata: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.47it/s]
message sent successfully: 1795be5d95e9bd0b
OK. Log says:
Sendmail: /usr/bin/gmi send -C ~/.mail/[email protected] -i [email protected] [email protected]
From: Guillaume Ranquet <[email protected]>
To: [email protected]
Cc: Guillaume Ranquet <[email protected]>
Subject: [PATCH 2/4] Add install script
Date: Tue, 11 May 2021 16:47:21 +0200
Message-Id: <[email protected]>
X-Mailer: git-send-email 2.26.3
In-Reply-To: <[email protected]>
References: <[email protected]>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

gitconfig:

[sendemail]
        smtpserver = /usr/bin/gmi
        smtpserveroption = "send"
        smtpserveroption = "-C"
        smtpserveroption = "~/.mail/[email protected]"
        suppresscc = self
        chainreplyto = false
        composeencoding = UTF-8
        transferencoding = 8bit

[granquet]

I spent a bit more time digging into this.
Although the In-Reply-To field of the subsequent messages mentions the Message-Id of the first message:

First message:
Message-Id: <[email protected]>
Subsequent messages:
In-Reply-To: <[email protected]>

The Message-Id of the first mail is rewrote by google into something like:

Message-Id: <CABnWg9tHjDu800FVqnH4nj5to2JTCOtDa4A24v4ZN4U9M06DDQ@mail.gmail.com>

Which, obviously, breaks the git sendmail thread feature.

some people seem to complain here and there that this is indeed happening to them when using the gmail API... but I couldn't find a reliable source of information on this "feature".

Not sure there's an easy fix in lieer... not sure there's a fix at all?

[gauteh]

Think a fix would require the In-Reply-To of the following messages to be updated with the new id. Seems like a lot of hassle? We could override it in gmi, but it would probably still be difficult to make git-send-email able to pass that information to gmi.

[granquet]

Think a fix would require the In-Reply-To of the following messages to be updated with the new id. Seems like a lot of hassle? We could override it in gmi, but it would probably still be difficult to make git-send-email able to pass that information to gmi.

Yeah, I do not see an easy way to fix it... unless google stops behaving likes this, I guess git-send-email + gmi is a no-go.

[mjg]

I just checked both with msmtp and with gmi send, both with my gmail address and with an address which configured as a sender address in GMail.

The result seems to be that no matter what, sending with smtp preserves message-IDs and sending via the API creates new ones.

So, in the process of ditching my app-specific passwords, I'll rather teach msmtp oauth. I'm wondering if it makes sense to teach lieer's remote.py to share the tokens so that there is only one config (using my own client secret, of course), rather than scripting around google's oauth.py for msmtp.

[gauteh]

It looks like this might be related to the format of the message-id:

• https://stackoverflow.com/questions/52013699/gmail-api-is-overwriting-the-custom-message-id-header-while-sending-emails
• https://groups.google.com/g/comp.mail.sendmail/c/gkRdBwka4iw- -> https://groups.google.com/g/comp.mail.sendmail/c/gkRdBwka4iw-/m/7DxqaxSzDwAJ

maybe git-send-email can be configured to do this?

[mjg]

My message-id's are of that form (as produced by alot, e.g. <[email protected]>). I'm afraid GMail recreates MIDs on "receive" when they are invalid; on "send by API" unconditionally; on "send by SMTP" probably when they are invalid (haven't checked).

The API returns the MID, but at that point the sent message is not in the notmuch db yet (only after sync). So, gmi could either:

• store a mapping original MID -> GMail MID somewhere or
• copy MID to a header X-Original-Message-ID before sending (and tell users to index that in notmuch)
  and then (on user's request) rewrite In-Reply-To headers before sending if they match a stored original header.

Or hope for GMail to change its behaviour :)

[gauteh]

Gmi adds the message (with the new id) to the db when it is sent, if that helps.

[mjg]

Gmi adds the message (with the new id) to the db when it is sent, if that helps.

Ah, sure, I completely missed that get_content() does that. So it adds the message as returned from the API. At that point send() still has the original e-mail object eml, can compare the MIDs and could store the original MID e.g. as a property in the notmuch db so that, in turn, it is available if send() fails to find a MID when it parses In-Reply-To headers.

I'm still wondering whether we have a confirmed example of a message sent through the API with unchanged MID, i.e. whether there is a simple way to "conform".

[gauteh]

søn. 17. okt. 2021, 16:45 skrev Michael J Gruber ***@***.***>:

Gmi adds the message (with the new id) to the db when it is sent, if that helps. Ah, sure, I completely missed that get_content() does that. So it adds the message as returned from the API. At that point send() still has the original e-mail object eml, can compare the MIDs and could store the original MID e.g. as a property in the notmuch db so that, in turn, it is available if send() fails to find a MID when it parses In-Reply-To headers.

Yes, or also printed out. The following messages need their in reply to modified.

I'm still wondering whether we have a confirmed example of a message sent through the API with unchanged MID, i.e. whether there is a simple way to "co From the comments on the stack overflow Q's it seems like it is not

possible when using the API.

…