new.tags not being honoured if not specified in database

[arvindsv]

First of all, thank you for writing and maintaining lieer! I've used it for a while, and it does what it does very well. :)

I've been trying to debug this issue for a little while now. The behaviour I see is described below. I'm not entirely sure when it started happening, because I also switched laptops -- but it seems like it started with the shift to the new notmuch2 bindings.

Observed behaviour

When gmi pull finishes, the tags it adds always includes unread and inbox even if new.tags in $NOTMUCH_CONFIG is set to something else.

Click / unhide this to see a shell session which demonstrates this issue

$ mkdir .mail

$ cd .mail

$ mkdir .notmuch

$ cd .notmuch

$ notmuch setup
...

$ cat notmuch-config
...
[database]
path=/Users/myuser/.mail

...
[user]
name=Aravind SV
[email protected]

...
[new]
tags=new

$ echo $NOTMUCH_CONFIG
/Users/myuser/.mail/.notmuch/notmuch-config

$ ln -s $NOTMUCH_CONFIG ~/.notmuch-config # For good measure

# After an edit to the config file to add the ignore line
$ grep -A2 '\[new\]' ~/.notmuch-config
[new]
tags=new
ignore=/.*[.](json|lock|bak)$/

$ notmuch new
Found 0 total files (that\'s not much mail).
No new mail.

$ cd ~/.mail; mkdir account.gmail && cd account.gmail

$ pwd
/Users/myuser/.mail/account.gmail

$ gmi init [email protected]
...
Authentication successful.
credentials stored in /Users/myuser/.mail/account.gmail/.credentials.gmailieer.json

$ gmi pull
pull: full synchronization (no previous synchronization state)
fetching messages: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  3.44it/s]
removing deleted: 0it [00:00, ?it/s]
receiving content:  50%|████████████████████████████████████████████████████████████████████████████████                                                                                | 1/2 [00:00<00:00,  5.36it/s]
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  9.64it/s]
receiving metadata: everything up-to-date.
pull: complete, removing resume file
current historyId: 1450, current revision: 6

$ notmuch count
2

$ notmuch search 'date:1d..'
thread:0000000000000001 13 mins. ago [1/1] Google; Security alert (inbox unread)
thread:0000000000000002 21 mins. ago [1/1] Google Community Team; Aravind, finish setting up your new Google Account (inbox unread)

# Sent a new email and then did ...
$ gmi pull
pull: partial synchronization.. (hid: 1450)
fetching changes: 2it [00:00, 8027.38it/s]
resolving changes: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 8073.73it/s]
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.80it/s]
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.63it/s]
current historyId: 1515

$ notmuch search 'date:1d..'
thread:0000000000000003  0 mins. ago [1/1] Aravind SV; Test 1 (inbox unread)
thread:0000000000000001 14 mins. ago [1/1] Google; Security alert (inbox unread)
thread:0000000000000002 23 mins. ago [1/1] Google Community Team; Aravind, finish setting up your new Google Account (inbox unread)

# Sent another email, and archived it via the GMail web front-end and then did ...
$ gmi pull
pull: partial synchronization.. (hid: 1515)
fetching changes: 5it [00:00, 12810.95it/s]
resolving changes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 15109.16it/s]
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.48it/s]
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.25it/s]
current historyId: 1588

$ notmuch search 'date:1d..'
thread:0000000000000004  1 mins. ago [1/1] Aravind SV; Test 2 (inbox unread)
thread:0000000000000003  7 mins. ago [1/1] Aravind SV; Test 1 (inbox unread)
thread:0000000000000001 22 mins. ago [1/1] Google; Security alert (inbox unread)
thread:0000000000000002 30 mins. ago [1/1] Google Community Team; Aravind, finish setting up your new Google Account (inbox unread)

# Sent another email, and archived it via the GMail web front-end + labeled it with "testlabel: and then did ...
$ gmi pull
pull: partial synchronization.. (hid: 1588)
fetching changes: 6it [00:00, 22469.49it/s]
resolving changes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 15382.53it/s]
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.79it/s]
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.62it/s]
current historyId: 1698

$ notmuch search 'date:1d..'
thread:0000000000000005  1 mins. ago [1/1] Aravind SV; Test 3 (inbox testlabel unread)
thread:0000000000000004  9 mins. ago [1/1] Aravind SV; Test 2 (inbox unread)
thread:0000000000000003 16 mins. ago [1/1] Aravind SV; Test 1 (inbox unread)
thread:0000000000000001 30 mins. ago [1/1] Google; Security alert (inbox unread)
thread:0000000000000002 39 mins. ago [1/1] Google Community Team; Aravind, finish setting up your new Google Account (inbox unread)

I made sure to create a new GMail account and a new notmuch database to verify this. As can be seen above, even archiving and changing a label does not prevent lieer from adding inbox and unread tags locally, and then pushing those back.

Potential issue causing this

I don't know the lieer codebase well enough. But, I poked around and it seems like the below lines might be relevant:

lieer/lieer/local.py

Lines 355 to 358 in 2cfbc5b

	# load notmuch config
	with notmuch2.Database() as db:
	self.new_tags = db.config.get("new.tags", "").split(';')
	self.new_tags = [t.strip() for t in self.new_tags if len(t.strip()) > 0]

I looked into the notmuch2 python bindings documentation (here) and it says: "Return a mutable mapping with the settings stored in this database" (emphasis mine).

The older notmuch bindings said much the same: "Note that only config values that are stored in the database are searched and returned. The config file is not read".

Till September 2022 (till this commit), lieer used to do the correct thing, and read from the config file. See:

lieer/lieer/local.py

Lines 303 to 311 in bf719d6

	# load notmuch config
	cfg = os.environ.get('NOTMUCH_CONFIG', os.path.expanduser('~/.notmuch-config'))
	if not os.path.exists (cfg):
	raise Local.RepositoryException("could not find notmuch-config: %s" % cfg)
	self.nmconfig = configparser.ConfigParser ()
	self.nmconfig.read (cfg)
	self.new_tags = self.nmconfig['new']['tags'].split (';')
	self.new_tags = [t.strip () for t in self.new_tags if len(t.strip()) > 0]

At this time, when I add print statements around this:

lieer/lieer/local.py

Lines 355 to 358 in 2cfbc5b

	# load notmuch config
	with notmuch2.Database() as db:
	self.new_tags = db.config.get("new.tags", "").split(';')
	self.new_tags = [t.strip() for t in self.new_tags if len(t.strip()) > 0]

... I only ever see unread and inbox and I have not been able to get it to show anything else.

Given that db.config.get seems to only read from the Xapian metadata, and not from the notmuch config file, it feels like this code should read directly from the config file, to fix this issue. Happy to be corrected.

[gauteh]

Maybe it should be reverted to the old behavior. Does notmuch still use the config file for that setting?

https://notmuchmail.org/doc/latest/man1/notmuch-config.html

[arvindsv]

They seem to. Maybe they're in the middle of a transition though? I used to use notmuch 0.27 before and notmuch-config set didn't have a way to add a value to the database (link to old man-page). They used to call out values which were to be stored in the database.

In the current documentation, it seems more fluid, though there are definite references to the config file. I guess I'll need to search through their mailing list or something to see if they're making changes (edit: their mailing list archives end in 2020). I did see that in their release notes for 0.36, they called out:

Use the config_pairs API in ConfigIterator. This returns all matching key-value pairs, not just those that happen to be stored in the database.

It doesn't seem well documented, but seems to be referring to https://github.com/notmuch/notmuch/blob/master/bindings/python-cffi/notmuch2/_config.py.

If I set the value in the database:

# Currently it's set only in the config file
$ notmuch config list | grep new.tags
new.tags=new

$ notmuch config set --database new.tags newmail

# It still shows the value in the config file
$ notmuch config list | grep new.tags
new.tags=new

# If I remove it from the config file, it shows the one in the DB
$ notmuch config list | grep new.tags
new.tags=newmail

Either way, the current README for lieer is out of sync with the behaviour I'm seeing.

[listx]

I too seem to be getting hit by the always-add-inbox-and-undread tags bug described in this thread. Is there a known workaround? This is a dealbreaker for me because it breaks my existing Gmail filters which remove the inbox tag for certain emails.

[tabuchid]

I too seem to be getting hit by the always-add-inbox-and-undread tags bug described in this thread. Is there a known workaround? This is a dealbreaker for me because it breaks my existing Gmail filters which remove the inbox tag for certain emails.

I ran into this and the fix for me was to install 1.3 and work off of that instead of latest main.

[euglevi]

I too seem to be getting hit by the always-add-inbox-and-undread tags bug described in this thread. Is there a known workaround? This is a dealbreaker for me because it breaks my existing Gmail filters which remove the inbox tag for certain emails.

I ran into this and the fix for me was to install 1.3 and work off of that instead of latest main.

I will do the same, but it seems to me an issue that should be solved. I spent a lot of time figuring out what was happening to the tags and why I was getting the 'unread' tag on all emails. Maybe for the moment a warning in the README?

By the way, thank you for the excellent utility.

[sstepashka]

I am experiencing the same issue, how to reproduce:

• Send an email to your account (Do NOT sync lieer)
• Open the email in Web-version (it would mark it unread)
• Sync lieer TWICE.

I'd assume the fist sync messes up with local labels by adding something like "inbox" and "unread". Then when it syncs second time, it pushes those changes as local changes? Hypothesis.... I was trying to look to the code, which does partial pull, but my limited understanding doesn't find any issues there.

Could it be the labels are added on notmuch sync automatically?

[sstepashka]

What I do not understand is why we use bitwise "and" operator there:

if not (set(mm.get('labelIds', [])) & self.remote.not_sync)

isn't it supposed to be logical "and"?

[sstepashka]

Can we revert it back? :)

[gauteh]

I think that's the same. Does it fix the issue?

[sstepashka]

I would be able to tests it in a few days, but it's bothering me a lot :) So, I wouldn't be able to avoid trying it...

I think, I can kick off the python in REPL and see what the not much binding returns and make some conclusions basing on that.

[gauteh]

I think this is set operations, the bitwise & is intersect if I remember correctly. If the intersection between those two are empty do the if-block.

[sstepashka]

% python
Python 3.11.6 (main, Oct  2 2023, 13:45:54) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import notmuch2
>>> with notmuch2.Database() as db:
...   db.config.get('new.tags', '')
...
'unread;inbox'
>>>

where

% notmuch config get new.tags
new

Seems like notmuch doesn't load the config my default...

[gauteh]

Does it work better if you also set that config in the database (taken from above):

notmuch config set --database new.tags newmail

I suspect that's everything lieer gets now. I'm not sure what notmuch wants us to do.

[sstepashka]

I think you're 100% correct. The

notmuch config set --database new.tags new

fixes the code as well:

% python -c "import notmuch2; print(notmuch2.Database().config.get('new.tags', ''))"
new

Weird that get operation doesn't accept --database argument:

SYNOPSIS
       notmuch config get <section>.<item>

       notmuch config set [--database] <section>.<item> [value ...]

       notmuch config list

[sstepashka]

I can confirm that notmuch config set --database new.tags new fixes the issue for me on the latest available code in the repo.

[gauteh]

Ok. So maybe the fix is to update the README, bug notmuch and add a line to caveats. If we're nice we could check for difference, but that means adding code for reading the config again, which would be nice to avoid.

[sstepashka]

I agree,

My mental model of the notmuch config is broken now and I can't find a good explanation of how it is supposed to work (having 2 configs: one in home directory, the other one in database). Cuz my assumption notmuch uses HOME config to find the database location, but then the rest of options ignore or overwritten by Database config or they're merged somehow... Confusing...

Looking into the python binding of notmuch2, it uses the home config ONLY to read database location and the create a config mapping from the database. And the config (seems like) is stored as some binary blob (not a text file), since I can't find it anywhere. Also, there is no way from shell to READ the database config...

I hope, I am wrong...

[gauteh]

According to #229 (comment) you can read db config from shell, if you remove text config value.

[sstepashka]

Right, I really didn't it through! I see it now!

[nickurak]

Is there a configuration that would go into the database that works in this regard if we don't want any tags added to new mail? (eg, if I'm letting gmail handle all labeling, and don't need to use any notmuch-side filtering/tagging)

[aspiers]

@nickurak I'm fairly sure this single space trick works, although not heavily tested yet:

notmuch config set --database new.tags new ' '

Note that it's critical to have the single space inside the quotes here. If there's no space then it defaults to unread;inbox which is exactly the original problem we want to avoid.

With this trick, it's also critical to not set new.tags in the config file, otherwise that takes precedence as noted above.

[aspiers]

@arvindsv commented on December 14, 2022 6:20 PM:

They seem to. Maybe they're in the middle of a transition though? I used to use notmuch 0.27 before and notmuch-config set didn't have a way to add a value to the database (link to old man-page). They used to call out values which were to be stored in the database.

In the current documentation, it seems more fluid, though there are definite references to the config file. I guess I'll need to search through their mailing list or something to see if they're making changes (edit: their mailing list archives end in 2020).

I did a small bit of archaeology and found notmuch/notmuch@dd9112e which is somewhat enlightening, but still doesn't explain why they split config between two places in the first place.