IMAP access cause CPU hogs on huge (>6Tb) mail account

[gardiol]

Hi!
i have installed Stalwart on my linux server (bare metal, 0.11.5 release) replacing the old stack of postfix + dovecot + DKIM/DMARC + spamassassin and bits.

The server is not the most performant, and it has mechanical HDD,but it's one domain with two active email address, and some 8GB of past emails that i have imap-migrated there. I never had significant speed issues with the old stack.

I use roundcube (but the same seems to apply over other IMAP clients) after an action on the account stalwart globs up al the CPU cores and often the client timesout. Everything has been configured with the defaults, specially the stores they are all on RocksDB.

In the logs i don't see anything out of place (at DEBUG level at least).

An example of top -H while hogging the CPU:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND 
19752 stalwart  20   0 2417184 518244  32420 R  94.8  13.0 151:16.93 stalwart-mail                                                                                                           
19753 stalwart  20   0 2417184 518244  32420 R  71.8  13.0  86:39.98 stalwart-mail                                                                                                           
19755 stalwart  20   0 2417184 518244  32420 S  18.4  13.0  37:57.79 stalwart-mail                                                                                                           
19754 stalwart  20   0 2417184 518244  32420 S  12.8  13.0  56:36.42 stalwart-mail    

Any suggestion on what to check or verify?

thank you.

edit: it seems to be linked to a huge account (>6GB mail), while a new account doesn't cause stalwart to slog down.

Update: i can reliably reproduce the issue when removing subscriptions to IMAP folders via roundcube.
Removing one substription works instantanously. Removing the second one will cause the CPU spike and timeouts. Again, removing one more immediately after works, and the next one not.

Update2: still experimenting with the imap folder subscription, it seems that if i unsubscribe from the first folder, it's immediate but after a few seconds tough, stalwart start hogging the CPU without any other action from my side. While hogging the CPU,stalwart will not react to another request of any kind over IMAP, while the admin interface seems responsive.

update3: using iotop, i verified that stalwart is not doing any significant disk I/O when this happen. Actually, it sits a 0.00 bytes read and write. It doesn't seems to be a swap/ram issue since my swap file is 0% used.

[enaut]

It might not be the issue but you could test another storage backend as Rocksdb is not made for spinning disks:

Optimized for Fast Storage

RocksDB is optimized for fast, low latency storage such as flash drives and high-speed disk drives. RocksDB exploits the full potential of high read/write rates offered by flash or RAM.

Source: their website

[gardiol]

I have migrated to a filesystem + SQLite storage and, while the import is still in progress, it seems that it actually solves the issue. Will report back again when the import will be finished.

[gardiol]

After more extensive testing, i can confirm that RocksDB definitely doesnt work for spinning disks. SQLite also is not suitable for 6+Gb of mail (it got over 55Gb of SQLite file after 24h of importing mail...). I finally switched to PostgreSQL and can report that it works fine, it's reasonably fast and does'nt hang.