feat: classify emails by importance based on subjects

[st3iny]

Partly addresses #3968

Closes #8257

Part 5: The great rebasing ...

Improvements

• Classify emails by importance using a new and improved classification pipeline
• Persist classifier models (and transformer pipelines) in memory cache only
• Drop the oc_mail_classifiers table

How to tests?

1. Have some important mails in your inbox.
2. Make sure to have the classification enabled: Account settings modal -> "Data collection consent" ✅ and "Mark as important" ✅
3. Train a model: occ mail:account:train -vvv <account-id> (extract account id with occ mail:account:export <user-id>)
4. Receive some mails and observe that some are marked as important and some aren't.

[ChristophWurst]

First impression: looks good 😎

[ChristophWurst]

I'll observe my account for a while. So far the stats are (suspiciously) excellent:

[debug] found 36 incoming mailbox(es)
[debug] found 1 outgoing mailbox(es)
[debug] found 300 messages of which 42 are important
WCF vocab size: 500
[debug] data set split into 240 (i: 33) training and 60 (i: 9) validation sets with 504 dimensions
[debug] classification report: {"recall":1,"precision":1,"f1Score":1}
[debug] classifier validated: recall(important)=1, precision(important)=1 f1(important)=1

[ChristophWurst]

I'm still not sure how I can judge the quality of the new classification. Is it overfitting?

[kesselb]

[debug] found 7 incoming mailbox(es)
[debug] found 1 outgoing mailbox(es)
[debug] found 300 messages of which 241 are important
[debug] data set split into 240 (i: 195) training and 60 (i: 46) validation sets with 504 dimensions
[debug] classification report: {"recall":1,"precision":0.8518518518518519,"f1Score":0.92}
[debug] classifier validated: recall(important)=1, precision(important)=0.85185185185185 f1(important)=0.92
[debug] Classifier for account 1 persisted

I will keep an eye on the classification ;)

However, I haven't cleaned my inbox in a while and fear the countless undeleted "update xyz announcements" or "calendar notifications" might influence the model 🙈

[kesselb]

1. Would adding an option to exclude or include specific mailboxes from the training improve the results? For instance, my Archive folder holds about 160 important messages, while my "Test Data" folder has around 5,000 emails that I keep for testing purposes. These test emails aren’t relevant at all, but I move them there instead of deleting them to maintain a large mailbox for tests.

2. While reviewing the code, I noticed that starring a message in Thunderbird sets the flag_flagged attribute. Should we consider this attribute when identifying important messages?

mail/lib/Service/Classification/ImportanceClassifier.php

Lines 157 to 159 in e2215d9

	$importantMessages = array_filter($messages, static function (Message $message) {
	return ($message->getFlagImportant() === true);
	});

[ChristophWurst]

Suggested change

	if ($cached == null) {
	if ($cached === null) {

[ChristophWurst]

Previously this condition was only met for new accounts where no training has happened yet. Now servers with no memory cache will always run into the rule-based classifier. I don't think this is desirable, because results will be bad. The rules based classifier is only meant to cold start the classification.
How about we skip classification all together if there is no distributed cache available?

[ChristophWurst]

Or, instead of skipping, we generate a pipeline on the fly? It's slow but gives good results