Summarize test run rule evaluations in a separate table.

[apognu]

This is a prototype of using the worker set up worked on in #855 to periodically save a test run rule execution statistics (for both the live and phantom versions) into a summary table, for faster querying.

[Pascal-Delange]

All clear and cool.
The only thing I have to say is, if we're doing this, we might as well complete it and also do the pre-accounting of:

• sanction check rule execution stats (goes inside the same stats table as the other rules, see a comment inline about the stable rule id)
• decisions by status stats (this one is less painful that the decisions/rules stats, but it's still liable to get significantly slow in the face of high decision volumes)
  WDYT?

[Pascal-Delange]

do we also want a define timeout on the job ? I'd say something in the ballpark of 20s to a minute would be reasonable

[apognu]

What would be the next success condition when we hit the timeout?

Except for transient errors, it is fair to say that a run hitting the timeout will never succeed since the next run would have even more data to process, and so on and so forth.

That raises a question this PR did not address yet which is that of monitoring. How do we raise alerts if a test run summary lags too far behind?

[Pascal-Delange]

Good question.
By default what would happen now is, if the job consistently fails, we would get notified on sentry (at least I think so - it's worth double checking if a timeout specifically triggers a sentry alarm in the job logger middleware).
Your remark about the "doom loop" where if it fails, it is expected to fail consistently is valid, and a good reason to have a long-ish value for the timeout, at least something longer than a duration that seems realistic to wait for it (a few minutes may do the job)

[Pascal-Delange]

That being said, I still think it's good to explicitly set the timeout, as not doing so will only implicitly inherit the default job timeout that is (not, in this case) set at the client level.

[apognu]

I set a 2-minute timeout for now. All timings will have to be defined, though (intervals and timeouts).

[Pascal-Delange]

A warning here about stable id if we also do the stats of sanction check rules: there is a bit of a dirty hack in the TestRunStatsByRuleExecution usecase method that computes a fake (uuid) "stable rule id" on the fly for sanction checks in live/test version. This is fine as long as we only compute them in real time, but we need to add a proper stable rule id on the sanction check if we also want to precompute them

[apognu]

All clear and cool. The only thing I have to say is, if we're doing this, we might as well complete it and also do the pre-accounting of:

* sanction check rule execution stats (goes inside the same stats table as the other rules, see a comment inline about the stable rule id)

* decisions by status stats (this one is less painful that the decisions/rules stats, but it's still liable to get significantly slow in the face of high decision volumes)
  WDYT?

Totally agree, let's talk about it today, I'll start looking at how it works.

[apognu]

I implemented the decision stats in the latest commit. I will have a look at sanctions stats later today.