Generalize BaseStatistics code a bit + document it

[eero-t]

Description

Generalize BaseStatistics code a bit + document it.

Fixes also fine-tuning code syntax error as I could not get CI passing otherwise.

Issues

n/a.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Others (enhancement, documentation, validation, etc.)

Dependencies

n/a.

Tests

Pylint likes the change, and I manually checked ChatQnA microservice before and after the change.

[eero-t]

Setting as draft, until I've done more testing on it.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
comps/cores/mega/base_statistics.py	100.00% <100.00%> (+57.89%)	⬆️

[eero-t]

Attention: Patch coverage is 14.28571% with 12 lines in your changes missing coverage. Please review.

Hm. It appears that statistics output is not tested, so none of the code I changed is covered.

Because my refactoring removed more code than it added, the overall coverage should have improved though.

[eero-t]

Was hard finding a microservice providing these metrics. None of "DocSum" ones provided them, but "ChatQnA" had one.

It works the same before and after the change:

$ curl --no-progress-meter 172.21.145.237:7000/v1/statistics | jq
{
  "opea_service@retriever_redis": {
    "p50_latency": 0.0011343955993652344,
    "p99_latency": 0.002569167613983175,
    "average_latency": 0.0012106266286638048,
    "p50_latency_first_token": null,
    "p99_latency_first_token": null,
    "average_latency_first_token": null
  }
}

[Spycsh]

Attention: Patch coverage is 14.28571% with 12 lines in your changes missing coverage. Please review.

Hm. It appears that statistics output is not tested, so none of the code I changed is covered.

Because my refactoring removed more code than it added, the overall coverage should have improved though.

Thanks for adding documentation. Could you also add the test to cover your code? There is a very simple test of this functionality here https://github.com/eero-t/GenAIComps/blob/stats/tests/cores/mega/test_base_statistics.py. You can add some unit test there to for coverage. (The original test init a server so the code is not detected by the coverage test I guess).

[eero-t]

Thanks for adding documentation. Could you also add the test to cover your code? There is a very simple test of this functionality here https://github.com/eero-t/GenAIComps/blob/stats/tests/cores/mega/test_base_statistics.py. You can add some unit test there to for coverage. (The original test init a server so the code is not detected by the coverage test I guess).

@Spycsh Rebased to main add improved code coverage to 100%. To get CI passing, I had to fix regex syntax error in unrelated code. Ready for merge?

(Sorry for the large number of (force) pushes, it took a while until I found other reason why CI failed to fail parse test Python code although it was pylint clean and ran fine locally.)

[eero-t]

Rebased on main in hopes it fixes the mystery CI issue.