Traffic model benchmark

[artikell]

from #368 (comment)

Is it necessary and how to construct testing standards for NoSQL products.

[zuiderkwast]

Thanks for opening this thread.

There is already some command statistics collected. See INFO commandstats, https://valkey.io/commands/info/

It may be possible to start with only this.

What more do we need? Key and value sizes? Something else?

It's a good point that statistics can be sensitive to share. In that case, companies can run our benchmarks on their side with their own traffic model.

[artikell]

@zuiderkwast I think there are three things that need to be implemented:

• Bench framework adaptation: There are 2 options: https://github.com/pingcap/go-tpc And https://github.com/mongodb-labs/py-tpcc
• Implementation of workload: Caching scenarios, flow limiting, distributed locks, etc，Choose one.
• Workload dataset distribution: Some traffic ratios need to be discussed, and the size of keys and values is reasonable

Of course, there are still many things to discuss, but it would be best to have a demo.
I will quickly implement a model and produce a document to support subsequent discussions to determine if this direction can meet the requirements.

[yairgott]

I think this makes sense to me in general but I would like to suggest some concrete requirements (some of which are obvious but stating to be on the safe side):

1. Enable A/B testing on local a linux VM. It's important that one may run the benchmarks on a common hardware on Linux. This is very useful when working on performance improvement change.
2. We need an extensible model were tests can be easily added.
3. We need the ability to group benchmarks into suits, essential/caching/networking/etc. This is useful when working on a change related to a specific domain.
4. We need to periodically, preferably on a nightly basis, run these benchmarks. In addition, we need the ability to be alerted of detected performance regressions.
5. To be able to maintain these benchmarks, I suggest that we would have an owner per benchmark who is responsible to address false positives/noise reported by the nightly runs. For real introduced regressions, the owner is responsible to identify the offending change and work with the change owner to address the regression.

[zuiderkwast]

Good points, but let's start with something simple. If we want all of this from the start, it may not happen at all.

[yairgott]

Agreed. Lets start with something and overtime reach some agreed north star.

Thanks for driving this!