Maximum known-good value for MaxMsgs in a JetStream stream

[asymmetric]

Hi,

I'm exploring using NATS JetStream to store data that should ~never be discarded, so that it can be picked up and replayed from the beginning by any new consumer.

The data is very small (~100B), so the metadata overhead would be quite large, but given its tiny size I was considering running a stream without a MaxMsgs -- or a very large one, that would allow me years of runway before hitting any hard limits.

The stream would have a LimitsPolicy and R3 replication.

The numbers we're talking about would be ~ 5M messages stored in one stream per year, amounting to ~0.6GB (considering data and overhead).

What are the practical limitations about this setup? How long before I'd hit a hard limit?

Are there people who have configured streams in such a way, and what were the pitfalls?

[jnmoyne]

Many people use streams with no limit or a very large one, and store equivalent (or larger) message sets without a problem. You should be able to test this for yourself easily by using nats bench to generate a large amount of small messages into a stream and then to see for example how fast you can get them back from the stream and at the same time monitor resource usage on your servers. There is no hard limit to the size of a stream, but practically you will eventually run out of some kind of resource, be it disk space or memory space on the servers (as they maintain some per subject indexing). At that point you can investigate ways to shard onto multiple streams (e.g. you could have a stream per year, or decade) and then deploy more servers in your cluster to continue scaling up.

For quick replay of a lot of messages (presented to you in order) consider using 'ordered' consumers.