kvserver: decide early to apply snapshot as a write

[pav-kv]

Snapshots can be applied to a Replica storage in two ways:

1. IngestAndExciseFiles excises a span and applies an arbitrary number of "regular", shared and external SSTs.
2. ConvertFilesToBatchAndCommit is a fast-path for small/simple snapshot ingestions (containing only "regular" SSTs and not exceeding 100 KiB).

The decision is made shortly before applying the snapshot. The fast-path (2) needs to read back the SST files that the MultiSSTWriter has already written/synced to storage.

However, the information needed for this decision (the total SST size, and the number of shared/external SSTs) is known way earlier at the sender (upd). If we pass this information in the snapshot header/handshake (we already have at least a proxy for that: size, shared/external), the receiver can avoid the MultiSSTWriter and the conversion step (2), instead generating the batch directly. This would result in faster small snapshots.

Other benefits of this approach:

• The slightly redundant tracking of cleared spans in the snapshot preparation code can be removed.
• Eliminating ConvertFilesToBatchAndCommit (which is a rather ad-hoc API and can change) removes the need for this special case to leak into storage (we need to persist ingestion call parameters into WAG). This one probably can be addressed independently: convert to batch and store this batch to the WAG, instead of the call parameters.

Jira issue: CRDB-55413

[pav-kv]

I am wrong in assuming this:

However, the information needed for this decision (the total SST size, and the number of shared/external SSTs) is known way earlier at the sender.

In the code, we first send the Header, and then discover the shared/external SSTs later when generating the content of the snapshot.

That doesn't preclude the approach though, just makes it more challenging, e.g.:

• In the Header, we could send an "estimate" of whether there are any shared/external SSTs, rather than say it precisely. In the same way the range size is approximate.
• Or we could unconditionally start off building a batch at the receiver, and convert it to SSTs if it becomes too big, or shared/external SSTs are discovered on the way.

So it would be good to understand whether the performance impact is worth it.