crosscluster/physical: run reader tenant AOST queries at user specified AOST #155369
Description
When a user runs an AOST query at t_user, it will grab the reader catalog at t_user. Then the index fetcher will grab t_AsOf from this historical descriptor, where t_AsOf < t_user. A much better UX for AOST queries would be to run the scan at t_user. This would allow the user to run the same AOST query on both the source and dest and get the same results.
To run the scan at t_user, we could instead pass t_user to the index_fetcher, but to do so, we need to validate that the relevant replicating tenant descriptors at t_user are identical to the historical version of t_AsOf. More concretely, we need to verify that descriptor version did not update from t3 to t5. Since schema changes are rare, the descriptor versions are likely the same.
As an example: imagine the user runs a select on foo at t5, and suppose descriptor foo at t5 in the reader catalog has an AsOF of t3. To run the AOST query at t5, foo@t3 and foo@t5 in the replicating tenant key space must have the same descriptor version.
This validation could be built into some oracle that lives in the reader tenant’s sql server that will take in the read txn’s descriptor collection, and validate the safety of the AOST. If the descriptor versions don’t match, we could throw an error, or run the query at the older AOST.
One side note: this implementation relies on the fact that we bump the AsOF time on each reader catalog descriptor during every catalog update. We could instead exclusively bump the system database descriptor on every reader catalog update and pass the AsOF from the system db descriptor to every scan.
Jira issue: CRDB-55411
Epic CRDB-52290