executors: Investigate cost reduction of registry mirror on Cloud

[eseliger]

Currently, every cloud instance has a docker registry mirror instance running at all times. Most of the time, these are not used and also they aren’t HA. If we could combine them for multiple MIs, the cost, operational burden, and additional cost for HA could be lowered. (Tracked as https://github.com/sourcegraph/sourcegraph/issues/42523)
This will save us a lot of money, and also operational burden. One mirror instance today costs us around $78 to run, and they are very underutilized, as load is spikey so we need ~big machines, but usually only a few minutes during the day.
Combining them will hopefully increase the utilization per machine, and also increase the cache hit rate. We can also add a docker service account to that machine to increase pull limits.

[eseliger]

https://sourcegraph.slack.com/archives/C02MR5PPMKJ/p1664884367067699

[eseliger]

This should do 2 things:

First, we discovered that mirror.gcr.io exists and that we could use it to offload some of the load from our registry mirrors. It sounds like this network traffic would not be counted towards the NAT cost, which would be important for this improvement to be cost effective: https://cloud.google.com/nat/docs/overview#interaction-pga

Second, we want to combine all the registry mirrors currently running (1 per customer) into one managed pool that ALL the MIs will be using. The proposal is for Cloud team to own a central "pull-through mirror registry" that can be used by all Cloud customers.
We'll definitely need an RFC here to explain the security/privacy implications, uptime requirement and anticipated operational / infrastructure, with someone from Cloud team and @eseliger or someone else who's working on executors collaborating on the plan (and maybe parts of implementation).

If we end up doing this, can we eagerly pull certain images to make this offer even better UX? When we share the registry, our "hit rate" should be better than today.

Also, we want to start tracking the hit rate. This is potentially something the registry exposes in its prometheus metrics.

[eseliger]

Another thing to investigate: Can a docker registry be run stateless in something like cloud run?

[eseliger]

This has been covered by Rafs Future of executors doc. Once a decision is made there, this can be implemented or closed.