Add initial cache for Spin URL component sources #1088
Conversation
Before this commit, for a `spin.toml` file that had a URL module source for a component (such as the file server and redirect templates), subsequent runs of `spin up` would always download the module from the URL before starting the application. spinframework#1014 added a content-addressed registry cache that can be used to store remote module sources. This commit updates the Spin local loader to use that cache to store the component sources referenced through URL in `spin.toml`. Before fetching a component, the loader will check the local cache first. If present, it will read the file and return its content; otherwise, it will fetch it, store it in the cache, then return its content. Currently, the default cache root is always used. Once `spin up` will have a way to configure it, that will be propagated here as well. Signed-off-by: Radu Matei <[email protected]>
| .await | ||
| .with_context(|| format!("Error loading source URL {}", self.url))?; | ||
| let bytes = body.into_iter().collect_vec(); | ||
| // TODO: when `spin up` integrates running an app from OCI, pass the configured |
There was a problem hiding this comment.
could you add an issue to track this after this PR is merged?
There was a problem hiding this comment.
@fibonacci1729 I for one had forgotten about this comment when you were doing spin up --from-registry - did you already deal to it, or should we resurrect it as an enhancement request
There was a problem hiding this comment.
@itowlson I completely missed this. We should about an issue. If you haven't got to it by your today, I'll open an issue during my tomorrow
There was a problem hiding this comment.
I got in there quick so I could assign it to you. flees #1188
|
We should add an e2e test for this scenario so we can capture the behavior somewhere and make sure it continues working as expected. |
| @@ -28,7 +28,7 @@ use spin_manifest::{ | |||
| }; | |||
| use tokio::{fs::File, io::AsyncReadExt}; | |||
|
|
|||
| use crate::{bindle::BindleConnectionInfo, digest::bytes_sha256_string}; | |||
| use crate::{bindle::BindleConnectionInfo, digest::bytes_sha256_string, oci::cache::Cache}; | |||
There was a problem hiding this comment.
Cache should be refactored out of the oci mod (up to spin_loader::cache, presumably).
There was a problem hiding this comment.
My assumption is that when we do end up refactoring the loader crate and cleanup any unused Bindle code, this refactor would happen.
ref #455
Before this commit, for a
spin.tomlfile that had a URL module source for a component (such as the file server and redirect templates), subsequent runs ofspin upwould always download the module from the URL before starting the application.#1014 added a content-addressed registry cache that can be used to store remote module sources.
This commit updates the Spin local loader to use that cache to store the component sources referenced through URL in
spin.toml.Before fetching a component, the loader will check the local cache first. If present, it will read the file and return its content; otherwise, it will fetch it, store it in the cache, then return its content.
Currently, the default cache root is always used. Once
spin upwill have a way to configure it, that will be propagated here as well.Signed-off-by: Radu Matei [email protected]