Restructure cache directory to use subdirs to avoid issues with long URL's #1940
Conversation
Alpine APK keys were not being cached properly in offline mode due to three issues: 1. fetchAlpineKeys() did not wrap the HTTP client with cache transport, bypassing offline cache entirely 2. cachePathFromURL() did not handle top-level URLs correctly (e.g., releases.json at the root path) 3. fetchOffline() could mistakenly select directories as cached files, causing "is a directory" errors This commit fixes all three issues by: - Wrapping the client with cache transport in fetchAlpineKeys() - Simplifying cachePathFromURL() to use full URL path structure uniformly - Filtering out directories in fetchOffline() when searching for cached files Note: The cache path structure change is not backwards compatible. Existing caches will need to be rebuilt, but old and new cache structures can coexist in the same cache directory.
The cache path structure was changed to use the full URL path instead of just the last two path components. This better handles top-level files like releases.json while maintaining proper cache organization. Updated test expectations to use the new structure: - Old: cache_root/encoded_full_repo_url/arch/file - New: cache_root/encoded_base_url/path/to/arch/file
|
Landing the cache structure change will be good, but possibly want a feature flag for this new behaviour; and/or allow opt out; or coordinate simultanious change across:
Because rules_apko has own cache implementation and yet it calls apko binary to do the builds in isolated environment. Separately terraform-provider-apko uses apko as a library and does the cache, and then mounts that cache into apko container the binary to do reproducibility checks and in that case the cache needs to be understood by both. Thus it would be nice if this new caching code, would also support looking up old cache paths, in addition to the new cache paths. |
javacruft
changed the title
In that case previous PR should also be rolled back to not break it when not using the feature flag.
Or indeed fallback to old cache if the paths are empty. |
There was no change to cache structure (i.e. using subdirs). We intentionally split it here. |
|
OK, just trying to point out the latest apko release is broken. When can we expect a fix? |
3 participants
Restructure the cache directory into top level domains, using the path component to create subdirectories.
This resolves an issue where long URL's generated with virtualapk is used in conjunction with other APK repo caching tools such as artifactory where the max filename length limit is hit for the top level cache directory name as it includes the full path.
This is not backwards compatible.