Skip to content

allow passing a Accept-Encoding header to PkgServer to fetch e.g. zstd compressed files from the storage server #220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

allow passing a Accept-Encoding header to PkgServer to fetch e.g. zstd compressed files from the storage server #220

wants to merge 2 commits into from

Conversation

@KristofferC
Copy link
Member

@KristofferC KristofferC commented Oct 27, 2025

for backwards compatibility, return gzipped files when no Accept-Encoding is provided

Written together with Claude code, but a lot of hand-holding and manual tweaking.

@KristofferC
allow passing a Accept-Encoding header to PkgServer to fetch e.g. z…
db76373 
…std compressed files from the storage server

for backwards compatibility, return gzipped files when no `Accept-Encoding` is provided
giordano
giordano reviewed Oct 27, 2025
View reviewed changes
@KristofferC KristofferC force-pushed the kc/accept_encoding_zstd branch from 520030d to 9a9cd02 Compare October 27, 2025 11:07
@KristofferC
Copy link
Member Author

KristofferC commented Oct 27, 2025

The startup timing test seems to hang after finishing?

1h 20m 20s
┌ Info: Successful connection:
└   duration = 64.83147096633911

Don't know what to do about docker:

Run echo "${DOCKER_PASSWORD}" | docker login --username "${DOCKER_USERNAME}" --password-stdin
Must provide --username with --password-stdin

@KristofferC
Copy link
Member Author

KristofferC commented Oct 27, 2025

(cc @staticfloat)

KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
8c5580c 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
773ce94 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
d436254 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
43d62af 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
1c17917 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
3ca05c0 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
585a426 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
e797553 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
@KristofferC KristofferC mentioned this pull request Oct 27, 2025
Merged
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
b29f71d 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
33bd78f 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
KristofferC added a commit to JuliaLang/Pkg.jl that referenced this pull request Oct 27, 2025
@KristofferC
ask the package server for zstd compressed data
e81cfa8 
Together with JuliaPackaging/PkgServer.jl#220 this allows Pkg to download registries, packages and artifacts as zst archives which based on benchmarks decompress significantly faster and also tend to be smaller.

Having the registry stored as a zst archive is not a backwards compatibility issue because even on older Pkgs it can decompress these with 7z
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@giordano giordano giordano left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@KristofferC @giordano