Upload arm64 docker images where relevant

[ashwinb]

🚀 The feature, motivation and pitch

This is especially important for a "non-heavyweight-GPU" focused distribution like ollama or tgi-cpu, etc.

Alternatives

There isn't a viable alternative.

Additional context

No response

[nathan-weinberg]

@cooktheryan is this related to some of the image publishing work you are doing?

[cooktheryan]

@nathan-weinberg no but wouldn't mind owning it if @ashwinb doesn't want to take it

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.

[codefromthecrypt]

let's try to not let this stale out, cc @bbrowning because the dev story is impacted if we relegate apple silicon docker to emulation

[bbrowning]

I've tried to use our published container images a few times, and agree it would be great for us to build and publish ship arm64 container images.

[bbrowning]

So, I have a local branch of our llama-stack-ops repo that builds and publishes arm64 images. The core bits in this llama-stack repo already support that. Things are a bit more complicated than they typically would be because we use llama stack build to create the container images instead of just relying on the normal GitHub Actions to do this for us with Dockerfiles. But, nothing we can't deal with there.

You can see an example of this output here - https://hub.docker.com/repository/docker/bbrowning/distribution-remote-vllm/general . You'll see I built and pushed single image test-0.2.9rc1-arm64 and test-0.2.9rc1-amd64 tags but also a multi-arch test-0.2.9rc1 tag that is an example of the manifest lists we'd point people to typically use. This lets me podman run --rm -it docker.io/bbrowning/distribution-remote-vllm:test-0.2.9rc1 from a Mac or Linux laptop and it automatically picks up the right image with everything working as expected.

However, not every distribution can build arm64 images. For example, the meta-reference-gpu distribution cannot build under arm64. Many of the others can. So, it's not as simple as just adjusting the existing release process to build and push arm64 images, and instead we need to keep track of which distributions are buildable under which architectures. We'll want to test and ensure those don't regress in this repository, so that we don't get hit with surprises when attempting to cut a release from the llama-stack-ops repo. It would be bad to be in the middle of cutting a release before we realized we accidentally introduced a dependency that breaks arm64 container builds for one or more distributions, for example.

[codefromthecrypt]

agree. @bbrowning so are you thinking of a filter in the automation where we choose "known working" or maybe the opposite? I'm not sure the routine build time, but if it were an allow list, CI could attempt to build and do a smoke test of each releaseable dist, then break the build if it can't.. well be built. of course we don't want to block PRs on a very long thing by default but perhaps some file-based triggers that taint it?

[bbrowning]

We currently have a CI job in this repo (providers-build.yml) that installs each distribution. I may prototype what it would look like to add supported architectures as an attribute of distributions, and then read that metadata from CI to build each provider based on the architecture it supports. That would ensure all the Python bits work to install on the given architecture, so we don't get surprised during releases later. I haven't fully though this through yet, but may at least start poking in that direction.

[bbrowning]

I had a few hours to work on this, so opened meta-llama/llama-stack-ops#33 that will get us building and pushing amd64 images for every distribution, arm64 images for all our distributions that build on arm64, and the existing image coordinations that users are used to using will be manifest lists so that the right architecture automatically gets selected based on the client pulling the image.

This doesn't attempt to keep a list of which providers work under amd64 or arm64 - it just tries to build all the distribution templates under both and gracefully handles any arm64 build failures so that we only push amd64 images for any distros that aren't compatible with arm64.

[bbrowning]

Testing the arm64 built starter distribution image on a mac with podman desktop, I'm able to pull the image and run the image by giving it a custom entrypoint. However, I'm unable to actually run our starter distro on a mac with podman desktop due to what looks like a sqlite-vec issue. It looks like that dependency is bundling in at least one .so prebuilt binary that's not actually arm64 compatible and trying to load it - sqlite3.OperationalError: /usr/local/lib/python3.10/site-packages/sqlite_vec/vec0.so: wrong ELF class: ELFCLASS32. So, we may need to figure out which providers actually work under arm64, which is a separate issue from which build under arm64 for those providers that just ship prebuilt binaries for the wrong architecture.

[anuraaga]

Thanks for the help on this @bbrowning - just did a bit of investigation into the sqlite_vec issue and I found upstream should already have a fix for it but it's not released yet

asg017/sqlite-vec#211

[bbrowning]

Thanks for the pointer! We may need to rethink the default vector store provider in the starter distribution so that it also works for Mac users...