Improve precompiled binary gem support #60
Conversation
text/0000-wheel-platform-support.md
Outdated
|
|
||
| The wheel platform format follows the pattern `whl-{abi_tags}-{platform_tags}`: | ||
|
|
||
| - `abi_tags`: Ruby ABI version (e.g., `rb33` for Ruby 3.3, `any` for pure Ruby, `cr3` for any CRuby with major version 3) |
There was a problem hiding this comment.
cr3for any CRuby with major version 3
CRuby doesn't provide a ABI compatibility level which specially handle Ruby major version. In other words, Ruby's major version is used for marketing purpose. Technically "3.3" is semantic "major version".
There was a problem hiding this comment.
Indeed, so I think cr3 should be removed/not supported as it can't be used in practice.
There was a problem hiding this comment.
👍🏻 cr3 has been removed
text/0000-wheel-platform-support.md
Outdated
|
|
||
| **`Gem::Platform::ELFFile`** - Analyzes ELF binaries on Linux | ||
| - Reads interpreter path to identify musl vs glibc | ||
| - Parses dynamic symbols to determine minimum libc version |
There was a problem hiding this comment.
Does this insists the minimum libc versions is automatically detected? If so, it only detects when a version is buggy or a feature is extended only when a specific argument is given.
There was a problem hiding this comment.
It is very difficult to reliably detect which libc is present and what the version is without compiling a program and executing it. The overall goal is to determine which libc is present (eg musl or glibc) and then which version is present, and use that version to look for a binary gem that was built against the same version or lower. The exact implementation of this system is not fixed, and can be changed and updated over time.
text/0000-wheel-platform-support.md
Outdated
|
|
||
| - Should rubygems.org automatically build wheel variants when gems with native extensions are pushed? | ||
| - What tooling should we provide to help gem authors build for multiple platforms? | ||
|
|
There was a problem hiding this comment.
How will this handle
- libraries other than libc for example openssl, libxml, libxslt, libyaml, libffi, and so on?
- CPU features like SSE and AVX
There was a problem hiding this comment.
Those are unhandled, same as today
There was a problem hiding this comment.
I understand, and I think it should be explicitly listed in this section because some of ruby core committers discussed about that. To avoid such confusion and postpone such discussion, it is a good practice to list these here.
There was a problem hiding this comment.
If this proposal focuses on only abi_tags, making them unhandled makes sense.
But this proposal includes platform_tags too. I think that platform_tags without handling them doesn't make sense. Because we can't create "portable Linux binaries" without them. If a Linux binariy static links to OpenSSL 1.1 and other Linux binary static links to OpenSSL 3.0, they can't be mixed.
Python's wheel also has this problem. So I think that we should not copy Python's wheel as-is. We should learn from Python's wheel and implement our better Python's wheel if we want to implement portable Linux binaries.
FYI: I think that static linking based binary gems approach can't resolve this problem.
There was a problem hiding this comment.
So I think that we should not copy Python's wheel as-is. We should learn from Python's wheel and implement our better Python's wheel if we want to implement portable Linux binaries.
💯
There was a problem hiding this comment.
As the protobuf maintainers made clear, it is not supported to statically link libprotobuf more than one time. As a result, any gem depending on libprotobuf will likely have trouble shipping a precompiled binary gem. Perhaps a future improvement to the tagging system introduced in this RFC could track shared locations and versions of libprotobuf to enable precompiled gems to share a dynamically linked libprotobuf.
There was a problem hiding this comment.
(The rust ecosystem has a pattern of using -sys libraries to represent only the raw linked library and not any wrappers, I wonder if that would be a direction to solve this problem?)
There was a problem hiding this comment.
If a Linux binariy static links to OpenSSL 1.1 and other Linux binary static links to OpenSSL 3.0, they can't be mixed.
FWIW OpenSSL is even more complicated, e.g. the ABI version of OpenSSL is not enough to ensure binary compatibility as e.g. there is FIPS and non-FIPS and other configure flags set e.g. by the OS which affect the ABI in practice but don't report a different ABI version.
Static linking it though should work, there used to be an issue with the openssl gem but that got fixed: ruby/openssl#517 (comment)
There was a problem hiding this comment.
(The rust ecosystem has a pattern of using -sys libraries to represent only the raw linked library and not any wrappers, I wonder if that would be a direction to solve this problem?)
It is "(3) Use the library installed through another gem" and similar to libv8's approach. If we use this, the version of the library needs to be coordinated. For example OpenSSL, an application needs to decide whether it uses 1.1 or 3.0 in the ruby process.
There was a problem hiding this comment.
In my opinion, "(3) Use the library installed through another gem" isn't a good idea.
If we choose the approach, it means that we re-implement a C/C++/... packaging system on RubyGems. There are many packaging systems such as Homebrew, vcpkg, Nix, ... Maintaining a packaging system need many resources. I think that we should not re-invent a packaging system to save resources of Ruby ecosystem.
RubyInstaller2 integrates with pacman and provides RubyGems integration https://github.com/oneclick/rubyinstaller2/wiki/For-gem-developers#-msys2-library-dependency . Can we use similar approach on other environments?
text/0000-wheel-platform-support.md
Outdated
| whl-rb33-manylinux_2_28_x86_64 | ||
| ``` | ||
|
|
||
| The choice impacts lockfile portability - more specific platforms ensure exact binary reproducibility but may cause unnecessary gem recompilation when moving between similar environments. |
There was a problem hiding this comment.
A binary will differ and maybe not compatible between compiled environment especially on Linux for example the ABI compatibility of libc. In this proposal, it is recorded as Gem::Platform::Manylinux.
There was a problem hiding this comment.
I think only option 3 is compatible with all existing, current gems, right?
There was a problem hiding this comment.
About existing, current gems, yes.
I meant x86_64-linux doesn't provide libc version information and practically doesn't ensure binary compatibility.
There was a problem hiding this comment.
As far as Gem::Platform is concerned, x86_64-linux implies glibc, but does not allow specifying a version. This is a limitation in Gem::Platform as it stands now
|
If RubyGems adds something like this, it seems the perfect occasion to also add the possibility to specify that a binary gem is for a given RUBY_ENGINE (e.g. ruby/jruby/truffleruby). |
text/0000-wheel-platform-support.md
Outdated
|
|
||
| # Summary | ||
|
|
||
| Add wheel-format platform support to RubyGems for more fine-grained matching for binary gem distribution through ruby/api tags and platform tags (`whl-{abi_tags}-{platform_tags}`). This enables "platform matching" based on Ruby ABI version, os, os version, cpu architecture, and libc implementation (for generic linux platforms). |
There was a problem hiding this comment.
It should also handle different RUBY_ENGINE, so e.g. one can have precompiled native extensions for TruffleRuby 24.2.x.
And also have the ability to mark some precompiled native extensions as CRuby-specific (for most extensions, so would be the default if using spec.extensions = for a binary gem) and some as platform-specific but not RUBY_ENGINE-specific (e.g. precompiled gems using ffi). More about that in https://github.com/orgs/rubygems/discussions/5988#discussioncomment-3882572
We could consider the RUBY_ENGINE to be part of the ABI in general potentially, but then we should explain how that looks like concretely (e.g. "#{RUBY_ENGINE}-#{RbConfig::CONFIG['ruby_version']}"). It seems a bit confusing though, because ABI version for Ruby normally means just RbConfig::CONFIG['ruby_version'].
(BTW Python does support wheels for alternative Python implementations, though I don't know exactly how they specify it)
There was a problem hiding this comment.
Just FTR, ruby_version is user defined string, despite what the name might imply:
https://github.com/ruby/ruby/blob/78b8ecd1ea79c777260c2d0221835ca2e5cdca31/configure.ac#L4385
There was a problem hiding this comment.
And for CRuby is by default in form of X.Y.0; to me for unknown reason, since per my understanding, there are no guarantees around this.
There was a problem hiding this comment.
And for CRuby is by default in form of X.Y.0; to me for unknown reason, since per my understanding, there are no guarantees around this.
It is the ABI, so it is compatible and guaranteed for all CRuby X.Y.Z versions sharing the same X.Y to have the same ABI (through RbConfig::CONFIG['ruby_version'] not changing for those versions).
For example Bundler with path will install gems based on that ABI version, and so will share the gems for e.g. 3.4.1 and 3.4.2.
Just FTR,
ruby_versionis user defined string, despite what the name might imply:
Good to know, OTOH the whole ecosystem relies on that being the ABI version (including current RubyGems and Bundler).
I guess it doesn't hurt if it's more precise, but I see no value to change it.
As in many configure flags, some values or combinations just generate a broken Ruby, I don't think we need to consider that here. If people change this they better understand the details of CRuby ABI and it will be at their own risk.
There was a problem hiding this comment.
Generally what eregon says is correct.
One point, as eregon says, Ruby X.Y.Z usually has the ABI version X.Y.0. But if something happen, it may use X.Y.3 for X.Y.3+ for example. As far as I understand one expected situation is security fix which affected the ABI.
There was a problem hiding this comment.
Are there any gems that need to select based on engine but don’t also need to select based on engine ABI version? If so, we can add “generic” engine tags to act as “this engine, but any version will work”.
There was a problem hiding this comment.
Are there any gems that need to select based on engine but don’t also need to select based on engine ABI version?
Yes, for gems which don't use a native extension but still need tweaks for a specific Ruby engine. For example JRuby already uses that via the java platform (see this comment), and TruffleRuby would use it as well, e.g. to avoid the unnecessary libv8-node dependency in mini_racer.
|
I was definitely assuming that RUBY_ENGINE would be one of the tags possible to select on. @segiddins, is it possible for us to get an exhaustive list of every possible tag we could expect to see under this system, rather than just a couple of "for instance" tags, so we know exactly what would be included? |
text/0000-wheel-platform-support.md
Outdated
|
|
||
| The wheel platform format follows the pattern `whl-{abi_tags}-{platform_tags}`: | ||
|
|
||
| - `abi_tags`: Ruby ABI version (e.g., `rb33` for Ruby 3.3, `any` for pure Ruby, `cr3` for any CRuby with major version 3) |
There was a problem hiding this comment.
Is there a different meaning between Ruby (from rb33 for Ruby 3.3) and CRuby (from cr3 for any CRuby with major version 3) on this line?
I think in both cases you mean the Ruby C API as implemented by CRuby X.Y, IOW the CRuby ABI.
There was a problem hiding this comment.
This has been updated in the RFC, but rb is just about RUBY_VERSION. cr is specific to RUBY_ENGINE=ruby
There was a problem hiding this comment.
The ABI version is AFAIK always RUBY_ENGINE-specific.
In theory a new Ruby implementation could try to support the exact same ABI as CRuby (there is none currently), but if so then RubyGems should just consider e.g. cr33 to be compatible with that new Ruby implementation.
IOW, I think this should be removed:
`rb33` for Ruby 3.3,
Because there is no such thing as a "Ruby" ABI 3.3 which is compatible with more than CRuby ABI 3.3.
There was a problem hiding this comment.
I saw #60 (comment), so maybe for wasm extensions it might be possible to be binary compatible across Ruby engines.
But then I think that ABI would be specific to the extension WASM ABI, and not necessarily related to RUBY_VERSION.
Do we have wasm extensions currently? I guess not.
So I think the rb33 should still be removed.
And mabe maybe a mention that something like wasm1.0 could be added for wasm extensions (where 1.0 is the wasm extension ABI version).
text/0000-wheel-platform-support.md
Outdated
| - Provides platform detection without external dependencies | ||
|
|
||
| **`Gem::Platform::Manylinux`/`Gem::Platform::Musllinux`** - Linux compatibility standards | ||
| - Implements PEP 600 (manylinux) and PEP 656 (musllinux) specifications |
There was a problem hiding this comment.
PEP 600 includes manylinux{1,2010,2014} for backward compatibility. We should not support them. We should accept only manylinux_${GLIBCMAJOR}_${GLIBCMINOR}.
There was a problem hiding this comment.
I was not specific enough, I only ever intended to support the new _${GLIBCMAJOR}_${GLIBCMINOR} pattern
|
We shouldn't use
|
I do agree. RubyGems deserves own naming to not confuse people. The wheel itself makes no sense in RubyGems context and the original idea behind wheel naming also makes no sense. https://discuss.python.org/t/where-the-name-wheel-comes-from/6708/2 |
text/0000-wheel-platform-support.md
Outdated
|
|
||
| # Summary | ||
|
|
||
| Add wheel-format platform support to RubyGems for more fine-grained matching for binary gem distribution through ruby/api tags and platform tags (`whl-{abi_tags}-{platform_tags}`). This enables "platform matching" based on Ruby ABI version, os, os version, cpu architecture, and libc implementation (for generic linux platforms). |
There was a problem hiding this comment.
Is it possible (in theory) to make wheel with only ruby code inside eligible for all platforms?
There was a problem hiding this comment.
I think only test assures the compatibility. I mean technically impossible, but I think it can expresss the intention of the author.
There was a problem hiding this comment.
The “any” tag will work for any ruby engine, on any platform, with any version. That should work for gems that are only ruby code, but in that case you also might as well not add any platform tags at all and just release a regular gem.
text/0000-wheel-platform-support.md
Outdated
| # Unresolved questions | ||
|
|
||
| - Should rubygems.org automatically build wheel variants when gems with native extensions are pushed? | ||
| - What tooling should we provide to help gem authors build for multiple platforms? |
There was a problem hiding this comment.
Let's provide guidance, not infrastructure. Simple usable official GitHub Action could be good starting point covering most of the needs and could be fitting also trusted publishing at the same time.
There was a problem hiding this comment.
I agree.
I fully agree the idea that rubygems.org provide build farm, but it is huge project. And also we also should provide a way to build wheel variants by author to debug it.
There was a problem hiding this comment.
Should we proceed this RFC without resolving these questions?
I think that we should resolve these questions before this RFC.
See also: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13293536
There was a problem hiding this comment.
As far as I understand, kou says if rubygems.org provide build farm, it can build binary gems just after newer ruby is released.
I agree it provides great value for ruby eco system and we should set it as a future goal.
There was a problem hiding this comment.
If Ruby Central is able to do this, that would be great. However, promising a rubygems.org build farm is outside the scope of this RFC, because it depends on Ruby Central deciding to spend both time and money on this build farm.
I believe the best option for this RFC is to agree to create a GitHub Action template that can be used to easily build and publish binary gems.
There was a problem hiding this comment.
Yes, a GitHub Action similar to cibuildwheel seems like the answer to “what kind of tools should we provide”. The exact details of the tool are outside the scope of this RFC, which is only about platforms, but we should definitely build such a tool (and hopefully that tool will also provide easy to set up scheduled builds or triggered builds when new Ruby is released).
There was a problem hiding this comment.
FYI: Related discussion: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13243371
|
Would you mind to share also some real-world example? What about nokogiri? Today it needs 9 gem platforms (including Ruby) to be released to cover most of the needs (Windows, Linux MUSL, Linux GNU, macOS). It bundles 4 Rubies into fat gems (3.1, 3.2, 3.3, 3.4). Fat bins are scoped with required ruby constraint to ensure it is not missing related build inside during runtime. Ruby platform only defines minimal Ruby version. What are current problems with this setup wheel will resolve? Can you provide the recommended setup (individual wheel tags, ...) to release Nokogiri? |
text/0000-wheel-platform-support.md
Outdated
| The wheel platform format follows the pattern `whl-{abi_tags}-{platform_tags}`: | ||
|
|
||
| - `abi_tags`: Ruby ABI version (e.g., `rb33` for Ruby 3.3, `any` for pure Ruby, `cr3` for any CRuby with major version 3) | ||
| - `platform_tags`: Platform specification (e.g., `x86_64_linux`, `manylinux_2_28_x86_64`, `arm64_darwin_23`) |
There was a problem hiding this comment.
I think that we should not use manylinux. I feel that its naming is a wheel failure.
We should use glibc_linux/linux_glibc or something if we assume glibc by manylinux.
There was a problem hiding this comment.
If we use the definition of certain manylinux set like manylinux_2_34, we should the name as is.
I assume it includes the set of a specific version of Linux distro, Linux kernel, glibc, and cpu arch.
There was a problem hiding this comment.
Old PEPs such as PEP 513 https://peps.python.org/pep-0513/ include such information. For example: https://peps.python.org/pep-0513/#the-manylinux1-policy
But PEP 600 https://peps.python.org/pep-0600/ doesn't include them. PEP 600 includes only glibc version.
There was a problem hiding this comment.
Ah, I understand your intention. Since Linux environment changes as time goes by, it is hard to define sustainable common linux environment as "manylinux" initially intended. Therefore newer manylinux definition refers major Linux distributions through the glibc version. If there is a edge case like libcurses 6 and OpenSSL, it is out-of-scope of abi_tags.
There was a problem hiding this comment.
As I mentioned above, perhaps a future improvement to platform tags can handle dynamic library locations or versions, but this RFC does not include a solution for dynamic, shared libraries.
text/0000-wheel-platform-support.md
Outdated
|
|
||
| The implementation adds four new classes to `lib/rubygems/platform/`: | ||
|
|
||
| **`Gem::Platform::Wheel`** - Parses wheel-format platform strings (`whl-{abi}-{platform}`) |
There was a problem hiding this comment.
I also think that we should not use "wheel".
There was a problem hiding this comment.
It makes sense to not use the same name as Python to prevent search result pollution. The proposal is updated to use the string “gb” (for gem binary) instead of “whl”.
text/0000-wheel-platform-support.md
Outdated
| - Provides platform detection without external dependencies | ||
|
|
||
| **`Gem::Platform::Manylinux`/`Gem::Platform::Musllinux`** - Linux compatibility standards | ||
| - Implements PEP 600 (manylinux) and PEP 656 (musllinux) specifications |
There was a problem hiding this comment.
PEP 600 includes manylinux{1,2010,2014} for backward compatibility. We should not support them. We should accept only manylinux_${GLIBCMAJOR}_${GLIBCMINOR}.
text/0000-wheel-platform-support.md
Outdated
| p.for platform: "x86_64_linux", abi: ["rb33", "rb32"] do | ||
| add_dependency "linux-specific-gem" |
There was a problem hiding this comment.
In general, we should use block variable instead of instance_eval for readability.
There was a problem hiding this comment.
The proposal has been updated to use block variables.
text/0000-wheel-platform-support.md
Outdated
| # Unresolved questions | ||
|
|
||
| - Should rubygems.org automatically build wheel variants when gems with native extensions are pushed? | ||
| - What tooling should we provide to help gem authors build for multiple platforms? |
There was a problem hiding this comment.
Should we proceed this RFC without resolving these questions?
I think that we should resolve these questions before this RFC.
See also: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13293536
text/0000-wheel-platform-support.md
Outdated
|
|
||
| The platform matching system now uses `Gem::Platform::Specific.local` instead of `Gem::Platform.local` for more precise matching. The `sort_priority` method in `Gem::Platform` now assigns: | ||
|
|
||
| 1. Ruby platform: priority -1 (highest, for pure Ruby gems) |
There was a problem hiding this comment.
Why -1 is the highest priority? It seems that it should be 3 or larger.
There was a problem hiding this comment.
I agree. This doesn't provide a new value.
|
What about WASM platform? Is it supported? |
text/0000-wheel-platform-support.md
Outdated
|
|
||
| # Drawbacks | ||
|
|
||
| - **Increased complexity** - The resolver must handle more platform variants, though this complexity is isolated to the new wheel classes. |
There was a problem hiding this comment.
Does this affect compact index? Is it needed to store more info?
There was a problem hiding this comment.
I believe adding more platform information should not change the compact index—there will be more versions, and those versions will have longer platform strings, but everything else should stay the same.
There was a problem hiding this comment.
Yes, the compact index will stay the same, and some gems will have more versions with longer platform strings. Backwards compatibility with the existing full & compact indices was the main constraint I was working around with this design shape.
indirect
changed the title
|
Before any further discussion takes place, I would like to take a moment to mention that I am strongly in favor of the overall goal of this RFC: Ruby should have better support for precompiled gems. I am happy to take advantage of the lessons that Python has learned in making wheels, and I want RubyGems to be at least that good. I think we can likely do even better, since we are able to create something new that learns from the issues that have come up with wheels. I hope we can use this RFC to agree on ways to make using gems faster and easier for everyone. <3 |
|
It seems the name discussion has split across several different inline comment threads, so I would like to gather it together: what should we call this new format? I think the easiest answer is "reels", because Ruby just puts an r at the front of everything. Another option could be "binary gems". We could also not give it a new name, and just call it "gems v2", since this design is intended to be backwards compatible with existing gem servers and clients (they will see that the gems exist, but ignore them since the old platform logic says to ignore them). |
I like descriptive name for example "binary gems". Since this will be used by many newbies, the name should be easy to understand and easy to google. |
|
I am a little bit worried about possible confusion between "binary gems" from this RFC and the "binary gems" that you can already make today, but maybe it is ok if this new kind of binary gem will fully replace the old kind. |
|
I think it's the same concept as existing precompiled/binary gems, I would phrase it like "Improved platform and ABI support for precompiled gems". |
|
I have just quickly asked Red Hat Python maintainers about their thoughts about wheels, mainly about their limitations and this was their answer:
|
|
Good news, the Bundler and RubyGems teams have already solved 3 of the 4 items listed on the wheelnext.dev site as the big problems with wheels. If we thoughtfully expand platform support with tags like this RFC is proposing, we can probably cover the last one too. |
Could you clarify the 4 items you refer? The following part?
|
|
Yes, that is the part I was referring to.
|
|
I'm interested in how to handle symlinks on Windows. |
|
@nurse sure, how about we move discussion of symlinks to either RubyGems Discussions or Bundler Slack. I would like to keep this thread limited to discussion of this RFC about platform tags. |
|
Adding some background from the JRuby perspective... We have "overloaded" the existing gem platform support by using "java" as our platform for over twenty years now. Initially that started with us defining RUBY_PLATFORM as "java", because that's what RubyGems looked at, but since then RG has moved to using RbConfig values. RUBY_PLATFORM is still used by a lot of CRuby-specific code to determine the native platform. This would actually be useful for us (we support FFI-based gems that could have pre-built libraries, like sassc for example), but after so many years, JRuby users depend on RUBY_PLATFORM to stay "java". So we have for a very long time needed more multi-dimensional ways to specify the target/supported runtime for a given gem. We need to be able to say we support "java" or "jruby" gems, but also "linux-arm" gems (when they don't use the CRuby extension API), or "java-21" for gems that require newer JDKs. JRuby's "java" gems typically fall into two categories:
We do not recommend or support having "java" gems build at install time, due to the complexities of shipping a working build system, making sure that toolchain is present, and producing different, unexpected results if parts of that toolchain vary across platforms. Users are strongly encouraged to always pre-build any JVM code (Java, Scala, Kotlin, etc) and ship that inside the gem. We also provide the "jar-dependencies" tool that lets gems specify additional external dependencies from Maven (Java's primary package repository) so that "java" gems don't re-package third-party libraries.
For example, gems that depend on a specific CRuby extension cannot be installed on JRuby, and cannot be modified to omit that dependencies when being installed on JRuby. So we have to release a separate "java" version of the gem just to change the dependencies. There's also several stdlib gems that use FFI when running on JRuby; there's no way to tell them not to generate and execute a Makefile so either we generate a dummy Makefile (which causes other problems, like requiring "make"), or we release a "java" version of the gem that has no extension attribute. Basically, the single dimension through which released gems can target individual platforms is simultaneously too specific (a single gem file can't support more than one platfom) and too general (there's many other dimensions we want to be able to specify). |
segiddins
changed the title
The more detailed platforms made possible by this RFC would mean nokogiri could ship separate “skinny” binaries for each Ruby ABI version. For example, |
WASM has its own platform triples, including wasm32-unknown-wasip1 and wasm32-unknown-emscripten. Combining those triples with the tags from this RFC like |
|
@headius It’s great to hear that those are the major problems, since this proposal both lets gems support more than one platform and also creates a basis for arbitrary dimensions to match on |
|
|
||
| The new binary gem platform format follows the pattern `gb-{abi_tags}-{platform_tags}`: | ||
|
|
||
| - `abi_tags`: Ruby ABI version (e.g., `rb33` for Ruby 3.3, `any` for pure Ruby, `cr33` for CRuby 3.3, `jr92` for JRuby 9.2, `tr241` for TruffleRuby 24.1) |
There was a problem hiding this comment.
We shouldn't transform the ABI version, we should take it as-is from RbConfig::CONFIG['ruby_version'].
As said in #60 (comment) the ABI version may evolve in non-predictable ways. Only exact match of RbConfig::CONFIG['ruby_version'] (and RUBY_ENGINE) are compatible.
Another problematic example with e.g. jr92 is then jr10 could mean JRuby 1.0 or 10.0.
There was a problem hiding this comment.
Regarding shortening the RUBY_ENGINE to two letters, I think that's fine (shouldn't cause issues), and I think it's good to use cr33 instead of ruby33 for clarity.
|
|
||
| The new binary gem platform format follows the pattern `gb-{abi_tags}-{platform_tags}`: | ||
|
|
||
| - `abi_tags`: Ruby ABI version (e.g., `rb33` for Ruby 3.3, `any` for pure Ruby, `cr33` for CRuby 3.3, `jr92` for JRuby 9.2, `tr241` for TruffleRuby 24.1) |
There was a problem hiding this comment.
| - `abi_tags`: Ruby ABI version (e.g., `rb33` for Ruby 3.3, `any` for pure Ruby, `cr33` for CRuby 3.3, `jr92` for JRuby 9.2, `tr241` for TruffleRuby 24.1) | |
| - `abi_tags`: Ruby ABI version (e.g., `any` for any engine and any version i.e. for pure-Ruby gems, `cr-3.3.0` for CRuby 3.3.x, `jr-3.4.0` for JRuby 10.0.1.0, `jr-any` for any JRuby version, `tr-3.2.4.24.1.0.1` for TruffleRuby 24.1.x, `tr-any` for any TruffleRuby version) |
So this uses the real values from RbConfig::CONFIG["ruby_version"], and matches then exactly when it should (same as Bundler already using "vendor/bundle/#{RUBY_ENGINE}/#{RbConfig::CONFIG["ruby_version"]}).
One concern is JRuby doesn't seem to set that to its ABI currently (but to the same value as CRuby, which I guess doesn't mean much on JRuby), i.e. that version doesn't represent the JRuby Java extension ABI version cc @headius. Maybe the minimum supported Java version should also be part of the JRuby ABI version? (unsure, haven't thought much about it)
And TruffleRuby's ABI version is a bit verbose (because better safe than sorry about ABI, specifically TruffleRuby is concerned to not clash between master and release branches ABI versions), but that's mostly an aesthetic concern and something that could be improved in future releases.
I also added - between engine and ABI version, to address #60 (comment)
BTW should it be any or any-any considering there is jr-any then?
| Multiple tags can be combined with dots to express compatibility with multiple versions: | ||
|
|
||
| ``` | ||
| gb-rb33.rb32-x86_64_linux # Works with Ruby 3.3 or 3.2 on Linux x86_64 |
There was a problem hiding this comment.
It will need to be something else than . to combine when using the ABI version as reported by the Ruby engine (since that contains dots) to not make parsing needlessly complex.
BTW, it's probably impossible to have a native extension compatible with both CRuby 3.2.0 ABI and CRuby 3.3.0 ABI, unless doing the fat gem approach and including two compiled extension libraries (one for 3.2, one for 3.3), but as far as I understand it's not something that would make much sense with the new binary gem support.
| ``` | ||
|
|
||
|
|
||
| The new binary gem platform format follows the pattern `gb-{abi_tags}-{platform_tags}`: |
There was a problem hiding this comment.
I'm fine with this format and grouping, but I wonder if it could be more flexible, a bit like in https://github.com/orgs/rubygems/discussions/5988#discussioncomment-3882572 where the tags would basically be pairs like:
{
engine: "any|cruby|jruby|truffleruby",
engine_abi: "any|RbConfig::CONFIG['ruby_version'] value",
platform: "any|arch-os", # I see no value to separate arch and os
libc: "any|glibc|musl",
libc_abi: "any|GLIBC_MAJOR-GLIBC_MINOR",
}
and potentially be extended with more keys in the future if necessary (e.g. maybe os_version or kernel_version would be useful?).
Thanks for clarifying the list. I think that the list doesn't include a solution of
in https://wheelnext.dev/#key-issues-identified . And it's the topic discussed in the #60 (comment) thread. I think that we should solve the topic if we want to include glibc/musl libc support in |
| ``` | ||
|
|
||
|
|
||
| The new binary gem platform format follows the pattern `gb-{abi_tags}-{platform_tags}`: |
There was a problem hiding this comment.
Why do we need the gh- prefix? It seems that we can keep compatibility without the gb- prefix.
| - **Format**: `jr{major}{minor}[_{suffix}]` | ||
| - **Examples**: `jr92`, `jr93`, `jr94_static` |
There was a problem hiding this comment.
Why do we need to use abbreviated RUBY_ENGINE as prefix?
It's a bit cryptic.
|
|
||
| #### GlibcLinux Tags (glibc-based) | ||
|
|
||
| - **Format**: `glibclinux_{glibc_major}_{glibc_minor}_{arch}` |
There was a problem hiding this comment.
Other platform tags use {arch}_{platform}_... format.
How about using {arch}_glibclinux_{glibc_major}_{glibc_minor}?
| - `glibclinux_2_17_aarch64` (glibc 2.17+, ARM64) | ||
| - `glibclinux_2_35_i686` (glibc 2.35+, 32-bit x86) | ||
| - **Compatibility**: Forward-compatible with newer glibc versions | ||
| - **Caveat**: Only the `_{major}_{minor}` format from the PEP is supported. The legacy formats (`_1`, `_2010`, `_2014`) are not recognized. |
There was a problem hiding this comment.
I think that we don't need this because we don't use "manylinux" here.
|
|
||
| #### Musllinux Tags (musl-based) | ||
|
|
||
| - **Format**: `musllinux_{musl_major}_{musl_minor}_{arch}` |
There was a problem hiding this comment.
muls libc works only on Linux. So "linux" is redundant.
| ### Backward Compatibility | ||
| ``` | ||
| gb-rb33.rb32.rb31-x86_64_linux # Ruby 3.1, 3.2, or 3.3 | ||
| gb-cr32-glibclinux_2_17_x86_64 # Any CRuby 3.2.x with old glibc |
There was a problem hiding this comment.
| gb-cr32-glibclinux_2_17_x86_64 # Any CRuby 3.2.x with old glibc | |
| gb-cr32-glibclinux_2_17_x86_64 # Any CRuby 3.2.x with old glibc |
| - Provides platform detection without external dependencies | ||
|
|
||
| **`Gem::Platform::GlibcLinux`/`Gem::Platform::MuslLinux`** - Linux compatibility standards | ||
| - Implements a pared down equivalent of the PEP 600 (manylinux) and PEP 656 (musllinux) specifications |
There was a problem hiding this comment.
Do we need to use the same format as PEP 600/656?
They use inconsistent format, musllinux_${MUSLMAJOR}_${MUSLMINOR}_${ARCH} not ${arch}_${platform}_${major}_${minor}.
| **`Gem::Platform::GlibcLinux`/`Gem::Platform::MuslLinux`** - Linux compatibility standards | ||
| - Implements a pared down equivalent of the PEP 600 (manylinux) and PEP 656 (musllinux) specifications | ||
| - Maps glibc/musl versions to compatibility tags | ||
| - Allows for compact representation of cross-distribution compatible binaries |
There was a problem hiding this comment.
I still doubt that we can create cross-distribution compatible binaries only with libc information...
|
|
||
| The gem binary format was chosen because: | ||
|
|
||
| - It's a proven solution in the Python ecosystem with years of real-world usage |
There was a problem hiding this comment.
Should we add a note that some people https://wheelnext.dev/ think that the current wheel have some problems?
|
|
||
| The implementation adds four new classes to `lib/rubygems/platform/`: | ||
|
|
||
| **`Gem::Platform::GemBinary`** - Parses gem binary-format platform strings (`gb-{abi}-{platform}`) |
There was a problem hiding this comment.
It seems that Gem is redundant: Gem::Platform::Binary
9 participants
For ruby/rubygems#8891
Rendered
Edit 2025-08-28:
Thank you everyone for the comments! I've spent the past week editing the RFC and making matching changes to the implementation. The implementation PR should be updated in the next couple of days.