rb_bundle_fetch not building extra gemspec jars needed by certain gems

[noahkawasakigoogle]

Repro: https://github.com/noahkawasakigoogle/rules_ruby_bug

I've run into an issue attempting to use rules_ruby with the psych Gem that I believe is due to rb_bundle_fetch for some reason not executing certain gem installation steps.

The psych gem for this example, should result in a JAR file being produced: https://github.com/ruby/psych/blob/master/psych.gemspec#L71

.gem/jruby/3.1.0/gems/psych-3.3.3-java/lib/org/yaml/snakeyaml/1.31/snakeyaml-1.31.jar

However an rb_bundle_fetch WORKSPACE rule does not result in this jar being produced, while a normal bundle install execution does result in this file being produced.

I've reproduced this behavior in a public repo. But TLDR is that this does not include the expected JAR files:

WORKSPACE:

rb_bundle_fetch(
    name = "bundle",
    gemfile = "//:Gemfile",
    gemfile_lock = "//:Gemfile.lock",
    srcs = [
      "//:.ruby-version"
    ]
)

BUILD.bazel:

load("@rules_ruby//ruby:defs.bzl", "rb_library")

rb_library(
    name = "repro",
    deps = [
      "@bundle"
    ]
)

Run this to see:

bazel build --enable_workspace //:repro

# Missing snakeyaml-1.31.jar in bazel outfiles
find "$(bazel info output_path)/k8-fastbuild/bin/external" -type f -name snakeyaml-1.31.jar

Running bundle install works though!

bazel run --enable_workspace @ruby//:bundle -- install --path "${BUNDLE_PATH}"

# Returns:
# ./rules_ruby_bug/.gem/jruby/3.1.0/gems/psych-3.3.3-java/lib/org/yaml/snakeyaml/1.31/snakeyaml-1.31.jar
find "${BUNDLE_PATH}" -type f -name snakeyaml-1.31.jar

You will see extra logs during the installation that you will not see using rb_bundle_fetch:

using maven for the first time results in maven
downloading all its default plugin and can take time.
as those plugins get cached on disk and further execution
of maven is much faster then the first time.

      org.yaml:snakeyaml:1.31:compile

Other information, I am using jruby-9.4.12.0 and bundler 2.4.22

[p0deje]

Hm, it's a pretty old version of Psych, does the issue reproduce if you upgrade to 5.3.3?

[noahkawasakigoogle]

Just tried with psych 5.2.3, same behavior. Running direct binary commands: bazel run --enable_workspace @ruby//:bundle -- update includes the snakeyaml compilation:

Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Fetching date 3.4.1 (java)
Fetching ruby-maven-libs 3.9.9
Fetching jar-dependencies 0.5.5
Installing date 3.4.1 (java)
Installing jar-dependencies 0.5.5
Fetching psych 5.2.3 (java)
Installing psych 5.2.3 (java)
  jar dependencies for psych-5.2.3-java.gemspec . . .
Installing gem 'ruby-maven' . . .
Installing ruby-maven-libs 3.9.9
Fetching ruby-maven 3.9.3
Installing ruby-maven 3.9.3
Installing gem 'ruby-maven-libs' . . .

using maven for the first time results in maven
downloading all its default plugin and can take time.
as those plugins get cached on disk and further execution
of maven is much faster then the first time.

      org.snakeyaml:snakeyaml-engine:2.9:compile
Bundle updated!
Post-install message from jar-dependencies:

if you want to use the executable lock_jars then install ruby-maven gem before using lock_jars

  $ gem install ruby-maven -v '~> 3.9'

or add it as a development dependency to your Gemfile

   gem 'ruby-maven', '~> 3.9'

While using rb_bundle_fetch bazel build --enable_workspace //:repro does not:

INFO: From Running bundle install (@@bundle//:bundle):
`/usr/local/google/home/noahkawasaki` is not writable.
Bundler will use `/tmp/bundler20250306-3-e1uwsz3' as your home directory temporarily.
Installing date 3.4.1 (java)
Installing jar-dependencies 0.5.5
Installing ruby-maven-libs 3.9.9
Installing psych 5.2.3 (java)
Installing ruby-maven 3.9.3
Bundle complete! 2 Gemfile dependencies, 6 gems now installed.
Bundled gems are installed into `./bazel-out/k8-fastbuild/bin/external/bundle/vendor/bundle`
Post-install message from jar-dependencies:

if you want to use the executable lock_jars then install ruby-maven gem before using lock_jars

  $ gem install ruby-maven -v '~> 3.9'

or add it as a development dependency to your Gemfile

   gem 'ruby-maven', '~> 3.9'

INFO: Found 1 target...
Target //:repro up-to-date (nothing to build)
INFO: Elapsed time: 24.457s, Critical Path: 10.63s
INFO: 5 processes: 3 internal, 2 linux-sandbox.
INFO: Build completed successfully, 5 total actions

[p0deje]

Interesting, we use psych in examples and it didn't cause any issues so far https://github.com/bazel-contrib/rules_ruby/blob/main/examples/gem/Gemfile.lock#L38-L43.

[noahkawasakigoogle]

It is odd, the only issue that comes up is if your code uses psych in a way where it needs to actually load the jar ./rules_ruby_bug/.gem/jruby/3.1.0/gems/psych-3.3.3-java/lib/org/yaml/snakeyaml/1.31/snakeyaml-1.31.jar so I suspect its not a common problem. I will do more digging and see if I can figure it out. But its weird that bundle install within the bazel world is working differently than bundle install outside

[noahkawasakigoogle]

There is a whole bunch of related issues from jruby and psych:

jruby/jar-dependencies#92

jruby/jar-dependencies#91

jruby/jruby#8606

Although its not obvious to me why bazel rules_ruby would fail to build the jars, while running bundle outside of the bazel rules builds them fine.

I've actually tested the suggestions in the bugs referred, specifically upgrading jar-dependencies to 0.5.5 and psych to 5.2.3 and this seems to fix my problem. It appears to be more of a jruby bug than a rules_ruby thing.

[p0deje]

Although its not obvious to me why bazel rules_ruby would fail to build the jars, while running bundle outside of the bazel rules builds them fine.

Likely because we run bundle install as an action w/o access to the network and restricting to only use cached gems (bundle install --standalone --local).

I've actually tested the suggestions in the bugs referred, specifically upgrading jar-dependencies to 0.5.5 and psych to 5.2.3 and this seems to fix my problem. It appears to be more of a jruby bug than a rules_ruby thing.

Good to know, I'm going to close this for now, feel free to re-open if you think we need to change the way ruby_rules work.

[noahkawasakigoogle]

Yeah, you are correct, removing --local from bundle install fixes the problem. I can reproduce this behavior outside of bazel:

> bundle install

using maven for the first time results in maven
downloading all its default plugin and can take time.
as those plugins get cached on disk and further execution
of maven is much faster then the first time.

      org.yaml:snakeyaml:1.31:compile

# See that snakeyaml exists
 > find ${GEM_HOME} -type f -name "*snakeyaml*"
.gem/gems/psych-3.3.3-java/lib/org/yaml/snakeyaml/1.31/snakeyaml-1.31.jar

# Delete snakeyaml
> rm .gem/gems/psych-3.3.3-java/lib/org/yaml/snakeyaml/1.31/snakeyaml-1.31.jar

# Run bundle how bazel runs it
> bundle install --local

Fetching gem metadata from https://rubygems.org/.......
Fetching ruby2_keywords 0.0.5
Installing ruby2_keywords 0.0.5
Fetching ipaddr 1.2.4
Installing ipaddr 1.2.4
Fetching nkf 0.2.0 (java)
Installing nkf 0.2.0 (java)
Bundle complete! 156 Gemfile dependencies, 256 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

# No snakeyaml jar found
find ${GEM_HOME} -type f -name "*snakeyaml*"

RubyGems.org psych page the Build File links to doing this step: https://github.com/ruby/psych/actions/runs/12831802395/workflow#L48

So somehow bundler knows to execute rake compile from the gem metadata.

What are the implications are relaxing --local flag or making that toggle-able? Although its possible for me to just change the gem versions to get around the need for the jar file to be compiled from that JRuby bug this would potentially still be a problem for other gems that have extra build steps.

[noahkawasakigoogle]

Ah, I see how psych does this now. It uses https://github.com/rake-compiler/rake-compiler here https://github.com/ruby/psych/blob/master/Rakefile#L15

Im not sure how to integrate this into rules_ruby the best but it seems like a limitation of certain gems that need to compile anything.

[p0deje]

What are the implications are relaxing --local flag or making that toggle-able?

It would break hermeticity during bundle install task, meaning it would be able to fetch new dependencies outside of Bazel. This means the build is no longer reproducible (we don't control what version of dependencies are downloaded during installation). Also, it would not work in RBE environments where usually build targets have no access to the internet.

Ideally we need to find a way to let these commands be done without access to the network or simulate them via Bazel.

[noahkawasakigoogle]

Thats what I thought, ill see if I can recreate the behavior needed with bazel java rules. Its only searching for a snakeyaml jar file, so I imagine that must be doable.

[p0deje]

Alternatively, if we know what gems/jars need to be downloaded and what commands need to be run, we can replicate that in rules_ruby.

[noahkawasakigoogle]

For this particular Psych gem, all that would be needed is rake compile: https://github.com/ruby/psych/blob/master/Rakefile#L34C20-L34C27

And we'd expect the jar file to be installed into the same directory as bundle install installs the gem's files. Though I dont know if there is an easy standardized way to implement something like this in rules_ruby. It may require some additional configuration options for a gem. I imagine this would come up more for any gem that needs extra extensions being built

[p0deje]

Maybe the issue is that psych gem should declare that it needs to build JAR as part of gem installation via extensions gemspec field?

For example, https://github.com/jruby/activerecord-jdbc-adapter/ declares default rake task as part of gem install https://github.com/jruby/activerecord-jdbc-adapter/blob/c56c7b476da8cb78c2184ac4a68175811038b631/activerecord-jdbc-adapter.gemspec#L28

    gem.extensions << 'Rakefile' # to support auto-building .jar with :git paths

If I read it correctly, it means that it would run rake during installation of the gem. The default Rake task would have to build the JAR file (https://github.com/jruby/activerecord-jdbc-adapter/blob/c56c7b476da8cb78c2184ac4a68175811038b631/Rakefile#L187).

[noahkawasakigoogle]

Yeah that would make sense, I was kind of confused that the gemspec file had no mention of the Rakefile