`py_binary` does not have runfiles created if called transitively through `sh_binary`

[engnatha]

🐞 bug report

Affected Rule

This issue discusses the behavior of a py_binary target.

Is this a regression?

Haven't done a thorough search of past versions yet, but this does work correctly in bazel 6.4.0 and we are now seeing issues in bazel 7.5.0. During the switch, both systems were running rules_python version 0.37.0.

Description

When a py_binary is invoked, the backend will create the necessary <target>.runfiles directory that includes the relevant dependencies for running the target. There are cases in our system where it useful to wrap the calling of this py_binary behind a sh_binary as a data dependency. One such example is to make use of console scripts like flake8 in the repository as if they were installed in a virtual environment. If one uses the py_console_script_binary target directly, all supplied paths to lint must be absolute since the current working directory is modified by bazel when running the target. By changing the working directory, we can get the benefit of having hermetic tooling as a drop-in replacement for something normally provided in a virtual environment.

🔬 Minimal Reproduction

Attached a zip of a repro. Instructions to reproduce the behavior are in the README.

🔥 Exception or Error

Traceback (most recent call last):
  File "/home/nathanael/.cache/bazel/_bazel_nathanael/e616dc8a3f01de94997c692cfb80f031/execroot/_main/bazel-out/k8-fastbuild/bin/src/hello_world", line 580, in 
    Main()
  File "/home/nathanael/.cache/bazel/_bazel_nathanael/e616dc8a3f01de94997c692cfb80f031/execroot/_main/bazel-out/k8-fastbuild/bin/src/hello_world", line 471, in Main
    module_space = FindModuleSpace(main_rel_path)
  File "/home/nathanael/.cache/bazel/_bazel_nathanael/e616dc8a3f01de94997c692cfb80f031/execroot/_main/bazel-out/k8-fastbuild/bin/src/hello_world", line 181, in FindModuleSpace
    raise AssertionError('Cannot find .runfiles directory for %s' % sys.argv[0])
AssertionError: Cannot find .runfiles directory for /home/nathanael/.cache/bazel/_bazel_nathanael/e616dc8a3f01de94997c692cfb80f031/execroot/_main/bazel-out/k8-fastbuild/bin/src/hello_world

🌍 Your Environment

Operating System:

  
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
(mono) ~/devel/rules_pyt
  

Output of bazel version:

  
bazel 7.5.0
  

Rules_python version:
0.37.0

rules_python_runfiles_repro-main.zip

Anything else relevant?

Thanks for making rules_python 😃 We're doing a lot of cool things with it!

[aignas]

Can you try the latest rules_python version to see if this is still present? I think we fixed this a long time ago.

[engnatha]

I replaced the rules python version with 1.2.0 in MODULE.bazel, reran bazel clean --expunge and received the same .runfiles exception.

[rickeylev]

From a quick look at the repro code, this looks a bit suspicious:

cmd=$(realpath $1)
exec $cmd

With all the symlinking bazel does, realpath could be landing in some other file location, from which the runfiles directory can't be figured out.

I think the thing to do is to cd in the directory realpath resolves to, and from there look around to see if its possible to divine the runfiles directory. i.e. given only the context of PWD=$(dirname $(realpath $1)) and sys.argv[0], is it possible to find the runfiles directory?

Setting `RULES_PYTHON_BOOTSTRAP_VERBOSE=1 might also help give insight into what the bootstrap is doing.

[engnatha]

I don't think it's a symlink problem since this all works fine if I force bazel build the py_console_script_binary or py_binary. realpath is also executed before the directory is changed, so I'm not sure how we could end up somewhere wrong here. To offer an opinion, I don't think a script should depend on being executed from the directory it is in to work. An absolute path should be sufficient.

I did try your suggestion about inspecting the directories before and it's pretty conclusive that the .runfiles directory (in bazel-bin and ~/.cache/bazel) are simply never created. I'll find some time to try out the more verbose settings as well.

[engnatha]

Would it be appropriate to remove the need:repro label on this?

[engnatha]

I've tested again with 1.6.0-rc0 and see the failure.