Build failed with updated dependency rules_python >= 1.5.0

[umaucher]

In the scope of eclipse-score/persistency#158 we wanted to update our dependencies to some latest versions, so also the inherited dependencies used in the scope of docs-as-code. When updating rules_python to any later version (starting from 1.5.0) we get an error in the bazel run -- //:docs rule:

✔ mymachine:~/Projects/git/eclipse-score/persistency [um_sdac_update|✚ 1⚑ 1] 
$ bazel run -- //:docs
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=272
INFO: Reading rc options for 'run' from /home/umaucher/Projects/git/eclipse-score/persistency/.bazelrc:
  Inherited 'common' options: --registry=https://raw.githubusercontent.com/eclipse-score/bazel_registry/main/ --registry=https://bcr.bazel.build
INFO: Reading rc options for 'run' from /home/umaucher/Projects/git/eclipse-score/persistency/user.bazelrc:
  Inherited 'common' options: --announce_rc --remote_cache=grpc://10.88.34.52:1985 --remote_timeout=3600 --bes_results_url=http://localhost:8080/invocation/ --bes_backend=grpc://10.88.34.52:1985
INFO: Reading rc options for 'run' from /home/umaucher/Projects/git/eclipse-score/persistency/.bazelrc:
  Inherited 'build' options: --java_language_version=17 --tool_java_language_version=17 --java_runtime_version=remotejdk_17 --tool_java_runtime_version=remotejdk_17 --@score_baselibs//score/json:base_library=nlohmann --@score_baselibs//score/mw/log/flags:KRemote_Logging=False --@score-baselibs//score/json:base_library=nlohmann --@score-baselibs//score/mw/log/flags:KRemote_Logging=False
WARNING: For repository 'rules_rust', the root module requires module version rules_rust@0.56.0, but got rules_rust@0.61.0 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'rules_cc', the root module requires module version rules_cc@0.1.1, but got rules_cc@0.1.2 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'platforms', the root module requires module version platforms@0.0.11, but got platforms@1.0.0 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
INFO: Analyzed target //:docs (353 packages loaded, 27792 targets configured).
ERROR: /<cache>/external/score_process+/BUILD:39:5: Sphinx building needs for @@score_process+//:needs_json failed: (Exit 3): sphinx_build failed: error executing SphinxBuildDocs command (from target @@score_process+//:needs_json) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_process+/sphinx_build @bazel-out/k8-fastbuild/bin/external/score_process+/needs_json/_build/needs-0.params
Unhandled error:
Request id: None
Traceback (most recent call last):
  File "/<cache>/<external>/score_process+/sphinx_build.runfiles/rules_python+/sphinxdocs/private/sphinx_build.py", line 75, in run
    response = self._process_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<cache>/<external>/score_process+/sphinx_build.runfiles/rules_python+/sphinxdocs/private/sphinx_build.py", line 167, in _process_request
    raise Exception(
Exception: Sphinx main() returned failure:   exit code: 2
========== STDOUT START ==========

========== STDOUT END ==========

ERROR: /<cache>/external/score_process+/BUILD:39:5: Sphinx building needs for @@score_process+//:needs_json [for tool] failed: (Exit 3): sphinx_build failed: error executing SphinxBuildDocs command (from target @@score_process+//:needs_json) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_process+/sphinx_build @bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_process+/needs_json/_build/needs-0.params
Unhandled error:
Request id: None
Traceback (most recent call last):
  File "/<cache>/<external>/score_process+/sphinx_build.runfiles/rules_python+/sphinxdocs/private/sphinx_build.py", line 75, in run
    response = self._process_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<cache>/<external>/score_process+/sphinx_build.runfiles/rules_python+/sphinxdocs/private/sphinx_build.py", line 167, in _process_request
    raise Exception(
Exception: Sphinx main() returned failure:   exit code: 2
========== STDOUT START ==========

========== STDOUT END ==========

Target //:docs failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 33.194s, Critical Path: 3.53s
INFO: 14 processes: 973 action cache hit, 14 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

the error should be fixed to support als later versions (rules_python 1.6.3) or the limitation should be documented.

[umaucher]

*) this only happens with the dependencies to score_platform and score_process (which is needed in our case to refer to requirements) … if we know further details we can also push the error there if it suits better

[a-zw]

I tried it with rules_python 1.5.0 and 1.6.3 (currently the latest non-release-candidate version) in the docs-as-code repo and I'm getting different errors:

[...]
  File "/home/xxx/.cache/bazel/_bazel_xxx/f5dacc4600ca9ae36ee268b8f085b9f2/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_process+/sphinx_build.runfiles/_main/src/extensions/score_plantuml.py", line 90, in setup
    app.config.plantuml = str(find_correct_path(get_runfiles_dir()))
                                                ^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.cache/bazel/_bazel_xxx/f5dacc4600ca9ae36ee268b8f085b9f2/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_process+/sphinx_build.runfiles/_main/src/extensions/score_plantuml.py", line 64, in get_runfiles_dir
    sys.exit(
SystemExit: Could not find runfiles_dir at /home/xxx/.cache/bazel/_bazel_xxx/f5dacc4600ca9ae36ee268b8f085b9f2/execroot/_main/bazel-bin/ide_support.runfiles. Have a look at README.md for instructions on how to build docs.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xxx/.cache/bazel/_bazel_xxx/f5dacc4600ca9ae36ee268b8f085b9f2/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_process+/sphinx_build.runfiles/rules_python+/sphinxdocs/private/sphinx_build.py", line 233, in <module>
    sys.exit(_worker_main(sys.stdin, sys.stdout, os.getcwd()))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.cache/bazel/_bazel_xxx/f5dacc4600ca9ae36ee268b8f085b9f2/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_process+/sphinx_build.runfiles/rules_python+/sphinxdocs/private/sphinx_build.py", line 214, in _worker_main
    with Worker(stdin, stdout, exec_root) as worker:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Worker.__exit__() takes 1 positional argument but 4 were given
---8<---8<--- End of log ---8<---8<---

The one at the end seems to be a bug within rules_python itself?

[a-zw]

With the newer rules_python the environment looks like this: ['TMPDIR', 'LC_CTYPE', 'PYTHONPATH', 'RUNFILES_MANIFEST_FILE', 'PYTHONSAFEPATH', 'DOCUTILSCONFIG']

The score_plantuml.py where the exception happens looks for "RUNFILES_DIR" which was present with rules_python 1.4.3.

In their docs, I saw this:

If env contains “RUNFILES_MANIFEST_FILE” with non-empty value, this method returns a manifest-based implementation. The object eagerly reads and caches the whole manifest file upon instantiation; this may be relevant for performance consideration.

Otherwise, if env contains “RUNFILES_DIR” with non-empty value (checked in this priority order), this method returns a directory-based implementation.

This is handled by bazel itself but I couldn't find any proper documentation.

[NEOatNHNG]

This issue currently breaks all module builds if they depend on rules_python >= 1.5.0!

@a-zw I tried to fix the issue in #337 but that only works for modules which do not define any needs (e.g. module_template) but it fails with another error for modules with needs. Can you have a look whether there would be any way to fix this?