Include debugpy in debug pex when configured as debugger

[marcuscaisey]

Problem

Debugging a pex with the debugpy debugger fails because the debugpy module is not found. For example:

please.nvim (master) plz debug -o=plugin.python.debugger:debugpy //experimental/python:ping --port 8765
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/marcus/scratch/please.nvim/plz-out/debug/experimental/python/./ping.pex/__main__.py", line 173, in <module>
    result = run(explode)
  File "/Users/marcus/scratch/please.nvim/plz-out/debug/experimental/python/./ping.pex/__main__.py", line 154, in run
    return main()
  File "/Users/marcus/scratch/please.nvim/plz-out/debug/experimental/python/./ping.pex/__main__.py", line 139, in main
    start_debugger()
    ~~~~~~~~~~~~~~^^
  File "/Users/marcus/scratch/please.nvim/plz-out/debug/experimental/python/./ping.pex/__main__.py", line 145, in start_debugger
    import debugpy
ModuleNotFoundError: No module named 'debugpy'
10:43:09.028   ERROR: Failed to execute //experimental/python:ping: exit status 1

Solution

Include debugpy in the debug pex. please_pex was already trying to include it but it wasn't included in please_pex itself.

Other changes:

• Upgrade debugpy to the latest version as the version in the repo wouldn't build (there was some python 2 code that was being compiled). The wheel couldn't be found using the any of the predictable URL schemes defined by WHEEL_NAME_SCHEME in third_party/python so I just used the wheel resolver instead.
• Add the .bootstrap directory in the exploded pex directory to sys.path. debugpy is not zip safe so it needs to be imported from the non-zip-safe location.

Tested my changes by overriding the pex tool to point to the new one:

please.nvim (master) plz debug -o=plugin.python.debugger:debugpy -o=plugin.python.pextool:/Users/marcus/scratch/python-rules/plz-out/bin/tools/please_pex/please_pex //experimental/python:ping --port 8765
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
^C10:56:14.322   ERROR: Failed to execute //experimental/python:ping: signal: terminated

I also hooked up an interactive debugger as a smoke test:

[toastwaffle]

Your title says when requested, but this looks fairly unconditional to me. What am I missing?

[marcuscaisey]

The "request" I was referring to was the user specifying which debugger they want either through .plzconfig or passing the -o=plugin.python.debugger:debugpy flag (since this is not the default). Updated the title to make this more clear 🙏

[toastwaffle]

The "request" I was referring to was the user specifying which debugger they want either through .plzconfig or passing the -o=plugin.python.debugger:debugpy flag (since this is not the default). Updated the title to make this more clear 🙏

But reading your changes, aren't we going to end up including debugpy into every single pex we build, whether we're requesting it for debugging or not?

[marcuscaisey]

aren't we going to end up including debugpy into every single pex we build

Nope. My changes unconditionally include debugpy into please_pex but please_pex only includes debugpy in the pex when -d debugpy is passed to it and -d debugpy is only passed when building the pex with --config=dbg.

Example:

please.nvim (master) unzip -l $(plz build -o=plugin.python.debugger:debugpy --config=dbg //experimental/python:ping) | rg debugpy | wc -l
     550
please.nvim (master) unzip -l $(plz build //experimental/python:ping) | rg debugpy | wc -l
       0

[toastwaffle]

I think the FreeBSD ports in https://github.com/please-build/python-rules/blob/master/third_party/cc/cpython/BUILD will need to be updated to get the tests to pass. You should be able to look them up on https://ports.freebsd.org/cgi/ports.cgi

[toastwaffle]

I suggest adding another target to tools/BUILD to get the prebuilt wheel_resolver to resolve the import cycle