Support more platforms in `ctypes.find_library`

[Samueru-sama]

Bug report

Bug description:

I make appimages that bundle glibc and as result are able to work on any linux system (even in musl libc systems).

I had this issue which wasn't fully fixed until I manage to find out that the problem was cpython running /sbin/ldconfig -p which caused everything to fail.

Interestingly enough the output of ldconfig is not needed here, since after doing some patching to bundle ldconfig the issue was fixed, that ldconfig is just outputting the libraries of the alpine system and none of the bundled libraries yet the application finds and uses the bundled libraries without issue.

so this should test if ldconfig -p is supported before attempting to use it.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-142020: ctypes docs: Note that find_library relies on the programs it runs #142134
• [3.14] gh-142020: ctypes docs: Note that find_library relies on the programs it runs (GH-142134) #142748
• [3.13] gh-142020: ctypes docs: Note that find_library relies on the programs it runs (GH-142134) #142749

[picnixz]

AFAICT, we do check this. Where is the ldconfig call being used? The only call I could find was in ctypes but we have a try-catch already.

[Samueru-sama]

AFAICT, we do check this. Where is the ldconfig call being used? The only call I could find was in ctypes but we have a try-catch already.

The call is coming from pyudev:

Traceback (most recent call last):
  File "/tmp/samuel/Volatile/AppDir/bin/oversteer", line 53, in <module>
    sys.exit(app.run(sys.argv))
             ~~~~~~~^^^^^^^^^^
  File "/tmp/samuel/Volatile/AppDir/lib/python3.13/site-packages/oversteer/application.py", line 63, in run
    self.device_manager.start()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/tmp/samuel/Volatile/AppDir/lib/python3.13/site-packages/oversteer/device_manager.py", line 54, in start
    context = pyudev.Context()
  File "/tmp/samuel/Volatile/AppDir/lib/python3.13/site-packages/pyudev/core.py", line 59, in __init__
    self._libudev = load_ctypes_library("udev", SIGNATURES, ERROR_CHECKERS)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/samuel/Volatile/AppDir/lib/python3.13/site-packages/pyudev/_ctypeslib/utils.py", line 51, in load_ctypes_library
    raise ImportError("No library named %s" % name)
ImportError: No library named udev

A similar issue happened with another python app that used pyusb.

Maybe the check for ldconfig -p stops any further lookup?

[picnixz]

In this case, i think it's an issue with pyudev and not CPython itself. load_ctypes_library is a pyudev function and pyudev/_ctypeslib/utils.py is not the ctypes from CPython.

[picnixz]

Also, the issue doesn't seem related to ldconfig. It looks like the host machine is missing the .so library for udev (that is the corresponding *-devel or *-dev package depending on the distro)

[picnixz]

More generally, what you could actually want is for ctypes to still find your library I guess? ctypes.find_library tries quite hard to find libs but it's not always possible to have a very generic solution for that, especially if it's for platforms that we don't support officially as per PEP-11. I would suggest to report this issue to pyudev instead and ask them if it's possible to try harder for finding libraries (it looks like they are doing their own machinery for that).

[Samueru-sama]

In this case, i think it's an issue with pyudev and not CPython itself. load_ctypes_library is a pyudev function and pyudev/_ctypeslib/utils.py is not the ctypes from CPython.

Isn't pyudev using find_library from here though?

Also, the issue doesn't seem related to ldconfig. It looks like the host machine is missing the .so library for udev (that is the corresponding *-devel or *-dev package depending on the distro)

It is related to ldconfig, after we added it and made sure to redirect all calls to /sbin/ldconfig to the bundled one that supports -p the issue was fixed.

And as previously mentioned this is weird that it even works, because that bundled ldconfig is outputting the libraries of the host, not the bundled ones, yet it eventually finds the bundled ones without problem.

In other words this is failing because /sbin/ldconfig -p fails, not because it cannot find the library.

[picnixz]

Isn't pyudev using find_library from here though?

I don't know. What I can tell you is that the call to ldconfig is protected by a try-catch. The exception being raised is not raised from CPython but from pyudev.

[picnixz]

Ref:

cpython/Lib/ctypes/util.py

Lines 297 to 310 in 3bbd669

	# XXX assuming GLIBC's ldconfig (with option -p)
	regex = r'\s+(lib%s\.[^\s]+)\s+\(%s'
	regex = os.fsencode(regex % (re.escape(name), abi_type))
	try:
	with subprocess.Popen(['/sbin/ldconfig', '-p'],
	stdin=subprocess.DEVNULL,
	stderr=subprocess.DEVNULL,
	stdout=subprocess.PIPE,
	env={'LC_ALL': 'C', 'LANG': 'C'}) as p:
	res = re.search(regex, p.stdout.read())
	if res:
	return os.fsdecode(res.group(1))
	except OSError:
	pass

[picnixz]

The exception being raised is here:

https://github.com/pyudev/pyudev/blob/7588bc8c84bf1e994b31020452124bc6618625ce/src/pyudev/_ctypeslib/utils.py#L49-L51

IOW, ctypes.find_library returns None which pyudev considers to indicate that the library is missing which is the case I guess. I would suggest reporting this to pyudev. It's not an issue in CPython (unless you want us to allow to specify the path to ldconfig?)

[picnixz]

cc @encukou as ctypes maintainer (I need to leave)

[Samueru-sama]

I thought the problem was from here because ctypes is the only thing that contains the /sbin/ldconfing and it attempts to execute /sbin/ldconfig -p

This problem gets fixed by bundling an ldconfig that supports -p, even though its output is totally useless here, this fixes the issue I'm having.

So I guess the exception stops pyudev from trying to keep looking up?

(this problem also affects pyusb btw)

[Samueru-sama]

Here is an LD_DEBUG=libs output of the working appimage with a bundled ldconfig: libs.log

Doesn't attempt any of the libraries reported by the bundled ldconfig -p. (which is good, it is all working fine btw)

So my current theory is that pyudev fails because find_library errors? stops? when ldconfig -p is not possible.