Ensure to Pre-Load imports in main thread

[stefanhahmann]

This draft PR is a suggestion to fix apposed/appose#13

This issue is a bit difficult to debug. As already found and communiated, it only can be reproduced in Windows environments. It could not be reproduced in Linux. Situation on Mac is unclear (not yet tested on Mac), cf: apposed/appose#13 (comment)

I tried to reproduce on Windows from the command line using the env from this code example (apposed/appose#13 (comment)) and the following command

micromamba run -p C:\Users\xxxx\.local\share\appose\stardist C:\Users\xxxx\.local\share\appose\stardist\python.exe -c "import appose.python_worker; appose.python_worker.main()"

and then interactively inputing this task to stdin

{"task":"c388f270-d6be-48df-9e02-e42b48cab98b","requestType":"EXECUTE","inputs":{},"script":"import numpy\n"}

This results in a perfectly working process:

[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","requestType":"EXECUTE","inputs":{},"script":"import numpy\n"}
[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","responseType":"LAUNCH"}
[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","responseType":"COMPLETION","outputs":{}}

However, when I try to run this from a Java process similar to the code example (apposed/appose#13 (comment)), I only get this response and then the whole process is stuck in a dead lock:

[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","requestType":"EXECUTE","inputs":{},"script":"import numpy\n"}
[SERVICE-0] {"task":"c388f270-d6be-48df-9e02-e42b48cab98b","responseType":"LAUNCH"}

The code added in this PR fixes this situation. It seems that loading imports like (e.g.) numpy can lead to deadlocks when being loaded in multiple threads.

I am not sure, if the proposed fix is a good one however. Since the problem occurs only on Windows, one could think about adding if sys.platform == 'win32': or something like this to only perform the extra code on Windows. Also, one could think about making this optional and controlable from outside the python_worker by perhaps only performing the new code, if some global variable is set.

@ctrueden curious to learn your opinion about this.

[ctrueden]

@stefanhahmann That is a clever workaround to the problem you describe in apposed/appose#13.

We definitely need to get this fix into the codebase. At first, I thought it would be good to tweak this idea to be more formal, to give better control over exactly what gets "preloaded" on the main thread. E.g., rather than simply extracting all import statements, we could somehow mark/decorate the code we want to be preloaded? Maybe look for a function with a particular special name like preinit() and run that? Or some kind of decorator? Or: Appose in general could add support for synchronous tasks, which don't start a new thread on the worker side, and therefore block the worker until the task completes. Then you could send a synchronous task that does nothing except first-time imports of finicky libraries, and a subsequent asynchronous task that can then (re)import them safely.

But then I thought: your solution is simple, already works, and does not require callers to do anything special. They won't bump into this problem accidentally and be stumped for a solution. The only downside is theoretical: it might break some scripts that would otherwise work. But let's cross that bridge when we come to it.