From 1cbce87d0e1718d27cbea57613ea0a5496f95432 Mon Sep 17 00:00:00 2001 From: Raffi Khatchadourian Date: Wed, 29 Apr 2026 15:26:51 -0400 Subject: [PATCH 1/7] Recover from Jython interpreter init failure during constant folding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Python3Interpreter.getInterp()` calls `new PythonInterpreter()`, which walks Jython's bootstrap path to set up `sys.importlib`. In some environments (e.g., OSGi consumers running under Tycho-surefire) the bootstrap resources aren't reachable from the current classloader and the constructor throws `NullPointerException` from `org.python.core.Py.importSiteIfSelected`. The exception was uncaught — it propagated through every recursive `ConstantFoldingRewriter.copyNodes` frame and aborted the entire module load, leaving the class hierarchy with no `.py` entries. Fix: catch the failure where it originates, memoize via `volatile boolean initFailed`, log once at WARNING, and return `null`. Update `Python3Loader`'s const-fold callback to treat a `null` interpreter as a folding miss. Memoizing avoids re-running the failing constructor on every const-fold attempt. Constant folding is a precision-only feature (it shrinks symbolic expressions to literals when possible); analysis remains correct without it, just less precise. ## Reproducer The downstream symptom is `IllegalStateException: Could not create a entrypoint callsites:` (with empty `Warnings`) at `PropagationCallGraphBuilder.makeCallGraph:238`. It surfaces in Hybridize-Functions-Refactoring's testAutoEncoder / testTensorFlowEagerExecution / testDatasetIteration4 against any Ariadne version after `30c15e58` (which removed the silently-swallowing `try { ... } catch (Throwable e) {}` block around the same code path). --- .../cast/python/loader/Python3Loader.java | 9 +++++- .../cast/python/util/Python3Interpreter.java | 29 +++++++++++++++++-- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java index 91448ecf6..74e1ce161 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java @@ -108,8 +108,15 @@ protected Object eval(CAstOperator op, Object lhs, Object rhs) { PyUnicode unicode = new PyUnicode(s); PyObject x; + org.python.util.PythonInterpreter ip = Python3Interpreter.getInterp(); + if (ip == null) { + // Jython init failed (memoized in Python3Interpreter). Skip constant folding + // for this expression; analysis remains correct, just less precise. + return null; + } + try { - x = Python3Interpreter.getInterp().eval(unicode); + x = ip.eval(unicode); } catch (PySyntaxError e) { // Handle syntax errors gracefully. logger.log(WARNING, e, () -> "Syntax error in expression: " + unicode); diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java index f29be6276..4c1405626 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java @@ -13,10 +13,35 @@ public class Python3Interpreter extends com.ibm.wala.cast.python.util.PythonInte private static PythonInterpreter interp; + /** + * Memoizes a failed Jython init so subsequent {@link #getInterp()} calls return {@code null} + * cheaply instead of re-running {@code new PythonInterpreter()} (which can be expensive when it + * fails — site-import walks the Jython resource path on every attempt). When a single failure has + * occurred, callers receive {@code null} and can degrade their behavior (e.g., {@link + * com.ibm.wala.cast.python.loader.Python3Loader} skips constant folding). + */ + private static volatile boolean initFailed = false; + public static PythonInterpreter getInterp() { + if (initFailed) return null; if (interp == null) { - PySystemState.initialize(); - interp = new PythonInterpreter(); + try { + PySystemState.initialize(); + interp = new PythonInterpreter(); + } catch (Throwable t) { + // Jython init can fail when bootstrap resources (e.g., the embedded + // _frozen_importlib bytecode used by org.python.core.imp) aren't reachable from the + // current classloader/working directory. This is environment-dependent (e.g., happens + // under Tycho-OSGi but not under plain Maven-surefire). Treat as a recoverable failure: + // log once, memoize, and let callers degrade gracefully. + initFailed = true; + LOGGER.log( + Level.WARNING, + t, + () -> + "Jython interpreter init failed; constant folding will be disabled for this run."); + return null; + } } return interp; } From 964c127e29ab2aea4b6482ba5e735621576881ce Mon Sep 17 00:00:00 2001 From: Raffi Khatchadourian Date: Wed, 29 Apr 2026 15:42:47 -0400 Subject: [PATCH 2/7] Address Copilot review: synchronize, narrow catch, null-check `evalAsInteger` Three concerns raised by the auto-review on the previous commit: 1. **Thread-safety**: `initFailed` and `interp` were checked/assigned without synchronization. Make `getInterp()` `synchronized` so the check and the constructor run as a unit through the static class monitor. 2. **`Throwable` is too broad**: catching `Throwable` swallows `Error` types (OOM, StackOverflow, LinkageError) and silently continues with `initFailed=true`, hiding genuine VM problems. Narrow to `Exception`. The Jython failure we're defending against is `RuntimeException` (NPE from `Py.importSiteIfSelected`), which is `Exception`'s subset, so the defensive intent still holds. 3. **`evalAsInteger` didn't null-check `getInterp()`**: with the new contract that `getInterp()` may return null after a memoized init failure, `evalAsInteger().eval(...)` would NPE. Add an `IllegalStateException` for that case so callers get a clear "interpreter unavailable" signal. No behavior change on the success path. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../cast/python/util/Python3Interpreter.java | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java index 4c1405626..46ac860de 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java @@ -22,18 +22,22 @@ public class Python3Interpreter extends com.ibm.wala.cast.python.util.PythonInte */ private static volatile boolean initFailed = false; - public static PythonInterpreter getInterp() { + public static synchronized PythonInterpreter getInterp() { if (initFailed) return null; if (interp == null) { try { PySystemState.initialize(); interp = new PythonInterpreter(); - } catch (Throwable t) { + } catch (Exception t) { // Jython init can fail when bootstrap resources (e.g., the embedded // _frozen_importlib bytecode used by org.python.core.imp) aren't reachable from the // current classloader/working directory. This is environment-dependent (e.g., happens // under Tycho-OSGi but not under plain Maven-surefire). Treat as a recoverable failure: // log once, memoize, and let callers degrade gracefully. + // + // We catch {@link Exception} (not {@link Throwable}) so that {@link Error} types + // (OOM, stack overflow, linkage errors) keep propagating to the caller — those signal + // serious VM problems we don't want to silently swallow and continue past. initFailed = true; LOGGER.log( Level.WARNING, @@ -47,8 +51,15 @@ public static PythonInterpreter getInterp() { } public Integer evalAsInteger(String expr) { + PythonInterpreter ip = getInterp(); + if (ip == null) { + throw new IllegalStateException( + "Jython interpreter unavailable (init failed earlier); cannot evaluate expression: " + + expr + + "."); + } try { - PyObject val = getInterp().eval(expr); + PyObject val = ip.eval(expr); if (val.isInteger()) { return val.asInt(); } else From 883d254c7e4f580d06ee8f1d7494d9aaf286682b Mon Sep 17 00:00:00 2001 From: Raffi Khatchadourian Date: Wed, 29 Apr 2026 21:19:42 -0400 Subject: [PATCH 3/7] Use import statement instead of FQN for `PythonInterpreter` Mirror of the fix-branch change applied to the fork PR (per review feedback on ponder-lab/ML#191). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../source/com/ibm/wala/cast/python/loader/Python3Loader.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java index 74e1ce161..140656a80 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java @@ -38,6 +38,7 @@ import org.python.core.PyObject; import org.python.core.PySyntaxError; import org.python.core.PyUnicode; +import org.python.util.PythonInterpreter; public class Python3Loader extends PythonLoader { @@ -108,7 +109,7 @@ protected Object eval(CAstOperator op, Object lhs, Object rhs) { PyUnicode unicode = new PyUnicode(s); PyObject x; - org.python.util.PythonInterpreter ip = Python3Interpreter.getInterp(); + PythonInterpreter ip = Python3Interpreter.getInterp(); if (ip == null) { // Jython init failed (memoized in Python3Interpreter). Skip constant folding // for this expression; analysis remains correct, just less precise. From ea58c7980b047e0d5738fdc34300971e2bbe266d Mon Sep 17 00:00:00 2001 From: Raffi Khatchadourian Date: Wed, 29 Apr 2026 21:26:03 -0400 Subject: [PATCH 4/7] Return null instead of throwing when Jython init failed in `evalAsInteger` Mirror of the fork-side change (per review on ponder-lab/ML#191). The previous `IllegalStateException` would abort callers that expect the nullable-`Integer` contract; null lets them degrade gracefully. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../cast/python/util/Python3Interpreter.java | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java index 46ac860de..5df3cda34 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java @@ -53,10 +53,19 @@ public static synchronized PythonInterpreter getInterp() { public Integer evalAsInteger(String expr) { PythonInterpreter ip = getInterp(); if (ip == null) { - throw new IllegalStateException( - "Jython interpreter unavailable (init failed earlier); cannot evaluate expression: " - + expr - + "."); + // Return {@code null} (the same "cannot evaluate" signal used elsewhere in this method's + // contract) rather than throwing, so callers like + // {@code com.ibm.wala.cast.python.util.PythonInterpreter#interpretAsInt} — which expect a + // nullable {@link Integer} and don't catch checked or runtime exceptions — degrade + // gracefully in the same OSGi-classloader environments that triggered the {@link + // #getInterp()} init failure in the first place. + LOGGER.log( + Level.WARNING, + () -> + "Jython interpreter unavailable (init failed earlier); cannot evaluate expression: " + + expr + + ". Returning null."); + return null; } try { PyObject val = ip.eval(expr); From d6d02462f095e5e10bc774c75705440e3d4d6d2b Mon Sep 17 00:00:00 2001 From: Raffi Khatchadourian Date: Wed, 29 Apr 2026 21:34:07 -0400 Subject: [PATCH 5/7] Address Copilot review: log noise, message accuracy, log ordering Mirror of the fork-side polish (per ponder-lab/ML#191). Three items: 1. Memoize per-call WARNING in `evalAsInteger` (first call WARNING, subsequent FINE). 2. Broaden init-failure message to mention all interpreter-based evaluation, not just const-fold. 3. Move `logger.info("Evaluating: ...")` after the `getInterp()` null check in `Python3Loader.eval`. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../cast/python/loader/Python3Loader.java | 14 ++++---- .../cast/python/util/Python3Interpreter.java | 34 +++++++++++++++---- 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java index 140656a80..b89bea9a6 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/loader/Python3Loader.java @@ -103,18 +103,20 @@ public ConstantFoldingRewriter createCAstRewriter(CAst ast) { @Override protected Object eval(CAstOperator op, Object lhs, Object rhs) { String s = lhs + " " + op.getValue() + " " + rhs; - logger.info(() -> "Evaluating: " + s); - - // Use the Python interpreter to evaluate the expression. - PyUnicode unicode = new PyUnicode(s); - PyObject x; PythonInterpreter ip = Python3Interpreter.getInterp(); if (ip == null) { // Jython init failed (memoized in Python3Interpreter). Skip constant folding - // for this expression; analysis remains correct, just less precise. + // for this expression; analysis remains correct, just less precise. Don't log + // an "Evaluating:" entry — nothing is actually evaluated, and the underlying + // init failure was already announced from getInterp(). return null; } + logger.info(() -> "Evaluating: " + s); + + // Use the Python interpreter to evaluate the expression. + PyUnicode unicode = new PyUnicode(s); + PyObject x; try { x = ip.eval(unicode); diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java index 5df3cda34..951d1e355 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java @@ -22,6 +22,15 @@ public class Python3Interpreter extends com.ibm.wala.cast.python.util.PythonInte */ private static volatile boolean initFailed = false; + /** + * Memoizes whether the "interpreter unavailable" warning has already been emitted from {@link + * #evalAsInteger(String)}. Without this, callers like {@code interpretAsInt} (invoked many times + * during shape inference) would flood logs with one WARNING per call after the first init + * failure. The first failure is already announced by the catch block in {@link #getInterp()}; + * subsequent calls log at FINE level only. + */ + private static volatile boolean unavailableWarned = false; + public static synchronized PythonInterpreter getInterp() { if (initFailed) return null; if (interp == null) { @@ -43,7 +52,9 @@ public static synchronized PythonInterpreter getInterp() { Level.WARNING, t, () -> - "Jython interpreter init failed; constant folding will be disabled for this run."); + "Jython interpreter init failed; all interpreter-based evaluation will be disabled" + + " for this run (constant folding in Python3Loader, shape-argument" + + " evaluation via interpretAsInt/evalAsInteger, etc.)."); return null; } } @@ -59,12 +70,21 @@ public Integer evalAsInteger(String expr) { // nullable {@link Integer} and don't catch checked or runtime exceptions — degrade // gracefully in the same OSGi-classloader environments that triggered the {@link // #getInterp()} init failure in the first place. - LOGGER.log( - Level.WARNING, - () -> - "Jython interpreter unavailable (init failed earlier); cannot evaluate expression: " - + expr - + ". Returning null."); + // + // Log the first such call at WARNING (so operators see that some shape inference is being + // skipped because of the earlier init failure); subsequent calls log at FINE only, since + // the underlying init failure has already been announced from {@link #getInterp()}. + if (!unavailableWarned) { + unavailableWarned = true; + LOGGER.log( + Level.WARNING, + () -> + "Jython interpreter unavailable (init failed earlier); evalAsInteger will return" + + " null for this and any subsequent calls. First skipped expression: " + + expr); + } else { + LOGGER.log(Level.FINE, () -> "evalAsInteger returning null (interp unavailable): " + expr); + } return null; } try { From 2b925c4578a90a57b54aaf13228a0611391df139 Mon Sep 17 00:00:00 2001 From: Raffi Khatchadourian Date: Wed, 29 Apr 2026 21:51:48 -0400 Subject: [PATCH 6/7] Address Copilot review: use `AtomicBoolean.compareAndSet` for `unavailableWarned` Mirror of the fork-side fix (per ponder-lab/ML#191). The previous `if (!unavailableWarned) { unavailableWarned = true; ... }` sequence isn't atomic; switch to `AtomicBoolean.compareAndSet(false, true)` so only the thread that flips the flag enters the WARNING branch. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../ibm/wala/cast/python/util/Python3Interpreter.java | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java index 951d1e355..b7c1bd25a 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java @@ -1,5 +1,6 @@ package com.ibm.wala.cast.python.util; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.logging.Level; import java.util.logging.Logger; import org.python.core.PyException; @@ -28,8 +29,13 @@ public class Python3Interpreter extends com.ibm.wala.cast.python.util.PythonInte * during shape inference) would flood logs with one WARNING per call after the first init * failure. The first failure is already announced by the catch block in {@link #getInterp()}; * subsequent calls log at FINE level only. + * + *

Uses {@link AtomicBoolean#compareAndSet} so the check-and-set is atomic — under concurrent + * call graph construction, multiple threads can race into the {@code if (ip == null)} branch + * simultaneously, and a non-atomic {@code volatile boolean} flag would let several of them all + * pass the check before any sets it, defeating the "log once" intent. */ - private static volatile boolean unavailableWarned = false; + private static final AtomicBoolean unavailableWarned = new AtomicBoolean(false); public static synchronized PythonInterpreter getInterp() { if (initFailed) return null; @@ -74,8 +80,7 @@ public Integer evalAsInteger(String expr) { // Log the first such call at WARNING (so operators see that some shape inference is being // skipped because of the earlier init failure); subsequent calls log at FINE only, since // the underlying init failure has already been announced from {@link #getInterp()}. - if (!unavailableWarned) { - unavailableWarned = true; + if (unavailableWarned.compareAndSet(false, true)) { LOGGER.log( Level.WARNING, () -> From b443209bd6581dd244b152de941a765300977e20 Mon Sep 17 00:00:00 2001 From: Raffi Khatchadourian Date: Wed, 29 Apr 2026 21:53:58 -0400 Subject: [PATCH 7/7] Use `@implNote` for the AtomicBoolean rationale Mirror of the fork-side change (per ponder-lab/ML#191 review feedback). The "uses AtomicBoolean.compareAndSet for atomicity" prose is an implementation detail, not contract; move it under `@implNote`. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../com/ibm/wala/cast/python/util/Python3Interpreter.java | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java index b7c1bd25a..d007a4e79 100644 --- a/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java +++ b/jython/com.ibm.wala.cast.python.jython3/source/com/ibm/wala/cast/python/util/Python3Interpreter.java @@ -30,10 +30,10 @@ public class Python3Interpreter extends com.ibm.wala.cast.python.util.PythonInte * failure. The first failure is already announced by the catch block in {@link #getInterp()}; * subsequent calls log at FINE level only. * - *

Uses {@link AtomicBoolean#compareAndSet} so the check-and-set is atomic — under concurrent - * call graph construction, multiple threads can race into the {@code if (ip == null)} branch - * simultaneously, and a non-atomic {@code volatile boolean} flag would let several of them all - * pass the check before any sets it, defeating the "log once" intent. + * @implNote Uses {@link AtomicBoolean#compareAndSet} so the check-and-set is atomic. Under + * concurrent call graph construction, multiple threads can race into the {@code if (ip == + * null)} branch simultaneously; a non-atomic {@code volatile boolean} flag would let several + * of them all pass the check before any sets it, defeating the "log once" intent. */ private static final AtomicBoolean unavailableWarned = new AtomicBoolean(false);