Release GIL during remote storage operations to prevent blocking

[beinan]

[Performance] Release GIL during graph execution and storage operations

Context

lance-graph relies on the underlying Lance format for storage and potentially remote object stores (S3, GCS) for persistence. Operations like executing Cypher queries (CypherQuery.execute) or persisting nodes/relationships involve significant I/O.

Problem

If the Python bindings for lance-graph use a "sync-over-async" bridge (e.g., rt.block_on) to call into the Rust core without explicitly releasing the Global Interpreter Lock (GIL), the Python interpreter will freeze during these operations.

This is critical when:

1. Remote Storage is used: Latency for S3/GCS operations (50ms - 500ms+) will block the main thread entirely.
2. Long-running Queries: Complex graph traversals or vector searches running in Rust will starve Python background threads (heartbeats, web server loops, UI) if the GIL is held for the duration of the query.

Proposed Solution

Audit the PyO3 bindings in python/ (likely python/src/lib.rs or similar) and ensure that any blocking runtime execution is wrapped in py.allow_threads.

Implementation Plan

1. Audit & Identification

Locate where Python methods call into async Rust code. Potential targets:

• CypherQuery::execute (if it spawns async tasks for scanning/filtering)
• GraphConfig / Storage initialization
• Any method using tokio::runtime::Runtime::block_on

2. Refactor Bindings

Refactor blocking calls to release the GIL.

Before (Blocking):

#[pyfunction]
fn execute_query(query: String) -> PyResult<PyArrowTable> {
    let rt = Runtime::new().unwrap();
    // Holds GIL while executing graph logic
    rt.block_on(async { engine.execute(&query).await }) 
}

After

#[pyfunction]
fn execute_query(py: Python<'_>, query: String) -> PyResult<PyArrowTable> {
    // 1. Prepare arguments (Clone/Move to Rust types)
    let query_clone = query.clone();

    // 2. Release GIL
    let result = py.allow_threads(move || {
        let rt = Runtime::new().unwrap();
        rt.block_on(async {
            // Expensive graph traversal / IO happens here
            engine.execute(&query_clone).await
        })
    });

    // 3. Handle Result (GIL re-acquired)
    result.map_err(|e| PyValueError::new_err(e.to_string()))
}

3. Verification

Add a concurrency test (python/tests/test_concurrency.py) to verify that a background thread (e.g., a simple counter or heartbeat) continues to run while a heavy graph query is executing.

import threading
import time
from lance_graph import CypherQuery

def test_gil_release_during_query():
    heartbeats = 0
    stop_event = threading.Event()

    def heartbeat():
        nonlocal heartbeats
        while not stop_event.is_set():
            heartbeats += 1
            time.sleep(0.01)

    t = threading.Thread(target=heartbeat)
    t.start()

    try:
        # Run a query that takes some non-trivial time
        # (Mock or use a large enough dataset/complex join)
        query = CypherQuery("MATCH (n) RETURN n")
        query.execute(...)
        
        assert heartbeats > 5, "Background thread was starved! GIL was not released."
    finally:
        stop_event.set()
        t.join()

Acceptance Criteria

• Audit performed on python/ bindings.
• py.allow_threads applied to heavy query/IO paths.
• Concurrency test confirms background thread liveness.

[jja725]

Do we potentially want to support async api using https://github.com/PyO3/pyo3-async-runtimes? I feel sometimes I get confused about whether I need to wait for the operation to finish or not if we release GIL.

[beinan]

I'd suggest:

1. Keep the current sync API - It's working correctly and is simpler for basic use cases
2. Add an optional async API alongside it using pyo3-async-runtimes
3. Document the difference clearly:
   - Sync API: Simple, blocks but releases GIL (current)
   - Async API: For async Python code, returns coroutines

[jezell]

Add async API definitely