Skip to content

Commit 49cbbbd

Browse files
keshavdandevalqiu96
authored andcommitted
refactor(bigquery-jdbc): migrate metadata thread management to connection-scoped executor (#13556)
b/520400589 This PR optimizes thread management for JDBC metadata methods, implementing a deadlock-free, bipartite thread pool design and unifying resource lifecycle management. ### Key Changes * **Bipartite Thread Pool Architecture (Deadlock Prevention)**: * Redirected all outer, orchestrating fetcher tasks (parent tasks) to the connection’s cached query executor (`connection.getExecutorService()`). Because the cached pool scales dynamically as needed, this structurally prevents pool-induced deadlocks (thread starvation). * Kept all inner, highly concurrent parallel network API calls (child tasks) on the connection's fixed-size metadata executor (`connection.getMetadataExecutor()`). This bounds parallel network traffic to prevent overwhelming the BigQuery backend. * *Result*: The driver is fully protected against pool-induced deadlocks, even under high concurrency or if `metadataFetchThreadCount` is configured to `1`. * **Connection-Scoped Shared Executor**: * Migrated all asynchronous metadata operations (`getSchemas`, `getTables`, `getColumns`, `getProcedures`, `getProcedureColumns`, `getFunctions`, and `getFunctionColumns`) from spawning raw threads or short-lived local pools to connection-managed executors. * Configured the shared metadata pool to be lazily instantiated on demand and gracefully shut down with the connection lifecycle. Idle threads are automatically reclaimed after 60 seconds. * **Lower CPU Overhead (Sequential Row-Building)**: * Completely removed secondary CPU-bound thread pools (such as `routineProcessorExecutor`, `processArgsExecutor`, `tableProcessorExecutor`, and `processParamsExecutor`) previously used for row-building. * Refactored row-building and parameter-processing tasks to execute sequentially on the background fetcher threads, removing context-switching overhead and preventing nested-pool deadlock risks. * **Teardown of Legacy Bridges**: * Deleted the obsolete `wrapThread(Thread)` compatibility adapter in `BigQueryDatabaseMetaData.java`. * Removed obsolete `testWrapThread_*` cases from `BigQueryDatabaseMetaDataTest.java`. Metadata queries now track and cancel background tasks natively via standard `Future<?>` handles. * **Test Suite Refactoring**: * Updated `BigQueryDatabaseMetaDataTest` setup to inject real, connection-scoped cached executor services, ensuring that background metadata tasks execute deterministically during testing. * Added `@AfterEach` teardown logic to prevent test-suite thread leaks. * Refactored argument-processing tests to verify row-building synchronously without mock executors.
1 parent 6ca6b2f commit 49cbbbd

4 files changed

Lines changed: 101 additions & 491 deletions

File tree

java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryConnection.java

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -362,15 +362,13 @@ public class BigQueryConnection extends BigQueryNoOpsConnection {
362362

363363
this.headerProvider = createHeaderProvider();
364364
this.bigQuery = getBigQueryConnection();
365-
// Fixed thread pool queues tasks to limit concurrent metadata calls and prevent API
366-
// throttling.
367-
this.metadataExecutor =
368-
BigQueryJdbcMdc.newFixedThreadPool(
369-
String.format("BQ-Metadata-%s", connectionId), metadataFetchThreadCount);
370365
// Cached pool executes queries immediately without queueing and reclaims all idle threads
371366
// when inactive, minimizing resources.
372367
this.queryExecutor =
373368
BigQueryJdbcMdc.newCachedThreadPool(String.format("BQ-Query-%s", connectionId));
369+
this.metadataExecutor =
370+
BigQueryJdbcMdc.newFixedThreadPool(
371+
String.format("BQ-Metadata-%s", connectionId), this.metadataFetchThreadCount);
374372
}
375373
}
376374

0 commit comments

Comments
 (0)