refactor(bigquery-jdbc): optimize and unify concurrent metadata methods

[keshavdandeva]

b/535543457

This PR significantly refactors and optimizes the concurrent execution model for JDBC metadata operations (e.g., getTables, getColumns, getProcedures, getExportedKeys) in BigQueryDatabaseMetaData.java.

Previously, these methods relied on bespoke, scattered threading logic that was susceptible to thread starvation deadlocks (where bounded pools wait on tasks submitted to the same pool) and suffered from sequential dataset listing bottlenecks.

This PR introduces a unified, robust Two-Tier Scatter-Gather Architecture, eliminating deadlocks, restoring parallel execution for broad dataset scans, and standardizing error handling across all metadata fetchers.

Key Changes

1. Unified Scatter-Gather Concurrency Framework:

   • Replaced custom Runnable implementations in each metadata method with shared helper functions (fetchAndPopulateQueueAsync, processTargetTablesConcurrently, processTargetRoutinesConcurrently).
   • Introduced a two-tier thread pool strategy to prevent thread pool starvation (deadlocks):
     • Tier 1 (Orchestrator): The overarching fetchAndPopulateQueueAsync task now runs on the unbounded queryExecutor (via connection.getExecutorService()). This thread safely blocks while coordinating API calls.
     • Tier 2 (API Workers): The bounded metadataExecutor is now exclusively used for concurrent BigQuery HTTP API calls (listTables, getTable, etc.). Workers never wait on other workers, completely removing the deadlock risk.

2. Parallelized Dataset Discovery (Phase 1):

   • Operations that require wildcard scanning across datasets now submit parallel dataset listing tasks (listTables / listRoutines) to the API pool.
   • The orchestrator gathers these listing results to construct a targeted list of parallel detail fetch tasks (Phase 2). This drastically speeds up broad schema discovery compared to the previous sequential listing behavior.

3. Robust Resource Management & Cancellation:

   • Centralized activeFutures tracking inside the new concurrent helpers.
   • If an InterruptedException or SQLException occurs, the finally blocks aggressively cancel all active futures, preventing runaway background API calls and memory leaks.

4. Specific Bug Fixes:

   • getExportedKeys Catalog Fallback: Fixed an issue where an exact schema match string without wildcards was incorrectly evaluated as a regex pattern during the fallback flow, by explicitly passing isPattern=false to getTargetDatasets().

5. Performance Benchmarks:

   • Rigorous benchmarking proves the new architecture eliminates thread starvation and maximizes throughput. While shallow targeted scans maintain parity (~1s), deep, full-catalog scans (e.g., getColumns(null, null, null, null)) are up to 2.6x faster (reducing latency from 23.5s to 8.8s on a 12,000-column project) because the API worker pool is no longer bottlenecked by blocked orchestrator threads.

[gemini-code-assist]

Code Review

This pull request refactors BigQueryDatabaseMetaData.java to streamline metadata retrieval methods (such as getProcedures, getTables, and getColumns) by introducing concurrent helper methods like processTargetRoutinesConcurrently and processTargetTablesConcurrently. The review feedback highlights several critical issues with this refactoring: an architectural regression where heavy metadata retrieval now blocks synchronously on the caller thread; a performance bottleneck due to sequential routine listing for wildcard patterns; a potential NullPointerException when handling execution exceptions; silent failures on thread interruption in waitForTasksCompletion; and a missing empty schema check that leads to redundant API calls.

[keshavdandeva]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors BigQueryDatabaseMetaData.java to streamline metadata fetching by introducing unified concurrent processing helpers and an asynchronous queue population mechanism, significantly reducing boilerplate code across various metadata methods. The review feedback highlights several critical issues where valid null parameters (such as columnNamePattern, schemaPattern, routineNamePattern, and tableNamePattern) are passed directly to compileSqlLikePattern, which would trigger NullPointerExceptions. Additionally, a bug was identified in getColumns where literal column name filtering is ignored due to conditional regex compilation. Lastly, an improvement was suggested to prevent redundant nesting of SQLExceptions during asynchronous error handling.