Skip to content

refactor(bigquery-jdbc): migrate metadata thread management to connection-scoped executor #13556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
keshavdandeva merged 5 commits into from
Jun 26, 2026
Merged

refactor(bigquery-jdbc): migrate metadata thread management to connection-scoped executor #13556

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
201a16b
refactor(bigquery-jdbc): migrate metadata thread management to connec…
keshavdandeva Jun 25, 2026
5aa8d67
redirect outer fetcher tasks to cached executor to prevent pool-induc…
keshavdandeva Jun 25, 2026
493e9c7
chore: address pr feedback
keshavdandeva Jun 25, 2026
1babe66
address pr feedback
keshavdandeva Jun 26, 2026
45a7f4f
Merge remote-tracking branch 'origin/main' into jdbc/metadata-thread-…
keshavdandeva Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryConnection.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,15 +362,13 @@ public class BigQueryConnection extends BigQueryNoOpsConnection {

this.headerProvider = createHeaderProvider();
this.bigQuery = getBigQueryConnection();
// Fixed thread pool queues tasks to limit concurrent metadata calls and prevent API

@logachev logachev Jun 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why changing this to lazy init? Init is cheap & it won't start threads unless required

@keshavdandeva keshavdandeva Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking to optimize it for the cases when connections only execute SQL statements (and never run metadata queries) so it takes absolutely zero memory or object allocations for the metadata pool.

But yeah you are right, I missed Init is cheap & it won't start threads unless required. Reverted.

// throttling.
this.metadataExecutor =
BigQueryJdbcMdc.newFixedThreadPool(
String.format("BQ-Metadata-%s", connectionId), metadataFetchThreadCount);
// Cached pool executes queries immediately without queueing and reclaims all idle threads
// when inactive, minimizing resources.
this.queryExecutor =
BigQueryJdbcMdc.newCachedThreadPool(String.format("BQ-Query-%s", connectionId));
this.metadataExecutor =
BigQueryJdbcMdc.newFixedThreadPool(
String.format("BQ-Metadata-%s", connectionId), this.metadataFetchThreadCount);
Comment on lines 365 to +371

@gemini-code-assist gemini-code-assist Bot Jun 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To avoid unnecessary performance and memory overhead when metadata methods are not called, initialize the metadataExecutor lazily inside getMetadataExecutor() instead of eagerly in the constructor. This also avoids any potential order-of-initialization issues with this.metadataFetchThreadCount during connection construction.

References
  1. Prefer lazy initialization over eager initialization for resource-intensive objects if they are not guaranteed to be used in all execution paths, to avoid unnecessary performance and memory overhead.

@keshavdandeva keshavdandeva Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No

}
}

Expand Down
Loading
Loading