feat(bigquery-jdbc): migrate getPrimaryKeys to use BQ API#13691
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the getPrimaryKeys method in BigQueryDatabaseMetaData to retrieve primary key metadata directly via the BigQuery API instead of executing a SQL query from a file. This change includes introducing concurrent processing of target tables and removing the obsolete SQL file and its associated tests. The code review feedback recommends wrapping potential BigQueryException runtime exceptions in SQLException to adhere to the JDBC API contract, and adding a finally block to cancel incomplete futures during concurrent execution to prevent thread leaks.
|
/gemini review |
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
|
/gemini review |
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
| List<String> projects = resolveTargetProjects(catalog); | ||
| List<DatasetId> datasetIds = new ArrayList<>(projects.size()); | ||
| for (String project : projects) { | ||
| datasetIds.add(DatasetId.of(project, schema)); |
There was a problem hiding this comment.
What happens if specific project doesn't have that dataset? e.g. have 3 projects, but only 2 have dataset named foo
There was a problem hiding this comment.
Good catch! I used the same logic here to throw exception (instead of logging/swallowing when there is an error with one of the catalogs) but it shouldn't be that way.
I've updated processSingleTable to catch BigQueryException. During catalog-wide searches (catalog == null), if a dataset/table is missing (404) or inaccessible (403), we now log it at INFO level and skip it instead of failing the entire metadata query. If a specific catalog was queried, these errors will still correctly propagate
|
Please check if we have integration tests covering this. |
Yes, we have |
2 participants
b/532245342
Migrates the
getPrimaryKeysmetadata methods inBigQueryDatabaseMetaDataaway from executing SQL queries onINFORMATION_SCHEMAtables, replacing them with native BigQuery Java Client SDK calls.This PR also introduces the shared thread-safe concurrent dataset scanning infrastructure that will be utilized by subsequent metadata migrations.
Key Changes:
getPrimaryKeys: Rewritten to fetch metadata usingbigquery.getTable(...)and parsing table constraints in-memory.processTargetTablesConcurrently(...)andprocessSingleTable(...)to handle concurrent metadata retrieval across target datasets using the metadata executor thread pool.resolveTargetProjects(...)to support wildcard schema matching.DatabaseMetaData_GetPrimaryKeys.sqlresource file.testReadSqlFromFileandtestMetadataMethodsDoNotInterfere) fromBigQueryDatabaseMetaDataTest.javaas they are no longer applicable.