docs: clarify what the BQ concept index is

[caohy1988]

Question

For the latest BQ AA SDK main, what is the BQ concept index? Is "concept" here meant in the RDFS/SKOS terminology sense? Is this a special kind of index that BigQuery materializes?

Answer

Short version: the BQ concept index is an SDK-generated normal BigQuery table, plus a __meta sibling table, used as a lookup surface for ontology entity resolution. It is not a native BigQuery index feature.

More precisely:

• gm compile --emit-concept-index --concept-index-table <project.dataset.table> emits SQL text for two CREATE OR REPLACE TABLE statements.
• The main table has one row per (entity_name, label, label_kind, language, scheme) tuple. Labels come from ontology entity names, synonyms, SKOS label annotations, and skos:notation.
• The __meta table stores compile provenance and fingerprints so a runtime can later verify that the table still matches the ontology + binding that produced it.
• BigQuery does not automatically maintain this table. The operator materializes it by executing the emitted SQL, and refreshes it by rerunning gm compile and executing the new SQL.

So "index" here is an SDK/domain term: a denormalized lookup table optimized for SQL joins and resolver queries. It is not CREATE SEARCH INDEX, CREATE VECTOR INDEX, or another native BigQuery-managed index structure.

What does "concept" mean here?

The term is related to SKOS/RDFS vocabulary, but the SDK uses it a little more broadly. In this context, a "concept" is a resolvable ontology entity candidate: something that free text can resolve to through labels, synonyms, notation, or scheme membership.

Relationship to SKOS/RDFS:

• Pure skos:Concept resources imported from TTL become abstract ontology entities, usually with a skos_ prefix. These are included in the concept index even though they are not bound to BigQuery tables.
• Mixed OWL/RDFS + SKOS resources can also appear. If a resource is an OWL class and also a SKOS concept, OWL provides the structural entity and SKOS contributes labels/notations/scheme metadata.
• Concrete ontology entities that are bound to BigQuery tables are also included, even if they did not originate as skos:Concept, because they can still be resolved from user/input text.

So the concept index is not only a table of SKOS concepts. It is a lookup table over the SDK ontology entities that are in resolver scope: abstract informational SKOS-derived entities are always included; concrete entities are included when they appear in the binding.

Why it exists

The concept index supports runtime/entity-resolution workflows such as:

• resolving free-text user inputs like geo, audience, or product values to declared ontology entities;
• joining production traces to ontology labels in bulk SQL;
• computing coverage by SKOS scheme or taxonomy;
• later supporting OntologyRuntime / resolver APIs on top of the same table.

Suggested docs clarification

docs/ontology/concept-index.md should probably state near the top:

The concept index is a normal BigQuery table emitted by the SDK compiler, not a BigQuery-native index object. "Concept" means a resolver-visible ontology entity, including but not limited to imported skos:Concept resources.

This would reduce confusion for readers who associate "index" with BigQuery Search/Vector indexes or "concept" strictly with SKOS.

[caohy1988]

Validation against current main

I cross-checked every claim against docs/ontology/concept-index.md, src/bigquery_ontology/concept_index.py, src/bigquery_ontology/owl_importer.py, and src/bigquery_ontology/docs/user_manual.md. The answer is accurate. Three small refinements below.

What's correct

Claim	Source
SDK-generated normal BigQuery table, not a native BQ index	docs/ontology/concept-index.md §1 — emitted via CREATE OR REPLACE TABLE T AS SELECT * FROM UNNEST(...), not CREATE SEARCH/VECTOR INDEX
Two CREATE OR REPLACE TABLE statements per emit	concept-index.md §1 line 15
One row per (entity_name, label, label_kind, language, scheme)	concept-index.md §4 + concept_index.py:179 (the _dedup_rows key is exactly this tuple)
Labels from name + synonyms + SKOS annotations + skos:notation	concept_index.py:232-259 (_emit(entity.name, "name", None) + _LABEL_PREFIX_MAP + synonyms + notation rows)
__meta stores provenance for runtime verification	concept-index.md §5
BigQuery does not auto-maintain; operator runs the SQL	concept-index.md §3 — "gm compile emits SQL text only; it does not call BigQuery"
Abstract entities always included; concrete entities included iff in binding	concept-index.md §4 "Scope rule" + concept_index.py:155-157
OWL+SKOS hybrid: OWL provides structure, SKOS contributes labels/notation/scheme	user_manual.md:461 verbatim

Minor refinements

1. "Usually with a skos_ prefix" — tighten to "always" for the standalone case. Per user_manual.md:461 and owl_importer.py:459:

   • Standalone skos:Concept (not typed as owl:Class) → always renamed skos_<name>, abstract.
   • owl:Class + skos:Concept → stays concrete and unprefixed — OWL structure dominates, SKOS contributes annotations only.

   The "usually" wording reads as if the prefix is heuristic. It isn't — it's a deterministic disambiguation rule applied by owl_importer._build_skos_only_concept_name_map().

2. Opt-in nature is worth mentioning. gm compile without --emit-concept-index is byte-identical to today and writes no extra DDL (concept-index.md line 6). Readers of the issue may assume the index is always materialized; it isn't — it's an explicit flag.

3. Provenance lives on every main-table row, not only in __meta. Each row of the main table carries both compile_id (12-hex) and compile_fingerprint (64-hex) — see concept_index.py:121-122 and concept-index.md §4 schema. That's why the §8 pair-consistency check works: the runtime can SELECT DISTINCT compile_fingerprint FROM main and compare against meta. The current answer says "the __meta table stores compile provenance and fingerprints," which is accurate but understates — runtimes can self-verify off the main table alone.

Also worth a sentence in the suggested docs blurb

The proposed clarification at the bottom of the issue is solid. Consider adding one line about the opt-in flag so readers don't conflate "ontology DDL exists" with "concept index exists":

Emitting the concept index requires the explicit gm compile --emit-concept-index --concept-index-table <fq> flags; without them, gm compile is byte-identical to its pre-A2 behavior.

Verdict

Accurate. Refinement 1 (the skos_ prefix rule) is the only one that risks reader confusion; refinements 2 and 3 are additive context. Nothing in the answer is wrong against current main.