[Bug] Reject ambiguous external metadata local-name mappings

[wenzhenghu]

Search before asking

I searched existing Apache Doris issues for duplicate external metadata local names, meta_names_mapping conflicts, and ambiguous remote-to-local name resolution. I did not find an issue covering this exact gap.

Version

Apache Doris master at 2215dc476a1f9a1f6edba8c7a056e450165f7001.

This problem was found while reviewing #65126, but the ambiguous mapping state is already possible on the master baseline and was not introduced by that PR.

What is wrong?

External metadata name loading does not unconditionally enforce that one exact local database/table name identifies only one remote object.

For example, a custom naming hook or mapping can produce:

RemoteA -> LocalX
RemoteB -> LocalX

This state is not resolvable: SQL can reference only LocalX, while the object cache key and generated Doris metadata ID are also derived from LocalX.

The generic conflict checks currently run only under selected case-insensitive/lower-case modes. In ordinary mode 0, a custom connector naming hook or an uncovered mapping collision can still produce the same exact local name for multiple remote objects.

Master baseline behavior

The legacy MetaCache keeps the complete remote/local pairs in a mutable list.

• MetaCache.listNames() returns both local entries, so database enumeration can expose duplicate LocalX values.
• MetaCache.getRemoteName(LocalX) uses findFirst(), so only RemoteA is selected.
• The metadata object cache is keyed only by LocalX, so RemoteB cannot be addressed independently.
• Incremental updateCache() appends another pair without replacing the existing local identity, making the result event-order dependent.

Relevant code:

• doris/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCache.java

  Lines 83 to 92 in 2215dc4

  	public List<String> listNames() {
  	return Objects.requireNonNull(namesCache.get("")).stream().map(Pair::value).collect(Collectors.toList());
  	}
  	public String getRemoteName(String localName) {
  	return Objects.requireNonNull(namesCache.getIfPresent("")).stream()
  	.filter(pair -> pair.value().equals(localName))
  	.map(Pair::key)
  	.findFirst()
  	.orElse(null);

• doris/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCache.java

  Lines 128 to 138 in 2215dc4

  	public void updateCache(String remoteName, String localName, T obj, long id) {
  	metaObjCache.put(localName, Optional.of(obj));
  	namesCache.asMap().compute("", (k, v) -> {
  	if (v == null) {
  	return Lists.newArrayList(Pair.of(remoteName, localName));
  	} else {
  	v.add(Pair.of(remoteName, localName));
  	return v;
  	}
  	});
  	idToName.put(id, localName);

• doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java

  Lines 240 to 267 in 2215dc4

  	// Check for conflicts when stored table names or meta names are case-insensitive
  	if (Boolean.parseBoolean(extCatalog.getLowerCaseMetaNames())
  	|| this.isStoredTableNamesLowerCase()
  	|| this.isTableNamesCaseInsensitive()) {
  	// Map to track lowercased local names and their corresponding remote names
  	Map<String, List<String>> lowerCaseToRemoteNames = Maps.newHashMap();
  	// Collect lowercased local names and their remote counterparts
  	for (Pair<String, String> pair : tableNames) {
  	String lowerCaseLocalName = pair.value().toLowerCase();
  	lowerCaseToRemoteNames.computeIfAbsent(lowerCaseLocalName, k -> Lists.newArrayList()).add(pair.key());
  	}
  	// Identify conflicts: multiple remote names mapping to the same lowercased local name
  	List<String> conflicts = lowerCaseToRemoteNames.values().stream()
  	.filter(remoteNames -> remoteNames.size() > 1) // Conflict: more than one remote name
  	.flatMap(List::stream) // Collect all conflicting remote names
  	.collect(Collectors.toList());
  	// Throw exception if conflicts are found
  	if (!conflicts.isEmpty()) {
  	throw new RuntimeException(String.format(
  	ExternalCatalog.FOUND_CONFLICTING + " table names under case-insensitive conditions. "
  	+ "Conflicting remote table names: %s in remote database '%s' under catalog '%s'. "
  	+ "Please use meta_names_mapping to handle name mapping.",
  	String.join(", ", conflicts), remoteName, extCatalog.getName()));
  	}
  	}

• doris/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java

  Lines 593 to 618 in 2215dc4

  	// Check for conflicts when lower_case_meta_names = true or lower_case_database_names = 2
  	if (Boolean.parseBoolean(getLowerCaseMetaNames()) || getLowerCaseDatabaseNames() == 2) {
  	// Map to track lowercase local names and their corresponding remote names
  	Map<String, List<String>> lowerCaseToRemoteNames = Maps.newHashMap();
  	// Collect lowercased local names and their remote counterparts
  	for (Pair<String, String> pair : remoteToLocalPairs) {
  	String lowerCaseLocalName = pair.second.toLowerCase();
  	lowerCaseToRemoteNames.computeIfAbsent(lowerCaseLocalName, k -> Lists.newArrayList()).add(pair.first);
  	}
  	// Identify conflicts: multiple remote names mapping to the same lowercase local name
  	List<String> conflicts = lowerCaseToRemoteNames.values().stream()
  	.filter(remoteNames -> remoteNames.size() > 1) // Conflict: more than one remote name
  	.flatMap(List::stream) // Collect all conflicting remote names
  	.collect(Collectors.toList());
  	// Throw exception if conflicts are found
  	if (!conflicts.isEmpty()) {
  	throw new RuntimeException(String.format(
  	FOUND_CONFLICTING + " database names under case-insensitive conditions. "
  	+ "Conflicting remote database names: %s in catalog %s. "
  	+ "Please use meta_names_mapping to handle name mapping.",
  	String.join(", ", conflicts), name));
  	}
  	}

Impact

The trigger is narrow because normal remote metadata normally has unique names and standard mapping implementations reject many direct conflicts. The remaining gap mainly affects custom connector naming hooks, general ExternalCatalog subclasses, or uncovered mode-0 mapping collisions.

Once triggered, the impact is a correctness problem:

• one remote database/table becomes inaccessible or is resolved through the wrong remote identity
• cached and cache-bypass paths may select different remote objects
• object-cache keys and deterministic IDs collide
• database enumeration may contain duplicate local names
• incremental events may change the selected remote object according to event order

Expected behavior

Within one catalog scope for databases, and within one database scope for tables:

• an exact local name must identify at most one remote object
• two different remote names mapping to the same exact local name must fail with a clear conflict error
• case-sensitive mode 0 must continue to allow distinct local names such as Foo and foo
• mode-specific case-insensitive conflict checks must remain in effect
• duplicate/replayed incremental events must remain idempotent

Suggested fix

1. Validate the exact local-name uniqueness invariant when building the complete external metadata names snapshot.
2. Report both conflicting remote names and the shared local name.
3. Add a defensive invariant in the shared names value abstraction so custom loaders cannot publish an ambiguous snapshot.
4. Keep mode-aware case-insensitive collision validation in the catalog/database loader.
5. Add database and table tests, including a custom naming hook in mode 0.

PR #65126 plans to fix this by validating the invariant in its immutable NameCacheValue snapshot before publication.

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_739ea86fc1ae

Initial assessment

Confirmed correctness bug on the reported 2215dc476a1f9a1f6edba8c7a056e450165f7001 baseline. The final remote-to-local mapping is not injective in all modes, although lookup, object-cache keys, and generated IDs require it to be. The trigger is narrow, but once present Doris can address or cache the wrong remote object, so this should be treated as an FE external-catalog correctness issue rather than an enumeration-only defect.

The issue currently has no labels or assignee. Maintainers should add the repository's applicable bug and external-catalog/FE labels.

Code findings

• Database and table snapshot loaders only run their generic collision checks under selected lowercase/case-insensitive settings. An exact RemoteA -> LocalX, RemoteB -> LocalX collision therefore passes in mode 0.
• A concrete standard-mapping trigger is a collision between an explicit mapping and an unmapped pass-through name: a remote list containing RemoteA and LocalX, with only RemoteA -> LocalX configured. The JDBC mapper validates explicit entries against one another but cannot detect the pass-through collision without the complete remote-name snapshot.
• There is an additional built-in database trigger: lower_case_database_names=1 lowercases the final local name at ExternalCatalog.java:583-585, while the conflict guard at ExternalCatalog.java:593-594 covers lower_case_meta_names and database mode 2, not database mode 1. Thus remote Foo and foo can both become local foo even without a custom hook.
• The legacy cache behavior is deterministic from the code: listNames() retains duplicate local values, getRemoteName() selects the first pair, the object cache overwrites by the shared local key, and updateCache() appends pairs. After conflicting incremental events, the reverse-name entry can therefore identify one remote object while the hot object entry contains the other. Replaying the same event also grows the names list.
• Cache-bypass table lookup independently scans the freshly loaded pairs with findFirst(), so it does not restore addressability and can disagree with a hot object-cache entry depending on list/event order.

Required fix properties

The suggested direction is correct: validate exact local-name uniqueness after all mapping and mode-specific transformations, for database scope and table-within-database scope. Keep that exact check separate from the existing mode-aware case-insensitive check so mode 0 continues to allow distinct Foo and foo.

For #65126, constructor/snapshot validation alone is not sufficient for incremental updates if NameCacheValue.withName() first removes the existing local name and then adds the new pair. That sequence hides RemoteA -> LocalX followed by RemoteB -> LocalX from constructor validation. withName() should instead:

1. accept an exact replay of the same remote/local pair without adding another entry;
2. reject a different remote name for an occupied exact local name;
3. reject one remote name being rebound to a different local name unless the rename path has explicitly removed the old identity first.

The conflict error should include the shared local name and both remote names. Validation must occur before publishing names, object, or ID state.

Tests needed before closing

• Database and table snapshot loads with a mode-0 custom naming hook producing two different remote names for one exact local name.
• Explicit-mapping-versus-pass-through collisions, not only two duplicate explicit mappings.
• The direct database mode-1 Foo/foo collision.
• Mode-0 Foo and foo remaining valid when their exact local names are distinct.
• Existing lower-case/case-insensitive conflict behavior remaining intact.
• Incremental exact replay remaining idempotent, while a different remote name targeting the same local name fails without partially changing names, object-cache, or ID state.
• Error assertions covering the shared local name and both remote names.

No FE log, query profile, or runtime reproduction is required to confirm the root cause because the failure follows directly from the supplied revision. A focused unit reproduction as above is the only missing evidence needed with the fix.

[wenzhenghu]

A complete reference implementation is available as a patch:

https://gist.github.com/wenzhenghu/532dd8c44fd5eb18f116acbde8f1bba6

The patch is based on the current PR #65126 head before scope reduction and includes:

• exact remote/local mapping uniqueness validation in NameCacheValue;
• consolidation of localNames and localNameSet into one ordered immutable set;
• the same validation for delegated-session/cache-bypass table and Iceberg REST database paths;
• unit tests for the snapshot, shared-cache, and cache-bypass paths.

SHA-256: 92fc2e9a5074bc3aec9f880d192ecbce374b0c5aaab449281ef7ec62735fab68

For PR #65126, the planned scope is intentionally smaller: only the shared-cache uniqueness invariant and collection consolidation will remain. The cache-bypass changes in this patch are retained here as follow-up reference code to reduce merge risk in the current PR.