Skip to content

feat(storage): inline polaris.storage.name overrides for namespace/table#4023

Open
sririshindra wants to merge 4 commits intoapache:mainfrom
sririshindra:phase1-inline-storage-name-overrides
Open

feat(storage): inline polaris.storage.name overrides for namespace/table#4023
sririshindra wants to merge 4 commits intoapache:mainfrom
sririshindra:phase1-inline-storage-name-overrides

Conversation

@sririshindra
Copy link

Summary

This PR adds inline storage-name overrides for namespaces and tables using the polaris.storage.name property, so credential vending can select named credentials per entity without introducing new management APIs.

It also refactors hierarchy lookup logic to remove duplication and adds focused test coverage for override behavior and credential vending.

Why this change is needed

Today, catalog-level storage credentials are often too coarse for multi-tenant or team-isolated workloads. This change enables namespace/table-scoped credential selection by storing an inline override and resolving it through the existing hierarchy at runtime.

What changed

  1. Added storage-name validation and normalization utilities.
  2. Added a helper to clone storage configuration while replacing only storageName.
  3. Allowed polaris.storage.name through reserved-property filtering.
  4. Implemented override handling for:
    1. namespace create
    2. namespace setProperties
    3. namespace removeProperties
    4. table metadata/property flows
  5. Refactored hierarchy traversal into shared utility logic to avoid duplicate implementations.
  6. Updated and expanded tests, including an end-to-end credential-vending assertion where leaf override credentials are vended.

Behavioral notes

  1. polaris.storage.name accepts alphanumeric, hyphen, underscore, up to 128 chars.
  2. Blank values normalize to null.
  3. Override resolution uses nearest ancestor storage config and replaces only storageName.
  4. Credential vending uses the resolved inline storage configuration for the target entity.

Testing

  1. Added/updated unit and integration tests for validation, inheritance, persistence, and vending behavior.
  2. Verified targeted runtime-service tests for override + vending paths passed.

Reviewer notes

  1. This PR intentionally keeps storage type inheritance unchanged: override changes storageName, not storage type.
  2. A follow-up PR is planned for cross-type override scenarios (for example AWS catalog overridden by Azure/GCS at leaf scope).

Checklist

  • 🛡️ Don't disclose security issues! (contact security@apache.org)
  • 🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
  • 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
  • 💡 Added comments for complex logic
  • 🧾 Updated CHANGELOG.md (if needed)
  • 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

…nd reserved property allowlist

Phase 0 of inline storage configuration spec:
- Add StorageNameValidator with format validation (alphanumeric + hyphens + underscores, 1-128 chars) and normalizeBlankToNull utility.
- Add PolarisStorageConfigurationInfo.withStorageName() static helper that clones a config with only storageName replaced, preserving all other fields across all four storage types (AWS, GCS, Azure, File).
- Allowlist polaris.storage.name in ReservedProperties so it passes through reserved property filtering.
- Add unit tests for validation, normalization, and field preservation round-trips.
…ierarchy lookup

Combines namespace/table override implementation with the hierarchy lookup dedupe and aligned tests.
@tokoko
Copy link
Contributor

tokoko commented Mar 19, 2026

This should definitely go behind a feature flag since there's currently no way to limit what a user can put in polaris.storage.name property.

I hate to be adding to the feature flag proliferation in polaris, but maybe there should be a way to configure at what level override is allowed to happen. I can see an admin wanting to allow override on namespace level, but not on table level since create table permissions are generally more liberally granted than create namespace ones. A more proper fix is to integrate polaris.storage.name property in the authorization step, but I'm not sure that's practical for internal rbac.

@sririshindra
Copy link
Author

This should definitely go behind a feature flag since there's currently no way to limit what a user can put in polaris.storage.name property.

I hate to be adding to the feature flag proliferation in polaris, but maybe there should be a way to configure at what level override is allowed to happen. I can see an admin wanting to allow override on namespace level, but not on table level since create table permissions are generally more liberally granted than create namespace ones. A more proper fix is to integrate polaris.storage.name property in the authorization step, but I'm not sure that's practical for internal rbac.

Thanks for the review, @tokoko! I'll go ahead and put this behind a feature flag. I intentionally kept authorization out of scope for this phase, but we can definitely address integrating polaris.storage.name into the authz step (and adding level-based configuration) in a follow-up PR.

@sririshindra sririshindra force-pushed the phase1-inline-storage-name-overrides branch from 8cdf152 to 7e84dce Compare March 19, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants