docs(site): add Delegation Service documentation#3990
docs(site): add Delegation Service documentation#3990jbonofre wants to merge 1 commit intoapache:mainfrom
Conversation
Describe how external services can integrate with Polaris using pull, push, and event-driven patterns for policy retrieval, table change notifications, and reactive event listeners.
jbonofre
added
the
documentation
|
@flyrain @singhpk234 @dimas-b @adnanhemani as we discussed together during the last Polaris Community Sprint, I tried to gather the different perspective about "external services" options, e.g. "Delegation Service". I'm trying first to provide an overview of the different "components" involved, with some use cases in perspective. I'm happy to update this documentation with your suggestions. Thanks ! |
|
|
||
| ``` | ||
| ┌──────────────────────┐ ┌─────────────────────┐ | ||
| │ External Service │ │ External Catalog │ |
There was a problem hiding this comment.
nit: misaligned vertical bars
| **Considerations:** | ||
| - Requires the external system to implement notification sending logic (e.g., a Hive hook or a CDC pipeline) | ||
| - The caller must manage monotonically increasing timestamps and handle `409 Conflict` rejections | ||
| - Only applies to table metadata synchronization — not for querying policies or configuration |
There was a problem hiding this comment.
Does Polaris already support this?
There was a problem hiding this comment.
Not yet, just a pure speculation from my side 😄
There was a problem hiding this comment.
ok 🙂 but it might be best to rephrase since users will read it and make assumptions followed by questions, I'm sure 😅
| | Mode | Header Value | Description | | ||
| |------|-------------|-------------| | ||
| | **Vended Credentials** | `vended-credentials` | Polaris generates short-lived, scoped credentials (e.g., AWS STS tokens) and sends them to the client. The client uses these credentials directly to access storage. | | ||
| | **Remote Signing** | `remote-signing` | The client sends data access requests back to Polaris, which signs them on behalf of the client. The client never receives raw storage credentials. | |
There was a problem hiding this comment.
AFAIK, Remote Signing is not yet fully supported: #2280 (comment)
There was a problem hiding this comment.
True, but it will (at some point) 😄
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| # | ||
| title: Delegation Service |
There was a problem hiding this comment.
suggestion: Integration with External Services?
There was a problem hiding this comment.
I used Delegation Service because it's the wording we used during the Polaris Community Sprint.
There was a problem hiding this comment.
That's fine too... It's just that for me personally Delegation does not ring right in this context... Maybe I was dealing with OAuth2 delegation too much 😅
There was a problem hiding this comment.
Let's see what the others are thinking. I'm fine either way 😄
| register a listener for `AFTER_ATTACH_POLICY` or `AFTER_UPDATE_POLICY` events and trigger a policy refresh in | ||
| the external service when relevant events fire. | ||
|
|
||
| ## Push Pattern — Sending Notifications to Polaris |
There was a problem hiding this comment.
I'm confused by this section. How is notification endpoint related to the delegation service?
There was a problem hiding this comment.
I wanted to provide a "landscape" of the possible interactions between external services and Polaris.
If it's confusing, I'm fine to remove this section.
3 participants
Describe how external services can integrate with Polaris using pull, push, and event-driven patterns for policy retrieval, table change notifications, and reactive event listeners.