[WIP] [Iceberg 1.11] Add support for remote signing (full feature)#3995
[WIP] [Iceberg 1.11] Add support for remote signing (full feature)#3995adutra wants to merge 3 commits intoapache:feature/iceberg-1.11from
Conversation
This PR introduces support for access delegation in the `registerTable` endpoint. Four new authorizable operations are introduced: - `REGISTER_TABLE_WITH_READ_DELEGATION`: requires `TABLE_CREATE` + `TABLE_READ_DATA` - `REGISTER_TABLE_WITH_WRITE_DELEGATION`: requires `TABLE_CREATE` + `TABLE_WRITE_DATA` - `REGISTER_TABLE_OVERWRITE_WITH_READ_DELEGATION`: requires `TABLE_FULL_METADATA`+ `TABLE_READ_DATA` - `REGISTER_TABLE_OVERWRITE_WITH_WRITE_DELEGATION`: requires `TABLE_FULL_METADATA` + `TABLE_WRITE_DATA` The implementation is inspired by the `loadTable` endpoint; the authorization checks are very similar.
| * href="https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L1858-L1859">Iceberg | ||
| * REST Catalog spec</a>. | ||
| */ | ||
| private AccessDelegationMode selectAccessDelegationMode( |
| * href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html">Create | ||
| * a signed AWS API request</a> | ||
| */ | ||
| private static final Set<String> UNSIGNED_HEADERS = |
There was a problem hiding this comment.
See apache/iceberg#15166, apache/iceberg#15171 and apache/iceberg#15428 for a thorough discussion on this topic.
adutra
force-pushed
the
request-signing-promoted-m2
branch
5 times, most recently
from
fa50884 to
d4103c6
Compare
|
Hi @adutra, thanks a lot for working on it. Will the new spec for s3 signing ship with Iceberg 1.11? |
Thanks for the input. Should we wait for that? A draft PR is completely fine though. |
adutra
force-pushed
the
request-signing-promoted-m2
branch
from
d4103c6 to
0b57ed4
Compare
What do you mean? I think it would be inappropriate to merge this PR before we have an Iceberg release that contains all the bits we need for remote signing. Unfortunately, this means we'll have to wait until 1.12, I'm afraid. |
I agree. Since the necessary Iceberg spec changes won't land in 1.11, merging this now would create a broken dependency state. Given this PR is currently targeting the 1.11 branch, should we keep it as a Draft for reference, or would you prefer to close this and reopen it against the main (or 1.12) branch once the upstream dependencies are released? |
I think I will reopen this PR against a future 1.12 branch. I can try to change this PR base, but from my experience that doesn't always work. |
2 participants
I have this branch waiting since last year for Iceberg 1.11 to bring the changes needed for remote signing, but unfortunately, it's likely that the changes won't make it into 1.11.
So, I decided to open this draft PR instead, and move on.
It contains the full remote signing feature, including the items in "milestone 2" (encrypted remote signing token, access location checks, and reverse proxy settings). Caching behavior has also been introduced.
Unfortunately it depends on many unreleased items:
Until apache/iceberg#15451 are apache/iceberg#15168 are released, you will need a custom Iceberg build containing both, in order to compile this branch.
The real commit to review is the last one.
Design doc: https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?tab=t.0
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)