From 2f909a22fc1787123a43f5e155f4612546406de4 Mon Sep 17 00:00:00 2001 From: gammazero Date: Wed, 21 Feb 2024 13:02:01 -0800 Subject: [PATCH 1/3] Add SeqNum field to advertisement specification --- IPNI.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/IPNI.md b/IPNI.md index 2ac3be3..d8fac47 100644 --- a/IPNI.md +++ b/IPNI.md @@ -155,7 +155,7 @@ its existence to the network. IPNI nodes are responsible for continuously listening to provider announcements. Once they receive an announcement, they fetch the advertisement and walk its chain to effectively construct the current list of content hosted by the provider. Because the advertisements themselves are immutable, -IPNI nodes can infer seem from unseen advertisements and only walk the portion of the chain that has +IPNI nodes can infer seen from unseen advertisements and only walk the portion of the chain that has not seen before. This property enables efficient traversal of the chain and allows IPNI nodes to tolerate very long ad chains as long as they continuously listen to advertisements and stay relatively close to the chain's _head_, i.e. the latest advertisement in the chain. @@ -197,6 +197,7 @@ type Advertisement struct { ContextID Bytes Metadata Bytes IsRm Bool + SeqNum optional Int ExtendedProvider optional ExtendedProvider } ``` @@ -219,6 +220,15 @@ type Advertisement struct { remaining format of metadata. The opaque data is send to the provider when retrieving content for the provider to use to retrieve the content. Storetheindex operators may limit the length of this field, and it is recommended to keep it below 100 bytes. +* **`SeqNum`** is a monotonically increasing integer that starts at 0 and is incremented for each + additional advertisement added to the chain. It is an optional field for the purpose of backward + compatibility with older advertisements that lack a `SeqNum` field. `SeqNum` is used for the + calculation of chain distance between different advertisements. An advertisement chain, whose + advertisements do not have a sequence number, can start adding advertisements with a `SeqNum` at + any time, but must continue including an increasing `SeqNum` field in advertisements onward. The + maximum value for `SeqNum` is 253-1. It is not expected for any chain to reach this + sequence, and restarting `SeqNum` may result in an inaccurate distance between advertisements + spanning the reset. * **`ExtendedProvider`** is an optional field; if specified, indexers which understand the `ExtendedProvider` extension should ignore the `Provider`, `Addresses`, and `Metadata` specified in the advertisement in favor of those specified in the `ExtendedProvider`. The values From 46cff535a5ef7ace2b7c59acbfe408d596e0ff80 Mon Sep 17 00:00:00 2001 From: gammazero Date: Thu, 22 Feb 2024 09:44:39 -0800 Subject: [PATCH 2/3] Add subsection describing SeqNum field --- IPNI.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/IPNI.md b/IPNI.md index d8fac47..f3abc06 100644 --- a/IPNI.md +++ b/IPNI.md @@ -226,9 +226,7 @@ type Advertisement struct { calculation of chain distance between different advertisements. An advertisement chain, whose advertisements do not have a sequence number, can start adding advertisements with a `SeqNum` at any time, but must continue including an increasing `SeqNum` field in advertisements onward. The - maximum value for `SeqNum` is 253-1. It is not expected for any chain to reach this - sequence, and restarting `SeqNum` may result in an inaccurate distance between advertisements - spanning the reset. + maximum value for `SeqNum` is 253-1. * **`ExtendedProvider`** is an optional field; if specified, indexers which understand the `ExtendedProvider` extension should ignore the `Provider`, `Addresses`, and `Metadata` specified in the advertisement in favor of those specified in the `ExtendedProvider`. The values @@ -310,6 +308,17 @@ Specified protocols are expected to be ordered in increasing order. If the `Metadata` field is not specified, the advertisement is treated as address update only. +#### SeqNum + +The sequence number field is optional only for backwards compatibility. It should be included as soon as possible by both new and existing index providers to enable indexer functionality that uses it. Omitting the `SeqNum` will result in degraded indexer functionality relating to distance calculation, reporting, indexer synchronization. Future indexer versions may choose to only accept new advertisements that have a sequence number. + +A `SeqNum` value higher then the maximum of 253-1 (9007199254740992) values will not be recognized and instead seen as 0. It is not expected for any chain to reach this + sequence as that would mean it has over nine quintillion advertisements. If this were reached, then `SeqNum` should be reset to 0, which may result in an inaccurate distance calculations between advertisements spanning the reset. + +If `SeqNum` is present then it is included in the data that the advertisement signature is calculated over. + +Indexers are not required to validate that `SeqNum` is monotonically increasing, but an incorrect sequence number may result in incorrect indexing information or may be ignored resulting in some indexing features being unavailable for that chain's provider. If an indexer depends on the sequence number for more than display and reporting purposes, it can choose not index advertisement chains that do not have a monotonically increasing `SeqNum`. + #### ExtendedProvider The `ExtendedProvider` field allows for specification of provider families, in cases where a From 790b26b8ce3eeae8a8ffbe45f3e093656f6b200f Mon Sep 17 00:00:00 2001 From: gammazero Date: Mon, 26 Feb 2024 08:47:20 -0800 Subject: [PATCH 3/3] Minor edits --- IPNI.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/IPNI.md b/IPNI.md index f3abc06..5a8bacd 100644 --- a/IPNI.md +++ b/IPNI.md @@ -220,13 +220,12 @@ type Advertisement struct { remaining format of metadata. The opaque data is send to the provider when retrieving content for the provider to use to retrieve the content. Storetheindex operators may limit the length of this field, and it is recommended to keep it below 100 bytes. -* **`SeqNum`** is a monotonically increasing integer that starts at 0 and is incremented for each - additional advertisement added to the chain. It is an optional field for the purpose of backward - compatibility with older advertisements that lack a `SeqNum` field. `SeqNum` is used for the - calculation of chain distance between different advertisements. An advertisement chain, whose - advertisements do not have a sequence number, can start adding advertisements with a `SeqNum` at - any time, but must continue including an increasing `SeqNum` field in advertisements onward. The - maximum value for `SeqNum` is 253-1. +* **`SeqNum`** is a monotonically increasing integer starting at 0 or at the current length of the + advertisement chain, and is incremented for each additional advertisement added to the chain. It + is an optional field for the purpose of backward compatibility with older advertisements that lack + a `SeqNum` field. An advertisement chain, whose advertisements do not yes have a `SeqNum`, can + start adding advertisements with a `SeqNum` at any time, but must continue including an increasing + `SeqNum` field in advertisements onward. The maximum value for `SeqNum` is 253-1. * **`ExtendedProvider`** is an optional field; if specified, indexers which understand the `ExtendedProvider` extension should ignore the `Provider`, `Addresses`, and `Metadata` specified in the advertisement in favor of those specified in the `ExtendedProvider`. The values @@ -310,15 +309,15 @@ If the `Metadata` field is not specified, the advertisement is treated as addres #### SeqNum -The sequence number field is optional only for backwards compatibility. It should be included as soon as possible by both new and existing index providers to enable indexer functionality that uses it. Omitting the `SeqNum` will result in degraded indexer functionality relating to distance calculation, reporting, indexer synchronization. Future indexer versions may choose to only accept new advertisements that have a sequence number. +The sequence number field is optional only for backwards compatibility. It should be included as soon as possible by both new and existing index providers to enable indexer functionality that uses it. Once included in an advertisement, all following advertisements must have a `SeqNum`. `SeqNum` is used for the calculation of chain distance between different advertisements. Delaying the includion of `SeqNum` will result in some indexer functionality being unavailable. + +Indexers are not required to validate that `SeqNum` is monotonically increasing, but an incorrect sequence number may result in incorrect indexing information for that chain's provider. An indexer can choose not index advertisement chains that do not have a monotonically increasing `SeqNum`. A `SeqNum` value higher then the maximum of 253-1 (9007199254740992) values will not be recognized and instead seen as 0. It is not expected for any chain to reach this sequence as that would mean it has over nine quintillion advertisements. If this were reached, then `SeqNum` should be reset to 0, which may result in an inaccurate distance calculations between advertisements spanning the reset. If `SeqNum` is present then it is included in the data that the advertisement signature is calculated over. -Indexers are not required to validate that `SeqNum` is monotonically increasing, but an incorrect sequence number may result in incorrect indexing information or may be ignored resulting in some indexing features being unavailable for that chain's provider. If an indexer depends on the sequence number for more than display and reporting purposes, it can choose not index advertisement chains that do not have a monotonically increasing `SeqNum`. - #### ExtendedProvider The `ExtendedProvider` field allows for specification of provider families, in cases where a