Support for auxiliary resources and manifest (#67)

[damooo]

Summary

This PR addresses the issue #67, and issue #86 by introducing support for auxiliary resources. And thus makes changes to logical resource organization, operations, etc.

The storage system organizes resources into two distinct layers: primary and auxiliary. The two layers are complementary. The primary layer organizes resources spatially and hierarchically using containment relations, while the auxiliary layer attaches supplementary resources to each node in that hierarchy using auxiliary relations.

The auxiliary resource layer is flat. Each primary resource owns a local set of auxiliary resources. An auxiliary resource is an LWS resource that is used to provide additional information or functionality related to a unique principal primary resource. The lifetime of an auxiliary resource is bound to the lifetime of its principal resource. There MUST be a link from aux resource to it's principal resource with rel value principal, and a link from principal to it's each auxiliary resource with specialized rel value for each auxiliary relation called auxiliary rel. This link also have auxiliary=true param set. auxiliary rels are functional. linkset is such an auxiliary rel, and linkset resources are auxiliary resources.

Most importantly, though server can manage (like linkset), or interpret (like acl) the auxiliary resources, the client too has freedom to create auxiliary resources with any aux-rel subject to functional constraint. Thus auxiliary resources are free axis , available for client as a tool for modelling auxiliarity interoperably across servers. For example, an app can choose to persist rights-info, or data-provenanace, etc using custom auxiliary relations and content media-types. Servers need not to pre-configured to support each relation specifically.

Auxiliary resources can be discovered from linkset by filtering links with parameter auxiliary=true.

Container Manifest resource.

Each container resource has a manifest auxiliary resource. The manifest describes basic resource metadata like type, etc. and provide contained member listing for containers. This manifest resource has the media-type lws+json. Manifest persists as a separate auxiliary resource. Thus also freeing containers to have arbitrary hypermedia representations.

Operations:

The Create operation is changed to support creation of auxiliary members as well. At the time of creation clients provide membership link, linking it to target via either up or principal rel corresponding to containment membership or auxiliary membership. The target resource is called as patron in both cases.

This is the gist. It is a significant change. But I believe it as obvious generalization, instead of forceful feature. Request to review the model first. The grammar, etc. can be updated if the direction find some resonance. Feel free to say no.

Thanks a lot for your time and consideration.

CC: @pchampin , @acoburn

---

Preview | Diff

[kaefer3000]

I see that you copied those words from the current LWS-core, so nothing to complain about in your PR. Maybe a separate discussion is to be had what the storage system is and what is its root? The file system where the resource representations are physically located? On Windows, that is C:\?

[damooo]

I updated the proposal for following changes:

1. Now only containers have container manifest resource. Listing it's contained members.
2. Auxiliary links from a principal to auxiliary have auxiliary=true param set, distinguishing them from other links. This is enough for auxiliary discovery through linkset. No need to have them listed again in a general manifest resource.

This makes only containers having manifest, and tidy ups auxiliary resource linking. This is final proposal. Request to re-review.

@kaefer3000 @pchampin

[pchampin]

Thanks a lot @damooo .
There are part of this PR (additional terminology, clear definition of auxiliary resources) that I like very much and, IMO, could easily reach consensus in the WG.
Unfortunately, there are other parts (the shift of membership metadata into an auxiliary resource) which are much more controversial.

I think it would be valuable to split this PR into several steps, so that at least the (least controversial) first ones could be merged more easily. I'm aware that this requires a lot of work (in addition to what you already did!)...

[damooo]

@pchampin Thanks for review, and sorry for late reply.

Can you specify which parts can be in this PR? I essentially care about having support for auxiliary resources. Container representation issue I am going to close.

Specifically:

Is it acceptable to WG to allow clients to create auxiliary resources with arbitrary aux relation? Is the proposed generalization of POST with ability to choose auxiliary membership Okay?

Or does WG wants to make them purely server managed (Like auto creating all server supported aux resources at the time of creation of principal resource, etc.)? But this case will be problematic, as clients cannot have custom aux relations. And for some auxiliary relations, creation of the aux resource MUST be deliberate decision of client (For eg: WAC inheritence algo makes decisions based on existence, and not just content of acl resources in inheritense chain. Thus acl creation MUST be at the explicit intent of client, and cannot be auto decision of the server)

[gibsonf1]

Thanks a lot @damooo . There are part of this PR (additional terminology, clear definition of auxiliary resources) that I like very much and, IMO, could easily reach consensus in the WG. Unfortunately, there are other parts (the shift of membership metadata into an auxiliary resource) which are much more controversial.

I think it would be valuable to split this PR into several steps, so that at least the (least controversial) first ones could be merged more easily. I'm aware that this requires a lot of work (in addition to what you already did!)...

For implementations where all resources are effectively data (with files accessed one step away via that data) such as TwinPod, it makes no sense at all to have another data resource about the data resource - all the data would be in the one resource. I think its fine to talk about different representations of the resource, but any kind of spec that is implementation specific (such as an implementation that has to have multiple files to handle things) and requiring additional resources that would make no sense in a given implementation, I don't think should be in the spec.

[damooo]

@gibsonf1
Thanks for review.

I noticed that, in many of your comments you conflate the notion of information resource / document with the subject resource in the content. That may be your constrained design to have information resources be rdf resources describing themselves alone. That need not be the case.

For example, an information resource /a/b can have subject resource wikidata:z1234, or multiple others. It can be a nano publication that describe many other subjects. In rdf world relations between the documents, and subjects are many-to-many. Some may constrain them.

In any case, the documents need not be rdf-sources either. They can be html, json, xml, images, videos,... They need to have separate description documents.

And it is the standard practice to have acls, provenanace, changelog, etc. of any info resource to be separate. For, they are not part of it's content.

[gibsonf1]

@gibsonf1 Thanks for review.

I noticed that, in many of your comments you conflate the notion of information resource / document with the subject resource in the content. That may be your constrained design to have information resources be rdf resources describing themselves alone. That need not be the case.

For example, an information resource /a/b can have subject resource wikidata:z1234, or multiple others. It can be a nano publication that describe many other subjects. In rdf world relations between the documents, and subjects are many-to-many. Some may constrain them.

In any case, the documents need not be rdf-sources either. They can be html, json, xml, images, videos,... They need to have separate description documents.

And it is the standard practice to have acls, provenanace, changelog, etc. of any info resource to be separate. For, they are not part of it's content.

I'm not suggesting that it is not possible to have an infinite number of resources about other resources. My point is that for the spec, in the case of implementations that are data first only such as TwinPod, it would make no sense to have another data resource about the data resource when that all is in one resource. Having multiple representations of that same resource is fine. So my point is that to keep the spec implementation details neutral, make this about an auxiliary representation, rather than an auxiliary resource, as then all implementations can comply rather than only specific file based implementations.

[pchampin]

@damooo what characterizes an auxiliary resource, IMO, is that its lifetime is bound by that of another resource; this is obviously the case for Linkset or ACL resource. In those two examples, their content is (at least partially, for Linkesets) client-managed. Whether clients should be allowed to create new auxiliary resources is an open question, AFAICT.

@gibsonf1 I don't think the discussion about auxiliary resources is in any way implementation-specific. The notion of auxiliary resource (not "representation") already exists in Solid. Are you suggesting we should depart from it?

[gibsonf1]

@gibsonf1 I don't think the discussion about auxiliary resources is in any way implementation-specific. The notion of auxiliary resource (not "representation") already exists in Solid. Are you suggesting we should depart from it?

Yes, I am suggesting we depart from any implementation ideas in the spec such that the spec does not assume how an implementation does things, such as the need for multiple resources for a more file based system, when a data first system would not want to do this. So if we use "representation" instead of "resource", then the problem is solved for both file based and data first systems where the implementation can decide on how to create the representation from their internal resources.

[ericPxKanda]

@gibsonf1 I don't think the discussion about auxiliary resources is in any way implementation-specific. The notion of auxiliary resource (not "representation") already exists in Solid. Are you suggesting we should depart from it?

Yes, I am suggesting we depart from any implementation ideas in the spec such that the spec does not assume how an implementation does things, such as the need for multiple resources for a more file based system, when a data first system would not want to do this. So if we use "representation" instead of "resource", then the problem is solved for both file based and data first systems where the implementation can decide on how to create the representation from their internal resources.

In accordance with Dimitri's reasoning at the end of the July 6 WG meeting, I believe that REST principles dictate that the metadata MR1 for a primary resource R1 must at least be modeled as distinct from R1, à la POWDER or LDP. Conflating the two introduces ambiguity that REST specifically addresses with separate resources.

Terms

• Primary resource - user-owned resource in LWS system -- R1
• Metadata resource - metadata for user-owned resource -- MR1
• Extensions - protocol and conventions not spec'd in LWS -- Ex1
• Naive client - LWS client that doesn't know about an extension -- NC1
• Prescient client - LWS client that does know about an extension -- PC1

Cost of conflation

Clients, libraries, and intervening proxies need to associate representations, status codes, and header data with these individual resources. Failure modes when MR1=R1 and the Profile prefers MR1 include:

GET => 403: is the primary resource or the metadata forbidden?

GET => 200: does an intervening proxy serve the body when NC1 GETs R1? In principle, Vary: profile should address this, but reliable Vary support is inconsistent across proxies and CDNs in ways that split into two distinct problems:

• Ignored by default. CloudFront does not consider request headers when building its cache key unless an operator explicitly configures that header per cache behavior — if the origin returns two different responses that differ only by a request header, CloudFront caches one and serves it regardless of the header on subsequent requests.¹ This isn't a bug, it's documented default behavior, and it directly reproduces the NC1 failure mode above with zero misconfiguration required — most self-managed CDN distributions and many commodity edge caches sit at this default.
• Recently added, imperfectly. HAProxy only added Vary handling in 2.4 (2021) via process-vary, which itself defaults to off — a response containing Vary simply isn't cached unless enabled.² When enabled, the initial implementation shipped cache-poisoning bugs: hash collisions and duplicate-header handling that let an unauthenticated client cause HAProxy to serve one client's cached response to another.^{3 4}
• Long-standing but brittle. Varnish has supported Vary for a long time, but matching is exact-string: a Vary'd header that differs only in whitespace or casing between two requests produces two separate cache entries (fragmentation) rather than a match, so operators are advised to normalize headers before varying on them.⁵

PUT => 200: (Naive) proxies don't know which cache entry to invalidate — and this isn't limited to naive proxies. Purge/ban APIs across the ecosystem (Varnish ban, CDN invalidation endpoints) are almost universally keyed on URL, not on URL-plus-Vary-value, so even a fully Vary-aware cache typically can't selectively invalidate "just the MR1 variant" of a shared URL without purging every variant.

Similarly, on the software side, browser Cache Storage API gets it right (modulo ignoreVary: true)⁶ but typically Vary header support in caching is usually opt-in or absent, e.g.:

• axios-cache-interceptor: shipped a real security advisory — the cache key was built from the URL alone, ignoring Vary, so a response varying by Authorization was served across different credentials until Vary handling was made automatic in v1.11.1.⁷
• guzzle-cache-middleware: Vary support was tracked as a missing feature rather than shipped by default.⁸
• curl --etag-save/--etag-compare: has no Vary concept at all — it's a single-slot, per-URL/per-file ETag store used purely for conditional revalidation, not a general cache.⁹
• HTTP/2 server push: browsers do not evaluate Vary at all for pushed responses, matching them against any request regardless of the header values the response varies on.¹⁰)

Web architecture assiduously preserves resource description when defining headers. For instance, when defining profile, RFC6906 §3 says

A profile MUST NOT change the semantics of the resource representation when processed without profile knowledge, so that clients both with and without knowledge of a profiled resource can safely use the same representation.¹¹

(Note: RFC6906 is an Informational, Independent-Submission RFC — not IETF-stream/Standards-Track — worth flagging if its normative weight is contested in-thread.) The underlying Vary matching semantics being circumvented here are themselves normatively defined in RFC9111 §4.1.

Most of the web tooling is keyed on URL. Content-based firewalls (those nosy proxies that poke their noses in your content), analytics, rate-limiting tools, crawlers/archivers, CORS (changes request from simple to something requiring an OPTIONS round-trip).

I mentioned "modeled as" in ¶1 with the intention of offering paths to address:

1. Wouter's proposal: save a round-trip if you are able to go directly to MR1 without having to HEAD R1 to get its location,
2. @gibsonf1's TwinPod implementation which saves MR1 in R1.
   One could address this with protocol or URL convention:

Extended protocol

(Usable on controlled networks and pre-vetted libraries.) Essentially, cheat and use Profile to select between R1 and MR1 but make sure that all errors on the metadata return a 406 to make sure that response isn't associated with the primary resource.

URL convention

Establish site-specific transformations between R1 and MR1 (prefix, suffix, rot13). In TwinPod's case, this is a null transformation.

One could have reasons to introduce heuristics to disambiguate between any set of conflated resources in any REST system, regardless of whether one is another's metadata, but one must address the above issues in the process.

---

Footnotes

1. CloudFront: Cache content based on request headers — "By default, CloudFront doesn't consider headers when caching your objects in edge locations. If your origin returns two objects and they differ only by the values in the request headers, CloudFront caches only one version of the object." ↩

2. Announcing HAProxy 2.4 — process-vary keyword, defaults to off. ↩

3. haproxy/haproxy#988 — Vary handling of cache affected by hash collisions ↩

4. haproxy/haproxy#987 — Vary handling of cache does not properly handle duplicated headers ↩

5. Varnish: Achieving a high hitrate — Vary ↩

6. MDN — Cache: match() method, ignoreVary option ↩

7. axios-cache-interceptor security advisory GHSA-x4m5-4cw8-vc44 ↩

8. Kevinrob/guzzle-cache-middleware#16 — Use and support Vary header ↩

9. curl manpage — --etag-save/--etag-compare; daniel.haxx.se — curl speaks etag ↩

10. Smashing Magazine — Understanding The Vary Header — "no browser actually respects [Vary] ... for resources pushed using HTTP/2 server push." ↩

11. RFC6906 §3 ↩