Allow URLs as process namespace

[m-mohr]

To be able to load external processes (UDPs) in process graphs easily, it would be valuable to have the namespace being a URL for read-only access. The process ID stays as it is.

[soxofaan]

FYI: the VITO backend already supports this (because this turned out a valuable solution in a lot of practical use cases).
It is also already documented in python client: https://open-eo.github.io/openeo-python-client/cookbook/udp_sharing.html#using-a-public-udp-through-url-based-namespace

[m-mohr]

I was also wondering whether the namespace should not include the process ID, but then I realized that the URL may happen to have (or not have) a .json file extension and that makes it ambiguous. So specifying the full url in the namespace (for URLs) makes sense.

[soxofaan]

FYI: in VITO implementation we currently support it as follows

if user provides process_id and namespace="https://.... we try the following URLs and take the first hit:

• {namespace}{process_id}
• {namespace}{process_id}.json
• {namespace}

e.g. with process_id="ndvi" and namespace="https://example.com/udp/" these two URLs are first attempted:

• https://example.com/udp/ndvi
• https://example.com/udp/ndvi.json

The last option is to allow specifying the UDP URL fully through the namespace (ignoring process_id)

[m-mohr]

Yeah, I think for namespace URLs we should simplify that the URL is the exact URL to the file and you don't need to do any try&error. The only thing that you need to check is the response. If it's a single process, then just use it. If it is a process list response, take the process from the list with the given process ID.

If not a URL, resolve as usual.

[jdries]

This is becoming a key issue for federated setups and for ESA APEx.

In APEx, the use case is to have the UDP's stored as json in github rather than being managed in a specific backend.
This should then make UDP invocation less backend dependant, and simplify the management.

@soxofaan @m-mohr We'll probably want to discuss what is still to be done here.

[m-mohr]

Community meeting: No objections, next step is creating a PR for review.

[soxofaan]

Yeah, I think for namespace URLs we should simplify that the URL is the exact URL to the file and you don't need to do any try&error.

yes that's fine for me

The only thing that you need to check is the response. If it's a single process, then just use it. If it is a process list response, take the process from the list with the given process ID.

I'm not sure that process listing support is even necessary.
I would work from the assumption that users prefer a 1-on-1 mapping between UDPs and files. For example, the python client has some helpers to store/load a UDP from/to a file. If you want to manage multiple UDPs in the same file, you have to do a lot more cumbersome housekeeping on your own.

So I would keep the API extension very basic for now:

• if namespace starts with https?://: assume this is a URL of JSON file containing a UDP representation and load the UDP directly form this URI
• else: proceed as usual

Also note that even with such a simple API spec, there is still room at the level of clients to provide a more rich UI (e.g. user passes a URL as process id -> client automatically converts that to actual process id + URL namespace)