When listening to server-sent events, regularly reload token from tokenfile

[onnozweers]

Ada has two commands that listen to server-sent events. These are operations that may last a few days. For instance, Ada listens for files that are restored from tape, and when they become online, compute jobs may be started.

Normally, dCache checks tokens only at the start of an operation. However, Ada may need to follow server-sent-events with more than one operation. In the help text for --events:

	      When --recursive is added, all subdirectories will be
	      followed as well, including any new subdirectories
	      that will be created.

For adding new subdirectories to the event channel, Ada needs to make separate API calls. At that moment, a valid token is needed.

OIDC tokens usually live for 1 hour. Ada should re-read tokens from the specified tokenfile. Then, a user can repeatedly run oidc-token and save the token in the tokenfile, to provide Ada with the necessary authorization to continue.

[onnozweers]

We should not forget curl authorization header files. They may be cleaned up after 12 hours, see #57. So, we not only will need to reload tokenfiles, but also may need to refresh authorization header files before they are too old.

[onnozweers]

I tested the change like this:

Got an x509 proxy that was mapped onto dteam

Start get-macaroon in a loop:

while true ; do get-macaroon --url https://dcachetest.grid.surfsara.nl:2884/groups/dteam --permissions LIST,DOWNLOAD,UPLOAD,MANAGE,READ_METADATA,UPDATE_METADATA --proxy --output rclone tokenfile_test --duration PT3M ; date ; echo Sleeping... ; sleep 1m ; done

This will give me a new macaroon every minute, valid for 3 minutes. So, the remaining lifetime should always be at least 2 minutes. Remember that Ada now refuses tokens that are valid for less than 60 seconds! See dfa1a05

Subscribing to events in the same dir:

ada/ada --tokenfile tokenfile_test.conf --events test-channel /groups/dteam --recursive

Create a new dir every 25 seconds:

for i in {1..100} ; do ada/ada --tokenfile tokenfile_test.conf --mkdir /groups/dteam/onno-tests/dir$i ; sleep 25s ; done

And with the ada --events command, I can see that the new dirs are added to the channel. So, this works nicely.

Example output:

...
inotify  /groups/dteam/onno-tests/dir49  IN_CREATE,IN_ISDIR
SYSTEM  {"type":"NEW_SUBSCRIPTION","subscription":"https://dcachetest.grid.surfsara.nl:20443/api/v1/events/channels/xQZoskUVW_3Fu60UrY9ZYg/subscriptions/inotify/AACKg9krPMROhZlTIE0wN0u9"}
inotify  /groups/dteam/onno-tests/dir49  IN_OPEN,IN_ISDIR
inotify  /groups/dteam/onno-tests/dir49/  IN_OPEN,IN_ISDIR
inotify  /groups/dteam/onno-tests/dir49  IN_CLOSE_NOWRITE,IN_ISDIR
inotify  /groups/dteam/onno-tests/dir49/  IN_CLOSE_NOWRITE,IN_ISDIR
inotify  /groups/dteam/onno-tests/dir50  IN_CREATE,IN_ISDIR
SYSTEM  {"type":"NEW_SUBSCRIPTION","subscription":"https://dcachetest.grid.surfsara.nl:20443/api/v1/events/channels/xQZoskUVW_3Fu60UrY9ZYg/subscriptions/inotify/AAB-yYNXNN9JNZiKewIUmXvV"}
inotify  /groups/dteam/onno-tests/dir50/  IN_OPEN,IN_ISDIR
inotify  /groups/dteam/onno-tests/dir50  IN_OPEN,IN_ISDIR
inotify  /groups/dteam/onno-tests/dir50/  IN_CLOSE_NOWRITE,IN_ISDIR
inotify  /groups/dteam/onno-tests/dir50  IN_CLOSE_NOWRITE,IN_ISDIR
...

[hailihu]

It appears we cannot make OIDC tokens that live longer than 1 hour. If this is a bug in oidc-agent, then it should be reported and fixed there. If this is expected behaviour, ADA should be able to deal with this by reloading tokens.

[onnozweers]

These commands show that the oidc-agent always gives an access token of 1 hour. Not longer, not shorter.

% export BEARER_TOKEN=$(oidc-token --time=600 dteam)
% export BEARER_TOKEN=$(oidc-token --time=7200 dteam)

% ada/ada --viewtoken