RoiWrapper load_shapes includes externalInfo

[will-moore]

Since it is useful to store data in the externalInfo of shapes, we want to be able to access that via the /rois endpoint.
Currently we have the option to load_shapes but this doesn't include externalInfo (added in this PR).

To test:

• Draw or find a ROI ID and a Shape ID
• These can be listed for an Image via /api/v0/m/rois/?image=IMAGE_ID
• Set an externalInfo on the Shape, e.g. For a Rectangle:

$ omero obj ext-info-set Rectangle:331301 3 com.glencoesoftware.ngff:multiscales https://livingobjects.ebi.ac.uk/idr/zarr/v0.4/idr0079A/idr0079_images.zarr/0/labels/0/

• Refresh /api/v0/m/rois/?image=IMAGE_ID and you should now see this info on the Shape.

[sbesson]

Performed some initial functional testing using a Mask with an associated ExternalInfo. Without this change, the api/v0/m/rois/ endpoint returns a shallow ExternalInfo object with only the type and the identifier. With this change included, the response includes a populated object.

This retrieval of ExternalInfo objects associated with a shape is consistent with the change proposed and integrated in #453. The performance implications associated with this query had been discussed in the previous PR and similar conclusions apply here. Functionally, it might be worth validating these still hold e.g. using an image with 100s of linked ROI/Shape objects.

How easy is it to add integration tests either at the gateway or at the OMERO.web API level to cover the new functionality?

[will-moore]

Added a test at ome/openmicroscopy#6455

[will-moore]

All tests passing today: https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/138/testReport/OmeroWeb.test.integration.test_api_rois/

Going to exclude this PR to check that the test fails...

[will-moore]

As expected, with this PR excluded the new ExternalInfo checks in test_rois.py are failing: https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/139/testReport/junit/OmeroWeb.test.integration.test_api_rois/TestContainers/test_image_rois/

Removing exclude now...

[sbesson]

The integration tests have been properly working over the last few weeks with this included - see https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/157/testReport/junit/OmeroWeb.test.integration.test_api_rois/

The main remaining item is to run some performance testing as mentioned in #495 (review).
On our side, we can certainly check typical examples of images with a few tens of linked label images i.e. Mask objects annotated with an ExternalInfo object. Do you have a good example of image with 1-10K linked ROI/Shape objects? And can we compute the exact HQL query that should being run with and without this PR?

[will-moore]

Created over 1000 ROIs (each with 10 Rectangles, with externalInfo set on every one) on image in merge-ci server:

Max limit of ROIs can be loaded with:
https://merge-ci.openmicroscopy.org/web/api/v0/m/rois/?image=1277&limit=500

Ran a bunch of fetch(url) calls to log response times.
Looking at server response time (green) rather than download time (blue):

E.g (secs)
3.9, 3.9, 3.6, 3.7, 4.1, 3.7, 3.7,

Let's exclude this and test tomorrow without this PR...

[will-moore]

full query generated is now:

select obj from Roi obj join fetch obj.details.owner as owner left outer join fetch obj.details.externalInfo
join fetch obj.details.creationEvent left outer join fetch obj.shapes as shapes
            left outer join fetch shapes.details.externalInfo

[will-moore]

Timings for the server response on merge-ci with this PR excluded today (secs):
2.7, 3.0, 2.8, 2.7, 3.3, 2.8, 2.9, 2.6

About 1 second faster than yesterday.

As expected, tests fail with this PR excluded https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/158/testReport/junit/OmeroWeb.test.integration.test_api_rois/TestContainers/test_image_rois/
Removing the exclude flag again now...

[will-moore]

I created similar shapes on a different image, this time with NO externalInfo.

Having re-deployed omero-web with this PR, we can test whether this PR has any performance hit in the absence of externalInfo.

With this query... https://merge-ci.openmicroscopy.org/web/api/v0/m/rois/?image=1276&limit=500
Times in secs:
2.6, 2.5, 2.7, 2.6, 2.4, 2.4, 2.6

So, performing the query itself has NO performance hit in the absence of externalInfo.

[sbesson]

Also retested using a sample image with 1213 associated ROI/Shape objects. Running the two following queries 10 consecutive time

>>> query1="select obj from Roi obj join fetch obj.details.owner as owner left outer join fetch obj.details.externalInfo join fetch obj.details.creationEvent left outer join fetch obj.shapes as shapes left outer join fetch shapes.details.externalInfo where obj.image.id=8134"
>>> query2="select obj from Roi obj join fetch obj.details.owner as owner left outer join fetch obj.details.externalInfo join fetch obj.details.creationEvent left outer join fetch obj.shapes as shapes where obj.image.id=8134"

yields the following execution times

• query1: 0.390s +/- 0.042s
• query2: 0.388s +/- 0.041s

I also don't see any performance degradation associated to the additional fetch. Any additional functional testing we would like to perform before integrating this?