Show R packages installed on runner

[david-mears-2]

Using the new orderly.runner /library/list endpoint, this adds a 'Packages' tab to the 'Runner' page, which just lists the packages orderly.runner tells packit about.

Manual testing

By default, when you run scripts/dev-start, the orderly runner image will look in the new ./fixtures/R directory for its reporting of R libraries. This is different from the path given in production (/library), since it's unlikely this will turn up any packages in development as it likely won't exist.

If you want to look in a different location for R libraries (perhaps one with more packages), you can pass the path to your R library of choice using the --r-lib-path option when running dependencies, e.g.

./scripts/dev-start --super-user --db-build-skip --r-lib-path $HOME/R/x86_64-pc-linux-gnu-library/4.4

or:

PACKIT_HOST_R_LIBRARY_PATH=$HOME/R/x86_64-pc-linux-gnu-library/4.4 ./scripts/run-dependencies

And you may ask yourself, 'does this definitely work in an actual environment, looking at a non-fixture R library?' in which case I would direct you to look at uat where this branch has been deployed: https://uat.montagu.dide.ic.ac.uk/packit/runner/packages

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.26%. Comparing base (7731645) to head (8f17692).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #257   +/-   ##
=======================================
  Coverage   99.26%   99.26%           
=======================================
  Files         201      203    +2     
  Lines        5820     5887   +67     
  Branches      974      985   +11     
=======================================
+ Hits         5777     5844   +67     
  Misses         42       42           
  Partials        1        1           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds visibility of R packages installed on the orderly.runner instance by wiring through the new runner /library/list endpoint and exposing it in the Packit “Runner” UI (plus fixtures/CI support so integration/e2e tests are deterministic).

Changes:

• Backend: add /runner/packages endpoint (API) that proxies to orderly.runner /library/list.
• Frontend: add “Packages” route/tab under Runner and fetch/render installed package metadata.
• Dev/test infra: add a checked-in minimal R package fixture and mount it into runner in dev/CI for deterministic integration/e2e tests.

Reviewed changes

Copilot reviewed 33 out of 35 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
scripts/run-dependencies	Mount optional host R library path into runner container at /library.
scripts/fixtures/R/minimalRPackage/Meta/package.rds	Binary fixture enabling installed.packages() detection in tests/dev.
scripts/fixtures/R/minimalRPackage/DESCRIPTION	Minimal package metadata fixture.
scripts/fixtures/R/README.md	Documents purpose/structure of R library fixtures.
scripts/dev-start	Adds --r-lib-path option and defaults to fixtures for local dev.
app/src/types.ts	Adds RunnerPackage type; renames internal Package → PacketPackage.
app/src/tests/mocks.ts	Adds mockRunnerPackages for MSW/UI tests.
app/src/tests/components/providers/UserProvider.test.tsx	Adds test coverage for isLoading being surfaced from SWR.
app/src/tests/components/contents/runner/PacketRunnerLayout.test.tsx	Adds route coverage and sidebar link visibility tests for Packages + loading state.
app/src/tests/components/contents/runner/PacketRunPackages.test.tsx	Adds unit test for rendering installed packages list.
app/src/msw/handlers/runnerHandlers.ts	Adds MSW handler for /runner/packages.
app/src/app/components/routes/Router.tsx	Registers runner/packages route.
app/src/app/components/providers/types/UserTypes.ts	Extends UserProviderState with isLoading.
app/src/app/components/providers/hooks/useGetUserAuthorities.ts	Returns isLoading from SWR.
app/src/app/components/providers/UserProvider.tsx	Plumbs isLoading into provider context.
app/src/app/components/contents/runner/index.ts	Exports PacketRunPackages.
app/src/app/components/contents/runner/hooks/useGetPackages.ts	New SWR hook for fetching runner packages from API.
app/src/app/components/contents/runner/PacketRunnerLayout.tsx	Adds Packages sidebar item (currently gated by packet.manage) and shows skeleton while authorities load.
app/src/app/components/contents/runner/PacketRunPackages.tsx	New packages page UI rendering the installed packages list.
app/src/app/components/Base/Skeleton.tsx	Adds screen-reader “Loading…” text inside skeleton.
app/e2e/runnerPage.generic.spec.ts	Adds Playwright e2e coverage for Packages tab using fixture package.
api/app/src/test/kotlin/packit/unit/service/RunnerServiceTest.kt	Adds unit test for RunnerService.getPackages().
api/app/src/test/kotlin/packit/unit/controllers/RunnerControllerTest.kt	Adds unit test for /runner/packages controller response.
api/app/src/test/kotlin/packit/integration/services/OrderlyRunnerClientTest.kt	Adds integration test for orderly runner client packages call.
api/app/src/test/kotlin/packit/integration/controllers/RunnerControllerTest.kt	Adds integration test for /runner/packages endpoint behavior.
api/app/src/main/kotlin/packit/service/RunnerService.kt	Adds getPackages() to service interface + implementations.
api/app/src/main/kotlin/packit/service/OrderlyRunnerClient.kt	Adds getPackages() calling /library/list.
api/app/src/main/kotlin/packit/model/dto/RunnerPackageDto.kt	New DTO for runner package metadata.
api/app/src/main/kotlin/packit/controllers/RunnerController.kt	Adds /runner/packages endpoint (inherits packet.run authorization).
api/app/src/main/kotlin/packit/config/RunnerConfig.kt	Clarifies comment about repository meaning git repository.
api/README.md	Documents running deps with PACKIT_HOST_R_LIBRARY_PATH for R package endpoint tests.
README.md	Updates root README to reflect what run-dependencies starts.
.gitignore	Ignores .nvmrc; normalizes demos draft ignore line formatting.
.github/workflows/frontend-test-and-build-and-e2e.yml	Ensures fixture library is mounted during CI e2e run.
.github/workflows/backend-test-and-build.yml	Ensures fixture library is mounted during CI backend integration tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[david-mears-2]

@copilot open a new pull request to apply changes based on this feedback and this feedback and this feedback

[Copilot]

@david-mears-2 I've opened a new pull request, #260, to work on those changes. Once the pull request is ready, I'll request review from you.

[david-mears-2]

Copilot's PR comments were good and valuable feedback, but the PR implementing those suggestions had some differences from how I would have done it, so I instead implemented those myself

[david-mears-2]

Closing this because the overall feature of installing packages would introduce a lot of complexity across the packit/orderly ecosystem, for little perceived gain to scientists.

Edit: reopened because this 'vertical slice' of the feature may have value for non-VIMC users of packit who don't have a yml-based workflow for R packages

[EmmaLRussell]

Looks great! Code looks good.

"fixtures" confused me a little as a top level folder name (though the README explanation is extremely helpful) - do you mean just as in test fixtures? I might be inclined to call that something a little more specific like "runnerDemoLib". Maybe even include it under the top level "demos" folder, if that won't collide with anything else?

This is different from the path given in production (/library), since it's unlikely this will turn up any packages in development as it likely won't exist.

Do you mean because there wont be any custom runner packages installed post-deployment? So is it the case that there are standard packages which are available to reports, but which aren't listed as they're preinstalled on the workers? (I think we did discuss, this, sorry can't recall the details.) I wonder if it would be good to eventually list those as well so that people don't wrongly think they are missing.

[EmmaLRussell]

A while ago I introduced the @demoPackets tag which was intended for tests running against local servers only, with the demo packet set. It seems like this would be a good candidate for this pattern too, although "demo packets" would then become a misleading tag since it would apply to more of the environment than just the packets.

This is why some fixtures have both "demo" and "generic" versions. You'll see that the demo ones have the demo packets tag, and are deliberately excluded from the test:e2e-dev and test:e2e-prod scripts.

So I think at minimum you could move this test into runnerPage.demo.spec and give it the demo packets tag. And in an ideal world we'd rename the tag to something more inclusive (@demoContext or @demoSystem?)

[david-mears-2]

Hm, I could make this demo-only, but that would prevent it from running in CI.

A while ago I introduced the @demoPackets tag which was intended for tests running against local servers only, with the demo packet set.

There is also a second goal of the @demoPackets tag, which, per the README, is testing against deployed dev instances:

We want to have e2e tests which test all packit features, but also to have a suite of tests which we can run against any packit server to test that its deployment has succeeded. In order to support detailed tests of all features, we want to have some tests which expect our standard demo dataset (available on localhost and perhaps on some dev instances)

I could add a demo-only test that tests for certain packages being listed as installed on dev environments, but I'm not sure there are any packages we can guarantee as being installed on any dev instance - until we start listing all installations, at which point we can check for a standard package.

[david-mears-2]

Just realised the point you're making is that the test I wrote will be picked up by any npm run test:e2e-prod which is not what we want.

But my test can't be expected to pass for either of test:e2e-dev or test:e2e-prod. Checking on https://reside-dev.packit-dev.dide.ic.ac.uk/runner/packages, there aren't any packages installed on the shared library volume. I could install some small R package there and then update the test to check for that. Does that sound OK?

Once the test is demo-only, I can remove PACKIT_HOST_R_LIBRARY_PATH from from the front-end github workflow.

[EmmaLRussell]

Hm, I could make this demo-only, but that would prevent it from running in CI.

A while ago I introduced the @demoPackets tag which was intended for tests running against local servers only, with the demo packet set.

There is also a second goal of the @demoPackets tag, which, per the README, is testing against deployed dev instances:

We want to have e2e tests which test all packit features, but also to have a suite of tests which we can run against any packit server to test that its deployment has succeeded. In order to support detailed tests of all features, we want to have some tests which expect our standard demo dataset (available on localhost and perhaps on some dev instances)

I think the "perhaps on some dev instances" part is causing more complications than it deserves to here. as we don't generally do this in practice (AFAIK). The @demoPackets tests are really just intended for local and CI testing.

Just realised the point you're making is that the test I wrote will be picked up by any npm run test:e2e-prod which is not what we want.

Yes, and I don't think we should necessarily be installing packages on the deployed instances for the sake of the e2e tests.

How about just adding another tag, @demoRPackages or whatever, and applying that to your test, and explicitly excluding it from the test:e2e-dev and test:e2e-prod scripts. That way we can treat the two separately if we need to, and we don't need to be misleading about what each test implies.

[david-mears-2]

OK, great. I've done the separate tag idea.

[david-mears-2]

"fixtures" confused me a little as a top level folder name (though the README explanation is extremely helpful) - do you mean just as in test fixtures? I might be inclined to call that something a little more specific like "runnerDemoLib". Maybe even include it under the top level "demos" folder, if that won't collide with anything else?

Yes, I mean as in "test fixtures". Originally I nested it under scripts/ and I can't remember why I thought that was not ideal. I think it would be confusing to mix this kind of demo in with the orderly demos under demos/. I can't really nest it under a tests folder since it's used by both the back end and front end tests. I think I'll put it back into scripts and call the folder runnerDemoLib as you suggest.

This is different from the path given in production (/library), since it's unlikely this will turn up any packages in development as it likely won't exist.

Do you mean because there wont be any custom runner packages installed post-deployment? So is it the case that there are standard packages which are available to reports, but which aren't listed as they're preinstalled on the workers? (I think we did discuss, this, sorry can't recall the details.) I wonder if it would be good to eventually list those as well so that people don't wrongly think they are missing.

Actually I just mean that, on your machine, there won't be any files or folders at the route /library, because your machine is not a dockerized/deployed environment where something is mounted to that address. Thus, there would be nothing listed on the packages page in development, unless we explicitly point your local (dev) orderly.runner server to some address where we know an R library can be found (whether the fixture or your real local library that you use).

Nevertheless in answer to your question, yes, there are standard packages available to reports but which are not listed - orderly.runner server (currently) only reports on the packages installed in the shared library (installed.packages takes an optional argument specifying the library location). It would be quite simple to start reporting all packages + their location of installation. I think technically the standard packages installed on the server (which is where we query installed.packages()) could diverge from those on a runner, which would potentially make reporting all the packages misleading, but that seems unlikely to happen in reality.

Here is a PR on orderly.runner for reporting all the packages regardless of location: mrc-ide/orderly.runner#40 - I would need to change this front-end to follow that.

[EmmaLRussell]

Actually I just mean that, on your machine, there won't be any files or folders at the route /library, because your machine is not a dockerized/deployed environment where something is mounted to that address. Thus, there would be nothing listed on the packages page in development, unless we explicitly point your local (dev) orderly.runner server to some address where we know an R library can be found (whether the fixture or your real local library that you use).

👍

Nevertheless in answer to your question, yes, there are standard packages available to reports but which are not listed - orderly.runner server (currently) only reports on the packages installed in the shared library (installed.packages takes an optional argument specifying the library location). It would be quite simple to start reporting all packages + their location of installation. I think technically the standard packages installed on the server (which is where we query installed.packages()) could diverge from those on a runner, which would potentially make reporting all the packages misleading, but that seems unlikely to happen in reality.

Here is a PR on orderly.runner for reporting all the packages regardless of location: mrc-ide/orderly.runner#40 - I would need to change this front-end to follow that.

I think that would be a good addition (future branch), if it was made clear in the UI what the distinction was between "core packages" and "installed packages" (don't think users will be interested in the location as such).