Github CI is not useful for external contributors

[PhilLab]

As I am not an expert with Github CI (I am more familiar with Gitlab), parts of these might be Github's fault, but nonetheless - the CI is not doing any favor to external contributors. IMHO, it should support them better, to motivate more outside contributions

1. Email flood

   • After a lot of quiet on the PR, I am suddenly bombarded with ~13 emails about failing jobs. Especially if I have multiple PRs open, it is quite hard to keep track of what is affected
   • Any chance of having one email per PR instead, reporting the CI failure once, per PR?
   • 

2. It is unclear when the CI runs and where

   • We should explain in the CONTRIBUTING.md that most jobs (but not all, I think?) run only when a maintainer starts them
   • There is also a Drone CI running somewhere?
   • It is unclear where I see my newly added tests actually running
   • I tried to get to the bottom of this here, but didn't get far: Minor improvements for automated testing #15424 (comment)

3. Some jobs simply fail by design, when it is on a fork

   • Example: https://github.com/nextcloud/android/actions/runs/20276837329/job/59030849217?pr=16145
   • this leads to an unnecessary and wrongful impression that there is something wrong. Rather, any CI failure should directly map to an action item - but in these cases, there is nothing you can do.

4. It is unclear what checks should be run locally, to ensure a smooth CI experience

   • E.g. gradlew lint is failing the CI, but it is not explained anywhere that you should run it. The pre-commit hook also doesn't execute it (probably because it is too expensive to run every time)
   • Example: https://github.com/nextcloud/android/actions/runs/20276837368/job/59030849289?pr=16145

5. Details of a CI failure are hard to understand

   • For example no full lint report downloadable on https://github.com/nextcloud/android/actions/runs/20276837368/job/59030849289?pr=16145
   • For example no access to details for screenshot test failures: https://github.com/nextcloud/android/actions/runs/20276837409/job/59030849251?pr=16145

6. Said linter job just always fails, unrelated to the changes in the PR

   • Example: https://github.com/nextcloud/android/actions/runs/20276837368/job/59030849289?pr=16145 fails due to issues in files the PR didn't even touch. So it is unclear, if and what I am supposed to do.

7. The detectWrongSettings job is destined to fail

   • Because the maintainer triggering of the CI job is happening weeks later than the original PR, at time of triggering the PR is almost always out-of-date, so it doesn't use one of the latest 10 android-library commits.
   • This issue is fixable by the contributor, by rebasing, but it does cause additional friction as well (yet another failure, yet another alert, ...)

cc @AndyScherzinger

[AndyScherzinger]

CC @tobiasKaminsky @alperozturk96

[AndyScherzinger]

Added the devs and teamlead.

As a quick fix for you, since you already contributed, sent you an invite for the Nextcloud org on Github, so you can don't need to work on a fork.

I agree that docs can be improve. Several CI checks terminate early when executing on a fork for security reasons to protect the original repo.

[PhilLab]

@tobiasKaminsky @alperozturk96 next to the initially stated problems, the test pipeline is always red for many many months now. So the CI is not useful for both external and internal contributors:

1. Screenshot tests always fail
   • Also see screenshottest are error prone #17143
2. Drone tests always fail. They are only successful when they are actually skipped (due to no changes)
   • Note that the drone tests were green in the past but only because they didn't actually run anything. Once this was fixed in Many test failures are not catched in the CI #16350, they finally were able to detect bugs again
   • But ever since then, because finally the tests run again, a timeout is reached and every test run runs into it

Is there any plan on addressing those now useless tests?

cc @AndyScherzinger

[hannesa2]

I agree to @PhilLab that always failing jobs makes no sense at all and just makes every pull request faulty.
If nobody takes care about it, then it makes even no sense at all ! --> just pointless overhead

What's about to get rid of them complete and (maybe at the same time) make a new pull request to re-introduce that again.
There the CI can be fixed separately.

[AndyScherzinger]

As mentioned in #16212 (comment)

@tobiasKaminsky @alperozturk96 would need to comment / look into it. I personally don't have the time at the moment.

[PhilLab]

@tobiasKaminsky can you take a look at this? This repository effectively has no kind of instrumented test anymore. Neither screenshot tests nor drone tests do what they are supposed to do.

1. This will amass more and more regressions in the app.
2. Contributors will draw the conclusion that they shall never again waste time on writing automated tests in this repository 😬

[tobiasKaminsky]

We are in transition to github action runners (#17038), that is why I did not spent that much love into Drone recently.

I will make this working again within this week.

Screenshot tests are a bit harder.
For me it seems that shot (https://github.com/pedrovgs/Shot) is a bit abandonded.
I tested it at some point in past any only got it working again with downgrading junit:

#16578