CI: Communicate changes to CI at least some period of time before failing CI

[danielfdickinson]

Currently, the majority of open PRs in https://github.com/openwrt/packages/pulls are reporting failing CI, due to recent breaking CI changes.

Can these changes please be made 'warn' instead of 'fail' for a reasonable length time, so that package maintainers and other contributors have a chance to adjust to the changes before it renders CI useless.

I have no problem with improving CI, but there is helpful way to do and an problematic way to do it. The current approach I feel is problematic.

@hauke @GeorgeSapkin @dangowrt

[danielfdickinson]

Or at least provide some advance notice! There's nothing like having something that has been working for ages suddenly blow up with no documentation or information on what just happened. Call it shell shock if you like, but this kind of abrupt hard failure is not reasonable, IMO.

[BKPepe]

Look, I'm not entirely sure where to reply to all this, since it's being discussed here: openwrt/packages#29390. If I'm not mistaken, just to be specific, run-time testing does exist in the packages feed. But as you might have noticed, very few packages actually use the test.sh file, so yeah. Some issues were, and still are, to be expected.

For over 6 years (!), there has been an ongoing discussion about enforcing this for all packages, rather than just keeping it opt-in. References: openwrt/packages#13785 and #60

Let's just be glad that someone is taking the time to improve the CI/CD state in their free time without getting paid for it. As has already been mentioned here, for example #109 (comment) it would be great if the necessary things got merged faster, but someone also has to review them first, right?

I really don't want to nitpick, but this doesn't affect the majority of pull requests. It can be fixed with a rebase over the time. And simply put, as I've mentioned elsewhere, GitHub Actions doesn't support a "warning" state https://github.com/orgs/community/discussions/11592. We could surface the warning in the generic tests, but is that actually what we want? Who is going to diligently go through every single warning when there are ten of them? That would be pretty counterproductive.

While you might find this problematic, I'll share my perspective: I actually go through the failing CI/CD logs before merging. And ideally, we need to figure out: is it an issue with the package itself? Or is it an issue with the test? If it's the package, we need to fix it. If it's the test, then the test needs to be fixed.

Bear in mind that there is an extreme number of packages in the packages feed, and users want to keep adding more and more. It's simply unsustainable like this, and expanding the CI/CD, thanks to @GeorgeSapkin, will help us tremendously.

And lastly, regarding the "breaking CI Changes"—if the CI/CD didn't compile the package at all and the other tests failed, then yeah, I'd be screaming too. But it does compile the package, right? Plus, you still have the artifacts available, so you can just download the package, upload it to your router, and install it.

So what actually broke? Yeah, just some test. So please, let's stay calm. We can handle this in a completely different way rather than writing in every single discussion thread. There's no need to lash out at each other or point fingers at other people's mistakes. That’s just not how we do things.

Personally, I'm surprised we aren't all collectively crying over the fact that you have to wait almost 6 hours for a runner to become available just to execute a test. It's a shame. In my opinion, that's a much higher priority.

[danielfdickinson]

@BKPepe

1. I wrote here a well as in the packages repo, because really the discussion should have been happening here (in the repo that is actually at issue, and can change things). if I was cross-posting all over the place, I would have posted on the mailing list, and/or an issue on the main repo.
2. I was under the impression a failed CI would prevent merging unless manually overridden (making more work for those with commit rights).
3. I personally go through warning when they are surfaced enough that I know about them (if I don't even know they exist, that is a little more difficult).
4. I get that this is unpaid work - I am not getting paid either for what I am doing here - but I think that slowing down a little and providing advance notice of changes in places where they are actually likely to get seen (this is the first I heard of the discussion threads on the referenced issue and PR.

It is one of the shortcomings of not having a centralized discussion area for this kind of thing. 'Discussions' are not enabled for any OpenWrt GitHub repo, the forum is not used for this, and neither is the mailing list. There is a communications gap.

5. I wasn't trying to be inflammatory - if I was doing that, I'd have been raging on the forum and other venues. I was upset, and perhaps did not express myself the best, for which I apologize.

[danielfdickinson]

Oh, and I do get frustrated with the long delays with queued CI builds. I just don't know how / where to file an issue or start a discussion to potentially see some changes on that. (E.g. are there any real opportunities for build farms or other ways of doing the CI builds that don't mean relying on the GitHub free open source tier?)

[BKPepe]

Phew, so many points and so many summaries. :)

I more or less agree with you. I'd also like to have Discussions enabled, but I think it was here before and then got disabled—see for example openwrt/packages#8147. And yes, communication is always an issue. I agree with that, but let's just try to improve it overall and not make a big deal out of it, because otherwise we'll discourage even the remaining people who still contribute to OpenWrt. :( And there really aren't that many of them left anymore.

It just felt to me like it was way too aggressive and at the same time unnecessary towards the people who are trying to improve the CI/CD situation.

As for the long delays, honestly, I don't know. I complain about it from time to time and nobody has done anything about it yet. :) Maybe Hauke might know?

[GeorgeSapkin]

I'm slowly going through the list of packages that fail the generics tests to see if the tests need fixing or if those packages need a version check override, and multiple packages are just broken. Missing dependencies, duplicate libraries, segfaults, etc. Perhaps the tests could be more verbose when checks fail.

[danielfdickinson]

@GeorgeSapkin Thank you. Verbosity would help, but obviously the package repo is a bit of a mess and needs the CI. Sorry again, for not handling the shock well.