[bug] github checks overwrite each other

[OmarTawfik]

Thanks for working on Bencher!

For monorepos that contain multiple projects, any PR that touches central benchmarks/test infra can trigger bencher reports for all projects. However, this is not distinguished in the results reported by bencher on the PR itself. See this recent PR for example, that touches benchmarks in four different bencher projects:

Unfortunately all four runs create the same GitHub check, overwriting each other. They should have resulted in four different checks, each with Bencher Report - {PROJECT_NAME} instead of all overwriting the same Bencher Report.

[epompeii]

Hey @OmarTawfik thank you for using Bencher, and I'm sorry things aren't working as expected. I think what you need should be supported by the --ci-id option as of v0.6.7 added for the changes here #894 (comment):

The ID is also appended to the GitHub Check name (ie Bencher Report (<ID>)), so each invocation gets a stable name that can be used as a separate required status check.

Is this option not working for you?

[OmarTawfik]

Thank you for the link. From the docs:

Setting a custom ID is useful when Bencher is being run multiple times in the same CI workflow for the same Project, Branch, Testbed, and Adapter combination.

Not sure this is the case here? In the PR I posted above, each of the four runs are indeed running with a different --project value (example), and one of them is even running with a different --adapter value (example).

Do I still to supply --ci-id in this case? and if so, would I need to include the same combination (Project, Branch, Testbed, and Adapter) in it? or would the project name be enough?

[epompeii]

Do I still to supply --ci-id in this case?

Yes, you would need to in order to get Bencher Report (<ID>) as your output instead of just Bencher Report. That's a bit of a hack though, and building your own unique ID is going to be a bit of a pain (and make for an ugly Check name).

Taking a step back though, why are you using 4 different projects in the same repo/CI run?