Conversation
|
|
||
| #### L4: Always-On PR Checks | ||
|
|
||
| `L4` uses the same callback and synchronization behavior as `L3`, but it does not wait for a label. The `L4` scenario is effectively the same as `L3` Scenario 1 except that the downstream result is represented in upstream PR Checks by default, without requiring explicit labeling. |
There was a problem hiding this comment.
Was there a plan to allow out of tree testing to block PRs? It seems like we'd need a very tight agreement on availability if out of tree tests can halt merging on pytorch/pytorch.
There was a problem hiding this comment.
Hi @groenenboomj, we've discussed this in detail here: pytorch/pytorch#175022 (comment). For L4 level repos, they have the capabilities to gate PyTorch PR, but there will be a very high standard to achieve L4 (and the PyTorch core team remains the right to choose whether a repo is L4 even though it meets all the requirements).
There was a problem hiding this comment.
Okay, I see that was added in the last level breakdown expansion in March. Has there been more discussion on the requirements for gating?
There was a problem hiding this comment.
The main requirement is going to be this one:
Very few accelerators would reach this level. One only becomes L4 if pytorch maintainers are highly invested in it and find it worthwhile to add the signal to every PR.
It's intentionally left somewhat fuzzy, since it's a balance of "how much value will this provide vs how much pain does this inflict" since the idea itself is somewhat subjective. The automatic downgrade will need a firm reliability metric, but even that will likely need fine tuning based on experimentation, so hard numbers here in the short term are likely to be unreliable.
Do you have concerns about the approach @groenenboomj ?
Sure! I will discuss this with @jewelkm89 ASAP and update it to the RFC as well. |
3 participants
As @ZainRizvi mentioned here, I am opening this PR for adding L1-L4 design in our RFC. This may help us better develop L2-L4 implementation.
cc @fffrog @ZainRizvi @albanD