[WIP][CRCR] Initial implementation of L2

[KarhouTam]

Author

• @KarhouTam
• @can-gaa-hou
• @fffrog

Summary

• This PR implements the L2 levels of the cross-repository CI relay described in [RFC] Cross-Repository CI Relay for PyTorch Out-of-Tree Backends rfcs#90.
• For the previous L1 implementation, please refer to Implement initial L1 cross-repo CI relay #7847.
• Please refer to [RFC] Cross-Repository CI Relay for PyTorch Out-of-Tree Backends rfcs#90 (comment) for the overall implementation.

Higher-level behaviors for L3 and L4 are intentionally left for follow-up work.

Architecture

The relay is split into two AWS Lambda functions:

• webhook lambda function (Updated)

  • receives GitHub webhook PR and push events from the upstream repo
  • validates webhook signatures and authenticates with AWS Secret Manager
  • reads the downstream whitelist from the URL and stores it in Redis
  • for opened/reopened/synchronized/closed actions, forwards repository_dispatch events to downstream repos

• callback lambda function (Added)

  • receives downstream callback payload through a public lambda function URL
  • validates callback payload with OIDC
  • reads the downstream whitelist from the URL and stores it in Redis
  • extracts CI result information from the payload and uploads to PyTorch HUD
  • records queue time and execute time for evolution to L3 repo

Changes

..github/
├── workflows/
│   └── _lambda-do-release-runners.yml     # Updates the Lambda release workflow to include cross-repo-ci-relay packaging/release
│
└── actions/
    └── cross-repo-ci-relay-callback/
        └── action.yml                     # Composite action used by downstream workflows to report status back to the relay/result endpoint

aws/lambda/cross_repo_ci_relay/
├── tests/                                 # Unit tests for allowlist/config/webhook/result/redis behavior
├── README.md                              # Project overview, local development, callback flow, and result-side validation steps
├── Makefile                               # Top-level local developer entrypoint for test / deploy / clean
├── local_server.py                        # FastAPI wrapper for local end-to-end testing of both webhook and result endpoints
├── requirements.txt                       # Python dependencies required by the relay Lambdas
│
├── utils/
│   ├── allowlist.py                       # Loads, parses, and queries the downstream allowlist by rollout level
│   ├── config.py                          # Shared runtime config loading and cached get_config() helper
│   ├── gh_helper.py                       # GitHub App, repository_dispatch, and GitHub file access helpers
│   ├── hud.py                             # HUD write helpers for downstream result reporting
│   ├── jwt_helper.py                      # Helpers for minting/verifying relay callback tokens
│   ├── redis_helper.py                    # Redis helpers for allowlist cache, OOT state, and timing data
│   └── misc.py                            # Shared TypedDict definitions and HTTPException
│
├── webhook/
│   ├── Makefile                           # Build/package/deploy commands for the webhook Lambda
│   ├── lambda_function.py                 # Webhook Lambda entrypoint: verifies GitHub webhook requests and routes events
│   └── event_handler.py                   # Handles PR/push events, resolves allowlist targets, and dispatches to downstream repos
│
└── callback/
    ├── Makefile                           # Build/package/deploy commands for the result Lambda
    ├── lambda_function.py                 # Result Lambda entrypoint: verifies callback token and GitHub OIDC token
    └── callback_handler.py                # Validates callback payloads, checks L2+ eligibility, stores state, and writes to HUD

Usage

See README.md for more details.

Verification

We performed the following scenario verification on our AWS Lambda instance:

• Test with Upstream PR create/reopen/synchronize and push events triggering webhook, then redispatching to the Downstream CI (different organization) workflow.
• Test with Downstream workflow send callback payload through the added action to the result lambda, then extract CI result information and send to PyTorch HUD.

Terraform configuration

• [WIP] Add L2 CRCR deployment configuration ci-infra#446

Unit Tests

• Unit Tests (Mock)

Security

• Callback payload carries full upstream webhook data back to HUD — action.yml builds the callback body by mutating github.event.client_payload (which contains the entire original webhook payload: PR metadata, commits, author info) and adding status/conclusion/workflow_name/workflow_url on top. This full blob is forwarded verbatim by hud.py to HUD with no relay-side filtering. HUD receives both relay-trusted verified_repo and an unvalidated body — if HUD trusts self-reported fields inside the body over verified_repo, a manipulated dispatch payload could tamper with HUD records.

• Lambda callback URL is public and hardcoded — The endpoint is hardcoded in `action.yml and exposed in a public action, making it trivially discoverable. OIDC verification blocks unauthorized HUD writes, but the endpoint has no rate limiting; request flooding can cause Lambda concurrency exhaustion or Redis connection saturation.

• Only OIDC is used for verification — The callback lambda relies solely on GitHub OIDC token verification for authentication, without additional application-level secrets or signatures. If an attacker compromises a downstream repo's GitHub Actions permissions, they could forge authenticated requests to the callback endpoint. Besides, OIDC has its own limitations (e.g., token expiration, potential misconfigurations) that could lead to unauthorized access if not carefully managed.

HUD Interaction

• Design Principle: Transparent Relay & Decoupling
  The Relay Server acts as a lightweight data passthrough layer. It does not define or parse specific CI data formats; instead, it offloads data interpretation and validation to the HUD. This ensures complete decoupling between the relay infrastructure and business-specific data.

• Security & Risk Mitigation
  The relay uses OIDC authentication to guarantee the authenticity of the data source (Verified Repo). Its core responsibility is to ensure the data originates from the claimed repository, while security filtering and content compliance are enforced at the HUD level.

[meta-cla]

Hi @KarhouTam!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[vercel]

@KarhouTam is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!