[DO NOT MERGE] Astro reorg scripts

.gitignore

@@ -301,6 +301,7 @@ assets/jsconfig.json
 # local build files
 .backup_path
 local/bin/py/build/*
+!local/bin/py/build/configurations/
 !local/bin/py/build/configurations/pull_config_preview.yaml
 !local/bin/py/build/configurations/pull_config.yaml
 !local/bin/py/build/configurations/integration_merge.yaml
@@ -338,3 +339,4 @@ playwright-report/
 .claude/*local*
 .superpowers/
 data/site_region_usage.json
+

REPO_REORG.md

@@ -0,0 +1,29 @@
+# `documentation` repository reorg
+
+# Reorg status
+
+Not yet merged.
+
+# Project overview
+
+We're gradually migrating the Datadog docs site from Hugo to Astro. To support this, we must reorganize the `documentation` repo into a monorepo that supports two sites (`hugo` and `astro`), instead of having the entire repo serve as our Hugo site.
+
+**This reorg causes conflicts in almost every open PR in the `documentation` repo.**
+
+## What does the reorg change?
+
+The most impactful change is that any Hugo-specific site files move to a `hugo` folder instead of being located at the top level of the repo. This change impacts thousands of files, and causes the majority of conflicts.
+
+For a full list of repo files and folders and their updated location, see [the configuration file for the reorg script](https://github.com/DataDog/documentation/blob/jen.gilbert/astro-reorg-scripts/astro_reorg/config.yaml).
+
+# Handling PR conflicts
+
+Once the reorg is merged to master, all open PRs are processed. Conflicts are fixed automatically where possible. When an automatic fix is not possible, we defer to the author's best judgment in resolving the conflicts, but offer escalation options in the PR comments in case they need help.
+
+## PR cases
+
+* **Skipped because it has no conflicts:**: We do not try to autofix the PR because it doesn't have any conflicts. We add a label as a guard against further processing.
+* **Skipped as a WIP:**: We do not try to autofix the PR because it has the `WORK IN PROGRESS` label. We leave a comment describing how to queue the PR for autofix processing if desired.
+* **Autofixed PR:** We fix the conflicts automatically, close the PR, and link a new PR for the author's review. 
+* **Stale PR:** We do not act on the PR, because it appears to be stale. We leave a comment describing how to queue the PR for autofix processing if desired.
+* **Manual fix required:** We cannot resolve the conflicts automatically. We leave a comment describing why the conflicts are occurring, and how to ask for help if needed.

astro_reorg/CLAUDE.md

@@ -0,0 +1,23 @@
+# Docs repository reorg context
+
+This repo is currently a Hugo site. We instead want it to contain a `hugo` and `astro` site side by side, with no overlap in their envs, `package.json` files, etc.
+
+`astro_reorg/config.yaml` describes the relocation target for every file and folder at the top level of the repo.
+
+`astro_reorg/execute_reorg.py` implements the file and folder path changes, and updates any dependencies on those paths, such as GitHub actions, CODEOWNERS, and Husky workflows.
+
+`astro_reorg/helpers.py` contains utilities (path manipulation, git/shell helpers, YAML config loading) used by `validate_reorg.py`. The other scripts predate it and each define their own small helpers inline.
+
+`astro_reorg/local_rollback.py` functions as an "undo" action for the reorg: removes `hugo/` and restores `.gitignore`, `.github/`, and `.husky/` from git.
+
+`astro_reorg/validate_reorg.py` verifies the functionality of affected entities where possible.
+
+`astro_reorg/resolve_pr_conflicts.py` finds open PRs with merge conflicts caused by the reorg, and either auto-fixes them (by replaying commits at post-reorg paths) or labels them for manual review. It defaults to a dry run against a mock base branch, so it can't touch real PRs by accident.
+
+`astro_reorg/create_test_prs.py` creates test PRs for exercising `resolve_pr_conflicts.py`, including the conflicts it can't auto-fix. Each spec in `TEST_PRS` becomes a PR that reproduces one conflict scenario.
+
+Both scripts share the `test:` section of `config.yaml`, which names the branches a test run uses: `mock_reorged_master_branch` (the stand-in for post-reorg `master` that test PRs target and the resolver treats as the base) and `branch_from` (the frozen snapshot the test PRs are cut from).
+
+For flags and the mechanics of each scenario, read the script's module docstring and the comments on `TEST_PRS` — that's the source of truth, so this file doesn't have to track them.
+
+You can ignore the `astro` folder, it's a remnant from another branch where an Astro site is being developed. It is completely out of scope.

astro_reorg/close_stale_prs.py

@@ -0,0 +1,170 @@
+#!/usr/bin/env python3
+"""
+Close open PRs that carry the `autolabeled-stale` label.
+Posts a comment before closing. Defaults to a dry run where it just reports
+what would be done.
+
+Use the GitHub label query to review which PRs are queued for closure:
+https://github.com/DataDog/documentation/pulls?q=is%3Aopen+label%3Aautolabeled-stale
+
+Usage:
+    python3 astro_reorg/close_stale_prs.py [--no-dry-run] [--pr NUMBER ...] [--limit N]
+
+Flags:
+    --no-dry-run          Actually close PRs instead of just reporting what would be done.
+    --pr NUMBER ...       Only process the given PR number(s).
+    --limit N             Stop after closing N PRs. PRs that are skipped don't count toward
+                          the limit. Use it to roll out gradually: start with --limit 1, review
+                          the results, then raise it as you gain confidence.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+from datetime import datetime, timedelta, timezone
+
+# ---------------------------------------------------------------------------
+# Configuration
+# ---------------------------------------------------------------------------
+
+REPO = "DataDog/documentation"
+
+LABEL_STALE = "autolabeled-stale"
+
+CLOSE_COMMENT = (
+    "Closing this PR because it has had no activity for six months. "
+    "If this was done in error, feel free to re-open the PR."
+)
+
+# ---------------------------------------------------------------------------
+# Shell helpers
+# ---------------------------------------------------------------------------
+
+def run(cmd: list[str]) -> subprocess.CompletedProcess:
+    return subprocess.run(cmd, capture_output=True, text=True)
+
+
+def gh_json(*args: str) -> object:
+    result = run(["gh", *args])
+    if result.returncode != 0:
+        raise RuntimeError(f"gh {' '.join(args[:3])}: {result.stderr.strip()}")
+    return json.loads(result.stdout) if result.stdout.strip() else []
+
+
+def gh_run(*args: str) -> str:
+    result = run(["gh", *args])
+    if result.returncode != 0:
+        raise RuntimeError(f"gh {' '.join(args[:3])}: {result.stderr.strip()}")
+    return result.stdout
+
+
+# ---------------------------------------------------------------------------
+# GitHub helpers
+# ---------------------------------------------------------------------------
+
+def post_comment(pr_number: int, body: str, dry_run: bool) -> None:
+    if dry_run:
+        print(f"  [dry-run] would comment on PR #{pr_number}:\n    {body[:120]}")
+        return
+    gh_run("pr", "comment", str(pr_number), "--repo", REPO, "--body", body)
+    print(f"  Posted comment on PR #{pr_number}")
+
+
+def close_pr(pr_number: int, dry_run: bool) -> None:
+    if dry_run:
+        print(f"  [dry-run] would close PR #{pr_number}")
+        return
+    gh_run("pr", "close", str(pr_number), "--repo", REPO)
+    print(f"  Closed PR #{pr_number}")
+
+
+# ---------------------------------------------------------------------------
+# Per-PR processing
+# ---------------------------------------------------------------------------
+
+def process_pr(pr: dict, dry_run: bool) -> bool:
+    """Comment and close a labeled PR. Return True if we acted, False if we skipped."""
+    pr_number = pr["number"]
+    title = pr["title"]
+
+    print(f"\nPR #{pr_number}: {title}")
+    print(f"  Last updated: {pr.get('updatedAt', 'unknown')}")
+
+    post_comment(pr_number, CLOSE_COMMENT, dry_run)
+    close_pr(pr_number, dry_run)
+    return True
+
+
+# ---------------------------------------------------------------------------
+# Entry point
+# ---------------------------------------------------------------------------
+
+def get_labeled_prs(only: list[int] | None = None, fetch_limit: int = 1000) -> list[dict]:
+    """Return open PRs carrying the stale label, optionally filtered to specific numbers."""
+    fields = "number,title,updatedAt"
+    if only:
+        prs = []
+        for n in only:
+            pr = gh_json("pr", "view", str(n), "--repo", REPO, "--json", fields)
+            prs.append(pr)
+        return prs  # type: ignore[return-value]
+    return gh_json(  # type: ignore[return-value]
+        "pr", "list", "--repo", REPO, "--state", "open",
+        "--label", LABEL_STALE,
+        "--json", fields, "--limit", str(fetch_limit),
+    )
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description=f"Close open PRs labeled {LABEL_STALE!r}.",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=__doc__,
+    )
+    parser.add_argument(
+        "--dry-run", action=argparse.BooleanOptionalAction, default=True,
+        help="Report what would be done without making any changes (default: on). "
+             "Use --no-dry-run to apply changes.",
+    )
+    parser.add_argument(
+        "--pr", type=int, action="append", dest="prs", metavar="NUMBER",
+        help="Only process this PR number (may be repeated).",
+    )
+    parser.add_argument(
+        "--limit", type=int, default=None, metavar="N",
+        help="Stop after closing N PRs. Skipped PRs don't count toward the limit. "
+             "Use it to roll out gradually: start at 1, review, then raise it.",
+    )
+    args = parser.parse_args()
+
+    if args.limit is not None and args.limit < 1:
+        parser.error("--limit must be a positive integer.")
+
+    if args.dry_run:
+        print("DRY-RUN mode — no PRs will be modified.\n")
+
+    prs = get_labeled_prs(args.prs)
+    print(f"Found {len(prs)} labeled PR(s) to close.")
+    if args.limit is not None:
+        print(f"Limit: will stop after closing {args.limit} PR(s).")
+
+    acted = 0
+    for pr in prs:
+        if args.limit is not None and acted >= args.limit:
+            print(f"\nReached --limit of {args.limit} closed PR(s) — stopping.")
+            break
+        try:
+            if process_pr(pr, args.dry_run):
+                acted += 1
+        except Exception as exc:
+            print(f"\nERROR processing PR #{pr.get('number', '?')}: {exc}",
+                  file=sys.stderr)
+            print("  Skipping to the next PR.", file=sys.stderr)
+
+    print(f"\nClosed {acted} PR(s). Done.")
+
+
+if __name__ == "__main__":
+    main()

astro_reorg/config.yaml

@@ -0,0 +1,139 @@
+# Files and folders that stay at the top level (not moved into hugo/).
+# Everything else moves into hugo/.
+
+top_level:
+  # Sites
+  - astro
+
+  # Repo scripts
+  - astro_reorg
+  - reorg_issues.md
+
+  # Created by reorg.py as the move target
+  - hugo
+
+  # Git / CI / repo-wide tooling
+  - .git
+  - .github
+  - .gitlab-ci.yml
+  - .gitignore
+  - .gitattributes
+  - .husky
+  - .claude
+
+  # Editor / IDE config
+  - .editorconfig
+  - .vscode
+
+  # Repo docs
+  - README.md
+  - LICENSE
+  - CONTRIBUTING.md
+  - CLAUDE.md
+  - REPO_REORG.md
+
+  # Linting / formatting (shared)
+  - .eslintrc
+  - .eslintignore
+  - .vale.ini
+  - .rubocop.yml
+  - .prettierignore
+  - prettier.config.js
+  - static-analysis.datadog.yml
+
+  # Datadog CI / synthetics
+  - datadog-ci.preview.json
+  - general.preview.synthetics.json
+  - repository.datadog.yml
+
+  # Docker
+  - docker-compose-docs.yml
+
+moves_to_hugo:
+  # Hugo core
+  - archetypes
+  - assets
+  - config
+  - content
+  - data
+  - i18n
+  - layouts
+  - static
+  - resources
+  - public
+
+  # Hugo build tooling
+  - Makefile
+  - Makefile.config
+  - Makefile.config.example
+  - local
+  - hugpython
+  - gradle
+  - _vendor
+
+  # Hugo build artifacts / state
+  - .hugo_build.lock
+  - logs
+  - integrations_data
+
+  # Node / JS (Hugo-specific)
+  - babel.config.js
+  - jest.config.js
+  - postcss.config.js
+  - markdoc.config.json
+  - customization_config
+
+  # Go module — powers Hugo Modules (imports websites-modules, vendors into
+  # _vendor/). Hugo requires go.mod in the project root; Astro uses no Go.
+  - go.mod
+  - go.sum
+
+  # Node / Yarn (Hugo site owns these; Astro has its own under astro/)
+  - package.json
+  - package-lock.json
+  - yarn.lock
+  - .yarnrc.yml
+  - .yarn
+  - node_modules
+  - .nvmrc
+
+  # Testing (Hugo-specific)
+  - playwright.config.ts
+  - playwright-report
+  - test-results
+  - e2e
+
+  # Content tooling
+  - translate.yaml
+  - .translate
+  - typesense.config.json
+  - .htmltest.yml
+  - .apigentools-info
+  - agent_config_types_list.txt
+
+  # Examples / docs
+  - examples
+  - docs
+  - plans
+
+  # Static assets (Hugo)
+  - usage-notifications-email.png
+
+ignore:
+  # macOS metadata
+  - .DS_Store
+  # Python bytecode cache
+  - __pycache__
+
+# Branches used to exercise the reorg tooling against a fake main instead of the
+# real master. create_test_prs.py opens test PRs against mock_reorged_master_branch, and
+# resolve_pr_conflicts.py treats that same branch as the post-reorg base by
+# default (pass --live to target the real master), so the two scripts always
+# agree on which branches to use during a test run.
+test:
+  # Stand-in for post-reorg master: the branch that has the reorg applied. Test
+  # PRs target it, and resolve_pr_conflicts.py treats it as the base by default.
+  mock_reorged_master_branch: mock-reorged-master-7-20-1231
+  # Frozen snapshot of master that test PRs branch off, so each PR's diff stays
+  # small (a PR shows every commit in its head that isn't in the base branch).
+  branch_from: jen.gilbert/astro-reorg-scripts