Skip to content

changelog: restructure S3 layout to artifact-root #3562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cotti wants to merge 7 commits into
base: main
Choose a base branch
from
+1,255 −477
Open

changelog: restructure S3 layout to artifact-root #3562

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c0413cb
changelog: restructure S3 layout to artifact-root (Option AD)
cotti Jun 24, 2026
0fc045f
changelog: address CodeRabbit review on S3 restructure
cotti Jun 25, 2026
820e382
changelog: name boolean arguments at call sites (style review)
cotti Jun 25, 2026
741da09
changelog: nest entry keys under {org}/{repo}/{branch}
cotti Jun 30, 2026
562bcab
changelog: align reference docs with {org}/{repo}/{branch} layout
cotti Jun 30, 2026
a8aeb9f
changelog: harden CDN entry pool resolution (review)
cotti Jun 30, 2026
95b7756
Merge remote-tracking branch 'origin/main' into feature/folder_structure
cotti Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions config/changelog.example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,9 @@ bundle:
# repo: elasticsearch
# Optional: default GitHub owner applied to all profiles that do not specify their own.
# owner: elastic
# Optional: branch whose CDN changelog pool (changelog/{org}/{repo}/{branch}/...) is sourced from when
# bundling entries from the CDN. Defaults to "main" when unset. Can be overridden per profile.
# branch: main
# Optional: control auto-population of release-date for all profiles by default.
# When true (default), auto-populate release dates. Profiles can override this setting.
# release_dates: true
Expand Down
30 changes: 29 additions & 1 deletion docs/cli-schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2779,6 +2779,13 @@
"required": false,
"summary": "GitHub repository owner for PR/issue numbers or --release-version. Falls back to bundle.owner or \u0022elastic\u0022. This option is not supported in profile-based commands. The equivalent configuration options are bundle.owner or bundle.profiles.\u003Cname\u003E.owner."
},
{
"role": "flag",
"name": "branch",
"type": "string",
"required": false,
"summary": "Branch whose CDN changelog entry pool (changelog/{org}/{repo}/{branch}/...) is sourced from. Falls back to bundle.branch or \u0022main\u0022. This option is not supported in profile-based commands. The equivalent configuration options are bundle.branch or bundle.profiles.\u003Cname\u003E.branch."
},
{
"role": "flag",
"name": "plan",
Expand Down Expand Up @@ -3923,7 +3930,7 @@
],
"name": "upload",
"summary": "Upload changelog entries or bundle artifacts to S3 or Elasticsearch.",
"notes": "Uses content-hash\u2013based incremental transfer \u2014 only changed files are uploaded.",
"notes": "Uses content-hash\u2013based incremental transfer \u2014 only changed files are uploaded.\n\n\nChangelog entries are uploaded once under changelog/{org}/{repo}/{branch}/{file}, keyed by the\nauthoring owner (--owner \u003E bundle.owner \u003E git remote), repository (--repo\n\u003E bundle.repo \u003E git remote), and branch (--branch \u003E the current checkout\u0027s\nbranch). The branch is stored verbatim, so a branch\u0027s / become real key separators. Bundles\nare uploaded under bundle/{product}/{file}, product-scoped from the bundle YAML, and do not\nrequire an owner/repo/branch.",
"usage": "docs-builder changelog upload --artifact-type \u003Cstring\u003E --target \u003Cstring\u003E [options]",
"examples": [],
"parameters": [
Expand Down Expand Up @@ -3982,6 +3989,27 @@
}
]
},
{
"role": "flag",
"name": "repo",
"type": "string",
"required": false,
"summary": "GitHub repository name, the second segment of changelog entry keys (changelog/{org}/{repo}/{branch}/...). Falls back to bundle.repo in changelog.yml, then the git remote origin. Required for changelog uploads; ignored for bundle uploads."
},
{
"role": "flag",
"name": "owner",
"type": "string",
"required": false,
"summary": "GitHub owner (org), the first segment of changelog entry keys (changelog/{org}/{repo}/{branch}/...). Falls back to bundle.owner in changelog.yml, then the git remote origin. Required for changelog uploads; ignored for bundle uploads."
},
{
"role": "flag",
"name": "branch",
"type": "string",
"required": false,
"summary": "Branch, the third segment of changelog entry keys (changelog/{org}/{repo}/{branch}/...), stored verbatim. Falls back to the current checkout\u0027s branch. Required for changelog uploads; ignored for bundle uploads."
},
{
"role": "flag",
"name": "log-level",
Expand Down
35 changes: 19 additions & 16 deletions docs/cli/changelog/cmd-upload.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Description

Upload changelog entries or bundle artifacts to S3 or Elasticsearch. The command discovers `.yaml` and `.yml` files in a local directory, maps each file to one or more product-scoped S3 keys, and uploads only files whose content hash changed since the last run.
Upload changelog entries or bundle artifacts to S3 or Elasticsearch. The command discovers `.yaml` and `.yml` files in a local directory and uploads only files whose content hash changed since the last run. Changelog entries are uploaded once under `changelog/{org}/{repo}/{branch}/{file}`, keyed by the authoring owner, repository, and branch; bundles are uploaded under `bundle/{product}/{file}`, product-scoped from the bundle YAML.

To create bundles first, use [](/cli/changelog/bundle.md).
For the end-to-end workflow, see [](/contribute/bundle-changelogs.md).
Expand Down Expand Up @@ -42,10 +42,8 @@ Your IAM policy must allow these S3 actions on the target bucket:

You can scope object-level permissions to the key prefixes the command writes:

- `{product}/bundle/*`
- `{product}/changelog/*`
- `{product}/registry.json`
- `{product}/changelog/registry.json`
- `bundle/*` (bundle YAML and `bundle/{product}/registry.json`)
- `changelog/*` (entry YAML and `changelog/{org}/{repo}/{branch}/registry.json`)

#### Local development

Expand Down Expand Up @@ -79,10 +77,10 @@ Use `--artifact-type` to choose what to upload:
| `bundle` | Consolidated bundle YAML files | `bundle.output_directory` from `changelog.yml`, or `docs/releases` |
| `changelog` | Individual changelog entry YAML files | `bundle.directory` from `changelog.yml`, or `docs/changelog` |

Each file is uploaded once per product listed in its YAML:
Keying differs by artifact type:

- Changelog entries use the `products[].product` field.
- Bundles use the `products[].product` field (product ID).
- **Changelog entries** are uploaded **once** under the authoring owner/repo/branch, regardless of how many products they list (or none). The owner is resolved from `--owner`, then `bundle.owner` in `changelog.yml`, then the git remote origin; the repo from `--repo`, then `bundle.repo`, then the git remote origin; the branch from `--branch`, then the current checkout's branch. The branch is stored verbatim, so a branch name containing `/` (for example `feature/foo`) becomes additional key segments.
- **Bundles** are uploaded once per product listed in the bundle's `products[].product` field (a bundle that declares multiple products is written under each product prefix).

## Upload targets

Expand All @@ -98,19 +96,21 @@ Use `--target` to choose the destination:
For each discovered file, the command writes to:

```text
s3://{bucket}/{product}/changelog/{filename} # --artifact-type changelog
s3://{bucket}/{product}/bundle/{filename} # --artifact-type bundle
s3://{bucket}/changelog/{org}/{repo}/{branch}/{filename} # --artifact-type changelog
s3://{bucket}/bundle/{product}/{filename} # --artifact-type bundle
```

A changelog or bundle that applies to multiple products is uploaded to multiple keys — one per product.
Changelog entries are written once under the authoring org/repo/branch. A bundle that applies to multiple products is uploaded to multiple keys — one per product.

After a successful upload, the command refreshes per-product `registry.json` manifests at:
After a successful upload, the command refreshes the relevant `registry.json` manifest:

```text
s3://{bucket}/{product}/registry.json # bundle uploads
s3://{bucket}/{product}/changelog/registry.json # changelog uploads
s3://{bucket}/changelog/{org}/{repo}/{branch}/registry.json # changelog uploads
s3://{bucket}/bundle/{product}/registry.json # bundle uploads
```

When several repositories publish bundles for the same shared product (for example `cloud-serverless`), use a `{repo}-{dateOrVersion}.yaml` bundle filename convention so they don't overwrite each other under `bundle/{product}/`.

The registry refresh is best-effort: upload failures block the run, but a stale manifest does not fail an otherwise successful upload.

:::{note}
Expand Down Expand Up @@ -142,13 +142,16 @@ docs-builder changelog upload \

### Upload changelog entries to S3

Upload individual changelog YAML files from the default changelog directory (`docs/changelog`):
Upload individual changelog YAML files from the default changelog directory (`docs/changelog`). Entries are written to `changelog/{org}/{repo}/{branch}/...`; pass `--owner`, `--repo`, and `--branch` when the authoring owner/repo can't be inferred from `bundle.owner`/`bundle.repo` or the git remote, or to override the current checkout's branch:

```sh
docs-builder changelog upload \
--artifact-type changelog \
--target s3 \
--s3-bucket-name my-changelog-bundles
--s3-bucket-name my-changelog-bundles \
--owner elastic \
--repo my-repo \
--branch main
```

### Override the source directory
Expand Down
19 changes: 8 additions & 11 deletions docs/contribute/configure-changelogs-ref.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,13 @@ These settings are relevant to one or all of the `changelog bundle`, `changelog

| Setting | Description |
| ------------------------- | ----------- |
| `bundle.branch` | Branch whose CDN changelog pool (`changelog/{org}/{repo}/{branch}/...`) entries are sourced from when bundling (default: `main`). Refer to [Entry sourcing](#bundle-entry-sourcing). |
| `bundle.directory` | Input directory containing changelog YAML files (default: `docs/changelog`). |
| `bundle.link_allow_repos` | List of `owner/repo` pairs whose PR/issue links are preserved. When set (including empty `[]`), links to unlisted repos become `# PRIVATE:` sentinels. Requires `bundle.resolve: true` |
| `bundle.output_directory` | Output directory for bundled files (default: `docs/releases`). |
| `bundle.owner` | Default GitHub repository owner (for example, `elastic`). |
| `bundle.owner` | Default GitHub repository owner (for example, `elastic`). Also the org segment of uploaded changelog-entry keys (`changelog/{org}/{repo}/{branch}/...`) and CDN entry sourcing. |
| `bundle.release_dates` | When `true`, bundles include a `release-date` field (default: true). |
| `bundle.repo` | Default GitHub repository name (for example, `elasticsearch`). Used by the `{changelog}` directive to generate correct PR and issue links. Only needed when the product ID doesn't match the GitHub repository name. |
| `bundle.repo` | Default GitHub repository name (for example, `elasticsearch`). Used by the `{changelog}` directive to generate correct PR and issue links, and to scope uploaded changelog-entry keys (`changelog/{org}/{repo}/{branch}/...`) and CDN entry sourcing. Only needed when the product ID doesn't match the GitHub repository name (or to override the git remote). |
| `bundle.resolve` | When `true`, changelog contents are copied into bundle (default: `true`). |
| `bundle.use_local_changelogs` | When `true`, always source entries from the local folder and never from the CDN (default: `false`). Refer to [Entry sourcing](#bundle-entry-sourcing). |

Expand All @@ -66,20 +67,16 @@ When `bundle.link_allow_repos` is omitted, no link filtering occurs.

### Entry sourcing [bundle-entry-sourcing]

`changelog bundle` reads the individual changelog entries it aggregates either from the local folder or from the public CDN. CDN sourcing is **opt-in per product** (declared-gate): an entry is pulled from the CDN only when its product is declared under `release_notes` in the docset's `docset.yml`.
`changelog bundle` reads the individual changelog entries it aggregates either from the local folder or from the public CDN. Entries are stored on the CDN per **authoring org/repo/branch** (`changelog/{org}/{repo}/{branch}/...`), not per product, so CDN sourcing keys off the resolvable authoring org/repo/branch rather than the bundle's target products.

```yaml
# docset.yml
release_notes:
- product: elasticsearch
```
The authoring repo is resolved with the same precedence as `changelog upload`: `--repo` > `bundle.repo` in `changelog.yml` > the git remote origin. The owner is resolved from `--owner` > `bundle.owner` (default `elastic`), and the branch from `--branch` > `bundle.branch` (default `main`).

Sourcing is decided per run:

- **Local folder (default).** Used when no product in scope is declared under `release_notes`, when `bundle.use_local_changelogs: true`, when `--directory` is passed, or when the filter resolves no concrete product (for example, `--all` or PR/issue-only filters). The folder must contain the changelog files.
- **CDN.** Used only when every product in scope is declared under `release_notes` and none of the local-sourcing conditions above apply. The product must also exist in [products.yml](https://github.com/elastic/docs-builder/blob/main/config/products.yml) with the `release-notes` feature enabled.
- **Local folder.** Used when `bundle.use_local_changelogs: true`, when `--directory` is passed, or when the authoring repo cannot be resolved. The folder must contain the changelog files.
- **CDN (default when a repo resolves).** Used when the authoring repo resolves, local sourcing is not forced, and a CDN base URL is configured (`DOCS_BUILDER_CHANGELOG_CDN`, defaulting to the public distribution). The command fetches `changelog/{org}/{repo}/{branch}/registry.json` and the entries it lists, then applies the bundle's own product/PR/issue filters to the downloaded set.

The product ID under `release_notes` matches the product format described in [](/cli/changelog/bundle.md#product-format). This is the same declaration the `{changelog}` directive's `:cdn:` mode consumes, so a repository that opts into CDN-sourced bundling and CDN-rendered release notes declares each product once.
Because entries are org/repo/branch-scoped, one repository can produce a bundle for a shared product (for example, `cloud-serverless`) while sourcing its own entries from `changelog/{org}/{repo}/{branch}/`, without that product appearing in the repository's `docset.yml`. The `{changelog}` directive's `:cdn:` mode still consumes product-scoped *bundles*, so a repository that also renders its own release notes declares each product under `release_notes` as before.
Comment on lines +70 to +79

@coderabbitai coderabbitai Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Clarify the owner fallback here.

This still says the owner comes from --owner/bundle.owner with default elastic, but the new flow derives the authoring org from the resolved repo/remote when no owner is set. As written, this can point CDN lookups at the wrong org for non-Elastic repos.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/contribute/configure-changelogs-ref.md` around lines 70 - 79, Clarify
the CDN owner resolution text so it matches the actual `changelog bundle` flow:
the authoring org should be derived from the resolved repo or git remote when
`--owner`/`bundle.owner` is not provided, not always default to `elastic`.
Update the `changelog bundle` and CDN sourcing description to reflect the same
precedence as `changelog upload`, and make sure the wording around
`DOCS_BUILDER_CHANGELOG_CDN` and `registry.json` points to the resolved
authoring org/repo/branch.


### Bundle descriptions [bundle-descriptions]

Expand Down
Loading
Loading