Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions docs/configurations/deployments/forge_tenant.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,35 +23,35 @@ Copy these templates and place them at the correct paths:

### Templates to Copy

- `examples/templates/tenant/_global_settings/_global.yaml`
- `examples/templates/tenant/_environment_wide_settings/_environment.yaml`
- `examples/templates/tenant/_vpc_wide_settings/_vpc.yaml`
- `examples/templates/tenant/_global_settings/_global.yml`
- `examples/templates/tenant/_environment_wide_settings/_environment.yml`
- `examples/templates/tenant/_vpc_wide_settings/_vpc.yml`

### Destination Paths

```
examples/deployments/forge-tenant/terragrunt/_global_settings/_global.yaml
examples/deployments/forge-tenant/terragrunt/_global_settings/_global.yml

examples/deployments/forge-tenant/terragrunt/environments/<aws_account>/_environment_wide_settings/_environment.yaml
examples/deployments/forge-tenant/terragrunt/environments/<aws_account>/_environment_wide_settings/_environment.yml

examples/deployments/forge-tenant/terragrunt/environments/<aws_account>/regions/<aws_region>/vpcs/<vpc_alias>/_vpc_wide_settings/_vpc.yaml
examples/deployments/forge-tenant/terragrunt/environments/<aws_account>/regions/<aws_region>/vpcs/<vpc_alias>/_vpc_wide_settings/_vpc.yml
```

### Edit the Config Files

Before editing your tenant's `config.yaml`, review and update these supporting configuration files:
Before editing your tenant's `config.yml`, review and update these supporting configuration files:

- **\_global.yaml**\
- **\_global.yml**\
Set global values such as team name, product name, AWS account prefix, GitHub organization, and contact email.\
*(Path: `_global_settings/_global.yaml`)*
*(Path: `_global_settings/_global.yml`)*

- **\_environment.yaml**\
- **\_environment.yml**\
Define environment-wide settings like environment name, AWS region, and account ID.\
*(Path: `environments/<aws_account>/_environment_wide_settings/_environment.yaml`)*
*(Path: `environments/<aws_account>/_environment_wide_settings/_environment.yml`)*

- **\_vpc.yaml**\
- **\_vpc.yml**\
Specify VPC-wide settings including VPC alias, VPC ID, subnet IDs, and cluster name.\
*(Path: `environments/<aws_account>/regions/<aws_region>/vpcs/<vpc_alias>/_vpc_wide_settings/_vpc.yaml`)*
*(Path: `environments/<aws_account>/regions/<aws_region>/vpcs/<vpc_alias>/_vpc_wide_settings/_vpc.yml`)*

These files provide the foundational settings used by your tenant and runner modules.\
**Be sure to replace all placeholder values (`<...>`) with your actual environment details.**
Expand Down
133 changes: 117 additions & 16 deletions docs/configurations/deployments/new_tenant.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Copy these templates and place them at the correct paths.
- `examples/templates/tenant/_global_settings/tenant.hcl`
- `examples/templates/tenant/tenant/terragrunt.hcl`
- `examples/templates/tenant/tenant/runner_settings.hcl`
- `examples/templates/tenant/tenant/config.yaml`
- `examples/templates/tenant/tenant/config.yml`

### Destination Paths

Expand All @@ -24,7 +24,7 @@ examples/deployments/forge-tenant/terragrunt/environments/<aws_account>/regions/

examples/deployments/forge-tenant/terragrunt/environments/<aws_account>/regions/<aws_region>/vpcs/<vpc_alias>/tenants/<tenant_name>/runner_settings.hcl

examples/deployments/forge-tenant/terragrunt/environments/<aws_account>/regions/<aws_region>/vpcs/<vpc_alias>/tenants/<tenant_name>/config.yaml
examples/deployments/forge-tenant/terragrunt/environments/<aws_account>/regions/<aws_region>/vpcs/<vpc_alias>/tenants/<tenant_name>/config.yml
```

### Example for tenant=`sbg`, account=`sec-plat`, region=`eu-west-1`, vpc_alias=`shared`
Expand All @@ -41,15 +41,15 @@ cp examples/templates/tenant/tenant/terragrunt.hcl \
cp examples/templates/tenant/tenant/runner_settings.hcl \
examples/deployments/forge-tenant/terragrunt/environments/sec-plat/regions/eu-west-1/vpcs/shared/tenants/sbg/runner_settings.hcl

cp examples/templates/tenant/tenant/config.yaml \
examples/deployments/forge-tenant/terragrunt/environments/sec-plat/regions/eu-west-1/vpcs/shared/tenants/sbg/config.yaml
cp examples/templates/tenant/tenant/config.yml \
examples/deployments/forge-tenant/terragrunt/environments/sec-plat/regions/eu-west-1/vpcs/shared/tenants/sbg/config.yml
```

______________________________________________________________________

## 2. Edit `config.yaml` — Tenant Configuration Fields
## 2. Edit `config.yml` — Tenant Configuration Fields

Controls GitHub integration, IAM roles, runner specs (EC2 & ARC).
Controls GitHub integration, IAM roles, EC2-wide runner settings, and runner specs (EC2 & ARC).

______________________________________________________________________

Expand Down Expand Up @@ -81,18 +81,40 @@ tenant:
github_logs_reader_role_arns: # (Optional) IAM role ARNs granted read (+ KMS decrypt) access to archived GitHub job/workflow logs
- arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>

ec2_config:
enable_dynamic_labels: <true|false> # Enable dynamic ghr-* labels for EC2 runners

ec2_runner_specs:
<runner_type>: # e.g. small, medium, gpu
ami_name: <AMI_NAME_PATTERN> # AMI name pattern, supports wildcard *, e.g. forge-gh-runner-v*
ami_owner: <ACCOUNT_ID> # AWS account ID owning AMI
ami_kms_key_arn: '' # Set to '' if AMI is unencrypted, else KMS ARN string
max_instances: <MAX_PARALLEL> # Max EC2 runners allowed in parallel
instance_types: # List of allowed instance types (prefer spot-compatible)
- <AWS_INSTANCE_TYPE> # e.g. t3.large, m5.large
pool_config: # Warm pool config for pre-warming runners; empty list [] disables
- size: <POOL_SIZE> # Number of instances to keep warm
<runner_type-alias>: # e.g. small, medium, gpu, mac
type: <runner_type> # Runner type label advertised to GitHub
ami_name: <AMI_NAME_PATTERN> # AMI name pattern, supports wildcard *, e.g. forge-gh-runner-v*
ami_owner: <ACCOUNT_ID> # AWS account ID owning AMI
ami_kms_key_arn: <KMS_ARN> # Set to '' if AMI is unencrypted, else KMS ARN string
runner_os: <OS> # linux, osx, or windows
runner_architecture: <ARCH> # x64 or arm64
runner_user: <RUNNER_USER> # OS user that runs the GitHub runner process
placement: # Required for macOS dedicated-host runners; omit otherwise
host_resource_group_arn: <HOST_RESOURCE_GROUP_ARN>
tenancy: host
availability_zone: <AVAILABILITY_ZONE>
max_instances: <MAX_PARALLEL> # Max EC2 runners allowed in parallel
license_specifications: <LICENSE_SPECIFICATIONS> # Optional License Manager config for dedicated hosts
use_dedicated_host: <true|false> # Set true for macOS EC2 runners
vpc_id: <VPC_ID> # Optional override; defaults to tenant VPC when omitted
subnet_ids: # Optional override; defaults to tenant subnets when omitted
- <SUBNET_ID>
instance_types: # List of allowed instance types
- <AWS_INSTANCE_TYPE> # e.g. t3.large, m5.large, mac2.metal
pool_config: # Warm pool config for pre-warming runners; empty list [] disables
- size: <POOL_SIZE> # Number of instances to keep warm
schedule_expression: <AWS_CRON_EXPR> # AWS cron expression (6 fields, use AWS docs)
schedule_expression_timezone: <TIMEZONE> # Optional timezone, e.g. UTC, America/New_York
volume:
size: <VOLUME_SIZE>
device_name: <VOLUME_DEVICE_NAME>
iops: <VOLUME_IOPS>
throughput: <VOLUME_THROUGHPUT>
type: <VOLUME_TYPE>

arc_runner_specs:
<runner_type>: # e.g. dependabot, k8s
Expand All @@ -108,6 +130,11 @@ arc_runner_specs:
container_requests_memory: <MEM> # Kubernetes memory requests, e.g. 1Gi (mandatory unit)
container_limits_cpu: <CPU> # Kubernetes CPU limits
container_limits_memory: <MEM> # Kubernetes memory limits
volume_requests_storage_type: <STORAGE_TYPE> # Storage class/type for runner workspace volume
volume_requests_storage_size: <STORAGE_SIZE> # Size for runner workspace volume

arc_cluster_name: <CLUSTER_NAME>
migrate_arc_cluster: <true|false>

```

Expand All @@ -116,19 +143,32 @@ ______________________________________________________________________
### Field Guidance & Gotchas

- **`ghes_url`**: empty for github.com, full URL for GHES.
- **`repository_selection`**: use `all` or `selected`, matching the GitHub App installation scope.
- **`iam_roles_to_assume`**: full ARNs only, no wildcards.
- **`ecr_registries`**: must be full URLs, including account and region.
- **`github_logs_reader_role_arns`**:
- Provide a list of IAM Role ARNs that need read (and KMS decrypt) access to archived GitHub job/workflow logs.
- Leave the list empty (or omit) if no external roles should access logs.
- Roles are added to the S3 bucket policy (GetObject/ListBucket) and KMS key policy (Decrypt/Describe/GenerateDataKey\*).
- Avoid granting organization-wide wildcard roles; principle of least privilege.
- **`ec2_config.enable_dynamic_labels`**: set to `true` to allow EC2 jobs to use dynamic `ghr-` labels, such as labels that override EC2 runtime options or add dynamic runner labels. Keep `false` unless the tenant needs this behavior.
- **`type`**: logical EC2 runner type used in generated GitHub labels.
- **`ami_kms_key_arn`**: must be explicitly set to `''` if AMI not encrypted; otherwise runner fails.
- **`runner_os`**: set the operating system for the runner AMI, for example `linux`, `osx`, or `windows`.
- **`runner_architecture`**: set the runner CPU architecture, for example `x64` or `arm64`.
- **`runner_user`**: OS user that runs the GitHub runner process on the EC2 instance.
- **`max_instances`**: check AWS EC2 quota before setting.
- **`vpc_id` / `subnet_ids`**: optional per-runner network overrides. Omit them to use the tenant-level VPC and subnets.
- **`instance_types`**: spot-compatible preferred for cost savings.
- **`volume`**: root volume settings for the runner AMI, including size, device name, IOPS, throughput, and EBS type.
- **`use_dedicated_host`**: set to `true` for macOS EC2 runners, because Mac instances require EC2 Dedicated Hosts. Pair it with `placement.tenancy: host`, a host resource group or host ID, and an availability zone that has matching Mac host capacity.
- **`license_specifications`**: include License Manager configuration ARNs when your dedicated host resource group requires them for macOS runners.
- **`pool_config.schedule_expression`**: AWS cron syntax with 6 fields, **not** standard cron. Example: `cron(0 8 * * ? *)`. See [AWS docs](https://docs.aws.amazon.com/eventbridge/latest/userguide/scheduled-events.html#cron-expressions).
- **`scale_set_type`**: only `dind` or `k8s`. Wrong values cause runtime errors.
- **`scale_set_labels`**: labels used in workflow `runs-on` matching for ARC runners. Include at least the intended runner type label.
- **`volume_requests_storage_type` / `volume_requests_storage_size`**: storage class/type and size used for ARC runner workspace volumes.
- **`arc_cluster_name`**: EKS cluster where ARC runners are deployed.
- **`migrate_arc_cluster`**: set to `true` only for an intentional ARC cluster migration flow; otherwise keep `false`.
- **Kubernetes CPU/memory fields**: units mandatory (e.g., `500m`, `1Gi`). Missing units break pods.

#### `github_webhook_relay` Guidance
Expand All @@ -140,12 +180,51 @@ ______________________________________________________________________
- All `destination_*` keys are ignored when `enabled: false` (can be left as placeholders).
- Typical use cases: central analytics, multi-account runner orchestration, or security event aggregation.

#### macOS EC2 Runner Guidance

macOS runners must run on EC2 Dedicated Hosts. Configure the runner spec with `use_dedicated_host: true`, host placement, and Mac instance types:

```yaml
ec2_runner_specs:
mac:
type: mac
ami_name: forge-gh-runner-macarm-v*
ami_owner: '123456789012'
ami_kms_key_arn: ''
runner_os: osx
runner_architecture: arm64
runner_user: ec2-user
placement:
host_resource_group_arn: arn:aws:resource-groups:<REGION>:<ACCOUNT_ID>:group/<HOST_RESOURCE_GROUP>
tenancy: host
availability_zone: <AVAILABILITY_ZONE>
license_specifications:
- license_configuration_arn: arn:aws:license-manager:<REGION>:<ACCOUNT_ID>:license-configuration:<LICENSE_CONFIGURATION_ID>
use_dedicated_host: true
vpc_id: <VPC_ID>
subnet_ids:
- <SUBNET_ID>
max_instances: <MAX_PARALLEL>
instance_types:
- mac2.metal
pool_config: []
volume:
size: <VOLUME_SIZE>
device_name: <VOLUME_DEVICE_NAME>
iops: <VOLUME_IOPS>
throughput: <VOLUME_THROUGHPUT>
type: gp3
```

Use a subnet in the same availability zone as the dedicated host placement. If your host resource group does not require License Manager, omit `license_specifications`.

______________________________________________________________________

### Common Pitfalls — Avoid These

- Wildcard or invalid IAM roles → runner startup failures.
- Forgetting `ami_kms_key_arn` = `''` when AMI isn’t encrypted → Terraform errors.
- Setting up macOS runners without `use_dedicated_host: true` or matching host placement → EC2 launch failures.
- Incorrect cron syntax → scheduled warm pools don’t trigger.
- Setting max runners beyond quotas → failures or throttling.
- Missing units in k8s resource requests/limits → pod rejection.
Expand Down Expand Up @@ -218,12 +297,13 @@ sec-plat-euw1-shared-sbg-cicd-forge

______________________________________________________________________

## 4. Minimal Working `config.yaml` Example
## 4. Minimal Working `config.yml` Example

```yaml
gh_config:
ghes_url: ''
ghes_org: cisco-sbg
repository_selection: selected
github_webhook_relay:
enabled: false
destination_account_id: ""
Expand All @@ -244,19 +324,35 @@ tenant:
github_logs_reader_role_arns:
- arn:aws:iam::123456789012:role/github_logs_reader

ec2_config:
enable_dynamic_labels: false

ec2_runner_specs:
small:
type: small
ami_name: forge-gh-runner-v*
ami_owner: '123456789012'
ami_kms_key_arn: ''
runner_os: linux
runner_architecture: x64
runner_user: ubuntu
max_instances: 10
vpc_id: vpc-0abc1234def567890
subnet_ids:
- subnet-0abc1234def567890
instance_types:
- t3.small
- t3.medium
pool_config:
- size: 2
schedule_expression: "cron(*/10 8 * * ? *)"
schedule_expression_timezone: "America/Los_Angeles"
volume:
size: 200
device_name: /dev/sda1
iops: 3000
throughput: 125
type: gp3

arc_runner_specs:
dependabot:
Expand All @@ -273,6 +369,11 @@ arc_runner_specs:
container_requests_memory: 1Gi
container_limits_cpu: '1'
container_limits_memory: 2Gi
volume_requests_storage_type: gp2
volume_requests_storage_size: 10Gi

arc_cluster_name: forge-arc-cluster
migrate_arc_cluster: false
```

______________________________________________________________________
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ inputs = {
lambda_vpc_id = local.config.locals.lambda_vpc_id
subnet_ids = local.config.locals.subnet_ids
vpc_id = local.config.locals.vpc_id
ec2_config = local.config.ec2_config.enable_dynamic_labels
runner_specs = local.config.locals.ec2_runner_specs
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ tenant:
- 123456789012.dkr.ecr.eu-west-1.amazonaws.com
github_logs_reader_role_arns:
- arn:aws:iam::123456789012:role/github_logs_reader
ec2_config:
enable_dynamic_labels: true
ec2_runner_specs:
small:
type: small
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ locals {
}
}

ec2_config = local.config.ec2_config

ec2_runner_specs = {
for size, spec in local.config.ec2_runner_specs :
size => {
Expand Down
11 changes: 6 additions & 5 deletions examples/templates/tenant/_global_settings/tenant.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,12 @@ inputs = {

# Runners (EC2/ARC)
ec2_deployment_specs = {
lambda_subnet_ids = local.config.locals.lambda_subnet_ids
lambda_vpc_id = local.config.locals.lambda_vpc_id
subnet_ids = local.config.locals.subnet_ids
vpc_id = local.config.locals.vpc_id
runner_specs = local.config.locals.ec2_runner_specs
lambda_subnet_ids = local.config.locals.lambda_subnet_ids
lambda_vpc_id = local.config.locals.lambda_vpc_id
subnet_ids = local.config.locals.subnet_ids
vpc_id = local.config.locals.vpc_id
enable_dynamic_labels = local.config.locals.ec2_config.enable_dynamic_labels
runner_specs = local.config.locals.ec2_runner_specs
}

arc_deployment_specs = {
Expand Down
2 changes: 2 additions & 0 deletions examples/templates/tenant/tenant/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ tenant:
- <ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com
github_logs_reader_role_arns:
- arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>
ec2_config:
enable_dynamic_labels: <ENABLE_DYNAMIC_LABELS>
ec2_runner_specs:
<runner_type-alias>:
type: <runner_type>
Expand Down
2 changes: 2 additions & 0 deletions examples/templates/tenant/tenant/runner_settings.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ locals {
}
}

ec2_config = local.config.ec2_config

ec2_runner_specs = {
for size, spec in local.config.ec2_runner_specs :
size => {
Expand Down
Loading