ref(pipedream): group regions into parallel jobs per pipeline

README.md

@@ -50,15 +50,15 @@ When `output-files=true` it'll output pipelines in the format:
 ```
 
 This is useful when you build using the `-m` flag in jsonnet as it'll output
-multiple files which makes reviewing pipeliens easier.
+multiple files which makes reviewing pipelines easier.
 
 ```bash
 jsonnet --ext-code output-files=true -m ./generated-pipelines ./example.jsonnet
 ```
 
 The GoCD plugin that can process jsonnet files directly doesn't support
 outputting multiple files, so GoCD is configured to have
-`--ext-code output-filaes=false` which will output the pipelines in a flattened
+`--ext-code output-files=false` which will output the pipelines in a flattened
 format:
 
 ```json
@@ -87,8 +87,40 @@ Pipedream will name the returned pipeline, add an upstream pipeline material and
 a final stage. The upstream material and final stage is needed to make GoCD
 chain the pipelines together.
 
-The end result will be a pipeline `deploy-<service name>` that starts the run of
-each pipeline, and a pipeline for each region.
+Regions are organized into **groups** (defined in `getsentry.libsonnet`). Each
+group produces a single GoCD pipeline where regions within the group run as
+parallel jobs. Groups are chained sequentially by default, or can fan out in
+parallel via `pipedream.render(config, pipeline_fn, parallel=true)`.
+
+The end result is a trigger pipeline `deploy-<service name>` and a pipeline per
+group (e.g. `deploy-example-s4s2`, `deploy-example-st`).
+
+`control` and `snty-tools` are excluded by default; use `include_regions` to
+opt in.
+
+Environment variables set at the pipeline or stage level in `pipeline_fn` are
+handled automatically: variables identical across all regions in a group stay at
+the stage level, while region-specific variables are cascaded to the job level.
+GoCD resolves precedence as job > stage > pipeline.
+
+### Targeting a subset of regions
+
+By default, triggering a group pipeline deploys to every region in the group.
+To deploy to a subset, override `PIPEDREAM_GROUP_REGIONS` on the Environment
+Variables tab in GoCD's "Trigger with options".
+
+For example, on `deploy-foo-st` whose default is
+`customer-1,customer-2,customer-4,customer-7`:
+
+| Override value          | Result                                |
+| ----------------------- | ------------------------------------- |
+| (no override)           | All four regions deploy               |
+| `customer-1`            | Only customer-1 deploys               |
+| `customer-1,customer-2` | Only customer-1 and customer-2 deploy |
+
+Setting a value that doesn't include any of the group's regions makes every
+job skip and the pipeline goes green having done nothing. If that happens,
+retrigger with a valid value.
 
 ### Example Usage
 

libs/getsentry.libsonnet

@@ -3,23 +3,27 @@
 */
 
 {
-  // These regions are user facing deployments
-  prod_regions: [
-    's4s2',
-    'de',
-    'us',
-    // 'control' is excluded by default and must be explicitly included
-    'control',
-    'prod-control',
-    // 'snty-tools' is excluded by default and must be explicitly included
-    'snty-tools',
-    'customer-1',
-    'customer-2',
-    'customer-4',
-    'customer-7',
-  ],
-  // Test regions will deploy in parallel to the regions above
-  test_regions: [
-  ],
+  group_order: ['s4s2', 'de', 'us', 'control', 'prod-control', 'snty-tools', 'st'],
+  // Empty for now — add future test groups here
+  test_group_order: [],
+  // These groupings consist of user facing deployments
+  pipeline_groups: {
+    s4s2: ['s4s2'],
+    de: ['de'],
+    us: ['us'],
+    control: ['control'],
+    'prod-control': ['prod-control'],
+    'snty-tools': ['snty-tools'],
+    st: ['customer-1', 'customer-2', 'customer-4', 'customer-7'],
+  },
+  // Test groups will deploy in parallel to the groups above
+  test_groups: {
+  },
+
+  group_names:: self.group_order,
+  test_group_names:: self.test_group_order,
+  get_targets(group)::
+    if std.objectHas(self.pipeline_groups, group) then self.pipeline_groups[group]
+    else self.test_groups[group],
   is_st(region):: std.startsWith(region, 'customer-'),
 }