From 1713edf12c1810111bb220034fe6fe4080abdca4 Mon Sep 17 00:00:00 2001 From: Laura Murphy Date: Thu, 30 Apr 2026 11:51:20 +0100 Subject: [PATCH 1/5] Update proprietary images to BIOIO workflow. Discussed in #4 --- episodes/01-opening-and-checking-an-image.md | 133 ++++++------------- 1 file changed, 39 insertions(+), 94 deletions(-) diff --git a/episodes/01-opening-and-checking-an-image.md b/episodes/01-opening-and-checking-an-image.md index c2aebbf..120aada 100644 --- a/episodes/01-opening-and-checking-an-image.md +++ b/episodes/01-opening-and-checking-an-image.md @@ -255,66 +255,46 @@ plt.show() ## Proprietary formats -Images can come in many formats, but for many of the common ones such as TIFF, -PNG and BMP, skimage is smart enough to know how to read each one. +Modern microscopes save files in vendor-specific formats (like .czi, .nd2, or .lif). While skimage can handle standard TIF(F)s, these complex files contain vital metadata, for example, pixel spacing information, that we need for accurate analysis. -Some image formats are associated with specific instruments or equipment and -require specialised packages to open. Depending on your system, these may -already be available via `import` the same as any other Python package. If -not, then these should be installed into whatever Python instance you are -using. -If using JupyterHub or JupyterLab, go to 'New' -> 'Terminal'. This will open -a shell session in a new browser tab, where you can run `pip install` commands. - -### Carl Zeiss .czi - -Images in .czi format can be opened with the `czifile` library. In the Terminal -you opened above: +### The Universal Adapter: BIOIO - $ pip install czifile +Instead of installing a different library for every microscope brand, we recommend BIOIO. It acts as a consistent interface for almost any biological image format, allowing you to use the same commands regardless of the file source. -### Nikon .nd2 - - $ pip install nd2 - -### Imaris .ims +If using JupyterHub or JupyterLab, go to 'New' -> 'Terminal'. This will open +a shell session in a new browser tab, where you can run `pip install` commands. - $ pip install imaris-ims-file-reader +::: callout +### Advanced: Scaling up with Dask +Once you are a bit more established as a Python user, the integration of BIOIO with Dask becomes a major advantage. It allows for "Lazy Loading," where the computer only reads the specific pixels you ask for. This is essential for analyzing massive datasets that are larger than your computer's RAM. +::: -### Leica .lif +:::: challenge +## Exercise 4: Reading with BIOIO - $ pip install readlif +Load the bioio package and use it to read the test file 'data/Ersi_organoid_WT2.nd2'. +1. Use print(img.dims) to check the dimensions. What does each letter represent? +2. Use get_image_data to extract a single 2D frame for the first channel (C=0) at the middle Z-slice (Z=13). -::::::::::::::::::::::::::::::::::::: challenge -## Exercise 4: Proprietary formats +::: solution +```python +from bioio import BioImage -Load the [nd2](https://www.talleylambert.com/nd2/#installation) package and use it to read -the test file 'Ersi_organoid_WT2.nd2'. Install it if you need to. +# Load the image object +img = BioImage('data/Ersi_organoid_WT2.nd2') -What axes are present in the image? Which look most likely to be the X and -Y axes? Use `imshow` to display a single frame from the image. +# Check dimensions explicitly +print(img.dims) +# Returns: Dimensions [T: 1, C: 3, Z: 27, Y: 512, X: 512] -:::::::::::::::::::::::: solution -If not already present, the nd2 package can be installed in the Terminal with -`pip install nd2`. According to its linked documentation, it has -an `imread` function that works in a similar way to the one in skimage, and returns a numpy -multidimensional array that we can work with the same way we have before: +# Grab a specific slice (Channel 0, Z-slice 13) +pixel_data = img.get_image_data("YX", C=0, Z=13) -```python -import nd2 -image = nd2.imread('data/Ersi_organoid_WT2.nd2') -image.shape -# returns: (27, 3, 512, 512) -``` - -`img.shape` shows that there are four axes, `(27, 3, 512, 512)`. The latter two numbers look like -the X and Y axes, while the second number looks like a number of colour channels. The first number -looks like either a time series or a Z axis. We can show a single frame with: +import matplotlib.pyplot as plt +plt.imshow(pixel_data) -```python -plt.imshow(image[0, 0, :, :]) ``` ![](fig/1_6_nd2_image_frame.jpg){alt='ND2 image'} @@ -322,22 +302,15 @@ plt.imshow(image[0, 0, :, :]) ::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::::::::::::: -As discussed above in exercise 4, it may be difficult to distinguish a time series from a Z axis. -You may also notice that here the X/Y axes are the latter two numbers, but in other examples -above, the X/Y axes were the first two. This demonstrates the diversity and general lack of -consistency in image formatting, and how it's usually a good idea to find out as much as you can -about the image from documentation and metadata before processing it. - ## Altering the lookup table -Now that we've been able to open an image, we can start to explore it and display -it in different ways. +Now that we have extracted a specific 2D slice of data using BIOIO, we can explore different ways to display it. -The `imshow()` function can take extra arguments in addition to the image to display. One -of these is called `cmap`, which can apply alternate lookup tables (a.k.a. colour maps): +The imshow() function can take an argument called cmap, which applies a "Lookup Table" (LUT). This maps the numerical intensity values in your image to specific colors on your screen. ```python -plt.imshow(image[0, 0, :, :], cmap='viridis') +# Using the pixel_data we extracted from the BIOIO object earlier +plt.imshow(pixel_data, cmap='viridis') ``` Skimage uses lookup tables from the plotting library matplotlib. A list of available tables @@ -427,45 +400,17 @@ position 0 (the start). ### Pixel size -Pixels are an approximation of an object - knowing that something is 50 pixels long and -50 pixels wide doesn't tell us anything about its actual size. To make any real-world -measurements on the image, we need the image's **pixel size**. - -To get this, it is necessary to read the image's **metadata**. For this we need a different -library, imageio. There are a couple of different places we can look: - +To make real-world measurements, we need the image's pixel size. While older methods required digging through complex metadata dictionaries, BIOIO makes this simple: ```python -import imageio -meta = imageio.v3.immeta('data/FluorescentCells_3channel.tif') -props = imageio.v3.improps('data/FluorescentCells_3channel.tif') -print('meta:', meta) -print('props:', props) +# Access the physical pixel sizes directly from the bioio object +print(img.physical_pixel_sizes) +# Returns physical sizes (e.g., X=0.1, Y=0.1, Z=0.5) usually in micrometres. ``` -`immeta()` gives us a dict summarising the image, and `improps` gives an object with the -property 'spacing'. However, there is no guarantee that wither of these will contain any -information on pixel size, and the BioimageBook notes that -[these numbers can be misleading](https://bioimagebook.github.io/chapters/1-concepts/5-pixel_size/python.html#imageio) -and require interpretation and cross-checking. - -Even if `immeta()` does return a key called 'unit', the value may be returned as escaped Unicode: - - 'unit': '\\u00B5m' - -This can be un-escaped with: - - >>> m['unit'].encode().decode('unicode-escape') - μm - -Here, this would indicate that the image is to be mesaured in micrometres. Combining this -with other information that may be available, e.g. `improps().spacing`, will help you figure -out the pixel size. - ::::::::::::::::::::::::::::::::::::: keypoints -- Common image formats can usually all be loaded in the same way with skimage -- Specialised proprietary formats may require specialised libraries -- Basic metrics of an image include histogram, shape and max/min pixel values -- These metrics can help tell us how the miage should be analysed -- Lookup tables can change how a single-channel image is rendered -- An RGB image contains 3 channels for red, green and blue +Standard image formats (TIFF, PNG) can be loaded as NumPy arrays using skimage. +Proprietary microscope formats are best handled by BIOIO to preserve dimensions and metadata. +Basic metrics like histograms, shape, and pixel ranges help determine the best analysis strategy. +Lookup tables (LUTs) change how data is rendered visually but do not change the underlying pixel values. +Lazy loading (via Dask in BIOIO) allows you to explore massive datasets without overwhelming your computer's memory. :::::::::::::::::::::::::::::::::::::::::::::::: From b5d6ab69027f619760cd9f2a476a67953b2bf4d1 Mon Sep 17 00:00:00 2001 From: Laura Murphy Date: Thu, 30 Apr 2026 12:05:09 +0100 Subject: [PATCH 2/5] Add link to BIOIO support formats --- episodes/01-opening-and-checking-an-image.md | 279 +++++++------------ 1 file changed, 106 insertions(+), 173 deletions(-) diff --git a/episodes/01-opening-and-checking-an-image.md b/episodes/01-opening-and-checking-an-image.md index 120aada..283e098 100644 --- a/episodes/01-opening-and-checking-an-image.md +++ b/episodes/01-opening-and-checking-an-image.md @@ -4,38 +4,31 @@ teaching: 60 exercises: 20 --- -:::::::::::::::::::::::::::::::::::::: questions -- How do I open an image file in Python for processing? -- How can I explore an image once it's open? -:::::::::::::::::::::::::::::::::::::::::::::::: +::: questions +- How do I open an image file in Python for processing? +- How can I explore an image once it's open? +::: -::::::::::::::::::::::::::::::::::::: objectives -- Open an image with skimage -- Discuss how to open proprietary image formats -- Display an opened image to the screen -:::::::::::::::::::::::::::::::::::::::::::::::: +::: objectives +- Open an image with skimage +- Discuss how to open proprietary image formats +- Display an opened image to the screen +::: ## Opening an image -At its core, an image is a multidimensional array of numbers, and as such can be opened -by programs and libraries designed for working with this kind of data. For Python, one -such library is scikit-image. This library provides a function called `imread` that we -can use to load an image into memory. In a new notebook cell: +At its core, an image is a multidimensional array of numbers, and as such can be opened by programs and libraries designed for working with this kind of data. For Python, one such library is scikit-image. This library provides a function called `imread` that we can use to load an image into memory. In a new notebook cell: -```python +``` python from skimage.io import imread image = imread('data/FluorescentCells_3channel.tif') ``` -If you saved your images to a different location, you will need to change the file path -provided to imread accordingly. Paths will be relative to the location of your .ipynb -notebook file. +If you saved your images to a different location, you will need to change the file path provided to imread accordingly. Paths will be relative to the location of your .ipynb notebook file. -To view things in Python, usually we use `print()`. However if we try to print this -image to the Jupyter console, instead of the image we get something that may be -unexpected: +To view things in Python, usually we use `print()`. However if we try to print this image to the Jupyter console, instead of the image we get something that may be unexpected: -```output +``` output > print(image) array([[[ 16, 50, 0], [ 15, 44, 0], @@ -46,16 +39,11 @@ array([[[ 16, 50, 0], [ 1, 15, 2]]], dtype=uint8) ``` -Python has loaded and stored the image as a Numpy `array` object of numbers, and -`print()` displays the string representation or textual form of the data passed -to it, which is why we get a matrix of numbers printed to the screen. +Python has loaded and stored the image as a Numpy `array` object of numbers, and `print()` displays the string representation or textual form of the data passed to it, which is why we get a matrix of numbers printed to the screen. -If we want to see what the image looks like, we need tell Python to display it -as an image. We can do this with the `imshow` function from matplotlib, which you -may already be familiar with as a library for drawing plots and graphs, but it can -also display images. +If we want to see what the image looks like, we need tell Python to display it as an image. We can do this with the `imshow` function from matplotlib, which you may already be familiar with as a library for drawing plots and graphs, but it can also display images. -```python +``` python import matplotlib.pyplot as plt plt.set_cmap('gray') # by default, single-channel images will now be displayed in greyscale @@ -64,93 +52,74 @@ plt.imshow(image) You should now see the image displayed below the current cell: -![](fig/1_1_imshow.jpg){alt='Displaying an image with imshow'} +![](fig/1_1_imshow.jpg){alt="Displaying an image with imshow"} -Since images are multi-dimensional arrays of numbers, we can apply statistical functions -to them and extract some basic metrics. Numpy arrays have methods for several of these -already, including the image's shape, data type and minimum/maximum values: +Since images are multi-dimensional arrays of numbers, we can apply statistical functions to them and extract some basic metrics. Numpy arrays have methods for several of these already, including the image's shape, data type and minimum/maximum values: -```python +``` python print(image.shape, image.dtype, image.min(), image.max()) # returns: (512, 512, 3) uint8 0 255 ``` -This shows that the image has a data type of uint8, it contains values between -0 and 255 and that it is in three dimensions. We can reasonably infer that the two `512` numbers -are the X and Y axes. The third axis in most cases will represent a number of channels. +This shows that the image has a data type of uint8, it contains values between 0 and 255 and that it is in three dimensions. We can reasonably infer that the two `512` numbers are the X and Y axes. The third axis in most cases will represent a number of channels. We can select a single channel by **indexing** the array: -```python +``` python plt.imshow(image[:, :, 2]) ``` -Here, we select the entire X and Y axes using `:` with no numbers around them, and -the last channel (remember that Python counts from 0). - +Here, we select the entire X and Y axes using `:` with no numbers around them, and the last channel (remember that Python counts from 0). ## Channels, series and stacks -Images can consist of more than two axes. The first two axes are usually X and Y, but if there -is a third axis, then this could be one of several things: +Images can consist of more than two axes. The first two axes are usually X and Y, but if there is a third axis, then this could be one of several things: -- **Channel** - the image shows different features in the same 2D space. One common example is - cell images with different staining for nuclei and membranes, expressed as different colours. -- **Time series** - the image is a collection of 2D frames taken at different points in time. -- **Z-stack** - essentially a series of 2D images piled up on top of each other in 3D space. +- **Channel** - the image shows different features in the same 2D space. One common example is cell images with different staining for nuclei and membranes, expressed as different colours. +- **Time series** - the image is a collection of 2D frames taken at different points in time. +- **Z-stack** - essentially a series of 2D images piled up on top of each other in 3D space. -It's usually easy enough to tell that you're looking at a colour channel from looking at the -image directly, but it may be more difficult to to distinguish a Z or a timepoint axis from -the data alone. If you don't know exactly how the images were generated, it's a good idea to -consult documentation or metadata. +It's usually easy enough to tell that you're looking at a colour channel from looking at the image directly, but it may be more difficult to to distinguish a Z or a timepoint axis from the data alone. If you don't know exactly how the images were generated, it's a good idea to consult documentation or metadata. -::::::::::::::::::::::::::::::::::::: challenge +:::: challenge ## Exercise 1: Loading an image Load the test image 'FluorescentCells_3channel.tif': -- Try the same as the example above, but display one of the other channels -- Save your single channel to a variable. What happens if you run `imshow` on - `channel.T`? -- How can we select part of the image, i.e. crop it? Remember that to do this, - we need to select a subset of the X and/or Y axes. +- Try the same as the example above, but display one of the other channels +- Save your single channel to a variable. What happens if you run `imshow` on `channel.T`? +- How can we select part of the image, i.e. crop it? Remember that to do this, we need to select a subset of the X and/or Y axes. -:::::::::::::::::::::::: solution -Other channels can be loaded with `image[:, :, x]`, where `image` is the variable -the image is saved to and `x` is the index of a channel to retrieve. +::: solution +Other channels can be loaded with `image[:, :, x]`, where `image` is the variable the image is saved to and `x` is the index of a channel to retrieve. -Next we can use `.T` to return a **transposed** version of the image. Running -`imshow()` on this results in an image that is flipped 90°: +Next we can use `.T` to return a **transposed** version of the image. Running `imshow()` on this results in an image that is flipped 90°: -```python +``` python channel = image[:, :, 1] plt.imshow(channel.T) ``` -![](fig/1_2_transpose.jpg){alt='Transposed image'} +![](fig/1_2_transpose.jpg){alt="Transposed image"} -Remember that Numpy arrays can be sliced and indexed the same way as lists, strings and -tuples. Up to this point we've been using `:` to select an entire axis, but we can -give it start and end bounds to select part of the X and Y axes, like: +Remember that Numpy arrays can be sliced and indexed the same way as lists, strings and tuples. Up to this point we've been using `:` to select an entire axis, but we can give it start and end bounds to select part of the X and Y axes, like: -```python +``` python image[:256, 128:384, 1] ``` -![](fig/1_3_2d_slice.jpg){alt='2-dimensional slice'} -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: - +![](fig/1_3_2d_slice.jpg){alt="2-dimensional slice"} +::: +:::: -Numpy arrays have many more methods available for checking them. Here are just a few to -start with: +Numpy arrays have many more methods available for checking them. Here are just a few to start with: ### Pixel value statistics -- [`image.mean()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.mean.html) -- [`image.min()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.min.html) -- [`image.max()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.max.html) -- [`image.std()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.std.html) +- [`image.mean()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.mean.html) +- [`image.min()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.min.html) +- [`image.max()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.max.html) +- [`image.std()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.std.html) ### Image size @@ -160,26 +129,23 @@ start with: [`image.nbytes`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.nbytes.html) -Note that some of these are functions and need to be called with brackets(`()`), whereas others are -simply attributes that do not. +Note that some of these are functions and need to be called with brackets(`()`), whereas others are simply attributes that do not. -::::::::::::::::::::::::::::::::::::: challenge +:::: challenge ## Exercise 2: Memory check How much memory did it take to load FluorescentCells_3channel.tif? -:::::::::::::::::::::::: solution -Running `image.nbytes` shows that it takes up 786432 bytes, or ~786 kilobytes, or ~0.7 megabytes. -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: - +::: solution +Running `image.nbytes` shows that it takes up 786432 bytes, or \~786 kilobytes, or \~0.7 megabytes. +::: +:::: ## Displaying one channel at a time -We've seen from exercise 1 that we can view single channels by indexing the array. We can also show all -channels together using a matplotlib figure: +We've seen from exercise 1 that we can view single channels by indexing the array. We can also show all channels together using a matplotlib figure: -```python +``` python import matplotlib.pyplot as plt plt.figure(figsize=(12, 6)) # figure size, in inches nchannels = 3 @@ -195,35 +161,27 @@ for i in range(nchannels): plt.show() ``` -![](fig/1_4_all_channels.jpg){alt='All image channels'} - +![](fig/1_4_all_channels.jpg){alt="All image channels"} ## Histograms -Another useful metric in image analysis is an image's histogram. This can be plotted by flattening -the image and passing it to matplotlib: +Another useful metric in image analysis is an image's histogram. This can be plotted by flattening the image and passing it to matplotlib: -```python +``` python plt.hist(image[:, :, 0].flatten(), bins=256) ``` -First, we need to select a single channel - since different channels may represent different cell organelles -or points in time, we need to ensure that we are comparing like with like. We also need `flatten()` because -we don't care about the arrangement of the pixels, we just want to sort their values values into bins. Finally, -we can use `bins` to control how many bins the data is split into. +First, we need to select a single channel - since different channels may represent different cell organelles or points in time, we need to ensure that we are comparing like with like. We also need `flatten()` because we don't care about the arrangement of the pixels, we just want to sort their values values into bins. Finally, we can use `bins` to control how many bins the data is split into. -::::::::::::::::::::::::::::::::::::: challenge +:::: challenge ## Exercise 3: Histograms -Combine the usage of `matplotlib.pyplot.hist()` and `matplotlib.pyplot.figure()` -introduced above above and plot a histogram of each of the three channels in -FluorescentCells_3channel.tif. - -:::::::::::::::::::::::: solution +Combine the usage of `matplotlib.pyplot.hist()` and `matplotlib.pyplot.figure()` introduced above above and plot a histogram of each of the three channels in FluorescentCells_3channel.tif. +::: solution Starting with displaying a single histogram for one channel: -```python +``` python channel_idx = 0 channel = image[:, :, channel_idx] plt.hist(channel.flatten(), bins=255) @@ -231,10 +189,9 @@ plt.title('Channel ' + str(channel_idx)) plt.show() ``` -We could call this three times, each with a different value for `channel_idx`, or we can use -a **for loop**: +We could call this three times, each with a different value for `channel_idx`, or we can use a **for loop**: -```python +``` python plt.figure(figsize=(12, 6)) # figure size, in inches nchannels = image.shape[-1] @@ -247,39 +204,37 @@ for i in range(nchannels): plt.show() ``` -![](fig/1_5_histograms.jpg){alt='All image channels'} - -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: +![](fig/1_5_histograms.jpg){alt="All image channels"} +::: +:::: ## Proprietary formats Modern microscopes save files in vendor-specific formats (like .czi, .nd2, or .lif). While skimage can handle standard TIF(F)s, these complex files contain vital metadata, for example, pixel spacing information, that we need for accurate analysis. - ### The Universal Adapter: BIOIO -Instead of installing a different library for every microscope brand, we recommend BIOIO. It acts as a consistent interface for almost any biological image format, allowing you to use the same commands regardless of the file source. +Instead of installing a different library for every microscope brand, we recommend BIOIO. It acts as a consistent interface for almost any biological image format, allowing you to use the same commands regardless of the file source. You can have a look at what file formats are supported by BIOIO [here](https://bioio-devs.github.io/bioio/OVERVIEW.html#reader-installation). -If using JupyterHub or JupyterLab, go to 'New' -> 'Terminal'. This will open -a shell session in a new browser tab, where you can run `pip install` commands. +If using JupyterHub or JupyterLab, go to 'New' -\> 'Terminal'. This will open a shell session in a new browser tab, where you can run `pip install` commands. ::: callout ### Advanced: Scaling up with Dask + Once you are a bit more established as a Python user, the integration of BIOIO with Dask becomes a major advantage. It allows for "Lazy Loading," where the computer only reads the specific pixels you ask for. This is essential for analyzing massive datasets that are larger than your computer's RAM. ::: :::: challenge ## Exercise 4: Reading with BIOIO -Load the bioio package and use it to read the test file 'data/Ersi_organoid_WT2.nd2'. +Load the bioio package and use it to read the test file 'data/Ersi_organoid_WT2.nd2'. -1. Use print(img.dims) to check the dimensions. What does each letter represent? -2. Use get_image_data to extract a single 2D frame for the first channel (C=0) at the middle Z-slice (Z=13). +1. Use print(img.dims) to check the dimensions. What does each letter represent? +2. Use get_image_data to extract a single 2D frame for the first channel (C=0) at the middle Z-slice (Z=13). ::: solution -```python +``` python from bioio import BioImage # Load the image object @@ -294,45 +249,39 @@ pixel_data = img.get_image_data("YX", C=0, Z=13) import matplotlib.pyplot as plt plt.imshow(pixel_data) - ``` -![](fig/1_6_nd2_image_frame.jpg){alt='ND2 image'} - -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: +![](fig/1_6_nd2_image_frame.jpg){alt="ND2 image"} +::: +:::: ## Altering the lookup table -Now that we have extracted a specific 2D slice of data using BIOIO, we can explore different ways to display it. +Now that we have extracted a specific 2D slice of data using BIOIO, we can explore different ways to display it. The imshow() function can take an argument called cmap, which applies a "Lookup Table" (LUT). This maps the numerical intensity values in your image to specific colors on your screen. -```python +``` python # Using the pixel_data we extracted from the BIOIO object earlier plt.imshow(pixel_data, cmap='viridis') ``` -Skimage uses lookup tables from the plotting library matplotlib. A list of available tables -can be obtained with: +Skimage uses lookup tables from the plotting library matplotlib. A list of available tables can be obtained with: -```python +``` python from matplotlib import colormaps print(sorted(colormaps)) ``` -::::::::::::::::::::::::::::::::::::: challenge +::::: challenge ## Exercise 5: Lookup tables -Go back to the FluorescentCells_3channel.tif image. Display each of its three channels side -by side in a matplotlib figure, each in a different colour using `cmap=`. Use the values in -`matplotlib.colormaps` to select a lookup table for each one. +Go back to the FluorescentCells_3channel.tif image. Display each of its three channels side by side in a matplotlib figure, each in a different colour using `cmap=`. Use the values in `matplotlib.colormaps` to select a lookup table for each one. -:::::::::::::::::::::::: solution -There will be many ways to do this (and many colour maps to choose from!), but here -is one possible solution: +::: solution +There will be many ways to do this (and many colour maps to choose from!), but here is one possible solution: -```python +``` python # you'll need to run this again if you overwrote your `image` variable image = imread('data/FluorescentCells_3channel.tif') plt.figure(figsize=(12, 6)) @@ -347,32 +296,23 @@ plt.subplot(1, 3, 3) plt.imshow(image[:, :, 2], cmap='YlOrBr') ``` -![](fig/1_7_colormaps.jpg){alt='ND2 image'} - -::::::::::::::::::::::::::::::::: +![](fig/1_7_colormaps.jpg){alt="ND2 image"} +::: -Load hela-cells_rgb.tif and try displaying it with different lookup tables. What results do -you get? Why might this be the case? +Load hela-cells_rgb.tif and try displaying it with different lookup tables. What results do you get? Why might this be the case? -:::::::::::::::::::::::: solution -The resulting image will not look as expected, and in Jupyter the image will appear unchanged. -In this case, since the lookup table is being ignored, this would imply that the pixel values -do not represent light intensities but rather are explicitly encoded colour values - i.e. it's -an RGB image. -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: +::: solution +The resulting image will not look as expected, and in Jupyter the image will appear unchanged. In this case, since the lookup table is being ignored, this would imply that the pixel values do not represent light intensities but rather are explicitly encoded colour values - i.e. it's an RGB image. +::: +::::: ## Other notes ### Rearranging channels -You may find yourself in a situation where the arrangement of dimensions in your image -is incorrect for the processing you wish to perform on it - maybe a function requires -that an image be oriented in a particular way. This is where it's useful to be able -to rearrange the dimensions of an image. To do this, we can use Numpy's `moveaxis` -function: +You may find yourself in a situation where the arrangement of dimensions in your image is incorrect for the processing you wish to perform on it - maybe a function requires that an image be oriented in a particular way. This is where it's useful to be able to rearrange the dimensions of an image. To do this, we can use Numpy's `moveaxis` function: -```python +``` python import numpy image = imread('data/FluorescentCells_3channel.tif') print(image.shape) @@ -384,33 +324,26 @@ print(image.shape) # returns: (800, 800, 3) ``` -We can see that calling `moveaxis` on an array gives us a rearranged version of the -array given to it - the channel axis that was at the end is now at the front. However, -we can see that the original value of the image is unchanged. This is because by default, -`moveaxis` returns a rearranged **copy** of the image. +We can see that calling `moveaxis` on an array gives us a rearranged version of the array given to it - the channel axis that was at the end is now at the front. However, we can see that the original value of the image is unchanged. This is because by default, `moveaxis` returns a rearranged **copy** of the image. The arguments supplied are: -- The image or array -- The current position of the dimension to move -- The position to move that dimension to +- The image or array +- The current position of the dimension to move +- The position to move that dimension to -In this case, we are moving the dimension at position -1 (i.e. the one at the end) to -position 0 (the start). +In this case, we are moving the dimension at position -1 (i.e. the one at the end) to position 0 (the start). ### Pixel size To make real-world measurements, we need the image's pixel size. While older methods required digging through complex metadata dictionaries, BIOIO makes this simple: -```python + +``` python # Access the physical pixel sizes directly from the bioio object print(img.physical_pixel_sizes) # Returns physical sizes (e.g., X=0.1, Y=0.1, Z=0.5) usually in micrometres. ``` -::::::::::::::::::::::::::::::::::::: keypoints -Standard image formats (TIFF, PNG) can be loaded as NumPy arrays using skimage. -Proprietary microscope formats are best handled by BIOIO to preserve dimensions and metadata. -Basic metrics like histograms, shape, and pixel ranges help determine the best analysis strategy. -Lookup tables (LUTs) change how data is rendered visually but do not change the underlying pixel values. -Lazy loading (via Dask in BIOIO) allows you to explore massive datasets without overwhelming your computer's memory. -:::::::::::::::::::::::::::::::::::::::::::::::: +::: keypoints +Standard image formats (TIFF, PNG) can be loaded as NumPy arrays using skimage. Proprietary microscope formats are best handled by BIOIO to preserve dimensions and metadata. Basic metrics like histograms, shape, and pixel ranges help determine the best analysis strategy. Lookup tables (LUTs) change how data is rendered visually but do not change the underlying pixel values. Lazy loading (via Dask in BIOIO) allows you to explore massive datasets without overwhelming your computer's memory. +::: From 445b56b875d8fa0fca83c404a7e51f954eba9911 Mon Sep 17 00:00:00 2001 From: lauracmurphy Date: Thu, 30 Apr 2026 12:14:45 +0100 Subject: [PATCH 3/5] Update .gitignore --- .gitignore | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 354578b..ac4a8e0 100644 --- a/.gitignore +++ b/.gitignore @@ -52,4 +52,5 @@ po/*~ # renv sandbox needed for renv version 0.16.0 renv/sandbox # OPTIONAL -.DS_Store \ No newline at end of file +.DS_Store +*.Rproj From 7ed9d2256b3c934c9cde1854c198460027c35874 Mon Sep 17 00:00:00 2001 From: Laura Murphy Date: Thu, 30 Apr 2026 12:19:27 +0100 Subject: [PATCH 4/5] infrastructure: update upload-artifact to v4 in PR workflows --- .github/workflows/pr-close-signal.yaml | 2 +- .github/workflows/pr-receive.yaml | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/pr-close-signal.yaml b/.github/workflows/pr-close-signal.yaml index 9b129d5..d20a299 100644 --- a/.github/workflows/pr-close-signal.yaml +++ b/.github/workflows/pr-close-signal.yaml @@ -16,7 +16,7 @@ jobs: mkdir -p ./pr printf ${{ github.event.number }} > ./pr/NUM - name: Upload Diff - uses: actions/upload-artifact@v3 + uses: actions/upload-artifact@v4 with: name: pr path: ./pr diff --git a/.github/workflows/pr-receive.yaml b/.github/workflows/pr-receive.yaml index 371ef54..d78f576 100644 --- a/.github/workflows/pr-receive.yaml +++ b/.github/workflows/pr-receive.yaml @@ -25,7 +25,7 @@ jobs: - name: "Upload PR number" id: upload if: ${{ always() }} - uses: actions/upload-artifact@v3 + uses: actions/upload-artifact@v4 with: name: pr path: ${{ github.workspace }}/NR @@ -107,20 +107,20 @@ jobs: shell: Rscript {0} - name: "Upload PR" - uses: actions/upload-artifact@v3 + uses: actions/upload-artifact@v4 with: name: pr path: ${{ env.PR }} - name: "Upload Diff" - uses: actions/upload-artifact@v3 + uses: actions/upload-artifact@v4 with: name: diff path: ${{ env.CHIVE }} retention-days: 1 - name: "Upload Build" - uses: actions/upload-artifact@v3 + uses: actions/upload-artifact@v4 with: name: built path: ${{ env.MD }} From ebda8908b799e7a0293c4acfb9cf964525cb9945 Mon Sep 17 00:00:00 2001 From: Laura Murphy Date: Thu, 30 Apr 2026 12:25:56 +0100 Subject: [PATCH 5/5] infrastructure: upgrade to latest Carpentries workflows --- .github/workflows/README.md | 318 +++++++++++-------- .github/workflows/docker_apply_cache.yaml | 229 +++++++++++++ .github/workflows/docker_build_deploy.yaml | 161 ++++++++++ .github/workflows/docker_pr_receive.yaml | 283 +++++++++++++++++ .github/workflows/pr-close-signal.yaml | 5 +- .github/workflows/pr-comment.yaml | 91 ++++-- .github/workflows/pr-post-remove-branch.yaml | 2 +- .github/workflows/pr-receive.yaml | 131 -------- .github/workflows/sandpaper-main.yaml | 61 ---- .github/workflows/sandpaper-version.txt | 1 - .github/workflows/update-cache.yaml | 147 ++++++--- .github/workflows/update-workflows.yaml | 98 ++++-- .github/workflows/workflows-version.txt | 1 + 13 files changed, 1108 insertions(+), 420 deletions(-) create mode 100644 .github/workflows/docker_apply_cache.yaml create mode 100644 .github/workflows/docker_build_deploy.yaml create mode 100644 .github/workflows/docker_pr_receive.yaml delete mode 100644 .github/workflows/pr-receive.yaml delete mode 100644 .github/workflows/sandpaper-main.yaml delete mode 100644 .github/workflows/sandpaper-version.txt create mode 100644 .github/workflows/workflows-version.txt diff --git a/.github/workflows/README.md b/.github/workflows/README.md index d6edf88..59a4861 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,198 +1,260 @@ -# Carpentries Workflows +# Workflow Documentation -This directory contains workflows to be used for Lessons using the {sandpaper} -lesson infrastructure. Two of these workflows require R (`sandpaper-main.yaml` -and `pr-recieve.yaml`) and the rest are bots to handle pull request management. +## Managing Workflow Updates -These workflows will likely change as {sandpaper} evolves, so it is important to -keep them up-to-date. To do this in your lesson you can do the following in your -R console: +By using prebuilt Docker containers that are managed by the Carpentries core Workbench maintainers, these workflows are designed to be rarely updated. + +However, is important to be able to keep them up-to-date when appropriate. +You can do this locally using your own R and Workbench installation, or via the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) GitHub Action. + +### Updating locally + +In a terminal/git bash, navigate to the lesson folder where you want to update the workflows. + +Then, start an R session and: ```r # Install/Update sandpaper -options(repos = c(carpentries = "https://carpentries.r-universe.dev/", - CRAN = "https://cloud.r-project.org")) +options(repos = c(carpentries = "https://carpentries.r-universe.dev/", CRAN = "https://cloud.r-project.org")) install.packages("sandpaper") # update the workflows in your lesson library("sandpaper") -update_github_workflows() +sandpaper::update_github_workflows() +quit() +``` + +And then in a bash prompt/git bash terminal: + +```bash +$ git add .github/workflows +$ git commit -m "Manual update to docker workflows" +$ git push origin main ``` -Inside this folder, you will find a file called `sandpaper-version.txt`, which -will contain a version number for sandpaper. This will be used in the future to -alert you if a workflow update is needed. +> [!NOTE] +> For non-renv lessons, this is all the setup you need! +> +> For renv-enabled lessons: +> - Cancel any "01 Maintain: Build and Deploy Site" workflow currently running +> - Run the "02 Maintain: Check for Updated Packages" workflow and merge any PR opened to update the renv lockfile +> - This should automatically run the "03 Maintain: Apply Package Cache" workflow to install packages and build the cache +> - A successful cache buid should then trigger the "01 Maintain: Build and Deploy Site" workflow -What follows are the descriptions of the workflow files: +### Updating using GitHub -## Deployment +#### Official lessons -### 01 Build and Deploy (sandpaper-main.yaml) +"Official" lessons are those in the lesson program repositories, Incubator, or Lab. +They need no extra setup as this is all managed for you as part of the Carpentries GitHub organisations. -This is the main driver that will only act on the main branch of the repository. -This workflow does the following: +To update the workflows, either: +- wait for the scheduled run of the "04 Maintain: Update Workflow Files" at approximately midnight every Tuesday +- go to the Actions tab on GitHub, click "04 Maintain: Update Workflow Files" on the left, then "Run Workflow" on the right - 1. checks out the lesson - 2. provisions the following resources - - R - - pandoc - - lesson infrastructure (stored in a cache) - - lesson dependencies if needed (stored in a cache) - 3. builds the lesson via `sandpaper:::ci_deploy()` +Once complete, this will raise a PR with any changes to the workflows that are needed. +If you are happy with the changes made, you can merge the PR into your lesson repository. -#### Caching +#### Your own lessons -This workflow has two caches; one cache is for the lesson infrastructure and -the other is for the the lesson dependencies if the lesson contains rendered -content. These caches are invalidated by new versions of the infrastructure and -the `renv.lock` file, respectively. If there is a problem with the cache, -manual invaliation is necessary. You will need maintain access to the repository -and you can either go to the actions tab and [click on the caches button to find -and invalidate the failing cache](https://github.blog/changelog/2022-10-20-manage-caches-in-your-actions-workflows-from-web-interface/) -or by setting the `CACHE_VERSION` secret to the current date (which will -invalidate all of the caches). +This presumes you: + - already have a lesson repository available on GitHub + - have enabled workflows in the lesson repo + - have set up a SANDPAPER_WORKFLOW personal access token (PAT) in the lesson repo -## Updates +To go through these steps, please follow the [Forking a Workbench Lesson](https://docs.carpentries.org/resources/curriculum/lesson-forks.html#forking-a-workbench-lesson-repository) +documentation. -### Setup Information +Once set up, run the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) action. -These workflows run on a schedule and at the maintainer's request. Because they -create pull requests that update workflows/require the downstream actions to run, -they need a special repository/organization secret token called -`SANDPAPER_WORKFLOW` and it must have the `public_repo` and `workflow` scope. +This will raise a PR with any changes to the workflows that are needed. +If you are happy with the changes made, you can merge the PR into your lesson repository. -This can be an individual user token, OR it can be a trusted bot account. If you -have a repository in one of the official Carpentries accounts, then you do not -need to worry about this token being present because the Carpentries Core Team -will take care of supplying this token. -If you want to use your personal account: you can go to - -to create a token. Once you have created your token, you should copy it to your -clipboard and then go to your repository's settings > secrets > actions and -create or edit the `SANDPAPER_WORKFLOW` secret, pasting in the generated token. +## Package Caches for RMarkdown Lessons -If you do not specify your token correctly, the runs will not fail and they will -give you instructions to provide the token for your repository. +In summary, generating a reusable package cache is achieved by running the "02 Maintain: Check for Updated Packages" workflow, and then the "03 Maintain: Apply Package Cache" workflow. -### 02 Maintain: Update Workflow Files (update-workflow.yaml) +> [!NOTE] +> Caching is only relevant for lessons that use Rmd files and renv to manage R packages. +> If you are building basic markdown documents, caching will not apply to you, and the only +> workflow that needs to be run is "01 Maintain: Build and Deploy Site". -The {sandpaper} repository was designed to do as much as possible to separate -the tools from the content. For local builds, this is absolutely true, but -there is a minor issue when it comes to workflow files: they must live inside -the repository. +### Caching -This workflow ensures that the workflow files are up-to-date. The way it work is -to download the update-workflows.sh script from GitHub and run it. The script -will do the following: +The two cache management workflows are separated to ensure that once you have a successful build with a working renv cache, this cache is stored and will be reused by the Workbench Docker container. +This means that lesson builds will be faster once an renv cache is created and reused by the Docker container. -1. check the recorded version of sandpaper against the current version on github -2. update the files if there is a difference in versions +Another major bonus of this setup is that you can keep using this cache indefinitely to build your lesson. +This is important if you need very specific versions of R packages ("pinning"). -After the files are updated, if there are any changes, they are pushed to a -branch called `update/workflows` and a pull request is created. Maintainers are -encouraged to review the changes and accept the pull request if the outputs -are okay. +If and when you want to perform an update to the cache, you can re-run the "02 Maintain: Check for Updated Packages" and verify that your lesson still builds with the new packages. +If all looks good, re-run the "03 Maintain: Apply Package Cache" workflow, and this will write a new renv cache file to GitHub. -This update is run ~~weekly or~~ on demand. +In any case, the renv cache is invalidated by new versions of the `renv.lock` file. +This happens: + - if you update your lockfile locally by using the `sandpaper::update_cache()` function, and then push it to the lesson repository + - when you run the "02 Maintain: Check for Updated Packages" and there are new packages to install -### 03 Maintain: Update Package Cache (update-cache.yaml) +More information on managing local renv caches for lessons can be found in the [Sandpaper packages vignettes](https://carpentries.github.io/sandpaper/articles/building-with-renv.html). -For lessons that have generated content, we use {renv} to ensure that the output -is stable. This is controlled by a single lockfile which documents the packages -needed for the lesson and the version numbers. This workflow is skipped in -lessons that do not have generated content. +#### Using different package cache versions -Because the lessons need to remain current with the package ecosystem, it's a -good idea to make sure these packages can be updated periodically. The -update cache workflow will do this by checking for updates, applying them in a -branch called `updates/packages` and creating a pull request with _only the -lockfile changed_. +There are times when you may want to go back to a previous renv package cache file: + - if you run "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" and the cache generation fails for some reason + - if there is a new R package that produces incorrect or broken lesson output + +Cache files will have the following name format, where IMAGE is the workbench-docker image version, and HASHSUM is the `renv.lock` lockfile MD5 hash: + +``` +IMAGE HASHSUM +[ | ] [ | ] +v0.2.4_renv-2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4 +``` + +Copy the hashsum part of the desired cache file you want to use, e.g. `2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4`. + +Then either: + 1. Add a repository variable called CACHE_VERSION, and paste in the hash + - Go to ... + 2. Run the "01 Maintain: Build and Deploy Site" manually, supplying the CACHE_VERSION input + - Go to ... + +If you have no caches listed, make sure to run the "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" to create a new renv cache file. + +> [!NOTE] +> If you are maintaining an official lesson, caches are saved in an AWS S3 bucket owned by the Carpentries. +> Once a successful cache has been saved, these will be listed in the outputs of the "01 Maintain: Build and Deploy Site" workflow. +> +> If you are developing a lesson in your own repository, caches are saved on GitHub. +> You can see available caches by going to the Actions tab, and clicking Caches on the left hand side. + + +## User Settings + +Input level variables are documented in the `carpentries/actions` repository READMEs for each composite action. + +Specific repository level variables can be set that will force particular options across all workflow runs. + +### 01 Maintain: Build and Deploy Site (docker_build_deploy.yaml) + +Repository-level variables for this workflow are: +- WORKBENCH_TAG + - The workbench-docker release version to use for a given build + - This can be set to a specific version number to force all builds to use a given container version + - Default is unset or `latest` +- BUILD_RESET + - Force a reset of previously build markdown files + - Setting this variable value to `true` will force sandpaper to delete any previously build markdown files + - Default is unset or `false` +- AUTO_MERGE_WORKBENCH_VERSION_UPDATE + - Control merge behaviour of the workbench-docker version update PR + - When a new workbench Docker image version is detected, usually after a sandpaper, varnish, or pegboard update, its version number will be incremented + - If a newer version is available, a PR will be auto-generated that updates the `.github/workbench-docker-version.txt` file, and this PR will be auto-merged + - To not auto-merge this PR and to choose when to update the Docker version used, set this to `false`. + - Default is unset or `true` +- LANG_CODE + - Two-letter language code that triggers the use of Joel Nitta's {dovetail} package for lesson translation + - This is used in the internationalisation repos of the main Carpentry lesson programs + - Default is unset or `''` + +### 02 Maintain: Check for Updated Packages (update-cache.yaml) + +Repository-level variables for this workflow are: +- LOCKFILE_CACHE_GEN + - Passed to the `generate-cache` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action + - A temporary renv cache is generated when this workflow runs + - If this option is set to `false`, no temporary cache will be generated + - Default is `true` +- FORCE_RENV_INIT + - Passed to the `force-renv-init` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action + - renv initialises a cache based on a given lockfile + - If this lockfile is particularly old or packages have broken/unresolvable dependencies, then builds will fail + - If this option is set to `true`, a full renv reinitialisation will occur, "wiping the slate clean" + - This option is useful if you're using Bioconductor packages which often break when new Bioconductor releases happen + - Default is `false` +- UPDATE_PACKAGES + - Passed to the `update` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action + - If set to `false` only package hydration will happen and no package update checks will occur + - Default is `true` + +### 03 Maintain: Apply Package Cache (docker_apply_cache.yaml) + +Repository-level variables for this workflow are: +- WORKBENCH_TAG + - The workbench-docker release version to use for a given build + - This can be set to a specific version number to force all builds to use a given container version + - Default is unset or `latest` + + +### 04 Maintain: Update Workflow Files (update-workflows.yaml) + +There are no repository variables for this workflow. -From here, the markdown documents will be rebuilt and you can inspect what has -changed based on how the packages have updated. ## Pull Request and Review Management -Because our lessons execute code, pull requests are a secruity risk for any -lesson and thus have security measures associted with them. **Do not merge any -pull requests that do not pass checks and do not have bots commented on them.** +Because our lessons execute code, pull requests are a security risk for any lesson and thus have security measures associted with them. +**Do not merge any pull requests that do not pass checks and do not have bots commented on them.** -This series of workflows all go together and are described in the following -diagram and the below sections: +This series of workflows all go together and are described in the following diagram and the below sections: ![Graph representation of a pull request](https://carpentries.github.io/sandpaper/articles/img/pr-flow.dot.svg) ### Pre Flight Pull Request Validation (pr-preflight.yaml) -This workflow runs every time a pull request is created and its purpose is to -validate that the pull request is okay to run. This means the following things: +This workflow runs every time a pull request is created and its purpose is to validate that the pull request is okay to run. +This means the following things: 1. The pull request does not contain modified workflow files -2. If the pull request contains modified workflow files, it does not contain - modified content files (such as a situation where @carpentries-bot will - make an automated pull request) -3. The pull request does not contain an invalid commit hash (e.g. from a fork - that was made before a lesson was transitioned from styles to use the - workbench). +2. If the pull request contains modified workflow files, it does not contain modified content files + (such as a situation where @carpentries-bot will make an automated pull request) +3. The pull request does not contain an invalid commit hash + (e.g. from a fork that was made before a lesson was transitioned from styles to use the Workbench). -Once the checks are finished, a comment is issued to the pull request, which -will allow maintainers to determine if it is safe to run the -"Receive Pull Request" workflow from new contributors. +Once the checks are finished, a comment is issued to the pull request, which will allow maintainers to determine if it is safe to run the "Receive Pull Request" workflow from new contributors. -### Recieve Pull Request (pr-recieve.yaml) +### Receive Pull Request (docker_pr_receive.yaml) -**Note of caution:** This workflow runs arbitrary code by anyone who creates a -pull request. GitHub has safeguarded the token used in this workflow to have no -priviledges in the repository, but we have taken precautions to protect against -spoofing. +**Note of caution:** This workflow runs arbitrary code by anyone who creates a pull request. +GitHub has safeguarded the token used in this workflow to have no privileges in the repository, but we have taken precautions to protect against spoofing. -This workflow is triggered with every push to a pull request. If this workflow -is already running and a new push is sent to the pull request, the workflow -running from the previous push will be cancelled and a new workflow run will be -started. +This workflow is triggered with every push to a pull request. +If this workflow is already running and a new push is sent to the pull request, the workflow running from the previous push will be cancelled and a new workflow run will be started. -The first step of this workflow is to check if it is valid (e.g. that no -workflow files have been modified). If there are workflow files that have been -modified, a comment is made that indicates that the workflow is not run. If -both a workflow file and lesson content is modified, an error will occurr. +The first step of this workflow is to check if it is valid (e.g. that no workflow files have been modified): +- If there are workflow files that have been modified, a comment is made that indicates that the workflow will not continue. +- If both a workflow file and lesson content is modified, an error will occur and the workflow will not continue. -The second step (if valid) is to build the generated content from the pull -request. This builds the content and uploads three artifacts: +The second step (if valid) is to build the generated content from the pull request. +This builds the content and uploads three artifacts: 1. The pull request number (pr) 2. A summary of changes after the rendering process (diff) 3. The rendered files (build) -Because this workflow builds generated content, it follows the same general -process as the `sandpaper-main` workflow with the same caching mechanisms. - -The artifacts produced are used by the next workflow. +The artifacts produced are used by the "Comment on Pull Request" workflow. ### Comment on Pull Request (pr-comment.yaml) -This workflow is triggered if the `pr-recieve.yaml` workflow is successful. +This workflow is triggered if the `docker_pr_receive.yaml` workflow is successful. The steps in this workflow are: -1. Test if the workflow is valid and comment the validity of the workflow to the - pull request. -2. If it is valid: create an orphan branch with two commits: the current state - of the repository and the proposed changes. +1. Test if the workflow is valid and comment the validity of the workflow to the pull request. +2. If it is valid: create an orphan branch with two commits: the current state of the repository and the proposed changes. 3. If it is valid: update the pull request comment with the summary of changes -Importantly: if the pull request is invalid, the branch is not created so any -malicious code is not published. +Importantly: if the pull request is invalid, the branch is not created so any malicious code is not published. -From here, the maintainer can request changes from the author and eventually -either merge or reject the PR. When this happens, if the PR was valid, the -preview branch needs to be deleted. +From here, the maintainer can request changes from the author and eventually either merge or reject the PR. +When this happens, if the PR was valid, the preview branch needs to be deleted. ### Send Close PR Signal (pr-close-signal.yaml) -Triggered any time a pull request is closed. This emits an artifact that is the -pull request number for the next action +Triggered any time a pull request is closed. +This emits an artifact that is the pull request number for the next action. ### Remove Pull Request Branch (pr-post-remove-branch.yaml) -Tiggered by `pr-close-signal.yaml`. This removes the temporary branch associated with -the pull request (if it was created). +Tiggered by `pr-close-signal.yaml`. +This removes the temporary branch associated with the pull request (if it was created). diff --git a/.github/workflows/docker_apply_cache.yaml b/.github/workflows/docker_apply_cache.yaml new file mode 100644 index 0000000..0f3a1ab --- /dev/null +++ b/.github/workflows/docker_apply_cache.yaml @@ -0,0 +1,229 @@ +name: "03 Maintain: Apply Package Cache" +description: "Generate the package cache for the lesson after a pull request has been merged or via manual trigger, and cache in S3 or GitHub" +on: + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + pull_request: + types: + - closed + branches: + - main + +# queue cache runs +concurrency: + group: docker-apply-cache + cancel-in-progress: false + +jobs: + preflight: + name: "Preflight: PR or Manual Trigger?" + runs-on: ubuntu-latest + outputs: + do-apply: ${{ steps.check.outputs.merged_or_manual }} + steps: + - name: "Should we run cache application?" + id: check + run: | + if [[ "${{ github.event_name }}" == "workflow_dispatch" || + ("${{ github.ref }}" == "refs/heads/main" && "${{ github.event.action }}" == "closed" && "${{ github.event.pull_request.merged }}" == "true") ]]; then + echo "merged_or_manual=true" >> $GITHUB_OUTPUT + else + echo "This was not a manual trigger and no PR was merged. No action taken." + echo "merged_or_manual=false" >> $GITHUB_OUTPUT + fi + shell: bash + + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest + needs: preflight + if: needs.preflight.outputs.do-apply == 'true' + permissions: + id-token: write + outputs: + renv-needed: ${{ steps.check-for-renv.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.check-for-renv.outputs.renv-cache-hashsum }} + renv-cache-available: ${{ steps.check-for-renv.outputs.renv-cache-available }} + steps: + - name: "Check for renv" + id: check-for-renv + uses: carpentries/actions/renv-checks@main + with: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG || 'latest' }} + token: ${{ secrets.GITHUB_TOKEN }} + + no-renv-cache-used: + name: "No renv cache used" + runs-on: ubuntu-latest + needs: check-renv + if: needs.check-renv.outputs.renv-needed != 'true' + steps: + - name: "No renv cache needed" + run: echo "No renv cache needed for this lesson" + + renv-cache-available: + name: "renv cache available" + runs-on: ubuntu-latest + needs: check-renv + if: needs.check-renv.outputs.renv-cache-available == 'true' + steps: + - name: "renv cache available" + run: echo "renv cache available for this lesson" + + update-renv-cache: + name: "Update renv Cache" + runs-on: ubuntu-latest + needs: check-renv + if: | + needs.check-renv.outputs.renv-needed == 'true' && + needs.check-renv.outputs.renv-cache-available != 'true' && + ( + github.event_name == 'workflow_dispatch' || + ( + github.event.pull_request.merged == true && + ( + ( + contains( + join(github.event.pull_request.labels.*.name, ','), + 'type: package cache' + ) && + github.event.pull_request.head.ref == 'update/packages' + ) + || + ( + contains( + join(github.event.pull_request.labels.*.name, ','), + 'type: workflows' + ) && + github.event.pull_request.head.ref == 'update/workflows' + ) + || + ( + contains( + join(github.event.pull_request.labels.*.name, ','), + 'type: docker version' + ) && + github.event.pull_request.head.ref == 'update/workbench-docker-version' + ) + ) + ) + ) + permissions: + checks: write + contents: write + pages: write + id-token: write + container: + image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_VERSION: ${{ needs.check-renv.outputs.renv-cache-hashsum }} + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 2 + steps: + - uses: actions/checkout@v6 + + - name: "Debugging Info" + run: | + echo "Current Directory: $(pwd)" + ls -lah /home/rstudio/.workbench + ls -lah $(pwd) + Rscript -e 'sessionInfo()' + shell: bash + + - name: "Mark Repository as Safe" + run: | + git config --global --add safe.directory $(pwd) + shell: bash + + - name: "Ensure sandpaper is loadable" + run: | + .libPaths() + library(sandpaper) + shell: Rscript {0} + + - name: "Setup Lesson Dependencies" + run: | + Rscript /home/rstudio/.workbench/setup_lesson_deps.R + shell: bash + + - name: "Fortify renv Cache" + run: | + Rscript /home/rstudio/.workbench/fortify_renv_cache.R + shell: bash + + - name: "Get Container Version Used" + id: wb-vers + uses: carpentries/actions/container-version@main + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }} + renv-needed: ${{ needs.check-renv.outputs.renv-needed }} + token: ${{ secrets.GITHUB_TOKEN }} + + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + uses: carpentries/actions/validate-org-workflow@main + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: "Configure AWS credentials via OIDC" + id: aws-creds + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + output-credentials: true + + - name: "Upload cache object to S3" + id: upload-cache + uses: tespkg/actions-cache@v1.10.0 + with: + accessKey: ${{ steps.aws-creds.outputs.aws-access-key-id }} + secretKey: ${{ steps.aws-creds.outputs.aws-secret-access-key }} + sessionToken: ${{ steps.aws-creds.outputs.aws-session-token }} + bucket: workbench-docker-caches + path: | + /home/rstudio/lesson/renv + /usr/local/lib/R/site-library + key: ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv-${{ needs.check-renv.outputs.renv-cache-hashsum }} + restore-keys: + ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv- + + record-cache-result: + name: "Record Caching Status" + runs-on: ubuntu-latest + needs: [check-renv, update-renv-cache] + if: always() + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: "Record cache result" + + run: | + echo "${{ needs.update-renv-cache.result == 'success' || needs.check-renv.outputs.renv-cache-available == 'true' || 'false' }}" > ${{ github.workspace }}/apply-cache-result + shell: bash + + - name: "Upload cache result" + uses: actions/upload-artifact@v7 + with: + name: apply-cache-result + path: ${{ github.workspace }}/apply-cache-result diff --git a/.github/workflows/docker_build_deploy.yaml b/.github/workflows/docker_build_deploy.yaml new file mode 100644 index 0000000..4baf306 --- /dev/null +++ b/.github/workflows/docker_build_deploy.yaml @@ -0,0 +1,161 @@ +name: "01 Maintain: Build and Deploy Site" +description: "Build and deploy the lesson site using the carpentries/workbench-docker container" +on: + push: + branches: + - 'main' + - 'l10n_main' + paths-ignore: + - '.github/workflows/**.yaml' + - '.github/workbench-docker-version.txt' + schedule: + - cron: '0 0 * * 2' + workflow_run: + workflows: ["03 Maintain: Apply Package Cache"] + types: + - completed + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + CACHE_VERSION: + description: 'Optional renv cache version override' + required: false + default: '' + reset: + description: 'Reset cached markdown files' + required: true + default: false + type: boolean + force-skip-manage-deps: + description: 'Skip build-time dependency management' + required: true + default: false + type: boolean + +# only one build/deploy at a time +concurrency: + group: docker-build-deploy + cancel-in-progress: true + +jobs: + preflight: + name: "Preflight: Schedule, Push, or PR?" + runs-on: ubuntu-latest + outputs: + do-build: ${{ steps.build-check.outputs.do-build }} + renv-needed: ${{ steps.build-check.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.build-check.outputs.renv-cache-hashsum }} + workbench-container-file-exists: ${{ steps.wb-vers.outputs.workbench-container-file-exists }} + wb-vers: ${{ steps.wb-vers.outputs.container-version }} + last-wb-vers: ${{ steps.wb-vers.outputs.last-container-version }} + workbench-update: ${{ steps.wb-vers.outputs.workbench-update }} + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: "Should we run build and deploy?" + id: build-check + uses: carpentries/actions/build-preflight@main + + - name: "Checkout Lesson" + if: steps.build-check.outputs.do-build == 'true' + uses: actions/checkout@v6 + + - name: "Get container version info" + id: wb-vers + if: steps.build-check.outputs.do-build == 'true' + uses: carpentries/actions/container-version@main + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }} + renv-needed: ${{ steps.build-check.outputs.renv-needed }} + token: ${{ secrets.GITHUB_TOKEN }} + + full-build: + name: "Build Full Site" + runs-on: ubuntu-latest + needs: preflight + if: | + needs.preflight.outputs.do-build == 'true' && + needs.preflight.outputs.workbench-update != 'true' + env: + RENV_EXISTS: ${{ needs.preflight.outputs.renv-needed }} + RENV_HASH: ${{ needs.preflight.outputs.renv-cache-hashsum }} + permissions: + checks: write + contents: write + pages: write + id-token: write + container: + image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 1 + steps: + - uses: actions/checkout@v6 + + - name: "Debugging Info" + run: | + cd /home/rstudio/lesson + echo "Current Directory: $(pwd)" + echo "RENV_HASH is $RENV_HASH" + ls -lah /home/rstudio/.workbench + ls -lah $(pwd) + Rscript -e 'sessionInfo()' + shell: bash + + - name: "Mark Repository as Safe" + run: | + git config --global --add safe.directory $(pwd) + shell: bash + + - name: "Setup Lesson Dependencies" + id: build-container-deps + uses: carpentries/actions/build-container-deps@main + with: + CACHE_VERSION: ${{ vars.CACHE_VERSION || github.event.inputs.CACHE_VERSION || '' }} + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG || 'latest' }} + LESSON_PATH: ${{ vars.LESSON_PATH || '/home/rstudio/lesson' }} + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + token: ${{ secrets.GITHUB_TOKEN }} + + - name: "Run Container and Build Site" + id: build-and-deploy + uses: carpentries/actions/build-and-deploy@main + with: + reset: ${{ vars.BUILD_RESET || github.event.inputs.reset || 'false' }} + skip-manage-deps: ${{ github.event.inputs.force-skip-manage-deps == 'true' || steps.build-container-deps.outputs.renv-cache-available || steps.build-container-deps.outputs.backup-cache-used || 'false' }} + lang-code: ${{ vars.LANG_CODE || '' }} + + update-container-version: + name: "Update container version used" + runs-on: ubuntu-latest + needs: [preflight] + permissions: + actions: write + contents: write + pull-requests: write + id-token: write + if: | + needs.preflight.outputs.do-build == 'true' && + ( + needs.preflight.outputs.workbench-container-file-exists == 'false' || + needs.preflight.outputs.workbench-update == 'true' + ) + steps: + - name: "Record container version used" + uses: carpentries/actions/record-container-version@main + with: + CONTAINER_VER: ${{ needs.preflight.outputs.wb-vers }} + AUTO_MERGE: ${{ vars.AUTO_MERGE_CONTAINER_VERSION_UPDATE || 'true' }} + token: ${{ secrets.GITHUB_TOKEN }} + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} diff --git a/.github/workflows/docker_pr_receive.yaml b/.github/workflows/docker_pr_receive.yaml new file mode 100644 index 0000000..486b4b4 --- /dev/null +++ b/.github/workflows/docker_pr_receive.yaml @@ -0,0 +1,283 @@ +name: "Bot: Receive Pull Request" +description: "Receive a pull request and build the markdown source files" +on: + pull_request: + types: + [opened, synchronize, reopened] + workflow_dispatch: + inputs: + pr_number: + type: number + required: true + +concurrency: + group: ${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + pull-requests: write + +jobs: + preflight: + name: "Preflight: md-outputs exists?" + runs-on: ubuntu-latest + outputs: + branch-exists: ${{ steps.check.outputs.exists }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v6 + + - name: "Check if md-outputs branch exists" + id: check + run: | + # 💡 Checking for md-outputs branch # + if [[ -n $(git ls-remote --exit-code --heads origin md-outputs) ]]; then + echo "exists=true" >> $GITHUB_OUTPUT + else + echo "exists=false" >> $GITHUB_OUTPUT + echo "::error::md-outputs branch required but does not exist." + echo "::error::Please merge any open package update PRs to trigger the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows." + + echo "## ❌ ERROR: md-outputs branch required" >> $GITHUB_STEP_SUMMARY + echo "Please merge any open package update PRs to trigger the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows." >> $GITHUB_STEP_SUMMARY + + exit 1 + fi + shell: bash + + test-pr: + name: "Record PR number" + if: | + github.event.action != 'closed' && + needs.preflight.outputs.branch-exists == 'true' + runs-on: ubuntu-latest + needs: preflight + outputs: + is_valid: ${{ steps.check-pr.outputs.VALID }} + pr_number: ${{ env.NR }} + pr_branch: ${{ env.PR_BRANCH }} + steps: + - name: "Grab PR" + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + if [[ "${{ github.event_name }}" == "pull_request" ]] ; then + PR_NUMBER=${{ github.event.number }} + elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]] ; then + PR_NUMBER=${{ inputs.pr_number }} + fi + + echo $PR_NUMBER > ${{ github.workspace }}/NR + echo "NR=$PR_NUMBER" >> $GITHUB_ENV + echo "PR_BRANCH=$(gh -R ${{ github.repository }} pr view $PR_NUMBER --json headRefName --jq '.headRefName')" >> $GITHUB_ENV + shell: bash + + - name: "Upload PR number" + id: upload + if: always() + uses: actions/upload-artifact@v7 + with: + name: pr + path: ${{ github.workspace }}/NR + + - name: "Get Invalid Hashes File" + id: hash + run: | + echo "json<> $GITHUB_OUTPUT + shell: bash + + - name: "Debug Hashes Output" + run: | + echo "${{ steps.hash.outputs.json }}" + shell: bash + + - name: "Check PR" + id: check-pr + uses: carpentries/actions/check-valid-pr@main + with: + pr: ${{ env.NR }} + invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} + + check-renv: + name: "Check If We Need {renv}" + runs-on: ubuntu-latest + outputs: + renv-needed: ${{ steps.renv-check.outputs.renv-needed }} + renv-cache-hashsum: ${{ steps.renv-check.outputs.renv-cache-hashsum }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v6 + + - name: "Is renv required?" + id: renv-check + uses: carpentries/actions/renv-checks@main + with: + CACHE_VERSION: ${{ inputs.CACHE_VERSION || '' }} + skip-cache-check: true + + build-md-source: + name: "Build markdown source files if valid" + needs: + - test-pr + - check-renv + runs-on: ubuntu-latest + if: needs.test-pr.outputs.is_valid == 'true' + env: + CHIVE: ${{ github.workspace }}/site/chive + PR: ${{ github.workspace }}/site/pr + GHWMD: ${{ github.workspace }}/site/built + PR_BRANCH: ${{ needs.test-pr.outputs.pr_branch }} + PR_NUMBER: ${{ needs.test-pr.outputs.pr_number }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + permissions: + checks: write + contents: write + pages: write + id-token: write + container: + image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }} + env: + WORKBENCH_PROFILE: "ci" + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: /home/rstudio/lesson/renv + RENV_PROFILE: "lesson-requirements" + RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library" + volumes: + - ${{ github.workspace }}:/home/rstudio/lesson + options: --cpus 2 + outputs: + workbench-update: ${{ steps.wb-vers.outputs.workbench-update }} + build-site: ${{ steps.build-site.outcome }} + steps: + - uses: actions/checkout@v6 + + - name: "Check Out Staging Branch" + uses: actions/checkout@v6 + with: + ref: md-outputs + path: ${{ env.GHWMD }} + + - name: Mark Repository as Safe + run: | + git config --global --add safe.directory $(pwd) + git config --global --add safe.directory /home/rstudio/lesson + shell: bash + + - name: "Ensure sandpaper is loadable" + run: | + .libPaths() + library(sandpaper) + shell: Rscript {0} + + - name: Setup Lesson Dependencies + run: | + Rscript /home/rstudio/.workbench/setup_lesson_deps.R + shell: bash + + - name: Get Container Version Used + id: wb-vers + if: needs.check-renv.outputs.renv-needed == 'true' + uses: carpentries/actions/container-version@main + with: + WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }} + renv-needed: ${{ needs.check-renv.outputs.renv-needed }} + token: ${{ secrets.GITHUB_TOKEN }} + + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + if: needs.check-renv.outputs.renv-needed == 'true' + uses: carpentries/actions/validate-org-workflow@main + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: Configure AWS credentials via OIDC + id: aws-creds + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + needs.check-renv.outputs.renv-needed == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + output-credentials: true + + - name: Get cache object from S3 + id: s3-cache + uses: tespkg/actions-cache/restore@v1.10.0 + if: needs.check-renv.outputs.renv-needed == 'true' + with: + # insecure: false # optional, use http instead of https. default false + accessKey: ${{ steps.aws-creds.outputs.aws-access-key-id }} + secretKey: ${{ steps.aws-creds.outputs.aws-secret-access-key }} + sessionToken: ${{ steps.aws-creds.outputs.aws-session-token }} + bucket: workbench-docker-caches + path: | + /home/rstudio/lesson/renv + /usr/local/lib/R/site-library + key: ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv-${{ needs.check-renv.outputs.renv-cache-hashsum }} + restore-keys: + ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv- + + - name: "Fortify renv Cache" + if: | + needs.check-renv.outputs.renv-needed == 'true' && + steps.s3-cache.outputs.cache-hit != 'true' + run: | + Rscript /home/rstudio/.workbench/fortify_renv_cache.R + shell: bash + + - name: "Validate and Build Markdown" + id: build-site + run: | + sandpaper::package_cache_trigger(TRUE) + sandpaper::validate_lesson(path = '/home/rstudio/lesson') + sandpaper:::build_markdown(path = '/home/rstudio/lesson', quiet = FALSE) + shell: Rscript {0} + + - name: "Generate Artifacts" + id: generate-artifacts + run: | + sandpaper:::ci_bundle_pr_artifacts( + repo = '${{ github.repository }}', + pr_number = '${{ env.PR_NUMBER }}', + path_md = '/home/rstudio/lesson/site/built', + path_pr = '/home/rstudio/lesson/site/pr', + path_archive = '/home/rstudio/lesson/site/chive', + branch = 'md-outputs' + ) + shell: Rscript {0} + + - name: "Upload PR" + uses: actions/upload-artifact@v7 + with: + name: pr + path: ${{ env.PR }} + overwrite: true + + - name: "Upload Diff" + uses: actions/upload-artifact@v7 + with: + name: diff + path: ${{ env.CHIVE }} + retention-days: 1 + + - name: "Upload Build" + uses: actions/upload-artifact@v7 + with: + name: built + path: ${{ env.GHWMD }} + retention-days: 1 + + - name: "Teardown" + run: sandpaper::reset_site() + shell: Rscript {0} diff --git a/.github/workflows/pr-close-signal.yaml b/.github/workflows/pr-close-signal.yaml index d20a299..de1f254 100644 --- a/.github/workflows/pr-close-signal.yaml +++ b/.github/workflows/pr-close-signal.yaml @@ -8,7 +8,7 @@ on: jobs: send-close-signal: name: "Send closing signal" - runs-on: ubuntu-latest + runs-on: ubuntu-22.04 if: ${{ github.event.action == 'closed' }} steps: - name: "Create PRtifact" @@ -16,8 +16,7 @@ jobs: mkdir -p ./pr printf ${{ github.event.number }} > ./pr/NUM - name: Upload Diff - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: pr path: ./pr - diff --git a/.github/workflows/pr-comment.yaml b/.github/workflows/pr-comment.yaml index bb2eb03..9ec78c6 100644 --- a/.github/workflows/pr-comment.yaml +++ b/.github/workflows/pr-comment.yaml @@ -1,36 +1,26 @@ name: "Bot: Comment on the Pull Request" - -# read-write repo token -# access to secrets +description: "Comment on the pull request with the results of the markdown generation" on: workflow_run: - workflows: ["Receive Pull Request"] + workflows: ["Bot: Receive Pull Request"] types: - completed -concurrency: - group: pr-${{ github.event.workflow_run.pull_requests[0].number }} - cancel-in-progress: true - - jobs: # Pull requests are valid if: # - they match the sha of the workflow run head commit # - they are open - # - no .github files were committed + # - no .github files were committed, except for .github/workbench-docker-version.txt test-pr: name: "Test if pull request is valid" runs-on: ubuntu-latest - if: > - github.event.workflow_run.event == 'pull_request' && - github.event.workflow_run.conclusion == 'success' outputs: is_valid: ${{ steps.check-pr.outputs.VALID }} payload: ${{ steps.check-pr.outputs.payload }} number: ${{ steps.get-pr.outputs.NUM }} msg: ${{ steps.check-pr.outputs.MSG }} steps: - - name: 'Download PR artifact' + - name: "Download PR artifact" id: dl uses: carpentries/actions/download-workflow-artifact@main with: @@ -50,12 +40,44 @@ jobs: run: | echo '::error::A pull request number was not recorded. The pull request that triggered this workflow is likely malicious.' exit 1 + + - name: "Checkout Lesson" + uses: actions/checkout@v6 + + - name: "Verify committed files" + id: changed-files + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + ## Get list of changed files in the PR ## + ONLY_VERSION=$(gh pr view ${{ steps.get-pr.outputs.NUM }} --json files --jq ' + .files | + length == 1 and + .[0].path == ".github/workbench-docker-version.txt" + ') + + if [[ "$ONLY_VERSION" == "true" ]]; then + echo "only_version_file=true" >> $GITHUB_OUTPUT + else + echo "only_version_file=false" >> $GITHUB_OUTPUT + fi + shell: bash + + - name: "Skip checks for Workbench version file updates" + if: steps.changed-files.outputs.only_version_file == 'true' + run: | + echo "# 🔧 Wait for Next Cache Update #" + echo "Only workbench-docker-version.txt changed." + exit 0 + shell: bash + - name: "Get Invalid Hashes File" id: hash run: | echo "json<> $GITHUB_OUTPUT + - name: "Check PR" id: check-pr if: ${{ steps.dl.outputs.success == 'true' }} @@ -67,6 +89,14 @@ jobs: invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} fail_on_error: true + - name: "Comment result of validation" + id: comment-diff + if: always() + uses: carpentries/actions/comment-diff@main + with: + pr: ${{ steps.get-pr.outputs.NUM }} + body: ${{ steps.check-pr.outputs.MSG }} + # Create an orphan branch on this repository with two commits # - the current HEAD of the md-outputs branch # - the output from running the current HEAD of the pull request through @@ -75,31 +105,31 @@ jobs: name: "Create Git Branch" needs: test-pr runs-on: ubuntu-latest - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + if: needs.test-pr.outputs.is_valid == 'true' env: NR: ${{ needs.test-pr.outputs.number }} permissions: contents: write steps: - - name: 'Checkout md outputs' - uses: actions/checkout@v3 + - name: "Checkout md outputs" + uses: actions/checkout@v6 with: ref: md-outputs path: built fetch-depth: 1 - - name: 'Download built markdown' + - name: "Download built markdown" id: dl uses: carpentries/actions/download-workflow-artifact@main with: run: ${{ github.event.workflow_run.id }} name: 'built' - - if: ${{ steps.dl.outputs.success == 'true' }} + - if: steps.dl.outputs.success == 'true' run: unzip built.zip - name: "Create orphan and push" - if: ${{ steps.dl.outputs.success == 'true' }} + if: steps.dl.outputs.success == 'true' run: | cd built/ git config --local user.email "actions@github.com" @@ -121,25 +151,25 @@ jobs: name: "Comment on Pull Request" needs: [test-pr, create-branch] runs-on: ubuntu-latest - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + if: needs.test-pr.outputs.is_valid == 'true' env: NR: ${{ needs.test-pr.outputs.number }} permissions: pull-requests: write steps: - - name: 'Download comment artifact' + - name: "Download comment artifact" id: dl uses: carpentries/actions/download-workflow-artifact@main with: run: ${{ github.event.workflow_run.id }} name: 'diff' - - if: ${{ steps.dl.outputs.success == 'true' }} + - if: steps.dl.outputs.success == 'true' run: unzip ${{ github.workspace }}/diff.zip - name: "Comment on PR" id: comment-diff - if: ${{ steps.dl.outputs.success == 'true' }} + if: steps.dl.outputs.success == 'true' uses: carpentries/actions/comment-diff@main with: pr: ${{ env.NR }} @@ -151,23 +181,25 @@ jobs: name: "Comment if workflow files have changed" needs: test-pr runs-on: ubuntu-latest - if: ${{ always() && needs.test-pr.outputs.is_valid == 'false' }} + if: | + always() && + needs.test-pr.outputs.is_valid == 'false' env: - NR: ${{ github.event.workflow_run.pull_requests[0].number }} + NR: ${{ needs.test-pr.outputs.number }} body: ${{ needs.test-pr.outputs.msg }} permissions: pull-requests: write steps: - - name: 'Check for spoofing' + - name: "Check for spoofing" id: dl uses: carpentries/actions/download-workflow-artifact@main with: run: ${{ github.event.workflow_run.id }} name: 'built' - - name: 'Alert if spoofed' + - name: "Alert if spoofed" id: spoof - if: ${{ steps.dl.outputs.success == 'true' }} + if: steps.dl.outputs.success == 'true' run: | echo 'body<> $GITHUB_ENV echo '' >> $GITHUB_ENV @@ -182,4 +214,3 @@ jobs: with: pr: ${{ env.NR }} body: ${{ env.body }} - diff --git a/.github/workflows/pr-post-remove-branch.yaml b/.github/workflows/pr-post-remove-branch.yaml index 62c2e98..9419e2b 100644 --- a/.github/workflows/pr-post-remove-branch.yaml +++ b/.github/workflows/pr-post-remove-branch.yaml @@ -9,7 +9,7 @@ on: jobs: delete: name: "Delete branch from Pull Request" - runs-on: ubuntu-latest + runs-on: ubuntu-22.04 if: > github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success' diff --git a/.github/workflows/pr-receive.yaml b/.github/workflows/pr-receive.yaml deleted file mode 100644 index d78f576..0000000 --- a/.github/workflows/pr-receive.yaml +++ /dev/null @@ -1,131 +0,0 @@ -name: "Receive Pull Request" - -on: - pull_request: - types: - [opened, synchronize, reopened] - -concurrency: - group: ${{ github.ref }} - cancel-in-progress: true - -jobs: - test-pr: - name: "Record PR number" - if: ${{ github.event.action != 'closed' }} - runs-on: ubuntu-latest - outputs: - is_valid: ${{ steps.check-pr.outputs.VALID }} - steps: - - name: "Record PR number" - id: record - if: ${{ always() }} - run: | - echo ${{ github.event.number }} > ${{ github.workspace }}/NR # 2022-03-02: artifact name fixed to be NR - - name: "Upload PR number" - id: upload - if: ${{ always() }} - uses: actions/upload-artifact@v4 - with: - name: pr - path: ${{ github.workspace }}/NR - - name: "Get Invalid Hashes File" - id: hash - run: | - echo "json<> $GITHUB_OUTPUT - - name: "echo output" - run: | - echo "${{ steps.hash.outputs.json }}" - - name: "Check PR" - id: check-pr - uses: carpentries/actions/check-valid-pr@main - with: - pr: ${{ github.event.number }} - invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} - - build-md-source: - name: "Build markdown source files if valid" - needs: test-pr - runs-on: ubuntu-latest - if: ${{ needs.test-pr.outputs.is_valid == 'true' }} - env: - GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} - RENV_PATHS_ROOT: ~/.local/share/renv/ - CHIVE: ${{ github.workspace }}/site/chive - PR: ${{ github.workspace }}/site/pr - MD: ${{ github.workspace }}/site/built - steps: - - name: "Check Out Main Branch" - uses: actions/checkout@v3 - - - name: "Check Out Staging Branch" - uses: actions/checkout@v3 - with: - ref: md-outputs - path: ${{ env.MD }} - - - name: "Set up R" - uses: r-lib/actions/setup-r@v2 - with: - use-public-rspm: true - install-r: false - - - name: "Set up Pandoc" - uses: r-lib/actions/setup-pandoc@v2 - - - name: "Setup Lesson Engine" - uses: carpentries/actions/setup-sandpaper@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Setup Package Cache" - uses: carpentries/actions/setup-lesson-deps@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Validate and Build Markdown" - id: build-site - run: | - sandpaper::package_cache_trigger(TRUE) - sandpaper::validate_lesson(path = '${{ github.workspace }}') - sandpaper:::build_markdown(path = '${{ github.workspace }}', quiet = FALSE) - shell: Rscript {0} - - - name: "Generate Artifacts" - id: generate-artifacts - run: | - sandpaper:::ci_bundle_pr_artifacts( - repo = '${{ github.repository }}', - pr_number = '${{ github.event.number }}', - path_md = '${{ env.MD }}', - path_pr = '${{ env.PR }}', - path_archive = '${{ env.CHIVE }}', - branch = 'md-outputs' - ) - shell: Rscript {0} - - - name: "Upload PR" - uses: actions/upload-artifact@v4 - with: - name: pr - path: ${{ env.PR }} - - - name: "Upload Diff" - uses: actions/upload-artifact@v4 - with: - name: diff - path: ${{ env.CHIVE }} - retention-days: 1 - - - name: "Upload Build" - uses: actions/upload-artifact@v4 - with: - name: built - path: ${{ env.MD }} - retention-days: 1 - - - name: "Teardown" - run: sandpaper::reset_site() - shell: Rscript {0} diff --git a/.github/workflows/sandpaper-main.yaml b/.github/workflows/sandpaper-main.yaml deleted file mode 100644 index 97eeebf..0000000 --- a/.github/workflows/sandpaper-main.yaml +++ /dev/null @@ -1,61 +0,0 @@ -name: "01 Build and Deploy Site" - -on: - push: - branches: - - main - - master - schedule: - - cron: '0 0 * * 2' - workflow_dispatch: - inputs: - name: - description: 'Who triggered this build?' - required: true - default: 'Maintainer (via GitHub)' - reset: - description: 'Reset cached markdown files' - required: false - default: false - type: boolean -jobs: - full-build: - name: "Build Full Site" - runs-on: ubuntu-latest - permissions: - checks: write - contents: write - pages: write - env: - GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} - RENV_PATHS_ROOT: ~/.local/share/renv/ - steps: - - - name: "Checkout Lesson" - uses: actions/checkout@v3 - - - name: "Set up R" - uses: r-lib/actions/setup-r@v2 - with: - use-public-rspm: true - install-r: true - - - name: "Set up Pandoc" - uses: r-lib/actions/setup-pandoc@v2 - - - name: "Setup Lesson Engine" - uses: carpentries/actions/setup-sandpaper@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Setup Package Cache" - uses: carpentries/actions/setup-lesson-deps@main - with: - cache-version: ${{ secrets.CACHE_VERSION }} - - - name: "Deploy Site" - run: | - reset <- "${{ github.event.inputs.reset }}" == "true" - sandpaper::package_cache_trigger(TRUE) - sandpaper:::ci_deploy(reset = reset) - shell: Rscript {0} diff --git a/.github/workflows/sandpaper-version.txt b/.github/workflows/sandpaper-version.txt deleted file mode 100644 index 54d1a4f..0000000 --- a/.github/workflows/sandpaper-version.txt +++ /dev/null @@ -1 +0,0 @@ -0.13.0 diff --git a/.github/workflows/update-cache.yaml b/.github/workflows/update-cache.yaml index 676d742..d182ac7 100644 --- a/.github/workflows/update-cache.yaml +++ b/.github/workflows/update-cache.yaml @@ -1,26 +1,45 @@ -name: "03 Maintain: Update Package Cache" - +name: "02 Maintain: Check for Updated Packages" +description: "Check for updated R packages and create a pull request to update the lesson's renv lockfile and package cache" on: + schedule: + - cron: '0 0 * * 2' workflow_dispatch: inputs: name: - description: 'Who triggered this build (enter github username to tag yourself)?' + description: 'Who triggered this build?' required: true - default: 'monthly run' - schedule: - # Run every tuesday - - cron: '0 0 * * 2' + default: 'Maintainer (via GitHub)' + force-renv-init: + description: 'Force full lockfile update?' + required: false + default: false + type: boolean + update-packages: + description: 'Install any package updates?' + required: false + default: true + type: boolean + generate-cache: + description: 'Generate separate package cache?' + required: false + default: false + type: boolean + +env: + LOCKFILE_CACHE_GEN: ${{ vars.LOCKFILE_CACHE_GEN || github.event.inputs.generate-cache || 'false' }} + FORCE_RENV_INIT: ${{ vars.FORCE_RENV_INIT || github.event.inputs.force-renv-init || 'false' }} + UPDATE_PACKAGES: ${{ vars.UPDATE_PACKAGES || github.event.inputs.update-packages || 'true' }} jobs: preflight: - name: "Preflight Check" + name: "Preflight: Manual or Scheduled Trigger?" runs-on: ubuntu-latest outputs: ok: ${{ steps.check.outputs.ok }} steps: - id: check run: | - if [[ ${{ github.event_name }} == 'workflow_dispatch' ]]; then + if [[ "${{ github.event_name }}" == 'workflow_dispatch' ]]; then echo "ok=true" >> $GITHUB_OUTPUT echo "Running on request" # using single brackets here to avoid 08 being interpreted as octal @@ -33,50 +52,44 @@ jobs: echo "ok=false" >> $GITHUB_OUTPUT echo "Not Running Today" fi + shell: bash - check_renv: - name: "Check if We Need {renv}" + check-renv: + name: "Check If We Need {renv}" runs-on: ubuntu-latest needs: preflight - if: ${{ needs.preflight.outputs.ok == 'true'}} + if: ${{ needs.preflight.outputs.ok == 'true' }} outputs: - needed: ${{ steps.renv.outputs.exists }} + renv-needed: ${{ steps.renv-check.outputs.renv-needed }} steps: - name: "Checkout Lesson" - uses: actions/checkout@v3 - - id: renv - run: | - if [[ -d renv ]]; then - echo "exists=true" >> $GITHUB_OUTPUT - fi + uses: actions/checkout@v6 - check_token: - name: "Check SANDPAPER_WORKFLOW token" - runs-on: ubuntu-latest - needs: check_renv - if: ${{ needs.check_renv.outputs.needed == 'true' }} - outputs: - workflow: ${{ steps.validate.outputs.wf }} - repo: ${{ steps.validate.outputs.repo }} - steps: - - name: "validate token" - id: validate - uses: carpentries/actions/check-valid-credentials@main + - name: "Is renv required?" + id: renv-check + uses: carpentries/actions/renv-checks@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + CACHE_VERSION: ${{ inputs.CACHE_VERSION || '' }} + skip-cache-check: true update_cache: - name: "Update Package Cache" - needs: check_token - if: ${{ needs.check_token.outputs.repo== 'true' }} - runs-on: ubuntu-latest + name: "Create Package Update Pull Request" + runs-on: ubuntu-22.04 + needs: check-renv + permissions: + contents: write + pull-requests: write + actions: write + issues: write + id-token: write + if: needs.check-renv.outputs.renv-needed == 'true' env: GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} RENV_PATHS_ROOT: ~/.local/share/renv/ steps: - - name: "Checkout Lesson" - uses: actions/checkout@v3 + uses: actions/checkout@v6 - name: "Set up R" uses: r-lib/actions/setup-r@v2 @@ -88,14 +101,60 @@ jobs: id: update uses: carpentries/actions/update-lockfile@main with: + update: ${{ env.UPDATE_PACKAGES }} + force-renv-init: ${{ env.FORCE_RENV_INIT }} + generate-cache: ${{ env.LOCKFILE_CACHE_GEN }} cache-version: ${{ secrets.CACHE_VERSION }} - - name: Create Pull Request + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + uses: carpentries/actions/validate-org-workflow@main + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: "Configure AWS credentials via OIDC" + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + + - name: "Set PAT from AWS Secrets Manager" + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + id: set-pat + run: | + SECRET=$(aws secretsmanager get-secret-value \ + --secret-id carpentries-bot/github-pat \ + --query SecretString --output text) + PAT=$(echo "$SECRET" | jq -r .[]) + echo "::add-mask::$PAT" + echo "pat=$PAT" >> "$GITHUB_OUTPUT" + shell: bash + + # Create the PR with the following roles in order of preference: + # - Carpentries Bot classic PAT fetched from AWS (will only work in official Carpentries repos) + # - repo-scoped SANDPAPER_WORKFLOW classic PAT (will work in all scenarios) + # - default GITHUB_TOKEN (will work suitably, but workflows need to be triggered) + - name: "Create Pull Request" id: cpr - if: ${{ steps.update.outputs.n > 0 }} + if: | + steps.update.outputs.n > 0 uses: carpentries/create-pull-request@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} delete-branch: true branch: "update/packages" commit-message: "[actions] update ${{ steps.update.outputs.n }} packages" @@ -123,3 +182,9 @@ jobs: [1]: https://github.com/carpentries/create-pull-request/tree/main labels: "type: package cache" draft: false + + - name: "Skip PR creation" + if: steps.update.outputs.n == 0 + run: | + echo "No updates needed, skipping PR creation" + shell: bash diff --git a/.github/workflows/update-workflows.yaml b/.github/workflows/update-workflows.yaml index 288bcd1..3510687 100644 --- a/.github/workflows/update-workflows.yaml +++ b/.github/workflows/update-workflows.yaml @@ -1,55 +1,105 @@ -name: "02 Maintain: Update Workflow Files" +name: "04 Maintain: Update Workflow Files" +description: "Update workflow files from the carpentries/sandpaper repository" on: + schedule: + - cron: '0 0 * * 2' workflow_dispatch: inputs: name: description: 'Who triggered this build (enter github username to tag yourself)?' required: true default: 'weekly run' + version: + description: 'Workflows version number (e.g. 0.0.1), branch name (e.g. main), or "latest"' + required: false + default: 'latest' clean: description: 'Workflow files/file extensions to clean (no wildcards, enter "" for none)' required: false default: '.yaml' - schedule: - # Run every Tuesday - - cron: '0 0 * * 2' jobs: - check_token: - name: "Check SANDPAPER_WORKFLOW token" - runs-on: ubuntu-latest - outputs: - workflow: ${{ steps.validate.outputs.wf }} - repo: ${{ steps.validate.outputs.repo }} - steps: - - name: "validate token" - id: validate - uses: carpentries/actions/check-valid-credentials@main - with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} - update_workflow: name: "Update Workflow" runs-on: ubuntu-latest - needs: check_token - if: ${{ needs.check_token.outputs.workflow == 'true' }} + permissions: + contents: write + pull-requests: write + id-token: write steps: - name: "Checkout Repository" - uses: actions/checkout@v3 + uses: actions/checkout@v6 + + - name: "Validate Current Org and Workflow" + id: validate-org-workflow + uses: carpentries/actions/validate-org-workflow@main + with: + repo: ${{ github.repository }} + workflow: ${{ github.workflow }} + + - name: Configure AWS credentials via OIDC + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.role-to-assume }} + aws-region: ${{ env.aws-region }} + + - name: Set PAT from AWS Secrets Manager + id: set-pat + env: + role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }} + aws-region: ${{ secrets.AWS_GH_OIDC_REGION }} + if: | + steps.validate-org-workflow.outputs.is_valid == 'true' && + env.role-to-assume != '' && + env.aws-region != '' + run: | + SECRET=$(aws secretsmanager get-secret-value \ + --secret-id carpentries-bot/github-pat \ + --query SecretString --output text) + PAT=$(echo "$SECRET" | jq -r .[]) + echo "::add-mask::$PAT" + echo "pat=$PAT" >> "$GITHUB_OUTPUT" + shell: bash + + - name: "Validate token" + id: validate-token + uses: carpentries/actions/check-valid-credentials@main + with: + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} + + - name: "No Token Found: Skipping Workflow Update" + if: ${{ steps.validate-token.outputs.wf == 'false' }} + run: | + echo "❗No valid SANDPAPER_WORKFLOW token or PAT from AWS found, cannot update workflows." + + echo "## ❌ Workflow Update Failed" >> $GITHUB_STEP_SUMMARY + echo "No valid SANDPAPER_WORKFLOW token or PAT from AWS found, cannot update workflows." >> $GITHUB_STEP_SUMMARY + shell: bash - name: Update Workflows id: update + if: ${{ steps.validate-token.outputs.wf == 'true' }} uses: carpentries/actions/update-workflows@main with: - clean: ${{ github.event.inputs.clean }} + version: ${{ github.event.inputs.version || 'latest' }} + clean: ${{ github.event.inputs.clean || '.yaml' }} - name: Create Pull Request id: cpr - if: "${{ steps.update.outputs.new }}" + if: | + steps.update.outputs.new && + steps.validate-token.outputs.wf == 'true' uses: carpentries/create-pull-request@main with: - token: ${{ secrets.SANDPAPER_WORKFLOW }} + token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }} delete-branch: true branch: "update/workflows" commit-message: "[actions] update sandpaper workflow to version ${{ steps.update.outputs.new }}" @@ -62,5 +112,5 @@ jobs: - Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }} [1]: https://github.com/carpentries/create-pull-request/tree/main - labels: "type: template and tools" + labels: "type: workflows" draft: false diff --git a/.github/workflows/workflows-version.txt b/.github/workflows/workflows-version.txt new file mode 100644 index 0000000..7dea76e --- /dev/null +++ b/.github/workflows/workflows-version.txt @@ -0,0 +1 @@ +1.0.1