Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/workflows/github-release.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: Create release

on:
pull_request_target:
branches:
- main
types:
- closed

jobs:
release:
if: github.event.pull_request.merged == true && !contains(github.event.pull_request.title, 'skip-release')
uses: libops/actions/.github/workflows/bump-release.yaml@ef667db8c16533a257d841e75df5c3388152b2d7 # main
with:
prefix: v
permissions:
contents: write
actions: write
secrets: inherit
10 changes: 10 additions & 0 deletions config.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,16 @@ iiif:
load_access: auto
# Cache info.json dimensions by identifier plus source mtime/size.
info_dimension_cache: true
# Optional protected route for invalidating all derivative variants for one
# identifier: POST {prefix}/{identifier}/cache/invalidate with
# Authorization: Bearer ${TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN}.
# If TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN is unset,
# TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN_FILE may point at a mounted secret
# file whose contents populate the token environment variable.
# Optionally restrict callers by client CIDR. When Triplet is behind a
# proxy, configure logging.trusted_proxy_cidrs so X-Forwarded-For is trusted.
# cache_invalidation_allowed_cidrs: [127.0.0.1/32, ::1/128]
# cache_invalidation_token: ${TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN}
presentation:
enabled: false
prefix: /presentation/v3
Expand Down
40 changes: 38 additions & 2 deletions docs/caching.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,42 @@ cache:
max_bytes: 1073741824
```

Derivative caches can be invalidated per identifier by configuring an image
cache invalidation token and calling the protected route:

```yaml
iiif:
image:
cache_invalidation_token: ${TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN}
cache_invalidation_allowed_cidrs:
- 127.0.0.1/32
- ::1/128
```

If `TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN` is unset,
`TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN_FILE` may point at a readable mounted
secret file. Triplet reads that file before expanding the YAML configuration and
uses its contents as `TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN`.

When `cache_invalidation_allowed_cidrs` is set, callers must match one of those
CIDRs in addition to presenting the bearer token. If Triplet runs behind a
reverse proxy, configure `logging.trusted_proxy_cidrs`; only trusted proxies are
allowed to supply the client IP via `X-Forwarded-For` or `X-Real-IP`.

```sh
curl -X POST \
-H "Authorization: Bearer ${TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN}" \
"https://iiif.example.edu/iiif/3/https%3A%2F%2Frepo.example.edu%2Fsystem%2Ffiles%2Fimage.tif/cache/invalidate"
```

The route writes an invalidation marker into the derivative cache. Subsequent
image requests for that identifier use a new cache namespace, so old derivative
objects are ignored even when the source bytes and metadata have not changed.
This is useful for local URL mappings with `auth_probe: true`, where repository
permission changes may need to take effect before the configured auth-probe TTL
expires. When the source backend supports per-identifier auth caching, the same
route also clears those cached auth-probe decisions for the identifier.

## Source cache

The optional source cache stores fetched source bytes, primarily for HTTP
Expand Down Expand Up @@ -71,10 +107,10 @@ sources:

| Layer | Configuration | What is cached | Invalidation / freshness |
|---|---|---|---|
| Derivative cache | `cache.root` or `cache.bucket_url`; optional `cache.max_bytes`, `cache.prefix` | Encoded IIIF image responses, keyed by identifier, source version, region, size, rotation, quality, and format. | A changed source version produces a new key. Filesystem caches can evict best-effort by size; GCS/object lifecycle is external. Failed transforms and HTTP error responses are not stored. |
| Derivative cache | `cache.root` or `cache.bucket_url`; optional `cache.max_bytes`, `cache.prefix`, `iiif.image.cache_invalidation_token` | Encoded IIIF image responses, keyed by identifier, source version, invalidation marker, region, size, rotation, quality, and format. | A changed source version produces a new key. The protected invalidation route bumps the per-identifier invalidation marker. Filesystem caches can evict best-effort by size; GCS/object lifecycle is external. Failed transforms and HTTP error responses are not stored. |
| HTTP source cache | `cache.source_root` or `cache.source_bucket_url`; optional `cache.source_max_bytes`, `cache.source_prefix`, `cache.source_stale_after` | Original source bytes fetched through the HTTP source backend. | Keys are source identifiers. When `source_stale_after` is set, stale hits are served immediately and refreshed in the background. Upstream 4xx/5xx responses are not stored. |
| `info.json` dimension cache | `iiif.image.info_dimension_cache` | Source dimensions used to build Image API `info.json`. | In-memory only. Entries are keyed by identifier plus source size/modtime metadata, so source changes with updated metadata miss the cache. |
| Local URL auth-probe cache | `sources.file.url_mappings[].auth_*` | Authorization probe results for local URL mappings with `auth_probe: true`. Anonymous and credentialed probes are cached separately. | In-memory only. Success, 403, and 404 results are cached for the configured TTL. Other upstream errors are not cached. 403/404 results with a `Last-Modified` newer than `auth_error_cache_min_age` are not cached. |
| Local URL auth-probe cache | `sources.file.url_mappings[].auth_*` | Authorization probe results for local URL mappings with `auth_probe: true`. Anonymous and credentialed probes are cached separately. | In-memory only. Success, 403, and 404 results are cached for the configured TTL. Other upstream errors are not cached. 403/404 results with a `Last-Modified` newer than `auth_error_cache_min_age` are not cached. The image cache invalidation route also clears matching auth-probe entries when the source backend supports it. |
| libvips operation cache | `vips.cache_max_mem`, `vips.cache_max_files` | libvips in-process operation results. | Disabled by default in the example config. This is process-local and separate from Triplet's derivative/source caches. |

Source caching improves performance but does not replace the HTTP source
Expand Down
43 changes: 42 additions & 1 deletion internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ type Image struct {
Enabled bool `yaml:"enabled"`
Prefix string `yaml:"prefix"`
AllowedOrigins []string `yaml:"allowed_origins"`
CacheInvalidationToken string `yaml:"cache_invalidation_token"`
CacheInvalidationAllowedCIDRs []string `yaml:"cache_invalidation_allowed_cidrs"`
MaxOutputPixels int64 `yaml:"max_output_pixels"`
AllowUnsafeUnlimitedOutputPixels bool `yaml:"allow_unsafe_unlimited_output_pixels"`
MaxSourcePixels int64 `yaml:"max_source_pixels"`
Expand Down Expand Up @@ -203,7 +205,11 @@ func Load(path string) (*Config, error) {
if err != nil {
return nil, fmt.Errorf("read config %q: %w", path, err)
}
expanded := os.ExpandEnv(string(b))
env, err := configEnv()
if err != nil {
return nil, fmt.Errorf("load image cache invalidation token file: %w", err)
}
expanded := os.Expand(string(b), env)
explicitUnlimitedOutput, err := explicitZeroYAMLField([]byte(expanded), "iiif", "image", "max_output_pixels")
if err != nil {
return nil, fmt.Errorf("parse config %q: %w", path, err)
Expand All @@ -224,6 +230,36 @@ func Load(path string) (*Config, error) {
return &c, nil
}

func configEnv() (func(string) string, error) {
token, err := imageCacheInvalidationTokenEnv()
if err != nil {
return nil, err
}
return func(key string) string {
if key == "TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN" && token != "" {
return token
}
return os.Getenv(key)
}, nil
}

func imageCacheInvalidationTokenEnv() (string, error) {
const tokenEnv = "TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN"
const fileEnv = tokenEnv + "_FILE"
if token := os.Getenv(tokenEnv); token != "" {
return token, nil
}
path := strings.TrimSpace(os.Getenv(fileEnv))
if path == "" {
return "", nil
}
b, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("read %s: %w", fileEnv, err)
}
return strings.TrimRight(string(b), "\r\n"), nil
}

func explicitZeroYAMLField(body []byte, path ...string) (bool, error) {
var root yaml.Node
if err := yaml.Unmarshal(body, &root); err != nil {
Expand Down Expand Up @@ -385,6 +421,11 @@ func (c *Config) validate() error {
if err := validateAllowedOrigins("iiif.image.allowed_origins", c.IIIF.Image.AllowedOrigins); err != nil {
return err
}
for _, cidr := range c.IIIF.Image.CacheInvalidationAllowedCIDRs {
if _, _, err := net.ParseCIDR(cidr); err != nil {
return fmt.Errorf("iiif.image.cache_invalidation_allowed_cidrs: invalid CIDR %q: %w", cidr, err)
}
}
if !strings.HasPrefix(c.IIIF.Presentation.Prefix, "/") {
return fmt.Errorf("iiif.presentation.prefix: must start with `/`, got %q", c.IIIF.Presentation.Prefix)
}
Expand Down
94 changes: 94 additions & 0 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -510,6 +510,21 @@ sources:
`,
wantErr: "iiif.image.allowed_origins",
},
{
name: "image cache invalidation cidr invalid",
body: `
server:
public_base_url: http://localhost:8080
iiif:
image:
cache_invalidation_allowed_cidrs: [not-a-cidr]
sources:
default: file
file:
root: /tmp
`,
wantErr: "iiif.image.cache_invalidation_allowed_cidrs",
},
{
name: "explicit unlimited output pixels requires unsafe opt-in",
body: `
Expand Down Expand Up @@ -546,6 +561,85 @@ sources:
}
}

func TestLoadImageCacheInvalidationTokenFromFile(t *testing.T) {
t.Setenv("TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN", "")
dir := t.TempDir()
tokenPath := filepath.Join(dir, "image-cache-token")
if err := os.WriteFile(tokenPath, []byte("file-token\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN_FILE", tokenPath)
path := writeConfig(t, `
server:
public_base_url: http://localhost:8080
iiif:
image:
cache_invalidation_token: ${TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN}
sources:
default: file
file:
root: /tmp
`)

c, err := Load(path)
if err != nil {
t.Fatal(err)
}
if got := c.IIIF.Image.CacheInvalidationToken; got != "file-token" {
t.Fatalf("cache_invalidation_token = %q", got)
}
if got := os.Getenv("TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN"); got != "" {
t.Fatalf("TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN was mutated to %q", got)
}
}

func TestLoadImageCacheInvalidationTokenEnvOverridesFile(t *testing.T) {
t.Setenv("TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN", "env-token")
dir := t.TempDir()
tokenPath := filepath.Join(dir, "image-cache-token")
if err := os.WriteFile(tokenPath, []byte("file-token\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN_FILE", tokenPath)
path := writeConfig(t, `
server:
public_base_url: http://localhost:8080
iiif:
image:
cache_invalidation_token: ${TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN}
sources:
default: file
file:
root: /tmp
`)

c, err := Load(path)
if err != nil {
t.Fatal(err)
}
if got := c.IIIF.Image.CacheInvalidationToken; got != "env-token" {
t.Fatalf("cache_invalidation_token = %q", got)
}
}

func TestLoadImageCacheInvalidationTokenFileMissing(t *testing.T) {
t.Setenv("TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN", "")
t.Setenv("TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN_FILE", filepath.Join(t.TempDir(), "missing"))
path := writeConfig(t, `
server:
public_base_url: http://localhost:8080
sources:
default: file
file:
root: /tmp
`)

_, err := Load(path)
if err == nil || !strings.Contains(err.Error(), "TRIPLET_IMAGE_CACHE_INVALIDATION_TOKEN_FILE") {
t.Fatalf("err = %v, want token file error", err)
}
}

func TestLoadAppliesDefaults(t *testing.T) {
path := writeConfig(t, `
server:
Expand Down
Loading
Loading