When the PostgreSQL cluster has a non-default tablespace, the backup will fail.

[marsqd]

What happened

When a tablespace is created in the PG Cluster, the pg_hardstorage backup fails.

ERROR backup.manifest_invalid: backup: refusing to commit local.full.20260626T012100Z.d8e0: manifest fails its own invariant check: manifest: backup_label is empty (required for restore)

If the non-default table spaces are deleted, pg_hardstorage can complete the backup successfully.

To reproduce

1. Create the tablespace in the PG Cluster:
2. Use pg_hardstoarge to back up the pg cluster.

Exact commands.

psql:
create tablespace tbs1 location '/data/postgresql/18/tablespaces/tbs1';
shell:
pg_hardstorage backup local

pg_hardstorage version

pg_hardstorage 1.0.4 (6bf20a6, built 2026-06-24T20:10:58Z)

Doctor report

pg_hardstorage doctor output:
{
"schema": "pg_hardstorage.v1",
"command": "pg_hardstorage doctor",
"generated_at": "2026-06-26T01:41:04.791113776Z",
"result": {
"mode": "user",
"paths": [
{
"domain": "config",
"value": "/var/lib/postgresql/.config/pg_hardstorage",
"source": "xdg",
"reason": "XDG default (XDG_CONFIG_HOME unset)",
"exists": true,
"is_dir": true,
"mode": "drwx------"
},
{
"domain": "config.drop_in",
"value": "/var/lib/postgresql/.config/pg_hardstorage/conf.d",
"source": "derived",
"reason": "<pg_hardstorage>/conf.d",
"exists": false
},
{
"domain": "config.deployments",
"value": "/var/lib/postgresql/.config/pg_hardstorage/deployments",
"source": "derived",
"reason": "<pg_hardstorage>/deployments",
"exists": false
},
{
"domain": "config.sinks",
"value": "/var/lib/postgresql/.config/pg_hardstorage/sinks",
"source": "derived",
"reason": "<pg_hardstorage>/sinks",
"exists": false
},
{
"domain": "config.skills",
"value": "/var/lib/postgresql/.config/pg_hardstorage/skills",
"source": "derived",
"reason": "<pg_hardstorage>/skills",
"exists": false
},
{
"domain": "config.keyring",
"value": "/var/lib/postgresql/.config/pg_hardstorage/keyring",
"source": "derived",
"reason": "<pg_hardstorage>/keyring",
"exists": true,
"is_dir": true,
"mode": "drwx------"
},
{
"domain": "state",
"value": "/var/lib/postgresql/.local/share/pg_hardstorage",
"source": "xdg",
"reason": "XDG default (XDG_DATA_HOME unset)",
"exists": false
},
{
"domain": "state.bookkeeping",
"value": "/var/lib/postgresql/.local/share/pg_hardstorage/bookkeeping",
"source": "derived",
"reason": "/bookkeeping (per-package JSON state)",
"exists": false
},
{
"domain": "state.inflight",
"value": "/var/lib/postgresql/.local/share/pg_hardstorage/inflight",
"source": "derived",
"reason": "<pg_hardstorage>/inflight",
"exists": false
},
{
"domain": "state.crashes",
"value": "/var/lib/postgresql/.local/share/pg_hardstorage/crashes",
"source": "derived",
"reason": "<pg_hardstorage>/crashes",
"exists": false
},
{
"domain": "cache",
"value": "/var/lib/postgresql/.cache/pg_hardstorage",
"source": "xdg",
"reason": "XDG default (XDG_CACHE_HOME unset)",
"exists": false
},
{
"domain": "logs",
"value": "/var/lib/postgresql/.local/state/pg_hardstorage",
"source": "xdg",
"reason": "XDG default (XDG_STATE_HOME unset)",
"exists": false
},
{
"domain": "runtime",
"value": "/run/user/102/pg_hardstorage",
"source": "xdg",
"reason": "synthesized: /run/user/102 (XDG_RUNTIME_DIR unset)",
"exists": false
},
{
"domain": "shared_data",
"value": "/var/lib/postgresql/.local/share/pg_hardstorage/share",
"source": "xdg",
"reason": "XDG default (XDG_DATA_HOME unset)",
"exists": false
}
],
"config": {
"configured": true,
"schema": "pg_hardstorage.config.v1",
"source_files": [
{
"path": "/var/lib/postgresql/.config/pg_hardstorage/pg_hardstorage.yaml",
"kind": "main",
"read_ok": true
}
],
"deployment_count": 1,
"deployment_names": [
"local"
],
"classifications": {
"local": "internal"
},
"sink_count": 0
},
"keystore": {
"keyring_dir": "/var/lib/postgresql/.config/pg_hardstorage/keyring",
"signing_key_exists": true,
"kek_exists": true
},
"airgap": {
"mode": "off"
},
"repos": [
{
"url": "file:///data/dbbak/18/repos/local",
"reachable": true,
"chain_event_count": 8,
"has_latest_anchor": true,
"latest_anchor_sequence": 2,
"anchor_behind_events": 4,
"anchor_fresh": false
}
],
"manifest_signatures": [
{
"deployment": "local",
"checked": 1,
"key_mismatch": 0,
"other_failures": 0
}
],
"issues": [
{
"severity": "notice",
"code": "audit.anchor_stale",
"message": "doctor: anchor for file:///data/dbbak/18/repos/local is 4 event(s) behind (latest anchor at sequence 2, chain head at sequence 6)",
"suggestion": {
"human": "run pg_hardstorage audit anchor to refresh the anchor",
"command": "pg_hardstorage audit anchor --repo file:///data/dbbak/18/repos/local"
}
}
],
"healthy": true
}
}

## Logs
**pg_hardstorage backup local -o ndjson** output:

{"schema":"pg_hardstorage.v1","severity":"info","severity_name":"info","component":"backup","op":"dedup_hints_loaded","subject":{"deployment":"local"},"body":{"hint_chunks":655},"generated_at":"2026-06-26T01:46:57.421852478Z"}
{"schema":"pg_hardstorage.v1","severity":"info","severity_name":"info","component":"backup","op":"pg_probed","body":{"pg_version":18,"raw":"18.4 (Debian 18.4-1.pgdg13+1)"},"generated_at":"2026-06-26T01:46:57.456418948Z"}
{"schema":"pg_hardstorage.v1","severity":"info","severity_name":"info","component":"backup","op":"identified","body":{"system_id":"7624454925457747509","timeline":1,"xlogpos":"0/23000168"},"generated_at":"2026-06-26T01:46:57.481733007Z"}
{"schema":"pg_hardstorage.v1","severity":"info","severity_name":"info","component":"backup","op":"started","subject":{"tenant":"default","deployment":"local","backup_id":"local.full.20260626T014657Z.fef8"},"generated_at":"2026-06-26T01:46:57.481800087Z"}
{"schema":"pg_hardstorage.v1","severity":"info","severity_name":"info","component":"backup","op":"stream_complete","body":{"bytes_received":32452096,"messages":2918,"tablespaces":2},"generated_at":"2026-06-26T01:46:58.324424738Z"}
{"schema":"pg_hardstorage.v1","command":"pg_hardstorage backup","generated_at":"2026-06-26T01:46:58.328213114Z","error":{"code":"backup.manifest_invalid","message":"backup: refusing to commit local.full.20260626T014657Z.fef8: manifest fails its own invariant check: manifest: backup_label is empty (required for restore)","severity":"error","suggestion":{"human":"the backup completed but the manifest is malformed and would not restore. Common cause: PG's BASE_BACKUP didn't emit backup_label in the streamed tarball — check the source PG version (PG 15+ required), confirm pg_backup_start/stop are the non-exclusive variants, and re-run the backup."}}}

## Environment

- OS: debian 13.5
- Architecture:  amd64
- PG: 18.4
- Repo backend: fs
- Coordination: single-host

## Anything else
The same problem also exists in PG 17.10.

[postgresql007]

Thanks for the really clear report — the exact error, the create tablespace repro, and the ndjson log made this easy to pin down. 🙏

You're right that the tablespace is the trigger. Here's what was happening: when a cluster has a user tablespace, PostgreSQL streams the base data directory last in the BASE_BACKUP — and that base archive is the one that carries backup_label. Our tar reader was only looking for backup_label/tablespace_map in the first archive it received, so with a non-default tablespace present they slipped past, the manifest ended up with an empty backup_label, and the safety check correctly refused to commit a backup that wouldn't have restored. (It refusing was the right call — it just shouldn't have gotten into that state.)

The fix makes the reader grab those files from whichever archive actually contains them, so multi-tablespace clusters back up and restore normally now. There's a regression test that reproduces your exact layout (a user tablespace plus the base streamed last) so this can't quietly come back.

Fix is in #18. Once it lands I'll follow up here. Thanks again for catching it!