New Catalog Configuration: Credentials via config file

[sfc-gh-npuka]

Description

This change adds a file-based credential source for REST catalog servers, sitting between server options and CREATE USER MAPPING in the resolution chain. A platform operator can place credentials in $PGDATA/catalogs.conf without requiring any per-user DDL, making it possible to provision catalog access at the infrastructure level rather than through SQL.

This branch builds on top of naisila/catalog_user_mapping (which moved credentials into CREATE USER MAPPING).

▶️ Usage

Platform-provided catalog

The operator creates the server via DDL and drops credentials into the config file — no user mapping is needed:

CREATE SERVER horizon TYPE 'rest'
    FOREIGN DATA WRAPPER iceberg_catalog
    OPTIONS (rest_endpoint 'https://horizon.example.com');

# $PGDATA/catalogs.conf
horizon.client_id     = 'platform_id'
horizon.client_secret = 'platform_secret'
horizon.scope         = 'PRINCIPAL_ROLE:ALL'

CREATE TABLE t (a int) USING iceberg WITH (catalog = 'horizon');

File location

The path is controlled by pg_lake_iceberg.catalogs_conf_path, a PGC_SIGHUP, superuser-only GUC. It defaults to catalogs.conf, which is resolved to $PGDATA/catalogs.conf. Absolute paths are used as-is.

ALTER SYSTEM SET pg_lake_iceberg.catalogs_conf_path TO '/etc/pg_lake/catalogs.conf';
SELECT pg_reload_conf();

▶️ Credential Resolution Order

With this change, the full resolution chain becomes:

Priority	Source	Mechanism
1 (highest)	CREATE USER MAPPING	pg_user_mapping syscache lookup for current user, fallback to PUBLIC
2	$PGDATA/catalogs.conf	File parsed via ParseConfigFp; dotted keys like server.client_id = '...'
3 (lowest)	GUCs	rest_catalog_client_id, rest_catalog_client_secret (set during initialization)

User mapping values overwrite whatever catalogs.conf provided. catalogs.conf values fill in only what server options didn't provide. GUCs are the implicit base layer from initialization. An error is raised if no credentials are found after all three sources.

scope follows the same layering: user mapping > catalogs.conf > server options > GUC.

The built-in pg_lake_rest_catalog (catalog 'rest') skips both the catalogs.conf and user mapping phases — its credentials live exclusively in GUCs.

▶️ Implementation

Two new static functions in rest_catalog.c:

ReadCatalogsConfCredentials reads the credentials file using PostgreSQL's ParseConfigFp. It matches entries where the dotted key prefix equals the server name and recognizes three keys: client_id, client_secret, and scope; unknown keys are silently ignored for forward compatibility. The file is re-read on every call — catalog operations are infrequent, and avoiding mtime tracking or SIGHUP wiring keeps the code simple. A missing file (ENOENT) is silently treated as "no credentials". I/O or permission errors are logged at WARNING and likewise treated as "no credentials" so one bad config file does not break the entire catalog path.

ApplyCatalogsConfOverrides calls ReadCatalogsConfCredentials and overlays any credentials found onto the RestCatalogOptions struct. Only fields actually present in the file are overwritten — unset entries leave whatever lower-priority layer (GUCs) populated.

ApplyCatalogsConfOverrides is called from BuildRestCatalogOptionsFromServer between ApplyServerOptionOverrides and ApplyUserMappingOverrides, guarded by !IsBuiltinCatalogServerName(serverName).