Add pg_lake_polaris extension for two-way Polaris sync

[sfc-gh-mslot]

Work in progress, opening as draft so the branch does not get lost.

Problem

pg_lake stores Iceberg metadata in lake_iceberg.tables_internal. When Apache Polaris runs co-resident in the same Postgres database (polaris_schema.entities), there is no path to keep the two catalogs in sync. Tables registered through pg_lake are invisible to Polaris clients, and metadata commits made through Polaris (external writers, REST clients) are invisible to pg_lake.

Solution

Add a new optional extension pg_lake_polaris that mirrors metadata between the two catalogs:

• Outbound (pg_lake to Polaris) is via triggers on tables_internal.
• Inbound (Polaris to pg_lake) piggybacks on sync_iceberg_metadata_from_external_write, which is also added in this branch as a generic external-write entrypoint in pg_lake_table.
• Loop prevention is a process-global suppress flag, mirroring the SkipIcebergDDLProcessing pattern.

v0.1 limitations (single-instance Polaris ID allocator, scalar-only inbound CREATE, flat namespaces) are documented inline in the SQL.

The branch also pulls in supporting changes in pg_lake_table: external Iceberg write support via catalog UPDATE, a deletion queue for files unreferenced after external sync, and snapshot/SPI fixes surfaced while wiring this up.

Setup

-- Optional: override the Polaris realm_id (default 'POLARIS').
SET pg_lake_polaris.realm_id = 'POLARIS';

-- Load the extension. pg_lake_iceberg and pg_lake_table are required.
CREATE EXTENSION pg_lake_polaris;

-- Pin this database to a Polaris catalog that already exists in
-- polaris_schema.entities. Installs the inbound trigger and mirrors
-- existing pg_lake Iceberg tables out to Polaris.
SELECT lake_polaris.register_catalog('mycat');

After register_catalog, pg_lake CREATE TABLE ... USING iceberg shows up as a table entity under the named Polaris catalog, and metadata commits made through Polaris are reflected back into lake_iceberg.tables_internal. To stop syncing, call lake_polaris.unregister_catalog().

Test plan

• pg_lake_polaris/tests/pytests/test_polaris_register.py, test_polaris_outbound.py, test_polaris_inbound.py, test_polaris_e2e.py
• pg_lake_table/tests/pytests/test_external_write.py
• Inbound INSERT path is not e2e-tested in v0.1; documented as a known limitation.