Merge tag 'REL_16_9' into Cloudberry

[chenjinbao1989]

Merge upstream PostgreSQL REL_16_9 into Cloudberry, upgrading the
kernel from PG 14.4 to PG 16.9. This spans two major versions (PG 15
and PG 16), bringing in 5730 commits.

Key features and changes, sorted by importance:

1. MERGE command

   • SQL-standard MERGE for INSERT/UPDATE/DELETE in a single statement.
   • ruleutils decompilation, PlaceHolderVar support in actions
   • Fixes for partitioned tables, cross-partition triggers, RLS,
   • EvalPlanQual, self-modified tuples, pg_stat_statements

2. Logical replication enhancements

   • Built-in support for prepared transactions (2PC)
   • Enable two-phase via replication protocol
   • Streaming changes after speculative aborts
   • New pg_create_subscription predefined role
   • Parallel apply workers (PG 16)
   • Row/column filtering, schema-level publication (PG 15)
   • Critical fix for data loss in logical replication
   • Slot invalidation, snapshot, and memory leak fixes

3. SQL/JSON

   • IS JSON predicate
   • Standard JSON constructor functions (json_array, json_object,
     json_scalar, json_serialize)
   • Auto-generated query jumbling via gen_node_support.pl
   • Fix json_array() subquery double transformation

4. WAL and storage subsystem

   • LZ4 compression for full-page writes (FPW)
   • Custom WAL resource managers
   • Skip WAL recycling/preallocation during archive recovery
   • XLOG_FPI_FOR_HINT honors full_page_writes setting
   • Direct I/O support on macOS
   • Buffer manager infrastructure refactor for faster relation extension
   • New smgrzeroextend(), FileZero(), FileFallocate()
   • Remove HeapBitmapScan skip_fetch optimization (correctness issue)
   • WAL/Recovery separation: xlogrecovery.c, xlogprefetcher.c (PG 15)
   • Backup sink architecture (PG 15)

5. Query optimizer / executor
   Performance:

   • Hash table acceleration for NOT IN(values)
   • Datum sorts for single-column sorts
   • Result Cache renamed to Memoize with ongoing fixes
   • Parallel Hash Full Join
   • Reduced planning cost for deeply-nested views
   • Fewer pallocs when building partition bounds
   • Outer join optimization enhancement (PG 16)
   • Symmetric hash join optimization (PG 16)
     Critical fixes:
   • Planner failure to identify multiple hashable ScalarArrayOpExprs
   • Consistent whole-row Var construction in parsing and planning
   • WindowAgg evaluation crash
   • WITH RECURSIVE UNION assert failure
   • ORDER BY / DISTINCT aggregates with FILTER
   • setrefs.c missing expression processing on prune steps
   • Parallel Hash Join extreme skew detection

6. pgstat monitoring enhancements
   pgstat shared memory rewrite (PG 15):

   • Stats collector process removed; stats now in shared memory
   • Monolithic pgstat.c split into 14 files under utils/activity/
     pg_stat_io (PG 16 new view):
   • Detailed I/O statistics: shared buffer hits, I/O times, writeback
     pg_stat_statements:
   • JIT counters, temp file block I/O timing
   • Utility query string normalization, MERGE support
   • 32-bit integer overflow fix
     Other:
   • pg_stat_wal time accumulation as instr_time
   • Macro-generated pg_stat_get*() functions for tables and databases
   • Enhanced pg_stat_reset_single_table_counters
   • SP-GiST index scans counted in pg_stat
   • New test helpers: pg_stat_force_next_flush(), pg_stat_have_stats()

7. libpq and client protocol

   • New PQsendFlushRequest
   • Pipeline mode state machine fix
   • SASL code refactored to generic interface
   • Escape function fixes for invalid encoding data (security)
   • PQescapeLiteral()/PQescapeIdentifier() length handling fix
   • New pg_encoding_set_invalid()
   • Build-time check that libpq doesn't call exit()/abort()

8. Security and privilege model

   • CREATEROLE privilege restrictions (major security improvement)
   • New GUCs: createrole_self_grant, reserved_connections
   • New predefined roles: pg_maintain, pg_create_subscription
   • Non-superuser predefined roles for vacuum/analyze
   • Revoke PUBLIC CREATE from public schema (now owned by pg_database_owner)
   • Security invoker views (SECURITY INVOKER)
   • session_authorization and role interaction fixes
   • scram_SaltedPassword() integer overflow fix

9. pg_dump / pg_upgrade / pg_basebackup toolchain
   pg_dump:

   • Generic compression API, zstd support, LZ4 frame-only format
     pg_basebackup:
   • Extended compression options, server-side compression with -Fp
   • Client-side LZ4 decompression, parallel zstd compression
     pg_upgrade:
   • Fix unintentional 'NULL' string literal
   • Fix XMLSERIALIZE(NO INDENT) cross-version upgrade
   • pg_dumpall handling of dangling OIDs in pg_auth_members

10. Partitioned table improvements

    • Self-referencing FKs in partitioned tables
    • Detach partition with top-level FK fix
    • Reset relhassubclass on ATTACH
    • Trigger rename consistency, preserve firing state on clone
    • Disallow partitionwise join/grouping on collation mismatch

11. ICU and collation

    • Build ICU support by default
    • initdb uses uloc_getDefault()
    • CREATE DATABASE LOCALE applies to all collation providers

12. Node support function auto-generation (FIXME)

    • gen_node_support.pl auto-generates copyfuncs/equalfuncs/outfuncs/readfuncs
    • Query jumbling code auto-generation
    • NodeTag ABI stability check

13. Global renames and structural changes

    • RelFileNode -> RelFileLocator (pervasive rename across hundreds of files)
    • GUC system split: guc.c -> guc.c + guc_tables.c + guc_funcs.c
    • varatt.h separated from postgres.h
    • Test framework: PostgresNode.pm -> PostgreSQL::Test::Cluster

14. SQL types and functions

    • Numeric scale allows negative or greater than precision
    • New ANY_VALUE aggregate function
    • unnest(multirange), range_agg with multirange inputs
    • pg_size_pretty/pg_size_bytes support petabytes
    • ALTER TABLE SET ACCESS METHOD
    • SYSTEM_USER function
    • numeric_mul() overflow fix

15. psql improvements

    • PSQL_WATCH_PAGER for \watch command
    • New \drg command for role grants
    • \copy from sends data in larger chunks
    • pg_waldump --rmgr multiple specification

16. Critical data corruption and crash fixes

    • Fix data corruption when relation truncation fails
    • Fail instead of corrupting page header on non-LP_NORMAL TID update
    • Fix unsafe BufferDescriptors access
    • GB18030 SIGSEGV from out-of-bounds read
    • Snowball stemmer null pointer dereference after OOM
    • Rare standby assertion failure on primary restart
    • catcache invalidation during list entry construction

17. Vacuum / Autovacuum

    • Parallel VACUUM (PG 16, vacuumparallel.c new file)
    • ON COMMIT DELETE ROWS avoids ERROR after relhassubclass=f
    • Use WaitLatch() instead of pg_usleep() at end-of-vacuum truncation
    • Prevent numeric overflow in parallel numeric aggregates
    • ANALYZE preserves relhasindex for partitioned tables

Co-authored-by: liushengsong lss602726449@gmail.com
Co-authored-by: reshke reshke@double.cloud
Co-authored-by: Hao Wu gfphoenix78@gmail.com
Co-authored-by: Jianghua.yjh yjhjstz@gmail.com
Co-authored-by: Dianjin Wang wangdianjin@gmail.com