You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PgQ-style direct set-based insert into current_event_table();
PgQ-style direct COPY into current_event_table().
This issue documents the numbers and the product/API conclusion: send_batch() is worth keeping as the safe high-level batch API, but COPY/direct table insert remains the expert maximum-throughput path.
Document send_batch() as set-based but not COPY-level.
Add a future docs/benchmark section for expert direct insert/COPY producer mode.
Do not remove send_batch(); it fills the ergonomic middle between single send() and raw PgQ table access.
Raw benchmark harness
The benchmark was run with a temporary Python/psycopg harness that creates equivalent legacy-loop functions for comparison and measures all paths in fresh queues.
Summary
Benchmarked PgQue producer paths to compare:
send();send_batch()implementation (PL/pgSQL loop overinsert_event());send_batch()implementation;insert into current_event_table();COPYintocurrent_event_table().This issue documents the numbers and the product/API conclusion:
send_batch()is worth keeping as the safe high-level batch API, but COPY/direct table insert remains the expert maximum-throughput path.Environment / method
fix/send-batch-set-based/ PR fix: make send_batch set-based #159 implementation.sql/pgque.sqlinstalled.psycopgbenchmark harness.{"x": i}style payloads.opaque-ipayloads.Results
N = 1,000
send()in client loopsend_batch(jsonb[])send_batch(jsonb[])send_batch(text[])send_batch(text[])insert into current_event_table()COPYintocurrent_event_table()N = 10,000
send()in client loopsend_batch(jsonb[])send_batch(jsonb[])send_batch(text[])send_batch(text[])insert into current_event_table()COPYintocurrent_event_table()N = 50,000
send_batch(jsonb[])send_batch(jsonb[])send_batch(text[])send_batch(text[])insert into current_event_table()COPYintocurrent_event_table()Interpretation
send_batch()is usefulEven the legacy loop
send_batch()beats client-sidesend()loops because it removes client round-trips.The set-based implementation in #159 is materially better:
send_batch()is not the absolute max-throughput pathDirect PgQ-style insertion into
current_event_table()and COPY remain faster, especially at 50k rows.That is expected: those paths bypass the safe SQL API wrapper and operate directly on queue event tables. They are expert tools.
Product/API conclusion
Keep a two-tier story:
send_batch()— safe high-level API for app developers and drivers.pgque_writer).current_event_table()+ directINSERT/COPY— expert maximum-throughput path.Recommendation
send_batch()as set-based but not COPY-level.send_batch(); it fills the ergonomic middle between singlesend()and raw PgQ table access.Raw benchmark harness
The benchmark was run with a temporary Python/psycopg harness that creates equivalent legacy-loop functions for comparison and measures all paths in fresh queues.