Problem
When running baton-sql in service mode, syncs fail with:
{"error": "grpc: received message larger than max (4915649 vs. 4194304)"}
The baton-sdk creates gRPC server and client connections without configuring grpc.MaxRecvMsgSize() or grpc.MaxCallRecvMsgSize(), so Go's default 4MB limit applies. As the number of synced resources grows, messages eventually exceed this limit with no workaround available.
There are no CLI flags, environment variables, or connector config options to override this.
Prior work
PR #357 attempted to increase the limit to 10MB but was closed with the note that it should only apply to service mode and one-shot, not hosted connectors.
Proposal
Expose a --grpc-max-recv-msg-size flag (or similar) that applies to service and one-shot modes, as suggested in #357. This would let connector operators set a limit appropriate for their data volume without affecting hosted connector behavior.
Alternatively, increasing the default for service/one-shot mode to 10-16MB (as #357 proposed) would address most cases without requiring configuration.
Problem
When running baton-sql in service mode, syncs fail with:
The baton-sdk creates gRPC server and client connections without configuring
grpc.MaxRecvMsgSize()orgrpc.MaxCallRecvMsgSize(), so Go's default 4MB limit applies. As the number of synced resources grows, messages eventually exceed this limit with no workaround available.There are no CLI flags, environment variables, or connector config options to override this.
Prior work
PR #357 attempted to increase the limit to 10MB but was closed with the note that it should only apply to service mode and one-shot, not hosted connectors.
Proposal
Expose a
--grpc-max-recv-msg-sizeflag (or similar) that applies to service and one-shot modes, as suggested in #357. This would let connector operators set a limit appropriate for their data volume without affecting hosted connector behavior.Alternatively, increasing the default for service/one-shot mode to 10-16MB (as #357 proposed) would address most cases without requiring configuration.