[BUG] GRPCFeatureClient sends invalid header keys that are incompatible with Python gRPC (metadata was invalid)

[AryAgarwal]

🐛 Describe the bug

The current Python implementation of GRPCFeatureClient uses uppercase and hyphenated gRPC metadata headers:

HEADER_CALLER_ID = "ONLINE-FEATURE-STORE-CALLER-ID"
HEADER_CALLER_TOKEN = "ONLINE-FEATURE-STORE-AUTH-TOKEN"

When these are passed to the grpcio Python client, it results in the following error:

E0000 00:00:1751799888.906104 110369931 filter_stack_call.cc:402] validate_metadata: INTERNAL:Illegal header key
Error persisting features: metadata was invalid: [('ONLINE-FEATURE-STORE-CALLER-ID', 'demo'), ('ONLINE-FEATURE-STORE-AUTH-TOKEN', 'demo_token'), ('job-id', 'demo'), ('job-token', 'demo_token')]

This is a known gRPC Python restriction — header keys must be lowercase ASCII without hyphens due to HTTP/2 requirements:

https://stackoverflow.com/questions/45071567/how-to-send-custom-header-metadata-with-python-grpc

Python clients should be able to use GRPCFeatureClient without modifying the SDK or subclassing it just to avoid metadata violations.

Where This Happens

py-sdk/grpc_feature_client/grpc_feature_client/client.py, during metadata construction for gRPC calls.

Would love it if someone could help solve my error.

[addloopy]

Thanks for raising this issue and providing detailed context, along with the stack trace and reference!
You're absolutely right, Python's grpcio enforces lowercase, ASCII-only metadata keys due to gRPC over HTTP/2 specifications. The current SDK uses uppercase and hyphens in the header names, which leads to this metadata validation error.

To fix it we need to change

1. In client.py

# gRPC authentication headers (from Go SDK)
HEADER_CALLER_ID = "ONLINE-FEATURE-STORE-CALLER-ID"
HEADER_CALLER_TOKEN = "ONLINE-FEATURE-STORE-AUTH-TOKEN"

to

# gRPC authentication headers (from Go SDK)
HEADER_CALLER_ID = "online_feature_store_caller_id"
HEADER_CALLER_TOKEN = "online_feature_store_auth_token"

2. On server side on middle-ware we need to handle accordingly for extracting and authenticating headers

onst (
	AuthToken       = "AUTH_TOKEN"
	callerIdHeader  = "online-feature-store-caller-id"
	AuthTokenHeader = "online-feature-store-auth-token"
)

func ServerInterceptor(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp any, err error) {
	startTime := time.Now()
	md, ok := metadata.FromIncomingContext(ctx)
	if !ok {
		return nil, status.Errorf(codes.InvalidArgument, "mandatory metadata are missing")
	}
	callerId, ok := md[callerIdHeader]
	if !ok {
		return nil, status.Errorf(codes.InvalidArgument, "%s header is missing", callerIdHeader)
	}
	authHeader, ok := md[AuthTokenHeader]
	if !ok {
		return nil, status.Errorf(codes.InvalidArgument, "%s header is missing", AuthTokenHeader)
	}

For backward compatibility with go client we need to assume

callerIdHeader  as "online-feature-store-caller-id" | "online_feature_store_caller_id"
AuthTokenHeader as "online-feature-store-auth-token" | "online_feature_store_auth_token"

[addloopy]

@AryAgarwal if you’re interested, feel free to raise a PR with the fix, it should mostly involve replacing the header keys options. We’ll be happy to review and help if needed!
If you’d rather not patch it yourself, let us know, we can take care of it immediately.

[AryAgarwal]

@addloopy Sure I'll raise a PR with the fix for you to review. Also I'd love to contribute in the future as well, so let me know.

[addloopy]

Amazing!!
For future keep an eye on issues list. Feel free to comment if you want to take up any issues. Refer Contribution Guide, to boot-up, let us know if you need any help. We are available here and on Discord Server to discuss and help.

[tanmay958]

Can you assign it to me

[addloopy]

@AryAgarwal assigning to @tanmay958

[addloopy]

@tanmay958 Ref PR

[arnav-meesho]

@tanmay958 hi, are you still working on this?