Skip to content

[FEATURE] Add support for KEY_VALUE Schema #164

@oneebhkhan

Description

@oneebhkhan

I am streaming CDC messages from PostGRES using the PostGRES Debezium Connector.

The topic has the following schema

"type": "KEY_VALUE",
    "timestamp": 1695369844027,
    "properties": {
      "key.schema.name": "KafKaJson",
      "key.schema.properties": "{\"__AVRO_READ_OFFSET__\":\"0\"}",
      "key.schema.type": "JSON",
      "kv.encoding.type": "SEPARATED",
      "value.schema.name": "KafKaJson",
      "value.schema.properties": "{\"__AVRO_READ_OFFSET__\":\"0\"}",
      "value.schema.type": "JSON"

I am successfully able to consume the messages on the Pulsar topic, however when I try to consume it using the Spark connector, I get the following error:
We do not support KEY_VALUE currently

I would like to request for this connector to support the KEY_VALUE schema so that Im properly correctly able to consume the messages.

As a stop-gap solution, I have enabled the allowDifferentTopicSchemas flag - which allows me to consume the messages since the Topic schema are not taken into account.

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/featureIndicates new functionality

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions