Kafka gateway: Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users

[ryerraguntla]

Part of #3560.

Description

Implement a foundation layer for the gateway for Kafka users to Iggy. The foundation consists of a TCP Listener receiving Kafka messages and acknowledges the messages for each of the scoped Kafka API key and respective version.

Kafka API key and versions are pretty comprehensive list. In this ticket, define the scope of supporting Kafka API keys and associated versions for each of the API Key.
The TCP Listener receives the message , validates the Kafka message format for each of the API Key and associated version.

Affected area / component

Other / not sure

Proposed solution

• Listens on predefined (port 9093 which is Kafka wire protocol)
• Decodes Kafka binary requests (ApiVersions, Produce, Fetch, etc.)
• Validates against the scoped API Keys and versions and the respective message formats
• Replies with return code. In case of expected data, for this ticket return predefined values.

To Do: Attach the scoped Kafka API keys and Versions.

Alternatives considered

Other considered approaches -

1. A layer on Iggy server with a dedicated port
2. Changes to each of the SDK so that Kafka clients can natively implement.

For (1) - Iggy server path will be affected and opening an additional port at this point of product life cycle makes product vulnerable and is discouraged.
For (2) - As users are spread across multiple SDKs, the changes need to be made in many SDKs. Also existing Kafka clients has learning curve in implementing the Iggy Binary protocol.

The above approach of having a gateway for Kafka users into Iggy eliminates the above issues. The con on having a separate gateway increases hosting cost. Alternate solutions can be considered to handle the hosting costs at a later point of time.

Contribution

• I'm willing to submit a pull request to implement this feature

Good first issue

• I think this could be a good first issue for a new contributor

[hubcio]

I dont think that this is good first issue, other than that green light from my side. At the beginning we want it to be a separate binary, not part of iggy server. is that fine @ryerraguntla ?

[ryerraguntla]

@hubcio Thats correct. It is a standalone process , interfacing Kafka client and iggy server. This initial ticket is a bare bones TCP listener, gets kafka message, if it is supported in this scope returns an acknowledgement or response. Later tickets will implement each of the functionality end to end in phases.

The proposed code structure looks something like this -

[ryerraguntla]

@spetz @hubcio @mmodzelewski @krishvishal @kparisa - I need your opinion and guidance on the below proposal. If it is acceptable, could one of you do the needful.

Proposed integration branch for multi-PR Kafka gateway work

Following up on #3421 — the Kafka gateway work is intentionally split across multiple successive PRs (foundation → Iggy bridge → additional APIs/behavior) like mentioned at #3253 (comment) . Each slice should stay reviewable on its own, but the community should be able to see cumulative progress in one place before anything lands on master.

Request

Could a maintainer please:

1. Create an integration branch on apache/iggy, e.g. iggy_gateway_for_kafka, from current master.
2. Confirm that sub-PRs for this effort should target that branch (not master) until the feature set is complete.
3. Confirm branch naming — happy to use iggy_gateway_for_kafka or whatever fits project convention.

Proposed workflow

master (stable)
│
└── iggy_gateway_for_kafka          ← integration branch (public, long-lived)
    │
    ├── PR1: TCP listener, wire codec, version firewall,
    │        stub responses, tests, kafka-tool
    │
    ├── PR2: … (next slice — TBD with maintainers)
    │
    ├── PR3: …
    │
    └── Final PR: iggy_gateway_for_kafka → master
                  (when complete; then delete branch)

• Each sub-PR: one purpose, linked to Kafka gateway: Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users #3421, with local checks per CONTRIBUTING.md.
• After each merge, iggy_gateway_for_kafka advances so anyone can browse the full in-progress tree.
• Final PR merges the integration branch into master once functionality and tests are in place.

PR1 scope (ready / nearly ready)

First sub-PR (~foundation from #3421):

• gateways/kafka/ — TCP listener, request decode, version firewall, stub responses (Produce, Fetch, ListOffsets, Metadata, ApiVersions, CreateTopics)
• Version-aware error responses (not bare 2-byte stubs for scoped APIs)
• KAFKA_ADVERTISED_HOST / broker advertisement validation
• Regression tests (103 tests) + kafka-message-gen tool for wire fixtures
• Docs: SCOPE.md, TEST_SUITE.md, MANUAL_TESTING.md
  Does not include Iggy backend bridge yet — that would be a follow-on PR on the same integration branch.

Questions for maintainers

1. Is an apache/iggy integration branch acceptable for this feature, or do you prefer sequential PRs directly to master?
2. Preferred branch name — iggy_gateway_for_kafka or other?
3. Should pre-merge CI run on PRs targeting the integration branch? Today workflows appear limited to PRs into master; I can run full local checks on each sub-PR regardless.
4. Any guidance on PR size/order beyond CONTRIBUTING (e.g. <500 LOC for new contributors, high-risk protocol areas)?

My setup

• Working branch locally: feat(gateways)/kafka_to_iggy_listener
• Fork: ryerraguntla/iggy
• Once the integration branch exists, I'll open PR1 as:
  ryerraguntla:<branch> → apache/iggy:iggy_gateway_for_kafka

[hubcio]

the one thing that actually matters for layout is workspace membership: it has to be listed in the root members and pull the sdk via iggy = { workspace = true }. that keeps it on a single lockfile with everything else (no second Cargo.lock that can quietly drift from the edge sdk it's tracking) and inside the same fmt/clippy/sort/test pipeline as the rest of the repo.

the directory itself i'm not fussed about, a top-level gateways/kafka/ is fine. just keep it a single flat crate to start (lib + thin bin, publish = false), we can grow it from there.

it shouldn't live in core/connectors/ though, even if it looks adjacent. connectors are dlopened .so plugins behind a stable ffi abi, whereas this is a standalone process speaking the kafka wire protocol and calling iggy over the network. different shape entirely.

one heads-up for the first pr:CIi components are matched by explicit path globs in .github/config/components.yml and there's no catch-all, so you'll need to add a rust-gateway component (paths: ["gateways/**"], depends_on: [rust-sdk]) and wire it into the rust aggregate. otherwise gateway-only prs match no component and quietly skip clippy/fmt/sort/test, which bites post-merge.

[hubcio]

couple more things, on your actual workflow questions (separate from the layout note above):

branch model: i'm fine with either an integration branch or sequential prs straight into master, let me get the rest of the core team's read before we lock one in. two practical notes either way: a long-lived integration branch on apache/iggy needs write/force-push rights for the rebases, which a fork contributor doesn't have, and our pre-merge ci only triggers on prs into master (branches: [master]), so sub-prs into the branch wouldn't be gated automatically. direct-to-master incremental avoids both, and it's safe here since the crate is a leaf behind an opt-in binary so master stays green between slices. branch name's a non-issue, we can settle it if we end up going that route. @spetz @numinnex any opinion on this?

pr size/order: no fixed loc cap from my side. just break it into independently-reviewable, ci-green slices and propose the boundaries, we'll iterate from there.

naming: heads-up that gateway isn't in the allowed pr-title scopes yet (.github/workflows/pr-title.yml), so add it there in the foundation pr and title things feat(gateway): .... for the binary, iggy-kafka-gateway fits the iggy-* convention (iggy-server, iggy-connectors).

[ryerraguntla]

@spetz @numinnex any opinion on this? - Could you chime in , so that I can take either one of the paths

[ryerraguntla]

Adjusting the naming convention should be iggy-gateway-kafka followed by component, which follows the iggy-connectors-* and iggy-sdk-*

[ryerraguntla]

@hubcio @spetz @numinnex Depending on the longevity of the branch and considering the potential drift from the master , effort needed to reconcile from branch to master, I considered raising PR against the master should be ok. I will make sure the time taken for CI is still in the current way without adding extra dependencies and additional complexity. With this I raised the initial PR #3519 . Though I started on the next iteration of code, I will wait till this PR is approved and merged before submitting the next PR.