feat(DOC-2012): document Kafka 4.x client compatibility and KIP-890

[mfernest]

Summary

Resolves DOC-2012.

Documents Kafka 4.x client compatibility on modules/develop/pages/kafka-clients.adoc and adds KIP-890 (Transactions Server-Side Defense) to the unsupported features list with engineering context.

This PR was originally opened by @mfernest, who is no longer with the team. @Feediver1 is taking it over to land the remaining work after the 2026-03 review round.

What changed

• Kafka 4.x compatibility statement — scoped to the Apache Kafka Java client per @mattschumpert's review note. Other clients in the validated list have not been broadly re-validated at 4.x and the page no longer claims they have.
• KIP-890 (Transactions Server-Side Defense) added to the unsupported features list with: (a) the Raft-based replication explanation @bashtanov asked for, and (b) the corrected fallback-vs-V2 wording (epoch bumping only applies to Transactions V2, which Redpanda doesn't implement).
• Frontmatter — added page-topic-type: reference, personas: developer, and two learning objectives.
• Style cleanup — Apache Kafka® trademark on first mention, removed "currently", direct imperative on the TIP, broke the SCRAM bullet's semicolon chain, monospaced pandaproxy, and resolved a CodeRabbit terminology inconsistency between "not validated" and "unsupported client".

Review history

Reviewer	Status
@bashtanov	Approved 2026-03-26 and 2026-03-27 after the KIP-890 wording was iterated to match V2 fallback semantics
@mattschumpert	Flagged the 4.x scope overclaim; addressed in this revision by narrowing the validated-at-4.x claim to the Java client (commit 69535fdf). Re-requesting review

Preview pages

• Kafka Compatibility — Self-Managed (updated)
• Kafka Compatibility — Cloud (updated)

Test plan

• Netlify preview builds without new errors
• Page renders the narrowed 4.x claim and the tightened KIP-890 bullet correctly in both cloud and non-cloud builds
• KIP-890 external link, SASL xref, and Slack link all resolve

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	3b2b884
🔍 Latest deploy log	https://app.netlify.com/projects/redpanda-docs-preview/deploys/6a19dfaca1d7a60008af5d18
😎 Deploy Preview	https://deploy-preview-1614--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request updates the Kafka clients documentation page with clarifications and additions. Changes include: adding trademark notation to "Apache Kafka®", expanding the compatibility note to include Redpanda and Kafka 4.x clients, reworded guidance for unsupported clients, and a new unsupported Kafka feature entry (KIP-890 Transactions Server-Side Defense) with details about per-transaction epoch bumping behavior. Environment-specific authentication guidance is added for both cloud and non-cloud deployments. Additional notes regarding HTTP Proxy limitations, delete.retention.ms behavior, and client quotas are incorporated with conditional visibility based on environment.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Suggested reviewers

• c4milo
• kbatuigas
• mattschumpert

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically summarizes the main changes: documenting Kafka 4.x client compatibility and KIP-890 support details, with the ticket reference for traceability.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	The PR description comprehensively documents the changes, includes the resolved JIRA ticket link, provides detailed context about the modifications, and explains the review history and testing approach.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/doc-2012-kafka4x-compatibility

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

modules/develop/pages/kafka-clients.adoc (1)

45-45: Consider aligning “unsupported client” with earlier “not validated” terminology.

Line 43 says unvalidated clients can still be compatible, while Line 45 asks users to report an “unsupported client.” “Unvalidated client” would read more consistently.

✏️ Suggested wording tweak

-If you find an unsupported client, contact the Redpanda team in the community https://redpanda.com/slack[Slack^].
+If you find an unvalidated client, contact the Redpanda team in the community https://redpanda.com/slack[Slack^].

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@modules/develop/pages/kafka-clients.adoc` at line 45, Replace the phrase
"unsupported client" with "unvalidated client" in the sentence "If you find an
unsupported client, contact the Redpanda team..." so terminology matches the
earlier "not validated" wording; update the copy that currently uses
"unsupported client" to read "If you find an unvalidated client, contact the
Redpanda team..." to maintain consistency with the phrase used on Line 43.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@modules/develop/pages/kafka-clients.adoc`:
- Line 45: Replace the phrase "unsupported client" with "unvalidated client" in
the sentence "If you find an unsupported client, contact the Redpanda team..."
so terminology matches the earlier "not validated" wording; update the copy that
currently uses "unsupported client" to read "If you find an unvalidated client,
contact the Redpanda team..." to maintain consistency with the phrase used on
Line 43.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ff2ed267-6e9a-4306-8ce8-b1bc9add75f1

📥 Commits

Reviewing files that changed from the base of the PR and between be11e77 and 061552a.

📒 Files selected for processing (1)

• modules/develop/pages/kafka-clients.adoc

[Feediver1]

Thanks for the quick turnaround @mfernest ! Reminder: be sure to include a preview link to the updated section(s). Thx!

[mfernest]

@bashtanov — gentle nudge on this one when you have a chance. Documents Kafka 4.x client compatibility and KIP-890 for DOC-2012.

[bashtanov]

Would it make sense to describe that the Kafka problem that KIP-890 solves is not present in Redpanda due to Raft's atomicity guarantees?

[mattschumpert]

cc @piyushredpanda @dotnwat

We can call out what we don't support (server side defense), butI dont think we can claim proper 4.x certification. My understanding is @bashtanov only tested Transactions with 4.x. The Aha! card got mistakedly closed.

We really need to fix failing tests with Kafka 4.x on a more broad basis before claiming victory.

[bashtanov]

My understanding is @bashtanov only tested Transactions with 4.x.

All ducktape and chaos tests that use kafka clients now use version 4.1 of it.

[mattschumpert]

Oh wow. @bashtanov is this true across all tests for ALL of these clients: https://docs.redpanda.com/current/develop/kafka-clients/ ?

[mattschumpert]

I guess we should clarify this is the Java client that is now fully certified. will have to check with the enterprise team about other clients.

[bashtanov]

@bashtanov is this true across all tests for ALL of these clients: https://docs.redpanda.com/current/develop/kafka-clients/ ?

What's this?

[Feediver1]

Re-opening this thread to close it out. To answer @bashtanov's question from 2026-03-28 — Matt was asking whether all clients in the validated table had been re-tested at Kafka 4.x. The conclusion from that thread was that only the Apache Kafka Java client at 4.x is fully validated (via ducktape and chaos test suites), and broader 4.x re-validation of the other clients hasn't happened.

I've narrowed the page's 4.x claim accordingly in commit 69535fdf so the docs no longer overstate the scope. Reposting @mattschumpert and @piyushredpanda for a final read.

[micheleRP]

PR Review (docs-team-standards)

What this PR does

Updates the Kafka clients compatibility reference page: narrows the Kafka 4.x compatibility claim to the Apache Kafka Java client (validated via ducktape/chaos suites), adds KIP-890 (Transactions Server-Side Defense) to the unsupported-features list with engineering rationale, and applies style cleanup (trademark on first mention, learning-objectives frontmatter, SCRAM bullet split, monospaced pandaproxy, imperative TIP). Single-sourced into cloud-docs. Audience: developers.

Jira ticket alignment

Ticket: DOC-2012. The PR addresses the intent — documenting Kafka 4.x client compatibility and the KIP-890 gap. Content-complete against the ticket, but blocked by open review feedback (below).

Critical issues (must fix)

1. [kafka-clients.adoc:78 — KIP-890 bullet] Disputed technical claim, flagged by an engineer today and unaddressed. The bullet currently says: "Redpanda's Raft-based replication is not susceptible to this class of errors." @bharathv commented: "I would remove 'raft-based replication' part. That has nothing to do with why Redpanda is not susceptible. A simpler rewording would be... 'Redpanda implementation of transactions is not susceptible..'"
   • Note the conflict: the Raft framing was added earlier specifically to satisfy @bashtanov's request (to explain it "due to Raft's atomicity guarantees"). Two engineers now disagree on the mechanism. This needs SME resolution before merge — recommend @bharathv and @bashtanov align rather than the docs team picking a side.
   • Likely fix (pending agreement): replace "Redpanda's Raft-based replication is not susceptible to this class of errors" with "Redpanda's transaction implementation is not susceptible to this class of errors."

Suggestions (should consider)

1. [kafka-clients.adoc:21 — Kafka client compatibility intro] @bashtanov (unaddressed): "I'm not sure I understand what it means. What is 're-validating a client at 4.x'?" The phrase "other clients in the table below have not been broadly re-validated at 4.x" is unclear to a reader who doesn't know what "validated at 4.x" means.
   • Current: "...other clients in the table below have not been broadly re-validated at 4.x but remain compatible with the Kafka protocol versions they have always supported."
   • Suggested: Clarify what validation means, e.g. "...Redpanda has not specifically tested the other clients in the table against Kafka 4.x, but they remain compatible with the Kafka protocol versions they support." Worth getting @bashtanov's sign-off on the rewording.

[micheleRP]

Final review — DOC-2012 Kafka 4.x compatibility

Thanks for landing the remaining work, @Feediver1. The content is in good shape and the engineering review has been well absorbed. One render issue to fix before merge, plus a few optional items.

🔴 Must fix before merge: learning objectives render as literal text in the Cloud build

The new learning objectives render correctly on Self-Managed but show literal {learning-objective-1} / {learning-objective-2} placeholders on Cloud:

• ✅ Self-Managed: https://deploy-preview-1614--redpanda-docs-preview.netlify.app/streaming/current/develop/kafka-clients/
• ❌ Cloud: https://deploy-preview-1614--redpanda-docs-preview.netlify.app/cloud-data-platform/develop/kafka-clients/

Why: This page is single-sourced into cloud-docs via include::...[tag=single-source]. The :learning-objective-1: / :learning-objective-2: attributes are defined in the page header, which is outside the tag::single-source[] region — so they don't travel with the include, and the cloud-docs stub doesn't define them. (Note: a docs-repo PR's own preview only renders the /streaming/ Self-Managed build, so this kind of issue isn't caught unless you also open the /cloud-data-platform/ URL.)

This is isolated to this page — every other single-sourced page that uses learning objectives already follows one of the two conventions below.

How to fix (pick one):

Option A — define inside the tag (recommended; single source of truth). In modules/develop/pages/kafka-clients.adoc, remove the two :learning-objective-N: lines from the header and re-add them immediately after the tag marker:

// tag::single-source[]
:learning-objective-1: Identify which Kafka clients are validated with Redpanda
:learning-objective-2: Identify unsupported Kafka features when integrating with Redpanda

Redpanda is compatible with Apache Kafka® versions 0.11 and later...

This matches leader-pinning.adoc, data-transforms/build.adoc, and the Iceberg pages.

Option B — duplicate in the cloud stub. Leave the docs header as-is and add the same attributes to cloud-docs/modules/develop/pages/kafka-clients.adoc frontmatter (this is what schema-reg-contexts.adoc and manage-throughput.adoc do):

= Kafka Compatibility
:page-aliases: development:kafka-clients.adoc
:description: Kafka clients, version 0.11 or later, are compatible with Redpanda. Validations and exceptions are listed.
:page-topic-type: reference
:personas: developer
:learning-objective-1: Identify which Kafka clients are validated with Redpanda
:learning-objective-2: Identify unsupported Kafka features when integrating with Redpanda

include::streaming:develop:kafka-clients.adoc[tag=single-source]

Either way, please confirm the fix on the /cloud-data-platform/ preview URL before merging. (The Pattern-B stubs also carry page-topic-type and personas for metadata parity — worth adding to the cloud stub regardless of which option you choose, though that's metadata only, not a render bug.)

🟡 Suggestions (optional)

• Kafka client compatibility intro: "...validated against Redpanda using the ducktape and chaos test suites..." — naming internal test infrastructure (ducktape, chaos) is fairly internal for a user-facing reference. Consider whether it informs the reader. "the table here" also reads slightly informally vs. "the following table." Low priority — the substance was approved by engineering.
• KIP-890 bullet: denser than its sibling bullets (Transactions V2, epoch bumping). Accurate and deliberately tightened, but if you want visual parity in the list you could move the protocol detail into a NOTE.

🟡 Pre-existing, worth a quick check

• The reworded SASL link xref:manage:security/authentication.adoc#sasl[Configure SASL authentication] keeps the #sasl fragment. I couldn't find a #sasl anchor on the target page (nearest heading is ==== SASL/SCRAM, auto-ID _sasl_scram). Antora doesn't validate cross-page fragments, so it won't fail the build, but the link may not jump to the SASL section. Pre-existing, but the new link text now promises SASL content — worth confirming the anchor.

🗨️ Unresolved review thread

• @bashtanov's first thread (mention Raft atomicity) is the only one still open. It looks superseded: @bharathv later asked to remove the Raft framing, and the final text complies ("Redpanda's implementation of transactions is not susceptible to this class of errors"). Might be worth a quick confirm + resolve so it's not left dangling.

✅ What works well

• Active voice, imperative TIP, present tense; no em dashes.
• Full module-prefixed xrefs; block-level ifdef/endif used correctly.
• Apache Kafka® trademark on first mention only.
• Valid Bloom's verb ("Identify"), capped at two objectives, correct checkbox format.
• KIP-890 wording reflects the engineering review iterations and reads accurately.
• CodeRabbit's terminology nit was addressed.

[Feediver1]

Companion cloud-docs PR opened for the Pattern B metadata mirror on the cloud stub: redpanda-data/cloud-docs#606. The two should land together so the new frontmatter (topic-type, personas, learning objectives) shows up consistently on both /current/develop/kafka-clients/ and /cloud-data-platform/develop/kafka-clients/.