Add document for setting up aws msk provisioned cluster. - #23361
Conversation
ti-chi-bot
Bot
added
contribution
|
Hi @ginkgoch. Thanks for your PR. I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
missing-translation-status
📝 WalkthroughWalkthroughAdds documentation for connecting TiDB Cloud Premium to Amazon MSK Provisioned through AWS PrivateLink, including AWS preparation, Kafka authorization, multi-VPC configuration, and the TiDB Cloud connection flow. ChangesMSK Provisioned PrivateLink integration
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant AWSMSKProvisioned
participant EC2KafkaClient
participant TiDBCloud
Operator->>AWSMSKProvisioned: Create VPC, SCRAM secret, and cluster
EC2KafkaClient->>AWSMSKProvisioned: Configure Kafka ACLs
Operator->>AWSMSKProvisioned: Enable multi-VPC connectivity and attach cluster policy
TiDBCloud->>AWSMSKProvisioned: Create PrivateLink connection using cluster ARN
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@sdojjy: adding LGTM is restricted to approvers and reviewers in OWNERS files. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 28cd1bbe-4d73-42f4-b432-88292887a4e9
📒 Files selected for processing (3)
TOC-tidb-cloud-premium.mdtidb-cloud/premium/set-up-sink-private-endpoint-premium.mdtidb-cloud/setup-aws-msk-provisioned-private-link-service.md
|
|
||
| 1. In the TiDB Cloud console, navigate to the overview page of your instance, and click **Settings** > **Networking**. | ||
| 2. In the **Private Link Endpoint For External Services** area, click **Create Private Endpoint for External Services**. | ||
| 3. In the dialog, switch the **Connection Type** to **AWS MSK Provisioned** and note the **AWS Account ID** and **availability zone IDs** (e.g., `use1-az1`). |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use “for example” instead of “e.g.,”.
This is a reported Vale failure. Replace e.g., with for example.
Proposed wording:
3. In the dialog, switch the **Connection Type** to **AWS MSK Provisioned** and note the **AWS Account ID** and **availability zone IDs** (for example, `use1-az1`).
As per path instructions, preserve the repository’s Markdown conventions.
🧰 Tools
🪛 GitHub Check: vale
[failure] 21-21:
[vale] reported by reviewdog 🐶
[PingCAP.Latin] Use 'for example' instead of 'e.g.,'.
Raw Output:
{"message": "[PingCAP.Latin] Use 'for example' instead of 'e.g.,'.", "location": {"path": "tidb-cloud/setup-aws-msk-provisioned-private-link-service.md", "range": {"start": {"line": 21, "column": 144}}}, "severity": "ERROR"}
🪛 LanguageTool
[style] ~21-~21: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...ndpoint for External Services**. 3. In the dialog, switch the **Connection Typ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
Sources: Path instructions, Linters/SAST tools
|
|
||
| - **Region**: Same AWS region as your TiDB Cloud Premium instance. | ||
| - **Availability Zones**: Must match your TiDB Cloud instance AZs (verify by AZ ID, not AZ name). | ||
| - **Authentication**: SASL/SCRAM must be enabled. IAM-only authentication does not work with PrivateLink. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
AWS MSK multi-VPC PrivateLink supports IAM TLS SASLSCRAM documentation
💡 Result:
Yes, Amazon MSK multi-VPC private connectivity (powered by AWS PrivateLink) supports IAM, TLS (mTLS), and SASL/SCRAM authentication schemes [1][2]. Unauthenticated clusters are not supported [1][2]. Key considerations and requirements for using these authentication schemes with multi-VPC private connectivity include: - Cluster Version: Multi-VPC private connectivity requires Apache Kafka 2.7.1 or higher [1][3]. - Configuration for SASL/SCRAM and TLS: If you are using SASL/SCRAM or mTLS, you must set Apache Kafka ACLs for your cluster and ensure that the cluster configuration property allow.everyone.if.no.acl.found is set to false [1][4]. - Managed VPC Connections: You must create a managed VPC connection for each client and authentication scheme [5]. Each managed VPC connection can be associated with only one authentication scheme [5]. - Enabling Connectivity: Cluster owners must explicitly turn on multi-VPC private connectivity for the specific authentication schemes they intend to use, which can be managed via the AWS Management Console or the UpdateConnectivity API [6][2]. - Authorization: When using IAM client authentication, you can leverage cluster policies to define granular data plane permissions for connecting clients [1][7]. For SASL/SCRAM or mTLS, standard Apache Kafka ACLs are used [1][4].
Citations:
- 1: https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html
- 2: https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cluster-owner-action-turn-on.html
- 3: https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-getting-started.html
- 4: https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cross-account-permissions.html
- 5: https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cross-account-user-action.html
- 6: https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cross-account-update-authschemes.html
- 7: https://aws.amazon.com/blogs/big-data/connect-kafka-client-applications-securely-to-your-amazon-msk-cluster-from-different-vpcs-and-aws-accounts/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Target file context:"
cat -n tidb-cloud/setup-aws-msk-provisioned-private-link-service.md | sed -n '1,80p'
echo
echo "Search related authentication terminology:"
rg -n "IAM-only|SASL/SCRAM|PrivateLink|private link|multi-VPC|Authentication" tidb-cloud/setup-aws-msk-provisioned-private-link-service.md resources/doc-templates --glob '!**/*.{png,jpg,jpeg,svg,ico}' | head -200Repository: pingcap/docs
Length of output: 9353
Do not claim IAM-only authentication is incompatible with MSK PrivateLink.
AWS MSK multi-VPC PrivateLink supports IAM, TLS (mTLS), and SASL/SCRAM. Restrict this to the TiDB Cloud-specific authentication requirement, or explicitly link TiDB Cloud’s SASL/SCRAM private endpoint limitation if relevant.
Committable replacement
- **Authentication**: Enable SASL/SCRAM for this TiDB Cloud connection. AWS MSK multi-VPC connectivity also supports IAM and TLS, but this setup uses SASL/SCRAM.Source: MCP tools
| - **Region**: Same AWS region as your TiDB Cloud Premium instance. | ||
| - **Availability Zones**: Must match your TiDB Cloud instance AZs (verify by AZ ID, not AZ name). | ||
| - **Authentication**: SASL/SCRAM must be enabled. IAM-only authentication does not work with PrivateLink. | ||
| - **Broker type**: `t4.small` is not supported. Choose a larger type. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate target file:"
fd -a 'setup-aws-msk-provisioned-private-link-service\.md$' . || true
echo
echo "Current line context:"
if [ -f tidb-cloud/setup-aws-msk-provisioned-private-link-service.md ]; then
sed -n '24,38p' tidb-cloud/setup-aws-msk-provisioned-private-link-service.md | cat -n
fi
echo
echo "Search for t4.small/t3.small mentions:"
rg -n 't[34]\.small|AWS Access|multi-VPC|private connectivity|Broker type' -S .Repository: pingcap/docs
Length of output: 11587
🌐 Web query:
AWS MSK Kafka broker types t3.small unsupported multi-VPC private connectivity docs.aws.amazon.com/msk latest developerguide aws-access-mult-vpc.html
💡 Result:
Amazon MSK multi-VPC private connectivity explicitly does not support the t3.small broker instance type [1][2][3]. According to official AWS documentation, this limitation is one of the key requirements and constraints for using the multi-VPC private connectivity feature [1]. To use multi-VPC private connectivity, your Amazon MSK cluster must meet several criteria, including [1][3]: - Broker Type: You must use supported instance types (such as M5 or M7g families); t3.small instances are unsupported [1][4][5]. - Apache Kafka Version: The cluster must be running Apache Kafka version 2.7.1 or higher [1][6]. - Authentication: The feature requires an authenticated cluster (IAM, SASL/SCRAM, or mTLS); it is not supported for unauthenticated (plaintext) clusters [1][6][7]. - Infrastructure: You must have an equal number of client subnets and cluster subnets, and the Availability Zone IDs must match [1][3]. This feature is designed for single-region, multi-VPC (including cross-account) connectivity using AWS PrivateLink technology [1][8][3]. If your workload currently runs on t3.small brokers and requires multi-VPC private connectivity, you will need to migrate your cluster to a supported instance type (e.g., M5 or M7g) [4][5].
Citations:
- 1: https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html
- 2: aws-msk-alpha: support multi-VPC private connectivity aws/aws-cdk#28640
- 3: https://www.automq.com/blog/msk-privatelink-secure-kafka-connectivity-and-cost
- 4: https://docs.aws.amazon.com/msk/latest/developerguide/broker-instance-sizes.html
- 5: https://docs.aws.amazon.com/msk/latest/developerguide/broker-instance-types.html
- 6: https://aws.amazon.com/msk/faqs/
- 7: https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cluster-owner-action-turn-on.html
- 8: https://docs.aws.amazon.com/msk/latest/developerguide/aws-access.html
Correct the unsupported broker type.
AWS documents t3.small, not t4.small, as the broker type that is unsupported for MSK multi-VPC private connectivity.
Committable replacement
- **Broker type**: `t3.small` is not supported. Choose a larger type.Source: MCP tools
|
|
||
| ## Step 2. Create a SCRAM Secret in AWS Secrets Manager | ||
|
|
||
| Create a secret in AWS Secrets Manager to store the SASL/SCRAM credentials that TiDB Cloud will use to authenticate against your MSK cluster. This secret requires a custom AWS KMS encryption key — the default AWS managed key will not work. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Replace e.g., with “for example”.
These are reported Vale failures.
Proposed wording:
- Line 47:
Create a symmetric KMS key and give it an alias for easy reference. Then create a secret whose name **must** start with \AmazonMSK_` (for example, `AmazonMSK_tidb_msk`). The secret value is a JSON object containing a `username` and `password`.` - Line 49:
Then create a secret whose name **must** start with \AmazonMSK_` (for example, `AmazonMSK_tidb_msk`).` - Line 57:
- **Kafka version**: Use a version supported by TiDB Cloud (for example, 3.7.x).
As per path instructions, preserve the repository’s Markdown conventions.
Also applies to: 49-49, 57-57
🧰 Tools
🪛 GitHub Check: vale
[failure] 47-47:
[vale] reported by reviewdog 🐶
[PingCAP.EmDash] Don't put a space before or after a dash.
Raw Output:
{"message": "[PingCAP.EmDash] Don't put a space before or after a dash.", "location": {"path": "tidb-cloud/setup-aws-msk-provisioned-private-link-service.md", "range": {"start": {"line": 47, "column": 195}}}, "severity": "ERROR"}
Sources: Path instructions, Linters/SAST tools
| wget https://download.java.net/java/GA/jdk22.0.2/c9ecb94cd31b495da20a27d4581645e8/9/GPL/openjdk-22.0.2_linux-x64_bin.tar.gz | ||
| tar -zxf openjdk-22.0.2_linux-x64_bin.tar.gz |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/sh
curl -fsSL 'https://kafka.apache.org/37/operations/java-version/' | grep -E 'Java (8|11|17)'Repository: pingcap/docs
Length of output: 1840
🏁 Script executed:
#!/bin/sh
set -eu
# Locate and inspect the referenced file around the JDK installation plus nearby Kafka context.
file="tidb-cloud/setup-aws-msk-provisioned-private-link-service.md"
if [ -f "$file" ]; then
wc -l "$file"
sed -n '60,110p' "$file"
else
echo "Referenced file not found"
fi
# Search for adjacent Kafka version mentions to establish tooling context.
printf '\nKafka version references:\n'
rg -n "Kafka 3\.7|Kafka 3\b|kafka|Java 22|openjdk|jdk" "$file" || trueRepository: pingcap/docs
Length of output: 3665
🌐 Web query:
Apache Kafka 3.7 Java Version page Java 8 Java 11 Java 17 supported
💡 Result:
Apache Kafka 3.7 supports Java 8, Java 11, and Java 17 [1]. However, there are important deprecation notices and nuances regarding which components these versions support [1][2][3]: - Java 8: Support for Java 8 is deprecated project-wide and has been since Kafka 3.0 [1][4][3]. - Java 11: Support for the Kafka broker and tools is deprecated in Kafka 3.7 [1][2][3]. It remains supported for other modules, such as clients, Streams, and Connect [2][5]. - Java 17: Kafka 3.7 supports Java 17 [1][4]. It is highly recommended due to performance improvements [1]. In Apache Kafka 3.7, both Java 8 and Java 11 (for the broker and tools) are planned for removal in Kafka 4.0 [1][4][3]. While Kafka is built and tested with Java 8, 11, 17, and 21, the project enforces compatibility with Java 8 or higher through build settings [3]. Users are advised to transition to newer Java versions to maintain compatibility with future releases [1][4].
Citations:
- 1: https://kafka.apache.org/37/operations/java-version/
- 2: https://kafka.apache.org/blog/2024/02/27/apache-kafka-3.7.0-release-announcement/
- 3: https://github.com/apache/kafka/tree/3.7
- 4: https://kafka.apache.org/37/getting-started/upgrade/
- 5: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
Use a Kafka 3.7-supported Java runtime.
Kafka 3.7.1 documents Java 8, 11, and 17 as supported for tools; Java 22 is not listed. Use Java 17 here, or document/verify Java 22 compatibility for these exact Kafka ACL tools before publishing.
Source: MCP tools
|
|
||
| ### 5.3 Retrieve the Bootstrap Brokers | ||
|
|
||
| In the AWS MSK console, open your cluster's **Summary** section. Click **View client information** in the upper-right corner. In the popup, locate the **SASL/SCRAM Private endpoint** entry — this is your bootstrap server URI. Copy it for use in the ACL commands below. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Remove spaces around em dashes.
These are reported Vale failures.
Proposed wording:
- Line 110:
In the popup, locate the **SASL/SCRAM Private endpoint** entry. This is your bootstrap server URI. Copy it for use in the ACL commands below. - Line 149:
This operation triggers a long-running cluster update. Expect it to take approximately 60 to 75 minutes. You can monitor progress in the MSK console under the **Cluster operations** tab. Wait for the cluster to return to **Active**.
As per path instructions, preserve the repository’s Markdown conventions.
Also applies to: 149-149
🧰 Tools
🪛 GitHub Check: vale
[failure] 110-110:
[vale] reported by reviewdog 🐶
[PingCAP.EmDash] Don't put a space before or after a dash.
Raw Output:
{"message": "[PingCAP.EmDash] Don't put a space before or after a dash.", "location": {"path": "tidb-cloud/setup-aws-msk-provisioned-private-link-service.md", "range": {"start": {"line": 110, "column": 189}}}, "severity": "ERROR"}
Sources: Path instructions, Linters/SAST tools
|
|
||
| Create the private link connection in TiDB Cloud using the ARN of your MSK cluster. | ||
|
|
||
| For more information, see [Create an Amazon MSK Provisioned private link connection](https://docs.pingcap.com/tidbcloud/). No newline at end of file |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Link to the actual TiDB Cloud connection instructions.
This points to the documentation homepage instead of the existing MSK Provisioned connection flow in tidb-cloud/premium/set-up-sink-private-endpoint-premium.md.
Committable replacement
For more information, see [Create an Amazon MSK Provisioned private link connection](/tidb-cloud/premium/set-up-sink-private-endpoint-premium.md#step-2-configure-the-private-endpoint-for-changefeeds).| - Reference | ||
| - [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) | ||
| - [Set Up Private Endpoint for Changefeeds](/tidb-cloud/premium/set-up-sink-private-endpoint-premium.md) | ||
| - [Set Up MSK Provisioned Cluster via Private Link in AWS](/tidb-cloud/premium/set-up-sink-private-endpoint-premium.md) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Point the TOC entry at the new MSK setup guide.
The current target duplicates line 140 and does not link to the document named by this entry.
Committable replacement
- [Set Up MSK Provisioned Cluster via Private Link in AWS](/tidb-cloud/setup-aws-msk-provisioned-private-link-service.md)2 participants
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Added guide for setting up aws msk provisioned for changefeed on premium.
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit