chore(docs): sparkconnect (spark) client support for kubeflow-mcp-server (kep#5) by vedhakoushik · Pull Request #52 · kubeflow/mcp-server

vedhakoushik · 2026-07-11T18:57:15Z

Summary

This PR adds KEP-5: SparkConnect (Spark) Client Support for kubeflow-mcp-server — a proposal-only PR (single doc, no code) mirroring the structure of #48 (KEP-34 / Katib), per the review guidance from @abhijeet-dhumal on #51.

It proposes the Spark client module: 5 MCP tools over kubeflow.spark.SparkClient's SparkConnect session lifecycle (list/get/logs + confirm-gated create/delete), mirroring the existing trainer module.

Compatibility metrics

Per the request to align with the project's stack-compatibility framework, the KEP documents the version baseline this work targets:

Component	Version
`kubeflow` (unified SDK)	>= 0.4.0 (`kubeflow.spark.SparkClient` landed here — kubeflow/sdk#225)
`kubeflow[spark]` / `pyspark`	must match cluster Spark major.minor (default 4.0.1)
Spark Operator (SparkConnect CRD)	`sparkoperator.k8s.io/v1alpha1`
Kubeflow Trainer	>= v2.2.0
Kubernetes	>= 1.27
Python	>= 3.10

...plus the SDK-method → MCP-tool mapping and the exact SDK_COMPATIBILITY dict entry, following KEP-34's format.

KEP numbering

KEP-34 takes its number from issue #34 ("Katib (HPO) Client Support"), the tracking issue it addresses. Following that same convention, this proposal is numbered KEP-5, after tracking issue #5 ("SparkClient module for Kubeflow MCP Server"). Happy to renumber if a different scheme is preferred.

Scope

SparkConnect session management only. Spark batch jobs (SparkJob / SparkApplication) are an explicit Non-Goal, deferred to a follow-up once kubeflow/sdk#521 (currently in progress) lands in a released SDK version.

…ver (kep#5) Proposal-only PR mirroring kubeflow#48 (KEP-34 / Katib), per review guidance on kubeflow#51. Proposes the Spark client module for kubeflow-mcp-server: 5 MCP tools over kubeflow.spark.SparkClient's SparkConnect session lifecycle (list/get/logs + confirm-gated create/delete), mirroring the existing trainer module. Documents the compatibility baseline the project tracks against (kubeflow SDK >= 0.4.0, Kubeflow Trainer >= v2.2.0, SparkConnect CRD sparkoperator.k8s.io/v1alpha1, K8s >= 1.27, Python >= 3.10) and the SDK method -> MCP tool mapping, plus Goals/Non-Goals (SparkJob batch jobs are out of scope, gated on the in-progress kubeflow/sdk#521), Risks, and a Testing Plan. KEP number 5 follows the same convention as KEP-34: the number is the tracking issue in this repo (issue kubeflow#5, "SparkClient module for Kubeflow MCP Server"). A reference implementation is available in kubeflow#51. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Bellam vedha kousik <bellam.vedhakoushik@gmail.com>

google-oss-prow · 2026-07-11T18:57:21Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kramaranya for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-07-11T18:57:24Z

🎉 Welcome to the Kubeflow MCP Server! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

Slack: Join our #kubeflow-ml-experience Slack channel
Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

Copilot

Pull request overview

Proposes KEP-5 for adding SparkConnect session lifecycle support to the Spark client module.

Changes:

Defines five SparkConnect MCP tools and persona/phase integration.
Documents SDK compatibility, architecture, risks, and testing plans.
Defers Spark batch jobs to follow-up work.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

vedhakoushik · 2026-07-11T19:12:02Z

+| `data-scientist` (inherits readonly) | `create_spark_session`, `delete_spark_session` |
+| `ml-engineer` / `platform-admin` | inherit the above / unrestricted |
+
+`delete_spark_session` is added to `DESTRUCTIVE_TOOLS`.


Confirmed and fixed in 1160d5e. Verified the trainer's pattern (is_mcp_managed() / MCP_MANAGED_LABEL in common/utils.py, enforced in trainer/api/lifecycle.py, label attached at creation in trainer/api/training.py) and mirrored it in the design: create_spark_session now labels sessions for ownership, and delete_spark_session is documented as ownership-guarded for non-platform-admin personas. Added a Risk row and a Testing Plan item for the accept/reject paths.

vedhakoushik · 2026-07-11T19:12:04Z

+
+| Tool | Description | SDK Method |
+|------|-------------|------------|
+| `get_spark_session_logs` | Bounded driver-pod logs (`tail_lines`) | `get_session_logs()` |


Confirmed and fixed in 1160d5e. Checked the SDK's backend.py: get_session_logs() calls read_namespaced_pod_log() with no server-side tail limit, so the full log is materialized before our tail_lines truncation applies. Reworded the tool descriptions to stop overclaiming "bounded," and added a Risk row proposing to bypass the SDK wrapper and call CoreV1Api.read_namespaced_pod_log(tail_lines=...) directly for a true server-side bound — mirroring the precedent already in trainer/api/discovery.py.

vedhakoushik · 2026-07-11T19:12:05Z

+
+| Component | Version | Notes |
+|-----------|---------|-------|
+| `kubeflow` (unified SDK) | >= 0.4.0 | Provides `kubeflow.spark.SparkClient` via the `[spark]` extra (landed in SDK 0.4 — [kubeflow/sdk#225](https://github.com/kubeflow/sdk/pull/225)) |


Confirmed and fixed in 1160d5e — good catch. I downloaded and inspected the actual kubeflow==0.4.0 and 0.4.1 wheels from PyPI: both name the field pod_name, not driver_pod_name. I'd read driver_pod_name from GitHub's default branch (unreleased main), not a tagged release. Updated the design to serialize pod_name first with a driver_pod_name fallback, and added a compatibility-table note. This also means the reference implementation on #51 has a real bug — getattr(info, "driver_pod_name", None) silently returns None against the declared >=0.4.0 baseline — which needs a follow-up fix there.

vedhakoushik · 2026-07-11T19:12:07Z

+  connection. The tool provisions the session, **releases the transient `SparkSession`**, and
+  returns session metadata + connect info; the data plane attaches separately. If `connect()`
+  fails *after* the CR is created (e.g. the MCP host can't reach the driver), the tool reports
+  the provisioned session with a warning rather than losing it.


Confirmed and fixed in 1160d5e. Verified in KubernetesBackend.connect(): outside the cluster it spawns a kubectl port-forward subprocess that SparkSession.stop() does not terminate. Documented this as a known limitation on create_spark_session with a recommended mitigation (in-cluster deployment as the supported mode), and added a Risk row.

All four verified against the actual code/SDK before fixing: - Serialize the released SDK field name, not main. Downloaded and inspected the real kubeflow==0.4.0 and 0.4.1 wheels from PyPI: SparkConnectInfo names the driver-pod field `pod_name` on both released versions; `driver_pod_name` only exists on unreleased SDK main (I had read GitHub's default branch, not a tagged release). Design now specifies pod_name first, driver_pod_name fallback, with a matching test. This also means the reference implementation on kubeflow#51 has a real bug (getattr(..., "driver_pod_name", None) silently returns None against the declared >=0.4.0 baseline) that needs a follow-up fix there. - Ownership guard on delete_spark_session. Confirmed the trainer module's precedent (is_mcp_managed() / MCP_MANAGED_LABEL in common/utils.py, enforced in trainer/api/lifecycle.py, label attached at creation in trainer/api/training.py). The Spark design had no equivalent, so any persona with delete access could remove any SparkConnect session in an allowed namespace, not just MCP-created ones. Design now mirrors the trainer pattern. - get_session_logs() log-bounding is client-side only. Confirmed via the SDK's backend.py: read_namespaced_pod_log() is called without a server-side tail limit, so the full log is materialized before our tail_lines truncation applies. Reworded tool descriptions to stop overclaiming "bounded" and added a mitigation (bypass the SDK wrapper for a true server-side tail, matching trainer's existing direct-CoreV1Api precedent in discovery.py). - Out-of-cluster port-forward leak on create_spark_session. Confirmed via the SDK's KubernetesBackend.connect(): outside the cluster it spawns a kubectl port-forward subprocess that SparkSession.stop() does not terminate. Documented as a known limitation with a recommended mitigation (in-cluster deployment). Updated: SDK Compatibility table + note, Goals, Key Design Decisions (2 new bullets + 1 extended), Non-Goals, MCP Tools tables, Persona Coverage, SDK Compatibility Update mapping, Risks and Mitigations (+3 rows), Testing Plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Bellam vedha kousik <bellam.vedhakoushik@gmail.com>

abhijeet-dhumal · 2026-07-12T08:40:49Z

Thanks for the KEP, @vedhakoushik.. this is very well-scoped draft. 🚀

abhijeet-dhumal · 2026-07-12T08:42:32Z

+- **`follow=True` streaming logs** — a stateless MCP tool returns a snapshot, truncated
+  client-side to the last `tail_lines` (see Risks for the retrieval-bound caveat).
+
+## Proposal


Can we consider adding a planning.py under api/ for a spark_pre_flight tool to check CRD existence (sparkoperator.k8s.io/v1alpha1), Spark Operator controller health, and whether kubeflow[spark] is importable. This follows the pattern in both the trainer (planning.py) and KEP-34 (katib_pre_flight).

Added in 143288a — a new Planning category with spark_pre_flight (CRD existence, Spark Operator controller health, kubeflow[spark] importability), mirroring the trainer's pre_flight()/check_compatibility() and KEP-34's katib_pre_flight. Included a scope note on how it's narrower (readiness only, no resource sizing — Spark sessions are user-sized via num_executors/executor_resources rather than estimated), a "Tools Beyond SDK Abstraction" entry, and threaded it through Persona Coverage, Tool Phase Categories, and the Testing Plan. Now 6 tools across 4 categories.

abhijeet-dhumal · 2026-07-12T08:44:13Z

+},
+```
+
+### Cross-Client Integration


This section could use concrete TOOL_NEXT_HINTS entries, for example:
create_spark_session - "Use connect info to attach PySpark, then pre_flight() to set up training"
or
wait_for_training - "Use create_spark_session() for data preparation"

wdyt?

Added in 143288a — a concrete TOOL_NEXT_HINTS table for all six tools under Cross-Client Integration. create_spark_session's hint points into the trainer's pre_flight(), surfacing the data-prep → train handoff. I noted the reverse-direction hint (a training-side hint back toward create_spark_session) as a possible small follow-up rather than committing to it here, since it'd mean editing the trainer module's existing hints — out of scope for this Spark-focused KEP. Happy to fold it in if you'd rather have it now.

abhijeet-dhumal · 2026-07-12T08:46:17Z

+  `delete_spark_session` for non-admin personas). The SDK is mocked, so tests run **without**
+  the `kubeflow[spark]` extra.
+- **Server load**: `create_server(clients=["trainer", "spark"])` registers all five tools in
+  both `full` and `progressive` modes.


The unit test plan is solid. Could you add a note on integration tests (Kind + Spark Operator), even if deferred?
The trainer and KEP-34 both include integration test plans against a live cluster.

Added in 143288a — a "Testing Plan > Integration Tests (deferred)" subsection mirroring the trainer's and KEP-34's plans: Kind cluster + Spark Operator (pre-flight → create → poll to Ready → logs → delete), the ownership-guard accept/reject paths, and the cross-client create → trainer pre_flight() handoff.

@abhijeet-dhumal

…est plan Addresses @abhijeet-dhumal's three review suggestions: 1. Add a Planning category with `spark_pre_flight` (6th tool): verifies the SparkConnect CRD (sparkoperator.k8s.io/v1alpha1), Spark Operator controller health, and kubeflow[spark] importability. Mirrors the trainer's pre_flight()/check_compatibility() and KEP-34's katib_pre_flight, including a scope note on how it differs (narrower — readiness only, no resource sizing) and a "Tools Beyond SDK Abstraction" entry + k8s_api_operations snippet entry, matching KEP-34's structure for the same kind of tool. Threaded the new tool through every affected section: Summary/Goals tool count, Module Structure, MCP Tools, Persona Coverage, Tool Phase Categories, Instruction Sections, Testing Plan. 2. Add a concrete TOOL_NEXT_HINTS table for all six tools under Cross-Client Integration, including create_spark_session's hint into the trainer's pre_flight() as the data-prep -> train handoff. Noted the reverse-direction hint (editing trainer's existing hints) as a possible follow-up rather than in scope here. 3. Add a Testing Plan > Integration Tests (deferred) subsection mirroring the trainer's and KEP-34's integration-test plans: Kind + Spark Operator, ownership-guard accept/reject, and the cross-client create -> pre_flight handoff. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Bellam vedha kousik <bellam.vedhakoushik@gmail.com>

Two Step-A fixes ahead of the KEP-5 design work landing: - kubeflow_mcp/spark/types/__init__.py: session_info_to_dict() read driver_pod_name via getattr, which silently returns None against the declared kubeflow[spark]>=0.4.0 baseline — the released 0.4.0/0.4.1 wheels name the field pod_name; driver_pod_name only exists on unreleased SDK main. Read pod_name first, fall back to driver_pod_name, so session info is correct today and keeps working if the rename ships in a future release. Added regression tests for both field shapes. - Removed docs/design/spark-client.md, superseded by the KEP-5 design proposal (kubeflow#52, docs/proposals/KEP-5.md in that PR). Repointed ARCHITECTURE.md's Spark row to kubeflow#52. Rebased onto upstream/main to resolve the README.md merge conflict (upstream added --otel-endpoint to the serve example; kept our copy/paste-safe table format and folded the new flag into it). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Bellam vedha kousik <bellam.vedhakoushik@gmail.com>

abhijeet-dhumal · 2026-07-18T18:33:23Z

+  `kubectl port-forward` subprocess to reach the driver; `SparkSession.stop()` does not
+  terminate it, so repeated calls in an out-of-cluster deployment can leak subprocesses and
+  listening ports (see Risks).
+- **Ownership guard on delete.** `delete_spark_session` mirrors the trainer's `is_mcp_managed()`


Please clarify that we reuse the pattern (label on create, guard on delete), not 'is_mcp_managed()' as implemented today... that helper is TrainJob-only (trainer.kubeflow.org/trainjobs). Spark needs a SparkConnect CR lookup (or a shared helper that takes group/version/plural) right?

On Kubeflow==0.4.0, label via options=[Labels(...)] on create, then on delete read the SparkConnect CR (sparkoperator.k8s.io/v1alpha1, plural sparkconnects). SparkConnectInfo has no labels field, so the guard has to hit the custom-objects API (same idea as trainer).

right, is_mcp_managed() is TrainJob-only. lets switch to reusing the pattern, not the helper: label the SparkConnect CR on create via options=[Labels(...)], guard on delete by reading the CR through CustomObjectsApi (sparkoperator.k8s.io/v1alpha1, sparkconnects) since SparkConnectInfo has no labels field. Noted a shared is_mcp_managed(group, version, plural, …) as the cleaner fix.

abhijeet-dhumal · 2026-07-18T18:38:11Z

+
+| Component | Version | Notes |
+|-----------|---------|-------|
+| `kubeflow` (unified SDK) | >= 0.4.0 | Provides `kubeflow.spark.SparkClient` via the `[spark]` extra (landed in SDK 0.4 — [kubeflow/sdk#225](https://github.com/kubeflow/sdk/pull/225)) |


Please pin verification to kubeflow[spark]==0.4.0 (floor). 0.4.1 is Spark-API-compatible today; don’t depend on unreleased main (driver_pod_name).

Also: the package is pyspark-connect, not plain pyspark. Worth fixing the wording here.

Done. Pinned verification to kubeflow[spark]==0.4.0, no dependence on unreleased main. Fixed the package name to pyspark-connect.

abhijeet-dhumal · 2026-07-18T18:40:13Z

+  under the same module.
+- **Proxying Spark RPCs** through the MCP server — the server manages session *lifecycle*; the
+  data plane attaches with PySpark using the returned connect info.
+- **Attaching to a pre-existing external Spark Connect server** (`connect(base_url=…)` returns a


On 0.4.0, session info has no ready-to-use sc://… URL.
Can we add a connect_url field when we serialize, using: sc://{service}.{namespace}.svc.cluster.local:15002
Otherwise agents can’t attach from what we return.. right?

Added a connect_url field to the serialization: sc://{service}.{namespace}.svc.cluster.local:15002, so responses are attachable.

abhijeet-dhumal · 2026-07-18T18:43:48Z

+|------|------|
+| `spark_pre_flight` | If checks pass, call `create_spark_session()` to provision a session |
+| `list_spark_sessions` | Call `get_spark_session(name)` for details on a specific session |
+| `get_spark_session` | Use `get_spark_session_logs(name)` for driver output, or `delete_spark_session(name)` to tear down |


nit: thinking log fetch pulls the whole log first, then we trim it. That can blow memory on a noisy driver.

Maybe we can mark “use Kubernetes tail_lines” as something we should do in the first implementation, not a maybe-later note.. wdyt?

Added first implementation. The function get_spark_session_logs now calls CoreV1Api.read_namespaced_pod_log(tail_lines=...) directly, bypassing the SDK wrapper. I updated the tool table, mapping, Risks row, and SDK snippet.

…nect_url, server-side log tail Signed-off-by: Bellam vedha kousik <bellam.vedhakoushik@gmail.com>

Copilot AI review requested due to automatic review settings July 11, 2026 18:57

google-oss-prow Bot requested review from andreyvelich, astefanutti and szaher July 11, 2026 18:57

google-oss-prow Bot added the size/L label Jul 11, 2026

Copilot started reviewing on behalf of vedhakoushik July 11, 2026 18:57 View session

vedhakoushik mentioned this pull request Jul 11, 2026

feat: add spark client module for SparkConnect sessions #51

Open

Copilot AI reviewed Jul 11, 2026

View reviewed changes

abhijeet-dhumal reviewed Jul 12, 2026

View reviewed changes

abhijeet-dhumal reviewed Jul 18, 2026

View reviewed changes

docs(spark): address KEP-5 review — ownership guard, version pin, con…

96b9fcf

…nect_url, server-side log tail Signed-off-by: Bellam vedha kousik <bellam.vedhakoushik@gmail.com>

Uh oh!

Conversation

vedhakoushik commented Jul 11, 2026

Summary

Compatibility metrics

KEP numbering

Scope

Related

Uh oh!

google-oss-prow Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhijeet-dhumal commented Jul 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhijeet-dhumal Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhijeet-dhumal Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abhijeet-dhumal Jul 18, 2026 •

edited

Loading

abhijeet-dhumal Jul 18, 2026 •

edited

Loading