fix(trainer): detect OpenShift pip permission error and document work…

[priyank766]

fix(trainer): detect OpenShift pip permission error and document workaround

Problem

fixes : #41
On OpenShift clusters with a restricted Security Context Constraint (SCC), the root filesystem is read-only. Running run_custom_training(packages=[...]) executes pip install --user (writing to /.local), which fails with a PermissionError: [Errno 13] Permission denied: '/.local' because user-defined volumes are not mounted during the pre-script installation step.

Changes

• Log Parser (kubeflow_mcp/trainer/api/monitoring.py): Added an OpenShift-specific pattern to _FAILURE_PATTERNS to catch permission errors on /.local and return the workaround suggestion.
• Agent Instructions (kubeflow_mcp/trainer/__init__.py): Added prompt guidance under INSTRUCTION_SECTIONS["training"] advising AI agents to avoid the packages parameter on OpenShift.
• Documentation (kubeflow_mcp/trainer/resources/platform-fixes.md): Documented the issue and the target-directory workaround.
• Tests (tests/unit/trainer/test_monitoring.py): Added a new test suite to verify the parser correctly matches and prioritizes the OpenShift error over generic permission errors.

Workaround

Do not use the packages parameter on OpenShift. Instead, run the installation within the training script using:

import subprocess, sys, os
lib_dir = '/workspace/lib'
os.makedirs(lib_dir, exist_ok=True)
subprocess.run([sys.executable, '-m', 'pip', 'install', '--target', lib_dir, '--quiet', 'transformers', 'peft', 'trl'], check=True)
sys.path.insert(0, lib_dir)

Verification

• All 147 unit tests pass.
• Ruff checks and formatting pass with zero violations.

[github-actions]

🎉 Welcome to the Kubeflow MCP Server! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience Slack channel
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[abhijeet-dhumal]

/lgtm

OpenShift pip permission detection is exactly what we need. Consider consolidating with #47 if both touch the same error path.

[abhijeet-dhumal]

/ok-to-test

[abhijeet-dhumal]

@priyank766 One doc nit: the detection table + "/.local emptyDir fixes most pip issues" still send agents to emptyDir for Permission denied on /.local, but new version says the packages= pre-script never gets those mounts.
Can you split that case in the table (runtime writes to emptyDir; packages= to /workspace/lib workaround) so agents don't mis-route?
WDYT?

[priyank766]

Yeah Makes sense. I've split the error cases in both platform-fixes.md and troubleshooting.md to separate runtime write issues from the packages pre-script limitation.
@abhijeet-dhumal

[abhijeet-dhumal]

/lgtm
/approve
Thanks @priyank766 for your contribution, this is great work 🚀

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhijeet-dhumal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [abhijeet-dhumal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment