fix(trainer): inject HF_HOME on fine_tune pods to fix OpenShift PermissionError#83
fix(trainer): inject HF_HOME on fine_tune pods to fix OpenShift PermissionError#83ptalgulk01 wants to merge 1 commit into
Conversation
…hift compatibility Signed-off-by: Prachiti Talgulkar <ptalgulk01@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
🎉 Welcome to the Kubeflow MCP Server! 🎉 Thanks for opening your first PR! We're happy to have you as part of our community 🚀 Here's what happens next:
Join the community:
Feel free to ask questions in the comments if you need any help or clarification! |
ptalgulk01
changed the title
There was a problem hiding this comment.
Pull request overview
Adds HF_HOME injection intended to make fine-tuning compatible with OpenShift’s restricted filesystem.
Changes:
- Injects
HF_HOME=/workspace/.hfthrough runtime patches. - Propagates environment variables to initializer containers.
- Documents the automatic cache configuration.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
kubeflow_mcp/trainer/api/training.py |
Adds and propagates HF_HOME. |
kubeflow_mcp/trainer/__init__.py |
Updates trainer instructions. |
kubeflow_mcp/trainer/resources/platform-fixes.md |
Documents OpenShift cache handling. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| affinity=affinity, | ||
| service_account_name=service_account_name, | ||
| image_pull_secrets=image_pull_secrets, | ||
| env=hf_home_env, |
There was a problem hiding this comment.
@ptalgulk01 this seems legit issue.. can we fix this?
This goes through RuntimePatch / spec.runtimePatches. Please put node env on spec.trainer.env and initializer env on the initializer env fields.
| if has_initializers and (volumes or volume_mounts or env): | ||
| for init_name in ("dataset-initializer", "model-initializer"): | ||
| init_containers = None | ||
| if volume_mounts: | ||
| init_containers = [ContainerPatch(name=init_name, volume_mounts=volume_mounts)] | ||
| if volume_mounts or env: |
There was a problem hiding this comment.
@ptalgulk01 can you try resolving this copilot review, this seems legit too?
|
|
||
| `fine_tune()` automatically sets `HF_HOME=/workspace/.hf` on all pods (node, dataset-initializer, model-initializer). This redirects HuggingFace cache writes from the default `/.cache/huggingface` (read-only under OpenShift's restricted SCC) to the writable `/workspace` PVC. | ||
|
|
||
| No user action is needed for `fine_tune()`. For `run_custom_training()` or `run_container_training()` that use HuggingFace libraries, pass: |
There was a problem hiding this comment.
Hmm.. /workspace must be mounted writable.. that path isn’t auto-injected for container training. Otherwise HF_HOME=/workspace/.hf can still fail on a read-only rootfs.
cc: @ptalgulk01
Description
PermissionError: [Errno 13] Permission denied: '/.cache'becausevHuggingFace defaults to writing to the read-only/.cache/huggingfaceunder restricted SCC.HF_HOME=/workspace/.hfon all fine_tune pods (node, dataset-initializer, model-initializer) via_build_runtime_patch()./workspaceis always writable (provided by the ClusterTrainingRuntime PVC).Changes
HF_HOME_PATHconstant and pass it as env to_build_runtime_patch()infine_tune()_build_runtime_patch()to propagateenvto initializer containers (previously onlyvolume_mountswere forwarded)Related Issue
Fixes #33
Checklist
make test-python)make verify)git commit -s)Testing