[Bug] [Worker] File parameter transfer feature is broken

[coldWord]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

File parameter transfer feature is completely broken in DolphinScheduler 3.4.1. The upstream task generates files with OUT type FILE parameters, but the downstream task cannot receive the files. The resourceItemMap={} in logs indicates that no files were downloaded to the downstream task.

After deep investigation, I found that commit 6f12d1ded9 ([Fix-17184] Fix varpool cannot use #17199) deleted the TaskFilesTransferUtils.java, which contains the critical file transfer logic:

• uploadOutputFiles() - uploads output files to S3 (DATA_TRANSFER directory)
• downloadUpstreamFiles() - downloads upstream files from S3 to local

Without these functions, files remain only in the local task working directory and downstream tasks cannot access them.

What you expected to happen

The file parameter transfer should work as documented:

1. Upstream task (ot) generates files and declares FILE type OUT parameters
2. System automatically uploads files to S3 (DATA_TRANSFER directory)
3. Downstream task (n2) declares FILE type IN parameters with reference ot.dir-data
4. System automatically downloads files from S3 to n2's working directory before execution
5. n2 can access the files locally

How to reproduce

1. Create a workflow with two Shell tasks (ot -> n2)
2. Configure ot task:
   • rawScript: mkdir -p data/test1 data/test2 && echo "test1 message" >> data/test1/text.txt && echo "test2 message" >> data/test2/text.txt && tree .
   • localParams: dir-data (OUT, FILE, value=data)
3. Configure n2 task:
   • rawScript: tree . && cat input_dir/test1/text.txt && cat input_dir/test2/text.txt
   • localParams: input_dir (IN, FILE, value=ot.dir-data)
4. Execute workflow
5. Result: n2 fails with "No such file or directory"

Anything else

MinIO bucket verification (empty):

$ mc ls myminio/dolphinscheduler/
[2026-04-10 11:40:25 CST]     0B dolphinscheduler/

No DATA_TRANSFER directory - files were never uploaded.

ot task (upstream) logs:

Initialize shell task params {
  "localParams" : [ {
    "prop" : "dir-data",
    "direct" : "OUT",
    "type" : "FILE",
    "value" : "data"
  } ],
  "varPool" : [ ],
  ...
}
Set taskVarPool: null successfully

n2 task (downstream) logs:

Download resources successfully: ResourceContext(resourceItemMap={})
Initialize shell task params {
  "localParams" : [ {
    "prop" : "input_dir",
    "direct" : "IN",
    "type" : "FILE",
    "value" : "ot.dir-data"
  } ],
  ...
}
cat: input_dir/test1/text.txt: No such file or directory

Version

3.4.1

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[SbloodyS]

Parameter passing currently only supports data, not files.

[coldWord]

Parameter passing currently only supports data, not files.

https://dolphinscheduler.apache.org/zh-cn/docs/3.4.1/guide/parameter/file-parameter
But the official documentation shows that this feature is still available.

[SbloodyS]

There may be some deviations, and this function is not implemented in the new version. I've reopened this issue.

[lui62233]

Hi there! I've been looking into this issue and it seems like a critical regression.

The root cause is quite clear: commit 6f12d1d deleted TaskFilesTransferUtils.java, which handled the core logic for uploadOutputFiles() and downloadUpstreamFiles(). Without this utility, the system has no way to move files from the upstream task's local directory to S3, and subsequently, the downstream task finds nothing to download, leading to the 'No such file or directory' error.

Reproduction Analysis:

1. Upstream Task: Generates files $\rightarrow$ calls uploadOutputFiles() (Now missing) $\rightarrow$ Files stay local.
2. S3 Bucket: Remains empty (as verified by your mc ls output).
3. Downstream Task: Calls downloadUpstreamFiles() (Now missing) $\rightarrow$
   esourceItemMap is empty $\rightarrow$ File access fails.

Proposed Fix:
The TaskFilesTransferUtils.java needs to be restored or its logic reimplemented in the current architecture. If the deletion was intentional for a refactor, the corresponding logic for S3 transfer was accidentally omitted.

I'd recommend checking if the logic was moved to a different utility class or if this was a genuine accidental deletion during the �arpool fix. This effectively breaks the fundamental 'FILE' parameter contract in 3.4.1.