Replace copy with move#224
Conversation
|
I added some logs to my local task-runner and ran this script using my local web API: import os
import pathlib
import shutil
import inductiva
dir_name = "input-dir"
os.makedirs(dir_name, exist_ok=True)
pathlib.Path(f"{dir_name}/hello.txt").write_text("Hello, world!\n")
custom_simulator = inductiva.simulators.CustomImage("docker://python:3.11-slim")
mg = inductiva.resources.machine_groups.get_by_name("hpmac")
task = custom_simulator.run(dir_name, on=mg,
commands=["touch test.txt"],
remote_assets=["test-print-dirs/dummy.py"])
shutil.rmtree(dir_name)
task.wait()The directory structure after | /workdir/hvo3je8bdf5ux1qcc1u3ck61f/
|---- sim_dir/
|----- dummy.py
|----- hello.txt
|---- output/
|----- artifacts/
|---- dummy.py
|---- system_metrics.csv
|---- hello.txtWe can't move directly |
This reverts commit c9194f5.
|
We download and extract the inputs into the task-runner/task-runner/task_runner/task_request_handler.py Lines 520 to 523 in 41d7028 Remote assets are downloaded into task-runner/task-runner/task_runner/task_request_handler.py Lines 480 to 486 in 41d7028 |
Issue: https://github.com/inductiva/tasks/issues/1323
Idea: First we download inputs and remote assets into
/workdir/<task-id>/sim_dirfolder, then rename the folder to/workdir/<task-id>/output/artifactswhen creating the artifacts to avoid extra copying.Folder structure before:
/workdir/hvo3je8bdf5ux1qcc1u3ck61f ├── sim_dir │ ├── dummy.py │ └── hello.txt └── output └── artifacts ├── dummy.py ├── system_metrics.csv └── hello.txtFolder structure after:
/workdir/ey3g6jtuaxvkujcqodc6y2kft └── output └── artifacts ├── dummy.py ├── system_metrics.csv ├── output_update.csv └── hello.txtI also deleted the
MPIExecuterclass since it is no longer used.