Skip to content

Replace copy with move#224

Merged
HenriquePreto merged 7 commits into
devfrom
hp-replace-copy-with-move
Nov 6, 2025
Merged

Replace copy with move#224
HenriquePreto merged 7 commits into
devfrom
hp-replace-copy-with-move

Conversation

@HenriquePreto

@HenriquePreto HenriquePreto commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

Issue: https://github.com/inductiva/tasks/issues/1323

Idea: First we download inputs and remote assets into /workdir/<task-id>/sim_dir folder, then rename the folder to /workdir/<task-id>/output/artifacts when creating the artifacts to avoid extra copying.

Folder structure before:

/workdir/hvo3je8bdf5ux1qcc1u3ck61f
├── sim_dir
│   ├── dummy.py
│   └── hello.txt
└── output
    └── artifacts
        ├── dummy.py
        ├── system_metrics.csv
        └── hello.txt

Folder structure after:

/workdir/ey3g6jtuaxvkujcqodc6y2kft
└── output
    └── artifacts
        ├── dummy.py
        ├── system_metrics.csv
        ├── output_update.csv
        └── hello.txt

I also deleted the MPIExecuter class since it is no longer used.

@HenriquePreto

Copy link
Copy Markdown
Contributor Author

I added some logs to my local task-runner and ran this script using my local web API:

import os
import pathlib
import shutil
import inductiva

dir_name = "input-dir"

os.makedirs(dir_name, exist_ok=True)

pathlib.Path(f"{dir_name}/hello.txt").write_text("Hello, world!\n")

custom_simulator = inductiva.simulators.CustomImage("docker://python:3.11-slim")

mg = inductiva.resources.machine_groups.get_by_name("hpmac")

task = custom_simulator.run(dir_name, on=mg, 
                            commands=["touch test.txt"],
                            remote_assets=["test-print-dirs/dummy.py"])

shutil.rmtree(dir_name)

task.wait()

The directory structure after copytree is like this:

| /workdir/hvo3je8bdf5ux1qcc1u3ck61f/
|---- sim_dir/
    |----- dummy.py
    |----- hello.txt
|---- output/
    |----- artifacts/
        |---- dummy.py
        |---- system_metrics.csv
        |---- hello.txt

We can't move directly sim_dir to artifacts/ because artifacts/ already exists and sim_dir/ will be created under artifacts/ (e.g, artifacts/sim_dir/dummy.py)

@HenriquePreto

Copy link
Copy Markdown
Contributor Author

We download and extract the inputs into the <task_workdir>/ folder. Since the inputs are contained within a folder named sim_dir/, they will be located at <task_workdir>/sim_dir:

unzip_duration = files.extract_zip_archive(
zip_path=tmp_zip_path,
dest_dir=task_workdir,
)

Remote assets are downloaded into <task_workdir>/sim_dir to keep all inputs together:

if self.input_resources:
download_duration = self.file_manager.download_input_resources(
input_resources=self.input_resources,
region=self.storage_region,
dest_path=sim_workdir,
workdir=self.workdir,
)

@HenriquePreto
HenriquePreto marked this pull request as ready for review November 5, 2025 16:42
@HenriquePreto
HenriquePreto merged commit b3057a7 into dev Nov 6, 2025
1 check passed
@HenriquePreto
HenriquePreto deleted the hp-replace-copy-with-move branch November 6, 2025 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants