Skip to content

local file to upload('-f' option) can be stdin ('-')#324

Open
cinsk wants to merge 1 commit into
TritonDataCenter:masterfrom
cinsk:master
Open

local file to upload('-f' option) can be stdin ('-')#324
cinsk wants to merge 1 commit into
TritonDataCenter:masterfrom
cinsk:master

Conversation

@cinsk

@cinsk cinsk commented Sep 8, 2017

Copy link
Copy Markdown

This small commit enables to use - for -f option value so that muntar can read from standard input instead from reading from real file. In short, users can able to use following one-liner to upload files in a directory to manta.

    $ tar -C SOME-DIRECTORY -cf - . | muntar -f - ~~/MANTA-DIRECTORY

Without this commit, users should use one of following to upload a directory:

    $ tar -cf ARCHIVE.tar SOME-DIRECTORY
    $ muntar -f ARCHIVE.tar ~~/public

Or, if the shell is bash or zsh, users can use process substitution feature like this:

    $ muntar -f <(tar -cf - SOME-DIRECTORY) ~~/public

@davepacheco

Copy link
Copy Markdown
Contributor

Thanks for the PR! FYI, this repository does not use pull requests. See CONTRIBUTING.md.

As for the change: it doesn't seem like this will actually work. We end up creating a bunch of workers, each of which reads the input file. I think the code relies on the fact that each worker will end up reading the entire tar file. If you use process.stdin (or any non-regular file), whatever one worker reads, others will miss, and I'd expect the result would be each worker reading totally invalid tar records. Is that not what happens?

@cinsk

cinsk commented Sep 11, 2017

Copy link
Copy Markdown
Author

Thanks for the quick response, davepacheco@.

Strangely, it was fine to use my patch in one of my mac, but in another mac it failed. I guess that there was only 1 worker when it was successful.

Still, there is an option to control the number of workers, so I'll working on to force 1 worker if standard input is used, and follow the process you suggested.

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants