fix CLI aguments

[ElectricRCAircraftGuy]

Hi, thanks for your tool!

Continuing the discussion from here: https://askubuntu.com/questions/388262/decrypt-axcrypt-encrypted-file/1536042#comment2702481_1524304

How arguments for command-line tools are expected to work in Linux; optional and positional args in Linux

At the top of a help menu, the tool usage should be described like this, for example. Arguments in [] are optional, and arguments in <> are required.

Here, we have a program called my_program that takes an input file and an output file. The output file is optional because, for example, you might auto-generate a name from the input file.

Here, input_file and output_file are both positional arguments. Their position relative to each other only, but not relative to the options, is important.

my_program [options] <input_file> [output_file]

Options are a special type of optionl argument. They are called "flags" or "options". The short form uses - plus a single, case-sensitive letter. The long form uses -- plus a case-sensitive word or phrase.

Example descriptions from a help menu:

-a, --all   Do blah blah blah
-B, --big   Do big blah blah blah

Single-letter options can be joined with a single -. Ex: -aB is equivalent to -a -B or -B -a.

Options/flags do not respect order. They can be placed anywhere in the command line. Positional args respect order and must be placed in the correct order relative to other positional args.

Examples: these are all exactly equivalent calls and the program parser should treat them as such.

my_program -a -B input_file output_file
my_program -B -a input_file output_file
my_program input_file -a --big output_file
my_program input_file -B -a output_file
my_program input_file output_file -a -B
my_program input_file output_file -B -a
my_program -aB input_file output_file
my_program -Ba input_file output_file
my_program -a input_file -B output_file
my_program -B input_file output_file -a
my_program input_file output_file -B --all
my_program input_file output_file --big --all

Notice, again, that the positional arguments must be in the correct order relative to each other. The options can be in any order or position.

If an optional argument has a parameter passed to it, however, then you treat the parameter as a positional argument relative to the option.

Example help menu description:

-o, --output-to <output_file>     Write output to <output_file>

You can also allow an = to separate the option from the parameter:

-o, --output-to <output_file>                   Write output to <output_file>.
-o=<output_file>, --output-to=<output_file>     Same as above. 

Identical usages:

my_program --output-to output_file input_file
my_program --output-to=output_file input_file
my_program input_file --output-to output_file
my_program input_file --output-to=output_file
my_progrm input_file -o output_file

Now put it all together. Identical usages:

my_program -a -B --output-to output_file input_file
my_program --output-to output_file -a -B input_file
my_program -a --output-to=output_file -B input_file
my_program --output-to=output_file -Ba input_file
my_program -o output_file -Ba input_file
my_program -o=output_file -Ba input_file
my_program -Bao output_file input_file
# etc.

NOT allowed, because now it makes output_file attach to the -a option instead of to the -o option:

my_program -Boa output_file input_file

An example bash script that I wrote that respects and handles all of the above (namely: options in any order, positional args in order relative to each other, and options with parameters), except for the = syntax and the clustered option syntax (like -aB), is here: https://github.com/ElectricRCAircraftGuy/eRCaGuy_hello_world/blob/master/bash/argument_parsing__3_advanced__gen_prog_template.sh

[xecrets]

Hi!

I appreciate any and all feedback, and very frequently it results in code changes, so let's dig in here - and thanks for taking the time!

First of all, I tried to find the reference for the quoted article "How arguments for command-line tools are expected to work in Linux; optional and positional args in Linux", but I failed to find it. I did previously find https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html . And this XecretsCli seems to adhere to mostly.

There are several things to consider.

One of them being that XecretsCli is cross platform across macOS/Windows/Linux, and I'd really like the syntax and command line argument parsing to work identically as far as allowed by the platform, thus mostly only adapting to path name syntax, and various quoting rules required by different platforms and their shells.

Another thing to consider is that XecretsCli is intended to be used by humans, scripts as well as other software via the SDK ( https://www.nuget.org/packages/Xecrets.Sdk ). This implies some efficiency considerations, one of them being to avoid multiple invocations of the command line executable when possible. Process creation is pretty expensive.

Yet another thing that affects how it operates is the availability of command line parsing libraries. Command line parsing is surprisingly complex, and surprisingly enough .NET does not (yet) have a standard library for this. There are various options, and one is in the process of being incorporated into .NET ( https://learn.microsoft.com/en-us/dotnet/standard/commandline/ ), but it has not yet made it there. The features and capabilities of the library will affect how the program can interpret it's arguments, and once again - it should be as similar as possible between the supported platforms. The currently used parser is https://ndesk.org/Options .

XecretsCli is intended to be very flexible and support complex compound operations in a single invocation. This by necessity requires some options to be interpreted in the order given, or at least in a defined order. For example, you may want to encrypt 'file.txt' to 'file.axx', and then wipe the original 'file.txt'. Obviously the encryption has to be done first, before the wipe. There are many other similar scenarios. Not all currently position dependent arguments may really need to be so, for example the --password option. But even there are are scenarios where the first given password has a higher priority than the second etc, and where additional passwords should not be specified until later.

The feedback given centers around the following issues as I understand it:

• Some options are position-dependent, notably for example the --password.
• Some options are difficult to understand or non-intuitively named, for example --encrypt-to.
• Arguments to options should follow immediately after. This I believe they do.

The position dependence could be eased, but not removed, without adversely affecting the flexibility of the command line. I do agree this could be stressed more clearly in the documentation, and will certainly do so.

Naming can definitely be discussed, but the given example is documented as --encrypt-to {clear} {encrypted} [{original}] Encrypt a file to given file path w/optional original name. . It should be read as Encrypt a clear text input to an encrypted output . This can certainly be clarified.

The final feedback on arguments following options, here I'd appreciate some elaboration, because I don't quite understand it.

Any suggestions on how to clarify and/or better conform to user expectations and conventions without seriously degrading the flexibility of the command line is much appreciated, and keeping the same syntax and semantics across platforms. The main challenge I think is how to maintain the "multi operation" functionality, while making it clearer and more conformant to expectations. The suggestions on how to interpret arguments in the article mentioned, really sort of implies a single operation invocation. We want a single invocation to be able to perform multiple operations. Ideas?

[ElectricRCAircraftGuy]

First of all, I tried to find the reference for the quoted article "How arguments for command-line tools are expected to work in Linux; optional and positional args in Linux", but I failed to find it. I did previously find https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html .

There's no quoted article. I simply added a heading to my comment above, and that's my heading name.

[xecrets]

There's no quoted article. I simply added a heading to my comment above, and that's my heading name.
Ok! It is written in such an assertive style that it seemed to be some form of authoritative declaration of style ;-) .

Since one of the points of feedback was that options are position dependent, I was looking for some guidance on how this should be done. And at least https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html seems to indicate that it's fine at least within the context of POSIX/Gnu, as it states: Options may be supplied in any order, or appear multiple times. The interpretation is left up to the particular application program.

As the positional importance of the options is such an integral part of the function of the program, and it's not entirely clear to me how else to achieve the desired functionality, I will at least to start with work with the documentation to make this aspect clearer, to reduce user confusion. I fully understand that it can be a little counter-intuitive at first that --password ... must precede the option that requires it, but I also think that once it's understood that arguments are processed in order of appearance it's also quite easy to use.

[ElectricRCAircraftGuy]

Hi @xecrets , I spoke very generally in the first post. Let me address specifics.

Problems with the existing interface

To encrypt notes.txt to notes.txt.axx, the usage is:

XecretsCli --password 'my_password' --encrypt-to notes.txt notes.txt.axx

The --encrypt-to option implies that the "to" file comes right after the --encrypt-to option, however, so it should be:

XecretsCli --password 'my_password' --encrypt-to notes.txt.axx notes.txt

Furthermore, putting the --password option at the end should work, because options have no order, but it does not work:

# this is broken
XecretsCli --encrypt-to notes.txt notes.txt.axx --password 'my_password' 

So, that's two inconsistencies right there.

A clearer interface

Typically, if an "option" has more than 1 parameter, then it is not an option. It is a subcommand. So, using subcommand style, it should like like this:

# Not like this
XecretsCli --password 'my_password' --encrypt-to notes.txt notes.txt.axx

# Do this instead, making `encrypt` a subcommand
XecretsCli encrypt --password 'my_password' notes.txt notes.txt.axx

# And if you want the output file to be an option instead of a positional parameter, it
# should look like this:
XecretsCli encrypt --password 'my_password' --output notes.txt.axx notes.txt
# And of course, options can be in any order, so any of these should then work too:
XecretsCli encrypt notes.txt --output notes.txt.axx --password 'my_password'
XecretsCli encrypt --output notes.txt.axx notes.txt --password 'my_password'
# etc.

Alternatively, this would also be an acceptable pattern shown in a help menu. Here, input is a required positional parameter, and output is an optional positional parameter, and all options are optional, not order-dependent options, not positional parameters:

Example help menu for a clearer interface

XecretesCli [options] <input> [output]

Positional parameters:
  input (required)      The path to the input file to encrypt/decrypt.
  output (optional)     The path to the output file to write the encrypted/decrypted content to.

Options:
  --password <password>  The password to use for encryption/decryption
  --encrypt              Encrypt the input file; if not provided, the input file will be decrypted
                             if its extension is '.axx' and encrypted otherwise.
  --decrypt              Decrypt the input file; if not provided, the input file will be decrypted
                             if its extension is '.axx' and encrypted otherwise.

Example usages:
    # Encrypt notes.txt to notes.txt.axx; these are all valid and the same
    XecretsCli --password 'my_password' --encrypt notes.txt notes.txt.axx
    XecretsCli --password 'my_password' --encrypt notes.txt
    XecretsCli --encrypt notes.txt --password 'my_password' notes.txt.axx
    XecretsCli --encrypt notes.txt --password 'my_password'
    XecretsCli --password 'my_password' notes.txt notes.txt.axx
    XecretsCli --password 'my_password' notes.txt
    XecretsCli notes.txt --password 'my_password' notes.txt.axx
    XecretsCli notes.txt --password 'my_password'
    # etc. 

[ElectricRCAircraftGuy]

I get the impression it would be helpful for me to provide my credentials so you can better trust I know what I'm talking about. So, here goes:

My credentials: I'm a professional software developer. I've been using Linux for 12 years. I've been using it as my primary OS at both home and work for the past 6 years. I'm in the top 125 people out of 23 million on Stack Overflow (top 0.00054%), and number 21 out of 1.3 million on Ask Ubuntu (top 0.0016%), and I've used thousands for CLI interfaces, and this is my advice. I've only come across a handful of "broken" (meaning: very non-standard) interfaces a handful of times. In all cases I've encountered, it's either:

1. when they are really old programs and use non-standard interfaces for historical differences, like find using whole-word options with a single dash like -path (non-standard) instead of --path (standard), or
2. when they are written by Windows developers (that applies here I think). In the case of Windows developers, the problems are generally they get options and positional parameters confused (that applies here), inputs are backwards as compared to what I'd expect (that applies here), and options which should not be position dependent are position dependent and don't work in other orders (that applies here).

I appreciate your tool. I appreciate your work. I'm providing this feedback to help you make it better and fit the expected Linux usage mold.

[xecrets]

Hi @xecrets , I spoke very generally in the first post. Let me address specifics.

Thanks for all the feedback!

The --encrypt-to option implies that the "to" file comes right after the --encrypt-to option, however, so it should be:

XecretsCli --password 'my_password' --encrypt-to notes.txt.axx notes.txt

This is a valid point, although I don't think it really makes it clear for all. Another thought is to rename it to --encrypt-from-to since the most common order is from -> to.

Furthermore, putting the --password option at the end should work, because options have no order, but it does not work:

This is more a matter of opinion I think. As mentioned, the GNU project and POSIX does seem to let the application determine the semantics of the order of options. It'll be a recurring theme in this discussion - the fact that XecretsCli is not a one-operation app. It should be able to do many things in one invocation.

A clearer interface

Typically, if an "option" has more than 1 parameter, then it is not an option. It is a subcommand. So, using subcommand style, it should like like this:

# Not like this
XecretsCli --password 'my_password' --encrypt-to notes.txt notes.txt.axx

# Do this instead, making `encrypt` a subcommand
XecretsCli encrypt --password 'my_password' notes.txt notes.txt.axx

The concept of subcommands may be the general way to go to handle some of your concerns, I'll need to dig into this a little deeper. There's still the issue of "options" - are they gobal across all subcommands? This won't really work here...

Example help menu for a clearer interface

XecretesCli [options] <input> [output]

Positional parameters:
  input (required)      The path to the input file to encrypt/decrypt.
  output (optional)     The path to the output file to write the encrypted/decrypted content to.

Options:
  --password <password>  The password to use for encryption/decryption
  --encrypt              Encrypt the input file; if not provided, the input file will be decrypted
                             if its extension is '.axx' and encrypted otherwise.
  --decrypt              Decrypt the input file; if not provided, the input file will be decrypted
                             if its extension is '.axx' and encrypted otherwise.
(cut for brevity)

As mentioned, this won't really fly here, since XecretsCli needs to do more than one operation per invocation.

[ElectricRCAircraftGuy]

There's still the issue of "options" - are they gobal across all subcommands?

No, a help menu might look like this:

XecretesCli [options] <subcommand> [subcommand options] <input> [output]

As mentioned, this won't really fly here, since XecretsCli needs to do more than one operation per invocation.

Can you give an example? I don't understand why it wouldn't work.

[xecrets]

2. when they are written by Windows developers (that applies here I think). In the case of Windows developers, the problems are generally they get options and positional parameters confused (that applies here), inputs are backwards as compared to what I'd expect (that applies here), and options which should not be position dependent are position dependent and don't work in other orders (that applies here).

Yes, I'm primarily a Windows developer, and XecretsCli is a .NET tool. Not a Windows tool. Not a Linux tool. Not a macOS tool. While most tools do indeed have position independent options, it's not as far as I can tell a documented community accepted best practice or standard in any of the supported platforms. Most tools are very simple tools usually conforming to the pattern "input" goes to "output", possibly affected by some options. This is not the case for XecretsCli, it's more of an action-interpreter invocable from the command line.

I think the concept of subcommands is possibly the best way to go forward, but since this is a .NET tool, not a bash script, nor a Linux tool that uses getopt et. al. but must use a .NET command line parser, I think this must wait until System.CommandLine is released (it's still in preview), https://learn.microsoft.com/en-us/dotnet/standard/commandline/ . This does support subcommands and a lot of other things.

While I try not to prioritize efficiency over usability and maintainability, XecretsCli is very much intended to be the backend of a GUI frontend ( https://www.axantum.com/xecrets-ez ), I do believe that it needs to be able to do more than one "input" -> "output" operation per invocation. For example it supports batching of fairly complex operations via the SDK ( https://www.nuget.org/packages/Xecrets.Sdk ), such as encrypting/decrypting many files in one invocation, with differing options depending on the file, such as whether to apply compression or not, and wiping of the source after successful transformation of each file etc. All within a single invocation. This requires dependence on the order of items on the command line.

For 100s of small files, requiring a separate invocation for each separate operation, will cause a very noticeable performance decrease unfortunately, even if it would be nice and indeed simplify XecretsCli quite a bit.

I appreciate your tool. I appreciate your work. I'm providing this feedback to help you make it better and fit the expected Linux usage mold.

Thanks!

Please consider though - XecretsCli is not a Linux tool. It's a cross platform .NET tool, that should have identical syntax and semantics as far as possible on Windows, Linux and macOS.

I will not make a special command line parser for Linux/macOS, so the command line works differently depending on platform. That being said, obviously I'd like to break as few expectations as possible on all platforms, and when it does be as clear as possible.

[ElectricRCAircraftGuy]

This is more a matter of opinion I think. As mentioned, the GNU project and POSIX does seem to let the application determine the semantics of the order of options.

I disagree. OpenGroups.org Guidelines 11 and 12 support my view:

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html:

Guideline 11:

The order of different options relative to one another should not matter, unless the options are documented as mutually-exclusive and such an option is documented to override any incompatible options preceding it. If an option that has option-arguments is repeated, the option and option-argument combinations should be interpreted in the order specified on the command line.

Guideline 12:

The order of operands may matter and position-related interpretations should be determined on a utility-specific basis.

[ElectricRCAircraftGuy]

but must use a .NET command line parser,

You could write your own manual parsing. My Bash program I've referenced twice above uses manual parsing--no parser.

I do believe that it needs to be able to do more than one "input" -> "output" operation per invocation.

Ok, I hear you.

Another thought is to rename it to --encrypt-from-to since the most common order is from -> to.

This would be clearer for sure.

[ElectricRCAircraftGuy]

I'll leave it up to you on how you want to make your tool. My 25 cents have been given (far more than 2 cents).

[xecrets]

I'll leave it up to you on how you want to make your tool. My 25 cents have been given (far more than 2 cents).

Indeed, and I'm very grateful! It certainly will result in changes, at first definitely to the documentation, and probably to the actual code once I figure out how to achieve it in the best way.

The main challenge being that the concept of "utility" as described in POSIX etc is a little to simple here. One way to go is just to invent a domain specific language for it, and let this be the main "operand". But while this strategy does work around the "formal" violations of the utility syntax conventions for Linux, it doesn't really make it easier to use...

One way for example to make it "conform" is to redefine all the "(global)" true "options" it has now, as just that, options, and then say that following that is a "--" pattern, and then the various position dependent actions and operands. But I'm not sure if that will really help. More thought is required!

If you ( or anyone else reading this ) has a good example of a software with similar complex functionality beyond the simple "input" -> "output" possibly affected by some options paradigm, I'd appreciate a heads up!

[xecrets]

but must use a .NET command line parser,

You could write your own manual parsing. My Bash program I've referenced twice above uses manual parsing--no parser.

Nah... As mentioned, it's surprisingly complex to parse command lines. This I will leave to a specialized library.

Another thought is to rename it to --encrypt-from-to since the most common order is from -> to.

This would be clearer for sure.

I will go through the naming and clarify this and perhaps others with similar issues. Thanks!

[xecrets]

As mentioned, this won't really fly here, since XecretsCli needs to do more than one operation per invocation.

Can you give an example? I don't understand why it wouldn't work.

This was said in the context of the suggested syntax:

XecretsCli [options] <input> [output]

The reason is that this is the syntax of a simple <input> -> [output] single operation tool. XecretsCli should be able to do several operations in one go, for example:

XecretsCli --password a-secret --encrypt-to a.txt a.xxx --wipe a.txt --encrypt-to b.txt b.axx --wipe b.txt --encrypt-to c.txt c.axx --wipe c.txt ...

Keep in mind that the consumer of XecretsCli may not only be a human writing the command line manually, it may be a script or a program using the SDK . The command line can easily reach several kilo-bytes in length when generated in this way (care should of course be taken not to exceed platform restrictions on command line length, which is 32K on Windows and generally longer on other platforms).