Allow long arguments to be provided via a file instead of only on the command line

[DanTup]

On Windows, there is a limit to the size of a command line (executable + arguments). It's easy to hit this limit with some operations like asking the IDE to re-run all test files that had failing tests from the last run, because each test file needs to be listed specifically:

One option to fix this for dart test is to support an argument like --tests-from-file testlist.txt, however this only solves the issue for that specific case, but it would be nice to solve this more generally.

@jakemac53 noted that Bazel supports a final argument of @filename that can be used to read arguments from a file which seems like a nice convention to follow (and I started implementing at dart-lang/test#2485), and this is also implemented in Dart compute_kernel.dart here.

When trying to add the equivalent to flutter test, it turned out to be easier to support for the whole flutter command because of how the CommandRunner works (I started this at flutter/flutter#167272). If doing this for all of flutter it would be nice if we also supported it for all of dart (instead of only pkg:test).

@bkonyi had some questions that would be worth discussing if we're going to implement this more generally across all tools/commands (and I added a few):

• Are we concerned about legit args that start with @ that aren't this file?
• Should we support replacing args-from-file that aren't in the final position of the command?
• Exactly what should the format of the file be?
  • Bazel seems to support a few formats but the Dart code here seems to just assume raw args on individual lines (I presume this makes it impossible to provide newlines in the values)

I don't personally have any real preference for what the solution is, my main goal is to be able to resolve bugs like the one shown up-top. I do think a general solution implemented once in dart ... and flutter ... would be better than having multiple implementations in individual sub-commands though.

[bwilkerson]

I asked Gemini that question just out of curiosity. The full answer is below, but based on it I would say that we shouldn't need to worry about using @.

We do need to decide whether arguments on the command-line override arguments from the file or whether they're all treated equally. I like the idea of being able to override what's in the file, but that might not be the right answer.

I'm not sure what advantage there would be of using spaces to separate arguments, but unless there are some I'd probably argue for one argument per line, given that the whole point of the file is to contain long lists of arguments (which would be hard to edit if they can't be broken across lines).

Probably unnecessary to say, but if we do provide this support we should add it to the args package so that it's available everywhere.

---

Q:
Is there a widely used convention for providing a file containing additional command line arguments to a command-line tool?

A:
While there isn't one universally enforced standard, there are common practices and conventions that command-line tools employ for handling additional arguments from files. Here's a breakdown:

Common Practices:

• @ symbol:

  • A frequent convention involves using the "@" symbol followed by the filename. When a command-line tool encounters "@filename," it reads the file and treats its contents as additional command-line arguments.
  • This is seen in some tools where for example "@myarguments.txt" would cause the program to read the arguments contained within "myarguments.txt".

• Configuration Files:

  • Many tools support configuration files, often in formats like:
    • Plain text files (where arguments are listed one per line or space-separated).
    • INI files.
    • JSON or YAML files.
  • These files allow for complex configurations and are often used when numerous or intricate arguments are needed.

• Environment Variables:

  • While not strictly "files," environment variables serve a similar purpose, allowing users to store and retrieve values that influence a program's behavior.

• Argument Parsing Libraries:

  • Libraries like Python's argparse provide functionality to handle various argument parsing scenarios, including the potential to create custom logic for reading arguments from files. This makes it easier for developers to implement these features.

Key Considerations:

• Syntax:

  • The syntax for how arguments are interpreted from a file can vary. Some tools expect one argument per line, while others allow space-separated arguments.

• Comments:

  • Some tools allow comments within argument files, often using characters like "#" to denote lines that should be ignored.

• Precedence:

  • It's crucial for tools to define the precedence of arguments provided via files versus those given directly on the command line. Typically, command-line arguments override those from files.

In essence, while a rigid standard is lacking, the "@" symbol and the use of configuration files are prevalent methods for supplying additional command-line arguments.

[DanTup]

We do need to decide whether arguments on the command-line override arguments from the file or whether they're all treated equally. I like the idea of being able to override what's in the file, but that might not be the right answer.

I like the idea of just saying that whatever is read from the file replaces the @file in-place and therefore all behaviour is the same as if you'd done it manually. Having a notion of overriding in the substitution could get confusing for args that can be supplied multiple times (are both supplied, or does one replace the other?).

I'm not sure what advantage there would be of using spaces to separate arguments, but unless there are some I'd probably argue for one argument per line, given that the whole point of the file is to contain long lists of arguments (which would be hard to edit if they can't be broken across lines).

Yeah, I think supporting spaces would also require args to be quoted/escaped if they contain spaces, which makes things more complicated on both sides (particularly as escaping across shells/platforms can differ and is a bit of a minefield as flutter/flutter#84270 / #59604 have shown 🙃). I think with line-separated we could probably just forbid args that contain newlines and therefore have no escaping at all.

[rmacnak-google]

gcc/clang/msvc call these response files.

[dcharkes]

Yes, this would be great! Being able to pass a file also was requested in dart-lang/native#39.

TL;DR: Use JSON as the format instead of plain text.

Format

Terminals support spaces and newlines with quotes and interpolation:

your_command "foo bar"

your_command $'line1\nline2'

your_command "line1
line2"

1. Plain text & is the argument separator

Verbatim what one would write in the command-line invocation but no terminal-like string interpolation.

This doesn't really work because it wouldn't support spaces inside arguments. Every space would be interpreted as a new String in List<String>

Pro:

• Verbatim how one would pass arguments on command-line.

Con:

• Doesn't work for spaces and new-lines.

2. Plain text & \n is the argument separator

@DanTup's suggestion IIRC.

dart main.dart @file arguments.txt

arguments.txt:

foo
bar baz

Would lead to ["foo", "bar baz"] as arguments.

New lines are the argument separator, spaces are spaces in arguments.

dart main.dart foo "bar baz"

I think with line-separated we could probably just forbid args that contain newlines and therefore have no escaping at all.

I believe this might end up biting us in the end. Users probably shouldn't be using new lines in command-line arguments, but they might. I'm assuming people are going to write abstractions over this mechanism, and it's likely someone will try to pass newlines in at some point.

Pro:

• Works for spaces.

Con:

• Not verbatim how one would pass arguments on command-line.
• Doesn't work for new-lines.

3. Plain text & terminal-like evaluation for strings/spaces/newlines

I like the idea of just saying that whatever is read from the file replaces the @file in-place and therefore all behavior is the same as if you'd done it manually.

We could try to support verbatim what's in the file is interpreted as if one was in the terminal.

However, this means we'll now need to duplicate the bash behavior inside Dart/Flutter. (We'd probably rule out other behavior that is not simply strings related such as $(pwd).)

Pro:

• Verbatim how one would pass arguments on command-line.
• Works for spaces and new-lines.

Con:

• Implementation complexity in Dart/Flutter.

4. Structured input: JSON / yaml

If the goal is pass something into List<String> args, then a way to completely skip parsing questions w.r.t. spaces and tabs is to use a format for which this is already properly defined, JSON:

[
  "--arg1",
  "my_value",
  "--arg2",
  "happens to contain spaces",
  "--arg3",
  "another\nargument\nwith\nmultiple\nnewlines"
]

Or if we expect these files to be edited by humans frequently, we could consider supporting YAML. (YAML would also support comments, JSON does not.)

Pro:

• Works for spaces and new-lines.
• No ambiguity for how parsing is done.

Con:

• Not verbatim how one would pass arguments on command-line.

I'm leaning towards option 4, which would rule out any ambiguity for how spaces and new-lines are parsed.

Issues with spaces in arguments and escaping are notoriously painful. Process.start have the runInShell option which changes how your arguments are escaped/parsed. And I've had weird things with spaces happen multiple times. Choosing a format in which the parsing for new-lines and spaces is well-understood avoids such pain.

@file or @filename

@file is shorter than @filename, so I prefer @file.

[DanTup]

I think I like the idea of a JSON string array.. it's a little less simple than verbatim lines in a text file, but it avoids the newline issue and makes the format more obvious/less ambiguous (probably requiring less documentation). I'm less sure about YAML.. having comments is nice, but the format is less simple (particularly to parse) and it'd be nice to have something that's easy to use across Dart/non-Dart (in this case, Dart-Code is TypeScript).

@file or @filename
@file is shorter than @filename, so I prefer @file.

I'm not sure I understand this - the idea was for the part after the @ to be an actual filename/path, so you would write something like:

dart test @foo/myargs.txt

So you wouldn't literally write @file or @filename. Of course if your interpretation was to do @file foo/myargs.txt that would work too, but I'm not sure if there's an additional value (besides reducing the scope of restricted legit args from anything starting with @ to only @file).

While searching for "repsonse files" as noted above, I found this:

mozilla/sccache#43

The options read are inserted in place of the original https://github.com/file option. If file does not exist, or cannot be read, then the option will be treated literally, and not removed.

So I guess there's a question of whether to do something like that, or just fail with an error if the file is missing/invalid. My feeling is that rejecting is better (assuming a missing file is because something is wrong, and not that it is a legit @ argument), but I can see an argument both ways.

[dcharkes]

dart test @foo/myargs.txt

Ah, I completely misunderstood that. Yeah dart test @path/to/file.json sgtm!

fail with an error if the file is missing [...] My feeling is that rejecting is better

+1

[jakemac53]

While JSON would be simpler on the edge cases we have existing use cases which follow the newline separated args pattern. Newlines in arguments are technically a thing but exceedingly rare.

Bazel also does not support Json as a format, and this is one of our primary use cases. Technically we could create our own params files using JSON but I think its a strong signal that this support is not necessary.

[natebosch]

+1 for using the same newline separated format as bazel if we are using the same @filename argument pattern.

[DanTup]

It sounds like there's a preference towards text instead of JSON. Is this enough of a consensus to implement something like this?

• a plain text file with one argument per line, literally (no kind of escaping - which means embedded newlines are not supported)
• support multiple @path/filename args in any position that will be replaced in-position with the args from the file
• the filename/path is relative to current working directory
• it's the responsibility of the caller to clean up the file (the processing parsing the arg will not delete it)

Some questions I'm not sure about:

1. What do we do if we get @foo and there is no file named foo (or we otherwise fail to read it)?
   a. throw an error (this will break anything with legit @ args)
   b. leave the arg as-is (this supports other @ args as long as they aren't real filenames we'd find)
2. Should we specific the file encoding? (for ex. UTF8 without BOM?)
3. Where is this best implemented?
   a. pkg:dartdev in bin/dartdev.dart main()
   b. pkg:dartdev in lib/dartdev.dart runDartdev()
   c. pkg:args? (note: DartdevRunner does some contains() checks on the args and parses VM experiments before it ever gets to pkg:args so this may either need some refactoring, or wouldn't support all args in the file)
   d. somewhere else (are there any paths that don't go through pkg:dartdev that should support this)?

[dcharkes]

It sounds like there's a preference towards text instead of JSON.

If everyone else prefers text, and no support for newlines, that's fine for me.

cc eco system @devoncarew @mosuem and dartdev flutter_tools @bkonyi in case they have an opinion.

RE 3: If it is implemented in dartdev, then it should also be implemented in flutter_tools. I'd say it should be implemented in dartdev, not package:args. And preferably as early in the code as possible. I'd consider it as "before any other code is running". E.g. whatever gets the List<String> args from main would only see the expanded version.

In addition to dartdev and flutter_tools it would be nice if we also implement it in the Dart VM. E.g. dart --disable-dart-dev @file/path.txt should also work. This must be implemented in runtime/bin/main_impl.cc most likely.

[bkonyi]

Some questions I'm not sure about:

1. What do we do if we get @foo and there is no file named foo (or we otherwise fail to read it)?
   a. throw an error (this will break anything with legit @ args)
   b. leave the arg as-is (this supports other @ args as long as they aren't real filenames we'd find)

I think B) is the best approach here.

2. Should we specific the file encoding? (for ex. UTF8 without BOM?)

We need to be able to consistently decode the contents of the file to do anything, so we should be explicit about what formats are supported and try and be consistent with what we support in other situations where the user can provide a file argument to an option.

3. Where is this best implemented?> a. pkg:dartdev in bin/dartdev.dart main()
   b. pkg:dartdev in lib/dartdev.dart runDartdev()
   c. pkg:args? (note: DartdevRunner does some contains() checks on the args and parses VM experiments before it ever gets to pkg:args so this may either need some refactoring, or wouldn't support all args in the file)
   d. somewhere else (are there any paths that don't go through pkg:dartdev that should support this)?

Ideally we'd add this support to package:args, even if this behavior needs to be explicitly enabled, as we'd get it for free across all of our tooling.

And preferably as early in the code as possible. I'd consider it as "before any other code is running". E.g. whatever gets the List args from main would only see the expanded version.

I'm not sure we should be modifying the contents of args as it's currently identical to what the user is providing via the CLI.

In addition to dartdev and flutter_tools it would be nice if we also implement it in the Dart VM. E.g. dart --disable-dart-dev @file/path.txt should also work. This must be implemented in runtime/bin/main_impl.cc most likely.

It'd be nice to support this for VM arguments, but I think we should leave processing of Dart arguments to the user program if we can.