Pitch [stdlib]: Command Line Argument Parsing

I agree. I'd just like to add that I believe I understand the desire to put things in the standard library:

  1. Making something like this widely available make it easier to adopt and use Swift in many contexts and improves developer experience.
  2. Because the relationship between Foundation, Swift Open Source, and Linux is a bit complicated (Work in progress, Owned by Apple, etc. etc.), it also complicates putting a feature like this in there.
  3. SwiftPM isn't where many developers would like it to be yet. There is no discoverability, and no native xcode integration, so putting this on there makes less accessible than if it were built in.

I don't mean to knock Foundation or SwiftPM, both are great and required a tremendous amount of work. Both also have a lot of potential to grow. I just see why there's a desire to and go straight to standard lib.

3 Likes

I am a huge +1 on a boost-like library and this seems most appropriate to include there for now. This approach would allow the community to work together on libraries in widely used domains without the constraints of source compatibility or Objective-C interop and Apple ownership (i.e. Foundation). Experimentation, innovation and use of bleeding edge language features would be explicitly encouraged and significant breaking changes expected. We would gain experience through the use of the libraries by early adopters who are willing to deal with the churn. Eventually as they mature we would find the right permanent home for them and commit to source stability.

A boost-like library written in Swift and built using the typical Swift toolchain would also be much easier for many of us to contribute to than the standard library and Foundation. This may be a good way to increase community involvement in implementation work.

3 Likes

-1.

Most other languages have a CLI parser in their standard libraries… and after a few years nobody uses them. Such things are improved on continuously and the state of the art shifts leading the base option unused but still a maintenance burden and a “swift is such a big dependency!” burden.

Swift packages are (relative to everything except maybe Rust) super-easy to consume, and provided we can pitch and get swift-sh into Swift itself†, even single-file scripts will be able to easily use them.

I could be swayed if the option parser was as SUPER awesome as eg. Codable (which feels like magic to me). I would want eg. to declare an enum with associated values for options and have the CommandLine parsed into that. But I doubt it's possible.

But even that’s not enough. There are many styles of parsing arguments, what I proposed above is really only adequate for simple scripts, not for mature command line tools with large numbers of possible options, flags and commands. A stdlib entry should be great for both. I don’t think it's possible. But a suite of packages across GitHub that cater to each and every kind of --help style there is: that's possible.

† I want to work on this pitch, but am super dang busy currently, I have a draft.

8 Likes

I seriously doubt this is true of the community as a whole.

Sure, some people may only write one or two scripts then be done, but at my job my coworkers and I write command line scripts frequently. Some in python, some in pure bash, some in C++. Working on headless servers means a lot of command line scripts since there are no gui’s. Since headless servers arent going away ever I would think that there will always be a need for CLI frameworks. Whether or not everyone uses them doesnt matter as long as enough people do. How many people is “enough” to make this effort worth it is up for debate :wink:

While I do agree this probably shouldnt go in the stdlib, there is definitely a need for a good CLI framework.
I write most my quick scripts in python because it’s so fast and easy. Swift is great but it cant even compete here. If one of swift’s goals is to be used as a scripting language then it needs a first class CLI library.

3 Likes

I think you misunderstood me. I'm not saying nobody uses option parsers, just that nobody uses the ones in the standard library. Third party ones come along and they become the “go to” choice.

If one of swift’s goals is to be used as a scripting language then it needs a first class CLI library.

Swift already has 4 perfectly serviceable option parsing libraries available as third party libraries.

6 Likes

I think this is the crux of the problem. I appreciate the enthusiasm for a Boost-alike for Swift, but I don't see that there's an actual problem to be solved there. Rather than asking permission to build such a thing, those who want to see it should just build it, and let their idea be tested in the hypothetical marketplace of Swift packages. (To be clear, I think it's a good idea, and I think its success is almost certainly assured, so it's a waste of time to discuss whether it should be done: someone just start the git repo already!)

I think the real problem comes for things where the community has built a consensus on a good implementation, but for whatever reason that implementation is ill-suited to living in the standard library.
I don't want to derail the discussion thread here, so I'll avoid elaborating on my thoughts, but I think it's a vital problem for the Swift community to tackle.

1 Like

I also feel that actual argument parsing (vs. just exposing arguments) seems like it doesn't necessarily have to be part of the standard library, though it straddles the line. That said, maybe a general-purpose mechanism could be slightly expanded to provide the basics?

At its core, command line arguments are array elements that are processed according to a pattern, so maybe a focus should be put on adding a kind of importer/filter to arrays in general that happens to be convenient for parsing command line arguments?

Seeing above list, maybe something for extracting array elements into properties? Maybe even employing Codable to do the brunt load of the work? Or simply a form of mapping function that calls a function with parameters labeled after a pattern extracted from an array of strings?

1 Like

One thing this should probably do is collect requirements for command line syntax to handle. The syntaxes I'm aware of is:

flags:
--foo --bar
-fb
-foo -bar
/FOO (Windows)

values:
--foo value
--foo=value
-foo value
-f value

And also excess unlabeled arguments (e.g. for filename lists following the arguments), and possibly a "--" indicator to disambiguate cases where an unlabeled parameter could be read as another option.

In a more abstract way:

  1. flags vs. values
  2. separate parameter for label and value, or argument label + delimiter + value in a single parameter
  3. short form vs. long form labels
  4. short flags that combine into a single parameter
  5. configurable label prefixes (--, - or /)
  6. labeled vs. unlabeled arguments (unlabeled = "the rest", like file names)
  7. Auto-generated help (this is a big one as it needs to cover localization)

Do we want to/need to cover all of these?

4 Likes

I would say yes - perhaps borrowing from the Codable prior art, such as JSONDecoder where we can set a flatStrategy and valueStrategy for how to parse the arguments.

What about a meta GitHub? Say at toolbag.swift.org which allowed anyone to submit git repo URLs (from github or anywhere else).

They would have to be tagged (e.g. ‘commandLine’) for browsability and registered users could upvote their favourites in each category. Then each tag would list packages sorted by a mix of number of downloads and votes.

Sort of like VersionTracker used to be for Mac Software.

I would prefer this over a monolithic library.

Perhaps if after a few years some collection of packages became obviously consistently popular you could wrap them together into a ‘boost-like’ umbrella package.

I think that if Swift.org hosted an official index of packages it would help the community see what's most used, upvoted, etc. We've already discussed this many times in these forums and the "Related Projects" category was a first good step but I think is time for the next step.

I much rather prefer put more efforts into improving the community invovelment on those "batteries" than agree in a theoretical good solution to include in the stdlib.

1 Like

I think there is another option between "thin stdlib" and "federated libraries". This is to make stdlib a collection of "packages" (I'm using "package" in an abstract sense, meaning something you can import).

Then we can have a series of packages that you import only if you need to use them, some examples could be: CommandLine, Compression, SHA, CSV, HTTP, etc. What is now known as "stdlib" will become the "builtin" package that is special in that you don't need to import it.

The idea here is to bundle some essential functionality and avoid situations like leftpad and event-stream. What is deemed essential can be decided by swift evolution. Other more specialized packages can of course be provided by SPM as expected.

3 Likes

I agree with not having this in the stdlib etc, but these discussions are going to continue to show up due to the main problem of discoverability.

As it stands there isn't an 'official' place to search for and discover packages that solve the issues that are being pitched for inclusion in the stdlib. I believe that if a solution for this existed where developers could submit their packages and other users could search for them, then there would be less concern over trying to include everything into the stdlib.

The beauty of the SPM is that it doesn't need a central location to pull packages from, and I believe this is one of its strengths. But in terms of looking for a package to help solve your issue, it's a bit hit and miss at the moment. There are some tools out there trying to solve the issue but they aren't official, or officially endorsed, so even trying to discover the discoverability tools is difficult.

2 Likes

It could also support sub-commands, like in git commit.

(hey Uli!)

I still think there is value in having a few batteries included. For example I don't think there are people arguing that Codable was a mistake and we would be better off letting the "market" decide between 4 competing JSON implementations.

1 Like

Sure, I totally agree, but we need to agree where it makes sense and where it doesn't.

1 Like

To be fair, JSONEncoder/Decoder are in Foundation

That's true. And furthermore, the big selling point for most people is the compiler integration that autogenerates it for most cases.

Even without that, Codable on its pure form is about "visiting" the properties of a type and being able to work with that. As pointed out, the specifics decoders are separated concepts.

A long way of saying that for me Codable (without the encoders/decoders) feels more like a language feature. Then you have the encoders/decoders as libraries (in foundation in this case). This and other pitches don't seem like that at all, as is shown by using "batteries included".

Don't get me wrong, I'm onboard with making things easier and getting more batteries. I just think that if we just accept specific things in the stdlib we don't really solve the problem. We should get into a point where spm + index is easy enough to work with that nobody really wants to pollute the stblib with more stuff.

3 Likes

For the purpose of this discussion, this is a distinction without a difference. These packages would still be constrained by the standard library evolution process, limited by its backwards compatibility requirements, unable to be versioned separately, and will strangle out substantial amounts of alternative development effort. The fact that they are not part of the default namespace (the only real difference here) does not really address any of the core concerns raised above.

Discoverability never solves this problem. There is always going to be a subset of people who want the standard library to grow to include more features. I say this from experience in the Python community, where discoverability is better but people still regularly propose to expand the standard library.

Discoverability does reduce the frequency of this issue, however, and is a worthy effort to consider. Again, though, it takes many forms. A motivated community member could begin maintaining a community-supported list or equivalent system that could help, perhaps beginning by scraping GitHub for Package.swift files.

As the comments below have demonstrated, Codable is a bad example here because it is agnostic to the JSON implementation. This means it encourages alternative implementations: if you don't like Foundation's JSONEncoder then you can go ahead and write your own, and the entire Swift ecosystem will keep working. That's great, it's really powerful! But it's not comparable to this proposal.

IMO, the Swift standard library is at its best when it provides "currency" data types and protocols. These are things that provide the boundary between different components, ensuring that different frameworks and libraries can be glued together without worrying too much about expressing the transformation. To me, command line parsing does not achieve this.

3 Likes

Well, most of these concerns are really non-issues in a lot of the cases. I'd rather my SHA1, compression, JSON, ... libraries to move slowly and not break things. And I don't find competition in the space of e.g. SHA1 digests to be meaningful. I'd rather we had collaboration than competition. The only controversial entry here is probably HTTP networking which is also Python's biggest failure. But Go doesn't seem to have this problem, so it seems to be more of a problem with how Python was managed than the model itself.

In turn, your answer doesn't address any of the issues that I raised, namely our software dependency problem [1]. What happens when libraries disappear, when sole maintainers get bored or move on, when malware is added (all of those have happened in the NPM world). More generally how you are asking for production apps and services to rely on and trust the work of tens (or hundrends) unpaid volunteers.

[1] https://research.swtch.com/deps

1 Like