DISCUSS: future of docopt-ng

[NickCrews]

This is a broad topic that I've been thinking about recently, that has come up in several places, such as #4 and #45

What does the community want for the future of docopt-ng?

1. A maintained and type-hinted port of the original docopt
2. that, plus more features
3. something else?

I have so far been pursuing 1. This was because

• I was estimating that most users are just porting over from the old docopt to here.
• There are now many more CLI libraries than when docopt was released, so I think any new applications are more likely to use one of the more modern frameworks, that provides type hints, input validation etc. Docopt inherently is limited in those two aspects because it operates on plain docstrings, so those would have to be done as a post-processing step, leading to code duplication. Perhaps we should add this to the README so that new users know the limitations coming in.
• I liked the scope that @keleshev started with, a function that takes a usage message and sys.argv, and returns a dict. It's well defined, has a narrow interface but a complex implementation, and composes well. In release 0.9.0 I even removed the magic stuff in pursuit of this (see the changelog for rationale)
• So far, additional features have been able to be implemented as extensions, rather than needing to change this repo
  • the magic stuff could have been an extension
  • docopt-dispatch could be implemented as an extension
• I have limited time and already am not giving this all the attention that it should probably be getting, so a smaller scope is more tenable for me.

I would love to hear people's opinions here. I don't have a lot of open source maintainership experience, so I apologize for any frustration that is caused by that.

Cheers,
Nick

[Sylvain303]

Hello @NickCrews,

I'm the maintainer of docopts with an S for shell in bash. Written in go. I'm in a huge pause on this project since last September.

My own plan was to totally rewrite the golang parser to fully handle parse errors of docopt language, hand have a really nice diagnostic behavior. AST and much more. This is an handcrafted parser and lexer. It's still at some PoC level.

I appreciate the original goal of docstring => to dict. In bash it makes even more sense. Then I handle the argument the way I want, in python too. If my docopt parser finally succeed to have a really nice error reporting (better achieved in handcrafted parser than in generated one) I was planing to port it back to python as well. The original docopt project is frozen but full of really nice issues (about my goal on error reporting).

Error reporting / diagnostic is mostly the only thing I need for a cli at my side.

Hope that helps.
Sylvain.

[NickCrews]

Hi @Sylvain303, nice to meet you!

If the multiple ports were able to share one grammar, I think that would help unify the whole ecosystem, and reduce bugs. If someone wrote that grammar, would you want to use it? I'm no expert in grammars so I don't know which one we would want (EBNF, PEG, etc?). I think the biggest drawback would be the generic errors, as you say. IDK how to address this, maybe in each language's grammar-parsing library there would be a way to catch the parsing errors and wrap them in more helpful error messages?

[Sylvain303]

If the multiple ports were able to share one grammar, I think that would help unify the whole ecosystem, and reduce bugs.

Basically that's what was done. On the code side, not a well defined grammar. Then the python development stopped. The python original docopt was mostly "blindly" ported into other some language. That was the case in golang and some other I looked at. The parser algorithm was almost verbatim in Go the same as in the original python one.

This issue discuss about grammar: docopt/docopt#330 (pointer to and language description used in F# port)

I made an analysis myself here: https://github.com/docopt/docopts/blob/dev-grammar/grammar/docopt_language.ebnf

If someone wrote that grammar, would you want to use it? I'm no expert in grammars so I don't know which one we would want (EBNF, PEG, etc?). I think the biggest drawback would be the generic errors, as you say.

I'm not a grammar parser expert myself, but I love it since my first met of the GNU lex & Yacc book 20 years ago...

The docopt language is not trivial (on grammar definition) and has ambiguity. I needed a state lexer to tokenise it the way I wanted.

What the book was saying, is that grammar generator greatly simplify the complexity of writing a parser and its maintenance. But the error handling is mostly limited. I tried some generator and I was limited. I finally looked at golang owns parser which is a dedicated code and lexer too, and decided to write my own for docopt too.

F# port was using some F# language trick which is friendly for parsing and matching rules. Other language has some similar facilities like in Rust I think.

would you want to use it?

I'm actually using the original golang one in doctops. This one also stopped, as mentioned it was a port. Go has now some interesting argument parser, that also adresses the command line completion. As in kubectl for example. Still docopt is the only one based on documentation parsing avoiding to remember the object side of other argument parser and their fixed model. As @keleshev was mentioning back in his original video presentation in 2012 at PyCon.

Yes I would use a shared grammar / parser and contribute to it. Obviously it need to satisfy my own need on error reporting too. And we may also need to agree on some behavior fix on some strange docopt original behavior (ex: dict mapping change when Options: is present with argument --long=<argument_for_long>)

IDK how to address this, maybe in each language's grammar-parsing library there would be a way to catch the parsing errors and wrap them in more helpful error messages?

I turn the problem in this way: what is the result I would like to have with error handling?
Then I'm experimenting with the parser and erroneous ambiguous docopt input.

When I would have the AST, obviously I will also generate the command line completion too. 😉

[FelixSchwarz]

Just a random Python developer: I always loved docopt but was worried due to the lack of updates. I considered switching away from docopt but now that I found this project I hope I can use it for much longer (also switching away from docopt would take a long time as we have hundreds of scripts using docopt).

The thing I liked most about docopt was that I could read the help text at the top and that text was in sync with the actual script. I don't need more features but better error messages and diagnostics would be nice.
Over the years I encountered some parsing bugs/shortcomings in docopt but I was able to work around these so I don't remember any specifics anymore.

[Cyleigh]

A colleague suggested docopt as a solution to a problem that I'm currently having. I'm building a library of command line tools, and templates about how to use the tool. Currently, there is a manual process of entering the command line with all its arguments etc, which is obviously not ideal. I'd like docopt to spit out my list of arguments with the name, flag, any choice options and the help text. This is a small part of a much larger project, so I was definitely hoping to find something already in the wild but haven't quite found what I need.

[NickCrews]

@Cyleigh that use case makes sense, thanks for the thoughts! Does docopt satisfy your needs, or is there something missing?

[Cyleigh]

@Cyleigh that use case makes sense, thanks for the thoughts! Does docopt satisfy your needs, or is there something missing?

I'm not necessarily the most experienced python user, so I am having some trouble. For example, the help string of options and arguments are not returned in ParsedOptions, and it very much assumes that I will want to parse a command line (which I don't want to do on creation).

[sebastian-philipp]

I continue to really enjoy docopt: it gives me a CLI without the need to thing about any APIs.

[jfolz]

Just found this and thought I'd chime in. I'm using docopt-ng in my project sqltrack. Specifically, I need to parse arguments without actually running the code, and put those arguments into a database. Think scheduling a job on an HPC cluster, where the software available on the head node is very limited. At least back then, I found no other option besides putting the argument parsing into a separate file that doesn't depend on any third-party libraries. Even then, there could be problems, since the compute environment most likely includes a different version of Python.

In practice, missing data types meant arguments read from the database were far less useful than they could've been, but I found a solution that works pretty well: Just guess. There are a number of rules based on suffix matching and users can add their own. E.g., an option that matches *-path would be converted to pathlib.Path. If no rule applies cleanly, it tries int, float, complex, and finally json.loads.
Overall, I'm pretty happy how this turned out. Argument parsing using docopt-ng is opt-in, and I haven't had any complaints about this scheme.

In conclusion, I'm happy to use docopt-ng as-is, and would appreciate if it remains functional with future Python versions :)

[dwt]

Having used the original docopt - so excuse me if you already support this.

I would really love for a special argument that just dumps a structured (json / yaml) representation of the argument parser, that easily allows me to check what is parsed and wether changes I did might have changed the way args will be parsed.

This could also allow diffing between the way stuff is parsed in versions (as a kind of acceptance tests).

That is something I always missed when using the original docopt, that I always just used very rarely and was having a lot of trouble remembering the intricacies of the docopt configuration language.

[NickCrews]

@dwt that feature makes sense, and should be theoretically be possible.

If you want to discuss this further, please make a new feature request issue, so this thread doesn't go off topic. But in short:
I am not going to implement that (don't have the time time or interest), but I would consider merging a PR if

• I didn't have to do huge amounts of edits/review on it
• no additional dependencies, or very lightweight ones
• didn't look like a huge maintenance burden. I'm skeptical this would be possible
• didn't break existing users
  I don't know of any common json schema that would work for this. I would love to not reinvent the wheel here, so would be worth it to explore what other. It also seems to me like "checking for a change in parsing behavior" would not be that useful on its own. If the behavior started off incorrect, you aren't actually catching that. It seems more useful to actually test assert my_parse("myprogram new --name foo") == <some dict>, even if it might be a bit more verbose. Also would serve as documentation.

[Sylvain303]

Having used the original docopt - so excuse me if you already support this.

I would really love for a special argument that just dumps a structured (json / yaml) representation of the argument parser, that easily allows me to check what is parsed and wether changes I did might have changed the way args will be parsed.

This could also allow diffing between the way stuff is parsed in versions (as a kind of acceptance tests).

That is something I always missed when using the original docopt, that I always just used very rarely and was having a lot of trouble remembering the intricacies of the docopt configuration language.

@dwt

Hello, I'm not very active yet on docopts coding (golang version for bash) watch-out the extra 'S'. But I
try to implement such thing: dumping what was actually parsed by docopt parser.

Could you create an new issue with some sample of what you would love to see outputted? That would be very useful. 🤩

[dwt]

@Sylvain303 @NickCrews Thanks for the feedback, I have opened a separate bug at #61

[nkakouros]

I am a heavy user of docopt. The selling points are, I think, three:

• it helps keep documentation and code in sync,
• it is a lightweight and straightforward dependency,
• it supports a familiar and intuitive help text syntax.

Personally, I would like for docopt to be maintained so that it follows newer versions of Python and its quality improves (there are reported bugs and ambiguities).

Regarding new features, it would be great to have them if they contribute to the selling points without compromising any of them. Given that there is an ecosystem of other projects around docopt, it would also be nice to maintain an up-to-date list of such projects.

[sebastian-philipp]

one thing that bothers me is that the API of argparse is super unintuitive, which is pretty much solved by docopt.

[frnhr]

Another random dev here, I keep returning to docopt.

Alternatives are just so much less intuitive and feel bulky. If I need validation, I don't mind a bit of duplication, for the benefits of KISS and the documentation-first approach.

I wish there was a sleek way of combining with pydantic (without duplication), though.

But definitely don't want to see docopt gone!

[Notmarrco]

Same here, I like the simplicity of docopt, and I wasn't even annoyed by docopt being unmaintained, but now I'll switch to docopt-ng.
I use it with commandopt for all my projects (disclaimer : I wrote it) to easily execute functions (commands) given docopts arguments.

The only thing I would dream to see is an autocomplete, so perhaps some external tool taking a docopt and turning it into bash completions, but imo that's out of the scope of docopt-ng.

[jbaber]

A maintained, tested, typehinted docopt that continues to work with updates to Python makes me very happy. A pure doc to dict converter, too, would be fine, too. I'm happy to parse the arguments myself. In particular, it makes parsing with custom functions an option.