Initial proposal for CLI plugin system.

[jeromekelleher]

Following on from sgkit-dev/sgkit-plink#8 and addressing #53, here's an initial proposal for how the plugin system might work.

So, basically sgkit needs to know about all the plugins it might support, but only loads the ones that are installed. I've included a sniffer function here, as this should then allow us to implement the generic sgkit import command.

With this, we get

$ python3 -m sgkit  --help
usage: __main__.py [-h] {list,ls,import-plink} ...

Utilities for manipulating sgkit datasets

positional arguments:
  {list,ls,import-plink}
    list (ls)           List the arrays stored in a dataset
    import-plink        Convert data in plink to sgkit

optional arguments:
  -h, --help            show this help message and exit

and

$ python3 -m sgkit import-plink --help
usage: __main__.py import-plink [-h] [--bim_sep BIM_SEP] [--fam_sep FAM_SEP]
                                input output

positional arguments:
  input              The input data in {plugin.format_name} format
  output             The path to store the converted dataset

optional arguments:
  -h, --help         show this help message and exit
  --bim_sep BIM_SEP  Separator used when parsing BIM files
  --fam_sep FAM_SEP  Separator used when parsing FAM files

It's not ideal, but seems like a reasonable compromise setup?

[tomwhite]

This looks like a good general direction. I wonder if using something like https://github.com/click-contrib/click-plugins would mean we can avoid having to write a plugin mechanism. What do you think @jeromekelleher?

[jeromekelleher]

I can see pros and cons to using a more general plugin mechanism @tomwhite. The main downside I see is that it's too general - we have a very well encapsulated problem here, which means that we can avoid writing any CLI code at all in the format repos. If we use a general CLI plugin system here, it will mean that we still have to write a bunch of CLI code across each of the repos. This would mean a fair bit of duplication, inevitably I think.

Something to talk about later, anyway.

[ravwojdyla]

I would be +1 to @tomwhite and click and not reinventing the wheel.

[tomwhite]

@jeromekelleher I agree there's a trade off, but I don't have a good sense where the boundary is in this case. I've only used Click for a very small project (and it didn't use the plugin part at all), but I found that it was mainly decorating functions with Click annotations - it was very low in terms of code overhead.

BTW I don't think there's a meeting today - Jeff has scheduled one for next week.

[jeromekelleher]

BTW I don't think there's a meeting today - Jeff has scheduled one for next week.

Ah, ok, thanks. I won't be around next week, so I guess we can pick this up again in a few weeks.

[dharhas]

I highly recommend click. I've used it in many projects and it is a mature and easy to use codebase and is much better than custom coding in argparse.

[jeromekelleher]

Click does look really nice. Maybe I should play with it a bit and see if we can do something similar as I've done here with it. I still think we should keep the plugin interface more limited, though, as there's only a handful of things that the file format needs to do and the CLI functionality can be done generally.

Also, I don't see how we could make sgkit import <dataset in whatever format> work without some tight integration and a well defined interface.

Thanks for the input all, I'll try out click and see how it goes.

[jeromekelleher]

I've updated this with a proposal using click (which is AWESOME - I am going to rip argparse out of my projects ASAP!).

There's a little bit of trickery involved in making the import/export commands work in a general way, but I don't think it's too bad. All the other commands we want in the CLI can be done using the standard click decorator syntax. The format repos need to provide function that converts the format into an sgkit dataset, and some options that can be provided to this function, via click.Option instances. This seems like a nice clean separation of duties to me, and will keep the data format repos nice and focused.

There's some subtlety about how we manage the execution, and we need some functionality in the main library around writing datasets to disk that we need, but I think the framework is basically sound.

Do we like this better, or is the idea of a specialised plugin system around import/export of data still unpalatable?

[tomwhite]

This looks great! I think it would be good to explore click's plugin system too, to see how it compares with the one you proposed @jeromekelleher.