Turning WDL files into executables using Cromexe

[prihoda]

Hi everyone,

I got an idea (read "stole it from another tool") that WDL files can easily be made executable using shebang lines.

Instead of calling:

java -jar ${CROMWELL_JAR} run example/hello.wdl -i example/hello.input.json

By using a #!/usr/bin/env cromwexe shebang line in our WDL file, we can call simply:

example/hello.wdl -i example/hello.input.json

See my working prototype:
https://github.com/prihoda/cromwexe

The only thing cromwexe is doing at the moment is passing all args to the regular java command:

cmd = ["java", "-jar", cromwell_jar, "run"] + sys.argv[1:]

It starts to get interesting once you realize that if we can parse the workflow script inside cromwexe, we can automatically generate a command-line interface that parses the args directly from the command line (and even provide a help screen with descriptions of all input args):

example/hello.wdl -i example/hello.defaults.json --sayHello.name World

We can also parse the job metadata output of cromwell to determine the location of our output files and move them over from the execution folder to the user's working folder, if the user provides the output arg as well:

example/hello.wdl -i example/hello.defaults.json --sayHello.name World --output.greeting greeting.txt

Edit: I see that this can be done by the use_relative_output_paths or final_workflow_outputs_dir args. Then I guess the added functionality is that output args could be used to define each output file path separately.

I have two questions :)

1. What do you think? I am excited to implement the full solution, it should not take more than a few hours.
2. Can I use this parser? https://github.com/TMiguelT/WdlParserPackaging

[cjllanwarne]

I think this is a really cool idea! It feels somewhat above the scope of Cromwell itself (since it's calling Cromwell rather than part of it) but as part of the openWDL ecosystem I think it's an awesome thing to have.

I suggest getting in touch with @mlin and @dinvlad for the following reasons:

What do you think? I am excited to implement the full solution, it should not take more than a few hours.

• I mention @dinvlad because he's been working on a plugin for VS Code with a WDL debugging option for WDL files. It could be that there's a fair amount of crossover you could take advantage of.

Can I use this parser? https://github.com/TMiguelT/WdlParserPackaging

• @mlin has a much friendlier WDL parsing library over at miniWDL. I suspect you'll have a much better time using that parser API.

[dinvlad]

Additionally, @rexwangcc has been working on https://cromwell-tools.readthedocs.io, which at least partially seems to implement what you proposed. We're also planning to move other functionality, like parsing of metadata.json for failures, into it over time, to be used together with miniWDL for WDL debugging support.

[gemmalam]

Closing this due to no actionable items for Cromwell at this time.

[mlin]

@prihoda

miniwdl has a little CLI wrapper to make it nicer to launch cromwell locally. It doesn't do the the shebang script which is a neat idea, however, it does implement versions of (i) parsing the task/workflow inputs to expose them as command-line arguments, and (ii) parsing the outputs to organize them more nicely after they come out. Here is a link to the CLI entrypoint for this where you can see how all this happens. I'd be happy to work with you on merging & fleshing out the ideas.

[prihoda]

Moving the discussion here: chanzuckerberg/miniwdl#158
Working draft: chanzuckerberg/miniwdl#164