Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change the list file argument expansion extension and add a new option #95

Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@
import org.apache.commons.lang3.tuple.Pair;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.broadinstitute.barclay.argparser.parseropt.CommandLineParserOption;
import org.broadinstitute.barclay.argparser.parseropt.ExpandCollectionExtensionOption;
import org.broadinstitute.barclay.argparser.parseropt.SwitchCommandLineParserOptions;
import org.broadinstitute.barclay.utils.Utils;

import java.io.BufferedReader;
Expand Down Expand Up @@ -62,8 +65,10 @@ public final class CommandLineArgumentParser implements CommandLineParser {
public static final String COMMENT = "#";
public static final String POSITIONAL_ARGUMENTS_NAME = "Positional Argument";

// Extension for collection argument list files
private static final String COLLECTION_LIST_FILE_EXTENSION = ".list";
/**
* Default options for the parser.
*/
public static final Set<CommandLineParserOption> DEFAULT_PARSER_OPTIONS = Collections.singleton(new ExpandCollectionExtensionOption());

private static final Logger logger = LogManager.getLogger();

Expand Down Expand Up @@ -115,7 +120,7 @@ private boolean inArgumentMap(ArgumentDefinition arg){
// and into which the values will be assigned.
private final Object callerArguments;

private final Set<CommandLineParserOptions> parserOptions;
private final Set<CommandLineParserOption> parserOptions;

// null if no @PositionalArguments annotation
private Field positionalArguments;
Expand Down Expand Up @@ -146,14 +151,16 @@ private String getUsagePreamble() {
}

/**
* <p>Note: uses {@link #DEFAULT_PARSER_OPTIONS}.
*
* @param callerArguments The object containing the command line arguments to be populated by
* this command line parser.
*/
public CommandLineArgumentParser(final Object callerArguments) {
this(
callerArguments,
Collections.<CommandLinePluginDescriptor<?>>emptyList(),
Collections.<CommandLineParserOptions>emptySet()
DEFAULT_PARSER_OPTIONS
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we now want to have more complex options for things like custom expansion file extensions, I think it would be cleaner to switch over to using a configuration object that is passed in by clients. Barclay can have a default implementation, with methods like "List getExpansionFileExtensions()", and custom implementations be provided by the clients.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So do you mean to use a BarclayConf interface (or mutable object) to pass to the parser? I think that passing the options as single elements makes clear which behaviour is changing/requested...

If we are going to that direction, I can either make another commit here or open a new PR, whatever you prefer...

Copy link
Collaborator

@cmnbroad cmnbroad Oct 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant that instead of the CommandLineParserOption enum or interface, Barclay should have a CommandLineParserConfig class, with methods like "List getExpansionFileExtensions()" and "boolean getAppendToCollections()", with default implementations. Any customizations would be implemented by passing a config object to the constructor which is a subclass of that default implementation. Barclay would just query it; hopefully the object itself won't need any mutators.

When we just had options that were toggles, it would have been overkill, but with the need to return a custom list, this is cleaner, and will eliminate the need for type casting, etc. Reusing this PR is fine, or make a new one if its easier - either way. Thanks!

);
}

Expand All @@ -168,7 +175,7 @@ public CommandLineArgumentParser(final Object callerArguments) {
public CommandLineArgumentParser(
final Object callerArguments,
final List<? extends CommandLinePluginDescriptor<?>> pluginDescriptors,
final Set<CommandLineParserOptions> parserOptions) {
final Set<CommandLineParserOption> parserOptions) {
Utils.nonNull(callerArguments, "The object with command line arguments cannot be null");
Utils.nonNull(pluginDescriptors, "The list of pluginDescriptors cannot be null");
Utils.nonNull(parserOptions, "The set of parser options cannot be null");
Expand Down Expand Up @@ -618,7 +625,7 @@ private void setArgument(ArgumentDefinition argumentDefinition, List<String> val
throw new CommandLineException("Argument '" + argumentDefinition.getNames() + "' cannot be specified more than once.");
}
if (argumentDefinition.isCollection) {
if (!parserOptions.contains(CommandLineParserOptions.APPEND_TO_COLLECTIONS)) {
if (!parserOptions.contains(SwitchCommandLineParserOptions.APPEND_TO_COLLECTIONS)) {
// if this is a collection then we only want to clear it once at the beginning, before we process
// any of the values, unless we're in APPEND_TO_COLLECTIONS mode, in which case we leave the initial
// and append to it
Expand Down Expand Up @@ -714,8 +721,16 @@ private void setArgument(ArgumentDefinition argumentDefinition, List<String> val
*/
private List<String> expandListFile(final List<String> originalValues) {
List<String> expandedValues = new ArrayList<>(originalValues.size());

// TODO - assume that there is only one ExpandCollectionExtensionOption?
final ExpandCollectionExtensionOption expandOption = new ExpandCollectionExtensionOption(
parserOptions.stream()
.filter(s -> s instanceof ExpandCollectionExtensionOption)
.flatMap(s -> ((ExpandCollectionExtensionOption) s).getExtensions().stream())
.collect(Collectors.toSet()));

for (String stringValue: originalValues) {
if (stringValue.endsWith(COLLECTION_LIST_FILE_EXTENSION)) {
if (expandOption.getExtensions().stream().anyMatch(stringValue::endsWith)) {
expandedValues.addAll(loadCollectionListFile(stringValue));
}
else {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
package org.broadinstitute.barclay.argparser.parseropt;

/**
* Marker interface for options used to control command line parser behavior.
*
* @author Daniel Gomez-Sanchez (magicDGS)
*/
public interface CommandLineParserOption {
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
package org.broadinstitute.barclay.argparser.parseropt;

import org.broadinstitute.barclay.utils.Utils;

import java.util.Collection;
import java.util.Collections;

/**
* Option for loading argument collections from files ending with concrete extensions.
*
* @author Daniel Gomez-Sanchez (magicDGS)
*/
public final class ExpandCollectionExtensionOption implements CommandLineParserOption {

/**
* Default xtension for collection argument list files.
*/
public static final String DEFAULT_COLLECTION_LIST_FILE_EXTENSION = ".args";

private final Collection<String> extensions;

/** Constructor for a collection of extensions. */
public ExpandCollectionExtensionOption(final Collection<String> extensions) {
this.extensions = Utils.nonNull(extensions);
}

/**
* Default constructor uses {@link #DEFAULT_COLLECTION_LIST_FILE_EXTENSION}.
*/
public ExpandCollectionExtensionOption() {
this(Collections.singleton(DEFAULT_COLLECTION_LIST_FILE_EXTENSION));
}

/**
* Load options for argument collections from files if the argument value ends with one of the
* provided extensions.
*/
public Collection<String> getExtensions() {
return extensions;
}
}
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package org.broadinstitute.barclay.argparser;
package org.broadinstitute.barclay.argparser.parseropt;

/**
* Options used to control command line parser behavior.
* Options to switch behaviour from the default parser.
*/
public enum CommandLineParserOptions {
public enum SwitchCommandLineParserOptions implements CommandLineParserOption {

/**
* The default behavior for the parser is to:
Expand All @@ -17,5 +17,5 @@ public enum CommandLineParserOptions {
* initial values of the collection, and allows the special value "null" to be used first to clear the initial
* values.
*/
APPEND_TO_COLLECTIONS // default behavior is "replace"
APPEND_TO_COLLECTIONS, // default behavior is "replace"
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
package org.broadinstitute.barclay.argparser;

import org.broadinstitute.barclay.argparser.parseropt.CommandLineParserOption;
import org.broadinstitute.barclay.argparser.parseropt.ExpandCollectionExtensionOption;
import org.broadinstitute.barclay.argparser.parseropt.SwitchCommandLineParserOptions;
import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
Expand All @@ -8,6 +11,7 @@
import java.io.IOException;
import java.io.PrintWriter;
import java.util.*;
import java.util.stream.Collectors;

/**
* Tests for arguments that are collections (not to be confused with ArgumentCollection).
Expand Down Expand Up @@ -46,7 +50,7 @@ public Object[][] uninitializedCollections() {
},
{
inputArgs,
Collections.singleton(CommandLineParserOptions.APPEND_TO_COLLECTIONS), // append mode
Collections.singleton(SwitchCommandLineParserOptions.APPEND_TO_COLLECTIONS), // append mode
makeList("L1", "L2"),
makeList("S1"),
makeList("HS1"),
Expand All @@ -58,7 +62,7 @@ public Object[][] uninitializedCollections() {
@Test(dataProvider="uninitializedCollections")
public void testUninitializedCollections(
final String[] args,
final Set<CommandLineParserOptions> parserOptions,
final Set<CommandLineParserOption> parserOptions,
final List<String> expectedList,
final List<String> expectedArrayList,
final List<String> expectedHashSet,
Expand Down Expand Up @@ -87,7 +91,7 @@ public Object[][] initializedCollections() {
return new Object[][]{
{
inputArgs,
Collections.singleton(CommandLineParserOptions.APPEND_TO_COLLECTIONS),
Collections.singleton(SwitchCommandLineParserOptions.APPEND_TO_COLLECTIONS),
makeList("foo", "bar", "baz", "frob") // original values retained
},
{
Expand All @@ -97,7 +101,7 @@ public Object[][] initializedCollections() {
},
{
inputArgsWithNullAtStart,
Collections.singleton(CommandLineParserOptions.APPEND_TO_COLLECTIONS),
Collections.singleton(SwitchCommandLineParserOptions.APPEND_TO_COLLECTIONS),
makeList("baz", "frob") // original values replaced
},
{
Expand All @@ -107,7 +111,7 @@ public Object[][] initializedCollections() {
},
{
inputArgsWithNullMidStream,
Collections.singleton(CommandLineParserOptions.APPEND_TO_COLLECTIONS),
Collections.singleton(SwitchCommandLineParserOptions.APPEND_TO_COLLECTIONS),
makeList("frob") // reset mid-stream via midstream null
},
{
Expand All @@ -117,7 +121,7 @@ public Object[][] initializedCollections() {
},
{
new String[]{},
Collections.singleton(CommandLineParserOptions.APPEND_TO_COLLECTIONS),
Collections.singleton(SwitchCommandLineParserOptions.APPEND_TO_COLLECTIONS),
makeList("foo", "bar")
},
{
Expand All @@ -131,7 +135,7 @@ public Object[][] initializedCollections() {
@Test(dataProvider="initializedCollections")
public void testInitializedCollections(
final String[] args,
final Set<CommandLineParserOptions> parserOptions,
final Set<CommandLineParserOption> parserOptions,
final List<String> expectedResult) {
final InitializedCollections o = new InitializedCollections();
final CommandLineParser clp = new CommandLineArgumentParser(
Expand All @@ -157,16 +161,21 @@ class CollectionForListFileArguments {
@DataProvider(name="listFileArguments")
public Object[][] listFileArguments() {
final String[] inputArgs = new String[] { "shmiggle0", "shmiggle1", "shmiggle2" };

final CommandLineParserOption defaultExpandCollection = new ExpandCollectionExtensionOption();

return new Object[][] {
{
// replace mode
Collections.EMPTY_SET, // parser options
inputArgs, // args
new String[] {"shmiggle0", "shmiggle1", "shmiggle2"}, // expected result
Collections.singleton(defaultExpandCollection), // parser options
inputArgs, // args
new String[] {"shmiggle0", "shmiggle1", "shmiggle2"}, // expected result
},
{
// append mode
Collections.singleton(CommandLineParserOptions.APPEND_TO_COLLECTIONS), // parser options
new HashSet<>(Arrays.asList(
defaultExpandCollection,
SwitchCommandLineParserOptions.APPEND_TO_COLLECTIONS)), // parser options
inputArgs, // args
new String[] {"foo", "bar", "shmiggle0", "shmiggle1", "shmiggle2"}, // expected result
},
Expand All @@ -176,7 +185,7 @@ public Object[][] listFileArguments() {
// Test that .list files populate collections with file contents, both mpdes
@Test(dataProvider="listFileArguments")
public void testCollectionFromListFile(
final Set<CommandLineParserOptions> parserOptions,
final Set<CommandLineParserOption> parserOptions,
final String [] argList,
final String[] expectedList) throws IOException
{
Expand All @@ -196,14 +205,34 @@ public void testCollectionFromListFile(
Assert.assertEquals(o.LIST, makeList(expectedList));
}

@Test
public void testDoNotExpandCollectionListFile() {
final CollectionForListFileArguments o = new CollectionForListFileArguments();
final CommandLineParser clp = new CommandLineArgumentParser(
o,
Collections.emptyList(),
Collections.emptySet()
);

final String noListFile = "no_arg_list_file" + ExpandCollectionExtensionOption.DEFAULT_COLLECTION_LIST_FILE_EXTENSION;

final String[] args = {"--LIST", noListFile};
Assert.assertTrue(clp.parseArguments(System.err, args));

Assert.assertEquals(o.LIST, makeList(noListFile));
}

@DataProvider(name="mixedListFileArguments")
public Object[][] mixedListFileArguments() {
final String[] inputArgList1 = { "shmiggle0", "shmiggle1", "shmiggle2" };
final String[] inputArgList2 = { "test2_shmiggle0", "test2_shmiggle1", "test2_shmiggle2" };

final CommandLineParserOption defaultExpandCollection = new ExpandCollectionExtensionOption();

return new Object[][] {
{
// replace mode
Collections.EMPTY_SET, // parser options
Collections.singleton(defaultExpandCollection), // parser options
inputArgList1, // args list 1
inputArgList2, // args list 2
new String[] {"shmiggle0", "shmiggle1", "shmiggle2"}, // expected result list 1
Expand All @@ -217,7 +246,9 @@ public Object[][] mixedListFileArguments() {
},
{
// append mode
Collections.singleton(CommandLineParserOptions.APPEND_TO_COLLECTIONS), // parser options
new HashSet<>(Arrays.asList(
defaultExpandCollection,
SwitchCommandLineParserOptions.APPEND_TO_COLLECTIONS)), // parser options
inputArgList1, // args list 1
inputArgList2, // args list 2
new String[] {"foo", "bar", "shmiggle0", "shmiggle1", "shmiggle2"}, // expected result list 1
Expand All @@ -236,7 +267,7 @@ public Object[][] mixedListFileArguments() {
// Test that .list files intermixed with explicit command line values populate collections correctly, both mpdes
@Test(dataProvider="mixedListFileArguments")
public void testCollectionFromListFileMixed(
final Set<CommandLineParserOptions> parserOptions,
final Set<CommandLineParserOption> parserOptions,
final String [] argList1,
final String [] argList2,
final String[] expectedList1,
Expand Down Expand Up @@ -281,7 +312,7 @@ public void testCollectionThatCannotBeAutoInitialized() {
// Helper methods

private File createListArgumentFile(final String fileName, final String[] argList) throws IOException {
final File listFile = File.createTempFile(fileName, ".list");
final File listFile = File.createTempFile(fileName, ".args");
listFile.deleteOnExit();
try (final PrintWriter writer = new PrintWriter(listFile)) {
Arrays.stream(argList).forEach(arg -> writer.println(arg));
Expand Down