Merge pull request #40 from nextflow-io/fix/support-$defs

support $defs
nextflow-io · Aug 5, 2024 · 5a24da2 · 5a24da2
2 parents eb12203 + dd6bd83
commit 5a24da2
Show file tree

Hide file tree

Showing 24 changed files with 61 additions and 53 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,11 @@
 # nextflow-io/nf-schema: Changelog
 
+# Version 2.1.0
+
+## JSON schema fixes
+
+1. The `defs` keyword is now deprecated in favor of the `$defs` keyword. This to follow the JSON schema guidelines. We will continue supporting `defs` for backwards compatibility.
+
 # Version 2.0.1 - Tsukemen
 
 ## Vulnerability fix

diff --git a/docs/migration_guide.md b/docs/migration_guide.md
@@ -65,15 +65,15 @@ e 's/from "plugin\/nf-validation"/from "plugin\/nf-schema"/g' {} +
 If you aren't using any special features in your schemas, you can simply update your `nextflow_schema.json` file using the following command:
 
 ```bash
-sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/defs/g' nextflow_schema.json
+sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/$defs/g' nextflow_schema.json
 ```
 
-This will replace the old schema draft specification (`draft-07`) by the new one (`2020-12`), and the old keyword `definitions` by the new notation `defs`.
+This will replace the old schema draft specification (`draft-07`) by the new one (`2020-12`), and the old keyword `definitions` by the new notation `$defs`.
 
 !!! note
 
     Repeat this command for every JSON schema used in your pipeline. e.g. for the default samplesheet schema in nf-core pipelines:
-    `bash sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/defs/g' assets/schema_input.json`
+    `bash sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/$defs/g' assets/schema_input.json`
 
 !!! warning
 

diff --git a/docs/nextflow_schema/create_schema.md b/docs/nextflow_schema/create_schema.md
@@ -51,7 +51,7 @@ nf-core schema build
     The current version of `nf-core` tools (v2.13.1) does not support the new schema draft used in `nf-schema`. Running this command after building the schema will convert the schema to the right draft:
 
     ```bash
-    sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/defs/g' nextflow_schema.json
+    sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/$defs/g' nextflow_schema.json
     ```
     A new version of the nf-core schema builder will be available soon. Keep an eye out!
 

diff --git a/docs/nextflow_schema/nextflow_schema_specification.md b/docs/nextflow_schema/nextflow_schema_specification.md
@@ -30,14 +30,14 @@ You can find more information about JSON Schema here:
 
 ## Definitions
 
-A slightly strange use of a JSON schema standard that we use for Nextflow schema is `defs`.
+A slightly strange use of a JSON schema standard that we use for Nextflow schema is `$defs`.
 
 JSON schema can group variables together in an `object`, but then the validation expects this structure to exist in the data that it is validating.
 In reality, we have a very long "flat" list of parameters, all at the top level of `params.foo`.
 
-In order to give some structure to log outputs, documentation and so on, we group parameters into `defs`.
+In order to give some structure to log outputs, documentation and so on, we group parameters into `$defs`.
 Each `def` is an object with a title, description and so on.
-However, as they are under `defs` scope they are effectively ignored by the validation and so their nested nature is not a problem.
+However, as they are under `$defs` scope they are effectively ignored by the validation and so their nested nature is not a problem.
 We then bring the contents of each definition object back to the "flat" top level for validation using a series of `allOf` statements at the end of the schema,
 which reference the specific definition keys.
 
@@ -47,7 +47,7 @@ which reference the specific definition keys.
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "type": "object",
   // Definition groups
-  "defs": { // (1)!
+  "$defs": { // (1)!
     "my_group_of_params": { // (2)!
       "title": "A virtual grouping used for docs and pretty-printing",
       "type": "object",
@@ -64,7 +64,7 @@ which reference the specific definition keys.
   },
   // Contents of each definition group brought into main schema for validation
   "allOf": [
-    { "$ref": "#/defs/my_group_of_params" } // (6)!
+    { "$ref": "#/$defs/my_group_of_params" } // (6)!
   ]
 }
 ```
@@ -77,7 +77,7 @@ which reference the specific definition keys.
 5. Shortened here for the example, see below for full parameter specification.
 6. A `$ref` line like this needs to be added for every definition group
 
-Parameters can be described outside of the `defs` scope, in the regular JSON Schema top-level `properties` scope.
+Parameters can be described outside of the `$defs` scope, in the regular JSON Schema top-level `properties` scope.
 However, they will be displayed as ungrouped in tools working off the schema.
 
 ## Nested parameters

diff --git a/examples/paramsHelp/pipeline/nextflow_schema.json b/examples/paramsHelp/pipeline/nextflow_schema.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -33,7 +33,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         }
     ]
 }
diff --git a/examples/paramsSummaryLog/pipeline/nextflow_schema.json b/examples/paramsSummaryLog/pipeline/nextflow_schema.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -33,7 +33,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         }
     ]
 }
diff --git a/examples/paramsSummaryMap/pipeline/nextflow_schema.json b/examples/paramsSummaryMap/pipeline/nextflow_schema.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -33,7 +33,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         }
     ]
 }
diff --git a/examples/validateParameters/pipeline/nextflow_schema.json b/examples/validateParameters/pipeline/nextflow_schema.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -34,7 +34,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         }
     ]
 }
diff --git a/examples/validationFailUnrecognisedParams/pipeline/nextflow_schema.json b/examples/validationFailUnrecognisedParams/pipeline/nextflow_schema.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -33,7 +33,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         }
     ]
 }
diff --git a/examples/validationWarnUnrecognisedParams/pipeline/nextflow_schema.json b/examples/validationWarnUnrecognisedParams/pipeline/nextflow_schema.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -33,7 +33,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         }
     ]
 }
diff --git a/parameters_meta_schema.json b/parameters_meta_schema.json
@@ -30,7 +30,7 @@
       "type": "string",
       "const": "object"
     },
-    "defs": {
+    "$defs": {
       "title": "Parameter groups",
       "type": "object",
       "patternProperties": {
@@ -139,7 +139,7 @@
         "properties": {
           "$ref": {
             "type": "string",
-            "pattern": "^#/defs/"
+            "pattern": "^#/$defs/"
           }
         }
       }

diff --git a/plugins/nf-schema/src/main/nextflow/validation/SchemaValidator.groovy b/plugins/nf-schema/src/main/nextflow/validation/SchemaValidator.groovy
@@ -243,14 +243,15 @@ class SchemaValidator extends PluginExtensionPoint {
         // Check for nextflow core params and unexpected params
         def slurper = new JsonSlurper()
         def Map parsed = (Map) slurper.parse( Path.of(Utils.getSchemaPath(baseDir, schemaFilename)) )
-        def Map schemaParams = (Map) parsed.get('defs')
+        // $defs is the adviced keyword for definitions. Keeping defs in for backwards compatibility
+        def Map schemaParams = (Map) (parsed.get('$defs') ?: parsed.get("defs"))
         def specifiedParamKeys = params.keySet()
 
         // Collect expected parameters from the schema
         def enumsTuple = collectEnums(schemaParams)
         def List expectedParams = (List) enumsTuple[0] + addExpectedParams()
         def Map enums = (Map) enumsTuple[1]
-        // Collect expected parameters from the schema when parameters are specified outside of "defs"
+        // Collect expected parameters from the schema when parameters are specified outside of "$defs"
         if (parsed.containsKey('properties')) {
             def enumsTupleTopLevel = collectEnums(['top_level': ['properties': parsed.get('properties')]])
             expectedParams += (List) enumsTupleTopLevel[0]
@@ -602,10 +603,11 @@ class SchemaValidator extends PluginExtensionPoint {
     private static LinkedHashMap paramsRead(Path json_schema) throws Exception {
         def slurper = new JsonSlurper()
         def Map schema = (Map) slurper.parse( json_schema )
-        def Map schema_defs = (Map) schema.get('defs')
+        // $defs is the adviced keyword for definitions. Keeping defs in for backwards compatibility
+        def Map schema_defs = (Map) (schema.get('$defs') ?: schema.get("defs"))
         def Map schema_properties = (Map) schema.get('properties')
         /* Tree looks like this in nf-core schema
-        * defs <- this is what the first get('defs') gets us
+        * $defs <- this is what the first get('$defs') gets us
                 group 1
                     title
                     description
@@ -623,7 +625,7 @@ class SchemaValidator extends PluginExtensionPoint {
                         parameter 1
                             type
                             description
-        * properties <- parameters can also be ungrouped, outside of defs
+        * properties <- parameters can also be ungrouped, outside of $defs
                 parameter 1
                     type
                     description

diff --git a/plugins/nf-schema/src/resources/META-INF/MANIFEST.MF b/plugins/nf-schema/src/resources/META-INF/MANIFEST.MF
@@ -1,6 +1,6 @@
 Manifest-Version: 1.0
 Plugin-Id: nf-schema
-Plugin-Version: 2.0.1
+Plugin-Version: 2.1.0
 Plugin-Class: nextflow.validation.ValidationPlugin
 Plugin-Provider: nextflow
 Plugin-Requires: >=22.10.0
diff --git a/plugins/nf-schema/src/testResources/nextflow_schema.json b/plugins/nf-schema/src/testResources/nextflow_schema.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -250,19 +250,19 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         },
         {
-            "$ref": "#/defs/reference_genome_options"
+            "$ref": "#/$defs/reference_genome_options"
         },
         {
-            "$ref": "#/defs/institutional_config_options"
+            "$ref": "#/$defs/institutional_config_options"
         },
         {
-            "$ref": "#/defs/max_job_request_options"
+            "$ref": "#/$defs/max_job_request_options"
         },
         {
-            "$ref": "#/defs/generic_options"
+            "$ref": "#/$defs/generic_options"
         }
     ]
 }
diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_draft7.json b/plugins/nf-schema/src/testResources/nextflow_schema_draft7.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "file_patterns": {
             "title": "Input/output options",
             "type": "object",
@@ -19,7 +19,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/file_patterns"
+            "$ref": "#/$defs/file_patterns"
         }
     ]
 }
diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_file_path_pattern.json b/plugins/nf-schema/src/testResources/nextflow_schema_file_path_pattern.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "file_patterns": {
             "title": "Input/output options",
             "type": "object",
@@ -19,7 +19,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/file_patterns"
+            "$ref": "#/$defs/file_patterns"
         }
     ]
 }
diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_required_numerics.json b/plugins/nf-schema/src/testResources/nextflow_schema_required_numerics.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -53,10 +53,10 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         },
         {
-            "$ref": "#/defs/numeric_options"
+            "$ref": "#/$defs/numeric_options"
         }
     ]
 }
diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_deeply_nested_samplesheet.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_deeply_nested_samplesheet.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",

diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_exists_false.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_exists_false.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -21,7 +21,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         }
     ]
 }
diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet.json
@@ -4,7 +4,7 @@
     "title": "nf-core/testpipeline pipeline parameters",
     "description": "this is a test",
     "type": "object",
-    "defs": {
+    "$defs": {
         "input_output_options": {
             "title": "Input/output options",
             "type": "object",
@@ -28,7 +28,7 @@
     },
     "allOf": [
         {
-            "$ref": "#/defs/input_output_options"
+            "$ref": "#/$defs/input_output_options"
         }
     ]
 }