Skip to content

Migration guide

Warning

nf-schema currently is not supported by the nf-core tooling. Using this plugin will break the linting and schema builder. See these issues for the progress on the nf-core migration to nf-schema:

  1. https://github.com/nf-core/tools/issues/2932
  2. https://github.com/nf-core/tools/issues/2784
  3. https://github.com/nf-core/tools/issues/2429

This guide is intended to help you migrate your pipeline from nf-validation to nf-schema.

Major changes in the plugin

Following list shows the major breaking changes introduced in nf-schema:

  1. The JSON schema draft has been updated from draft-07 to draft-2020-12. See JSON Schema draft 2020-12 release notes and JSON schema draft 2019-09 release notes for more information.
  2. The fromSamplesheet channel factory has been converted to a function called samplesheetToList. See updating fromSamplesheet for more information.
  3. The unique keyword for samplesheet schemas has been removed. Please use uniqueItems or uniqueEntries now instead.
  4. The dependentRequired keyword now works as it's supposed to work in JSON schema. See dependentRequired for more information.
  5. All configuration parameters have been converted to Nextflow configuration options. See Updating configuration for more information.
  6. Help messages are now created automatically instead of using the paramsHelp() function. (v2.1.0 feature)

A full list of changes can be found in the changelog.

Updating your pipeline

Updating your pipeline can be done in a couple simple steps.

Updating the name and version of the plugin

The name and the version of the plugin should be updated from nf-validation to nf-schema@2.0.0:

plugins {
    id 'nf-validation@1.1.3'
}

plugins {
    id 'nf-schema@2.0.0'
}

Additionally, all includes from nf-validation should be updated to nf-schema. This can easily be done with the following command:

find . -type f -name "*.nf" -exec sed -i -e "s/from 'plugin\/nf-validation'/from 'plugin\/nf-schema'/g" -e 's/from "plugin\/nf-validation"/from "plugin\/nf-schema"/g' {} +

Updating the JSON schema files

If you aren't using any special features in your schemas, you can simply update your nextflow_schema.json file using the following command:

sed -i -e 's/https?:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/$defs/g' nextflow_schema.json

This will replace the old schema draft specification (draft-07) by the new one (2020-12), and the old keyword definitions by the new notation $defs.

Note

Repeat this command for every JSON schema used in your pipeline. e.g. for the default samplesheet schema in nf-core pipelines: bash sed -i -e 's/https?:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/$defs/g' assets/schema_input.json

Warning

This will not update changes to special fields in the schema, see the guide for special JSON schema keywords on how to update these

Update the samplesheet conversion

The .fromSamplesheet channel factory should be converted to the samplesheetToList function. Following tabs shows how to use the function to get the same effect as the channel factory:

include { fromSamplesheet } from 'plugin/nf-validation'
Channel.fromSamplesheet("input")

include { samplesheetToList } from 'plugin/nf-schema'
Channel.fromList(samplesheetToList(params.input, "path/to/samplesheet/schema"))

Note

This change was necessary to make it possible for pipelines to be used as pluggable workflows. This also enables the validation and conversion of files generated by the pipeline.

Updating configuration

The configuration parameters have been converted to a Nextflow configuration option. You can now access these options using the validation config scope:

validation.<option> = <value>

OR

validation {
    <option1> = <value1>
    <option2> = <value2>
}

See this table for an overview of what the new configuration options are for the old parameters:

Old parameter New config option(s)
params.validationMonochromeLogs = <boolean> validation.monochromeLogs = <boolean>
params.validationLenientMode = <boolean> validation.lenientMode = <boolean>
params.validationFailUnrecognisedParams = <boolean> validation.failUnrecognisedParams = <boolean>
params.validationShowHiddenParams = <boolean> validation.showHiddenParams = <boolean>
params.validationIgnoreParams = <string> validation.defaultIgnoreParams = <list> and validation.ignoreParams = <list>

Note

defaultIgnoreParams is meant to be used by pipeline developers to set the parameters which should always be ignored. ignoreParams is meant for the pipeline user to ignore certain parameters.

Updating special keywords in JSON schemas

If you are using any special features in your schemas, you will need to update your schemas manually. Please refer to the JSON Schema draft 2020-12 release notes and JSON schema draft 2019-09 release notes for more information.

However here are some guides to the more common migration patterns:

Updating unique keyword

When you use unique in your schemas, you should update it to use uniqueItems or uniqueEntries instead.

If you used the unique:true field, you should update it to use uniqueItems like this:

{
    "$schema": "http://json-schema.org/draft-07/schema",
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "sample": {
                "type": "string",
                "unique": true
            }
        }
    }
}

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "sample": {
                "type": "string"
            }
        }
    },
    "uniqueItems": true
}

If you used the unique: ["field1", "field2"] field, you should update it to use uniqueEntries like this:

{
    "$schema": "http://json-schema.org/draft-07/schema",
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "sample": {
                "type": "string",
                "unique": ["sample"]
            }
        }
    }
}

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "sample": {
                "type": "string"
            }
        }
    },
    "uniqueEntries": ["sample"]
}

Updating dependentRequired keyword

When you use dependentRequired in your schemas, you should update it like this:

{
    "$schema": "http://json-schema.org/draft-07/schema",
    "type": "object",
    "properties": {
        "fastq_1": {
            "type": "string",
            "format": "file-path"
        },
        "fastq_2": {
            "type": "string",
            "format": "file-path",
            "dependentRequired": ["fastq_1"]
        }
    }
}

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "fastq_1": {
            "type": "string",
            "format": "file-path"
        },
        "fastq_2": {
            "type": "string",
            "format": "file-path"
        }
    },
    "dependentRequired": {
        "fastq_2": ["fastq_1"]
    }
}

Updating the help message

v2.1.0 feature

The creation of the help message now needs to be enabled in the configuration file. Using --help or --helpFull will automatically print the help message and stop the pipeline execution. paramsHelp() is still available in nf-schema and can still be used like before. This could be helpful to print the help message in specific cases. Mind that this function now automatically emits a deprecation warning. This warning can be disabled using the hideWarning:true option of the function.

main.nf
if (params.help) {
    log.info paramsHelp("nextflow run my_pipeline --input input_file.csv")
    exit 0
}
nextflow.config
validation {
    help {
        enabled = true
        command = "nextflow run my_pipeline --input input_file.csv"
    }
}