From c487aaa86196402bd83abf237b33b38736391ba8 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Thu, 24 Apr 2025 19:57:55 +1200 Subject: [PATCH 01/29] Update plugins docs Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 134 ++++++++ docs/index.md | 13 + docs/migrating-gradle-plugin.md | 134 ++++++++ docs/plugins/developing-plugins.md | 489 +++++++++++++++++++++++++++++ docs/plugins/example-nf-hello.md | 52 +++ docs/plugins/plugins.md | 67 ++++ docs/plugins/using-plugins.md | 78 +++++ 7 files changed, 967 insertions(+) create mode 100644 docs/gradle-plugin.md create mode 100644 docs/migrating-gradle-plugin.md create mode 100644 docs/plugins/developing-plugins.md create mode 100644 docs/plugins/example-nf-hello.md create mode 100644 docs/plugins/plugins.md create mode 100644 docs/plugins/using-plugins.md diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md new file mode 100644 index 0000000000..31f341f0ee --- /dev/null +++ b/docs/gradle-plugin.md @@ -0,0 +1,134 @@ +# Using the Gradle plugin + +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. + +:::{note} +Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +## Creating a plugin + +The nf-hello plugin uses the Gradle plugin and is a valuable starting point for developers. + +To create a Nextflow plugin with the Gradle plugin: + +1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. + * See Example: nf-hello for more information about the nf-hello plugin. +2. Rename the forked `nf-hello` directory with your plugin name. +3. Replace the contents of `settings.gradle` with the following: + + ```groovy + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. Replace the contents of `build.gradle` with the following: + + ```groovy + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + github { + repository = '' + userName = project.findProperty('github_username') + authToken = project.findProperty('github_access_token') + email = project.findProperty('github_commit_email') + indexUrl = '' + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (optional) your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` your plugin version—for example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: the minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. + - `PROVIDER`: your name or organization—for example, `nextflow`. + - `CLASS_NAME`: your plugin class name—for example, `nextflow.hello.HelloPlugin`. + - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. + - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. + - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). +5. Develop your plugin extension points: + - See Extension points for descriptions and examples. +6. In the plugin root directory, run `make assemble`. + + +## Installing a plugin + +Plugins can be installed locally without being packaged, uploaded, and published. + +To install a plugin locally: + +1. In the plugin root directory, run `make install`. + + :::{note} + Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. + ::: + +2. Configure your plugin. + * See Using plugins for more information. +3. Run your pipeline: + + ```bash + nextflow run main.nf + ``` + +## Unit testing a plugin + +Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. + +To run unit tests: + +1. Develop your unit tests. + * See HelloDslTest.groovy in the nf-hello plugin for unit test examples. +2. In the plugin root directory, run `make test`. + + +### Packaging, uploading, and publishing a plugin + +The Gradle plugin for Nextflow plugins simplifies publishing your plugin. + +To package, upload, and publish your plugin: + +1. Fork the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins). +2. In the plugin root directory, open `build.gradle` and ensure that: + * `github.repository` matches the plugin repository. + * `github.indexUrl` matches your fork of the plugins index repository. +3. Create a file named `$HOME/.gradle/gradle.properties` and add the following: + + ```bash + github_username= + github_access_token= + github_commit_email= + ``` + + Replace the following: + * `GITHUB_USERNAME`: your GitHub username granting access to the plugin repository. + * `GITHUB_ACCESS_TOKEN`: your GitHub access token with permission to upload and commit changes to the plugin repository. + * `GITHUB_EMAIL`: your email address associated with your GitHub account. +4. Run `make release`. +5. Create a pull request against the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins) from your fork. \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index fe8098efc7..7943bd88b7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -149,11 +149,24 @@ developer/packages developer/plugins ``` +```{toctree} +:hidden: +:caption: Plugins +:maxdepth: 1 + +plugins/plugins +plugins/using-plugins +plugins/developing-plugins +plugins/example-nf-hello +``` + ```{toctree} :hidden: :caption: Guides :maxdepth: 1 +gradle-plugin +migrating-gradle-plugin updating-spot-retries metrics flux diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md new file mode 100644 index 0000000000..b424ff4233 --- /dev/null +++ b/docs/migrating-gradle-plugin.md @@ -0,0 +1,134 @@ +# Migrating to the Gradle plugin for Nextflow plugins + +This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. + + +## Improvements to the plugin framework + +The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying plugin development, streamlining publishing and discovery, and improving how plugins are loaded into workflows. These improvements make plugins more accessible, maintainable, and interoperable with Nextflow. + +### Gradle plugin for Nextflow plugins + +The Gradle plugin for Nextflow plugins simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. + +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. + +### Nextflow plugin registry + +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. + +Developers can upload plugins to the plugin registry. The Gradle plugin for Nextflow plugins supports this process with Gradle tasks that package, upload, and publish plugins. + +## Impact on users and developers + +The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. + +### Plugin Users + +If you are a plugin user, no immediate actions are required. The plugin configuration has not changed. + +### Plugin developers + +Developers are encouraged to migrate to the Gradle plugin for Nextflow plugins and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. + +To migrate an existing Nextflow plugin: + +1. Remove the following files and folders: + - `buildSrc/` + - `nextflow.config` + - `launch.sh` + - `plugins/build.gradle` +2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. \ + + :::{note} + Plugin sources should be in `src/main/groovy` or `src/main/java`. + ::: + +3. Replace the contents of `settings.gradle` with the following: + + ``` + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. In the project root, create a new `build.gradle` file with the following configuration: + + ``` + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + github { + repository = '' + userName = project.findProperty('github_username') + authToken = project.findProperty('github_access_token') + email = project.findProperty('github_commit_email') + indexUrl = '' + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. + - `PROVIDER`: Your name or organization—for example, `nextflow`. + - `CLASS_NAME`: Your plugin class name—for example, `nextflow.hello.HelloPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. + - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. + - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). + +5. Replace the contents of `Makefile` with the following: + + ``` + # Build the plugin + assemble: + ./gradlew assemble + + clean: + rm -rf .nextflow* + rm -rf work + rm -rf build + ./gradlew clean + + # Run plugin unit tests + test: + ./gradlew test + + # Install the plugin into local nextflow plugins dir + install: + ./gradlew install + + # Publish the plugin + release: + ./gradlew releasePlugin + ``` + +6. Update `README.md` with information about the structure of your plugin. +7. In the plugin root directory, run `make assemble`. + +The Gradle plugin for Nextflow plugins also supports publishing plugins. See Packaging, uploading, and publishing for more information. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md new file mode 100644 index 0000000000..07b3c3fe2a --- /dev/null +++ b/docs/plugins/developing-plugins.md @@ -0,0 +1,489 @@ +(developing-plugins-page)= + +# Developing plugins + +The Nextflow plugin framework streamlines plugin development by providing the structure and tools needed to extend Nextflow functionality. The Gradle plugin for Nextflow plugins simplifies development by configuring default Nextflow dependencies and incorporates custom Gradle tasks that streamline building and publishing plugins. + +:::{note} +Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +(developing-plugins-framework)= + +## Framework + +Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. + +(developing-plugins-gradle)= + +## Gradle plugin for Nextflow plugins + +[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. + +(developing-plugins-structure)= + +### Structure + +Plugins built with the Gradle plugin for Nextflow plugins follow a standard project layout. This structure includes source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. + +A typical plugin built with the Gradle plugin for Nextflow plugins has the following structure: + +``` +. +├── COPYING +├── Makefile +├── README.md +├── build.gradle +├── gradle +│ └── wrapper +│ └── ... +├── gradlew +├── settings.gradle +└── src + ├── main + │ └── ... + └── test + └── ... +``` + +This structure contains the following key files and folders: + +- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. +- `src/main/`: The main source directory containing the plugin's implementation and resources.​ +- `src/test/`: The main source directory containing the plugin's unit testing. +- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ +- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ +- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ +- `build.gradle`: The primary build script for Gradle. +- `gradlew`: A script for executing the Gradle Wrapper. +- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. + +See {ref}`nf-hello-page` for an example of a plugin built using this structure. + +:::{note} +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. See nf-hello for an example of a plugin with the old structure. +::: + +(developing-plugins-make)= + +### make commands + +The Gradle plugin for Nextflow plugins defines tasks that can be executed with `./gradlew`. For example: + +```bash +./gradlew assemble +``` + +For convenience, the more important tasks are wrapped in a makefile and can be executed with the `make` command. For example: + +```bash +make assemble +``` + +The following `make` commands are available: + +`assemble` +: Compiles the Nextflow plugin code and assembles it into a zip file. See Creating a plugin for more information. + +`test` +: Runs plugin unit tests. See Unit testing for more information. + +`install` +: Installs the plugin into the local nextflow plugins directory. See Running locally for more information. + +`release` +: Publishes the plugin. See Packaging, uploading, and publishing for more information. + +(developing-plugins-versioning)= + +### Versioning + +The Gradle plugin is versioned and published to a [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: + +```nextflow +plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' +} +``` + +See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. + +(developing-plugins-extension)= + +## Extension points + +Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how they can be used. + +### Commands + +Plugins can define custom CLI commands that are executable with the `nextflow plugin` command. + +To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entry point (the class that extends `BasePlugin`). Alternatively, implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement the `getCommands()` and `exec()` methods. For example: + +```groovy +import nextflow.cli.PluginAbstractExec +import nextflow.plugin.BasePlugin + +class MyPlugin extends BasePlugin implements PluginAbstractExec { + @Override + List getCommands() { + [ 'hello' ] + } + + @Override + int exec(String cmd, List args) { + if( cmd == 'hello' ) { + println "Hello! You gave me these arguments: ${args.join(' ')}" + return 0 + } + else { + System.err.println "Invalid command: ${cmd}" + return 1 + } + } +} +``` + +The command can be run using the `nextflow plugin` command: + +``` +nextflow plugin my-plugin:hello --foo --bar +``` + +See the {ref}`cli-plugin` for usage information. + +### Configuration + +Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option safely. If the option isn’t defined, it will return null. + +A common practice is to use a custom config scope to define any configuration for your plugin. For example: + +```groovy +import nextflow.Session +import nextflow.trace.TraceObserver + +class MyObserver implements TraceObserver { + + @Override + void onFlowCreate(Session session) { + final message = session.config.navigate('myplugin.createMessage') + println message + } +} +``` + +This option can then be set in your configuration file: + +```groovy +// dot syntax +myplugin.createMessage = "I'm alive!" + +// block syntax +myplugin { + createMessage = "I'm alive!" +} +``` + +:::{versionadded} 25.02.0-edge +::: + +Plugins can declare their configuration options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation. For example: + +```groovy +import nextflow.config.schema.ConfigOption +import nextflow.config.schema.ConfigScope +import nextflow.config.schema.ScopeName +import nextflow.script.dsl.Description + +@ScopeName('myplugin') +@Description(''' + The `myplugin` scope allows you to configure the `nf-myplugin` plugin. +''') +class MyPluginConfig implements ConfigScope { + + MyPluginConfig(Map opts) { + this.createMessage = opts.createMessage + } + + @ConfigOption + @Description('Message to print to standard output when a run is initialized.') + String createMessage +} +``` + +This approach is not required to support plugin config options. However, it allows Nextflow to recognize plugin definitions when validating the configuration. + +### Executors + +Plugins can define custom executors that can be used with the `executor` process directive. + +To implement an executor, create a class in your plugin that extends the [Executor](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: + +```groovy +import nextflow.executor.Executor +import nextflow.util.ServiceName +import org.pf4j.ExtensionPoint + +@ServiceName('my-executor') +class MyExecutor extends Executor implements ExtensionPoint { + + // ... + +} +``` + +You can then use this executor in your pipeline: + +```groovy +process foo { + executor 'my-executor' + + // ... +} +``` + +:::{tip} +See the source code of Nextflow's built-in executors for examples of how to implement various executor components. +::: + + +### Filesystems + +Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see Remote files. + +To implement a custom filesystem, create a class in your plugin that extends [FileSystemProvider](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: + +```groovy +import java.nio.file.spi.FileSystemProvider + +class MyFileSystemProvider extends FileSystemProvider { + + @Override + String getScheme() { + return 'myfs' + } + + // ... +} +``` + +You can then use this filesystem in your pipeline: + +```nextflow +input = file('myfs://') +``` + +See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for examples of custom filesystems. + +:::{tip} +Custom filesystems are an advanced plugin extension. Before creating a new filesystem, check that your use case cannot already be supported by an existing filesystem such as HTTP or S3. +::: + +### Functions + +:::{versionadded} 22.09.0-edge +::: + +Plugins can define custom functions that can be included in Nextflow pipelines. + +To implement a custom function, create a plugin class that extends the `PluginExtensionPoint` class and implement your function with the `Function` annotation. For example: + +```groovy +import nextflow.Session +import nextflow.plugin.extension.Function +import nextflow.plugin.extension.PluginExtensionPoint + +class MyExtension extends PluginExtensionPoint { + + @Override + void init(Session session) {} + + @Function + String reverseString(String origin) { + origin.reverse() + } +} +``` + +You can then add this function to your pipeline: + +``` +include { reverseString } from 'plugin/my-plugin' + +channel.of( reverseString('hi') ) +``` + +Alternatively, you can use an alias: + +```nextflow +include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' +``` + +### Operators + +:::{versionadded} 22.04.0 +::: + +Plugins can define custom channel factories and operators that can then be included in pipelines. + +To implement a custom channel factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class and implement your function with the `Factory` or `Operator` annotation. For example: + +```groovy +import groovyx.gpars.dataflow.DataflowReadChannel +import groovyx.gpars.dataflow.DataflowWriteChannel +import nextflow.Session +import nextflow.plugin.extension.Factory +import nextflow.plugin.extension.Operator +import nextflow.plugin.extension.PluginExtensionPoint + +class MyExtension extends PluginExtensionPoint { + + @Override + void init(Session session) {} + + @Factory + DataflowWriteChannel fromQuery(Map opts, String query) { + // ... + } + + @Operator + DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) { + // ... + } + +} +``` + +You can then use the custom channel factories or operators in your pipeline: + +```nextflow +include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' + +def sql = 'select * from FOO' +channel + .fromTable(sql, db: 'test', emitColumns: true) + .sqlInsert(into: 'BAR', columns: 'id', db: 'test') +``` + +:::{note} +The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +::: + +### Process directives + +Plugins that implement a custom executor will likely need to access process directives that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. + +Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the ext directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. + +For example, a custom executor can use existing process directives and a custom setting through the `ext` directive: + +```groovy +class MyExecutor extends Executor { + + @Override + TaskHandler createTaskHandler(TaskRun task) { + final cpus = task.config.cpus + final memory = task.config.memory + final myOption = task.config.ext.myOption + + println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" + + // ... + } + + // ... + +} +``` + +### Trace observers + +A *trace observer* is an entity that can listen and react to workflow events, such as when a workflow starts, a task is completed, or a file is published. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. + +Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver` and implement the `create()` method in your observer factory. For example: + +```groovy +import java.nio.file.Path + +import nextflow.processor.TaskHandler +import nextflow.trace.TraceObserver +import nextflow.trace.TraceRecord + +class MyObserver implements TraceObserver { + + @Override + void onFlowBegin() { + println "Okay, let's begin!" + } + + @Override + void onProcessComplete(TaskHandler handler, TraceRecord trace) { + println "I completed a task! It's name is '${handler.task.name}'" + } + + @Override + void onProcessCached(TaskHandler handler, TraceRecord trace) { + println "I found a task in the cache! It's name is '${handler.task.name}'" + } + + @Override + void onFilePublish(Path destination, Path source) { + println "I published a file! It's located at ${path.toUriString()}" + } + + @Override + void onFlowError(TaskHandler handler, TraceRecord trace) { + println "Uh oh, something went wrong..." + } + + @Override + void onFlowComplete() { + println 'All done!' + } +} +``` + +You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option: + +```nextflow +myplugin.enabled = true +``` + +See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. + + +## Environment variables + +The following environment variables are available when developing and testing plugins: + +`NXF_PLUGINS_MODE` +: The plugin execution mode. Either `prod` for production or `dev` for development. + +`NXF_PLUGINS_DIR` +: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production and `./plugins` in development). + +`NXF_PLUGINS_DEFAULT` +: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: true). + + +`NXF_PLUGINS_DEV` +: Comma-separated list of development plugin root directories. + +`NXF_PLUGINS_TEST_REPOSITORY` +: :::{versionadded} 23.04.0. + ::: +: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry. + +: The URI should refer to a plugin repository JSON file or a specific plugin JSON meta file. In the latter case, it should match the pattern `https://host.name/some/path/-X.Y.Z-meta.json`. For example: + + ```bash + # custom plugin repository at https://github.com/my-org/plugins + export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json" + + # custom plugin release + export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json" + + nextflow run main.nf -plugins nf-hello + ``` + +: This variable is useful for testing a plugin release before publishing it to the main registry. diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md new file mode 100644 index 0000000000..0b96fb4d07 --- /dev/null +++ b/docs/plugins/example-nf-hello.md @@ -0,0 +1,52 @@ +(nf-hello-page)= + +# Example: nf-hello + +`nf-hello` is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. + +The nf-hello plugin has the following structure: + +``` +nf-hello +├── COPYING +├── Makefile +├── README.md +├── build.gradle +├── gradle +│ └── wrapper +│ ├── gradle-wrapper.jar +│ └── gradle-wrapper.properties +├── gradlew +├── settings.gradle +└── src + ├── main + │ └── groovy + │ └── nextflow + │ └── hello + │ ├── HelloConfig.groovy + │ ├── HelloExtension.groovy + │ ├── HelloFactory.groovy + │ ├── HelloObserver.groovy + │ └── HelloPlugin.groovy + └── test + └── groovy + └── nextflow + └── hello + ├── HelloDslTest.groovy + └── HelloFactoryTest.groovy +``` + +It includes examples of different plugin extensions: + +- A custom trace observer that prints a message when the workflow starts and when the workflow completes. +- A custom channel factory called reverse. +- A custom operator called goodbye. +- A custom function called randomString. + +It also includes several classes that demonstrate different plugin functionality: + +- `HelloConfig`: An example of how to handle options from the Nextflow configuration. +- `HelloExtension`: An example of how to create custom channel factories, operators, and functions that can be included in pipeline scripts. +- `HelloFactory`: An example of a workflow event with custom behavior. +- `HelloObserver`: An example of a workflow event with custom behavior. +- `HelloPlugin`: An example of a plugin entry point. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md new file mode 100644 index 0000000000..259d1e9e0c --- /dev/null +++ b/docs/plugins/plugins.md @@ -0,0 +1,67 @@ +(plugins-page)= + +# Overview + +## What are plugins + +Nextflow plugins are extensions that enhance the functionality of the Nextflow workflow framework. They allow users to add new capabilities and integrate with external services without modifying core Nextflow code. + +There are two types of Nextflow plugins: + +* Core plugins +* Third-party plugins + +The main features of each plugin type are described below. + +

Core plugins

+ +Core plugins do not require configuration. The latest versions of core plugins are automatically installed when a Nextflow pipeline requests them. Core plugins include: + +* `nf-amazon`: Support for Amazon Web Services. +* `nf-azure`: Support for Microsoft Azure. +* `nf-cloudcache`: Support for the cloud cache. +* `nf-console`: Implement Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). +* `nf-google`: Support for Google Cloud. +* `nf-tower`: Support for [Seqera Platform](https://seqera.io/platform/). +* `nf-wave`: Support for [Wave containers service](https://seqera.io/wave/). + +Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. + +:::{note} +The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See Environment variables for more information. +::: + +

Third-party plugins

+ +Third-party plugins must be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: + +``` +plugins { + id 'nf-hello@0.5.0' +} +``` + +To configure plugins at runtime, use the `-plugins` option. For example: + +``` +nextflow run main.nf -plugins nf-hello@0.5.0 +``` + +See {ref}`using-plugins-page` for more information. + +## Nextflow plugin registry + +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. Developers can upload plugins to the registry, with built-in support from the Gradle plugin for Nextflow plugins. + +## Versioning + +Nextflow plugins follow the Semantic Versioning format: MAJOR.MINOR.PATCH (e.g., 0.5.0). This helps developers communicate the nature of changes in a release. + +Version components: + +* MAJOR: Increment for incompatible changes. +* MINOR: Increment for backward-compatible feature additions. +* PATCH: Increment for backward-compatible bug fixes. + +Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. +See Semantic Versioning for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md new file mode 100644 index 0000000000..8aef78fc7e --- /dev/null +++ b/docs/plugins/using-plugins.md @@ -0,0 +1,78 @@ +# Using plugins + +Nextflow core plugins require no additional configuration. When a pipeline uses a core plugin, Nextflow automatically downloads and uses the latest compatible plugin version. In contrast, third-party plugins must be explicitly declared. When a pipeline uses one or more third-party plugins, Nextflow must be configured to download and use the plugin. + +## Identifiers + +A plugin identifier consists of the plugin name and version, separated by an `@` symbol: + +```console +nf-hello@0.5.0 +``` + +The plugin version is optional. If it is not specified, Nextflow will download the latest version of the plugin that meets the minimum Nextflow version requirements specified by the plugin. + +:::{note} +Plugin versions are required to run plugins in an Offline environment. +::: + +:::{versionadded} 25.02.0-edge. +::: + +The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. + +:::{tip} +It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. +::: + +## Configuration + +Plugins can be configured via nextflow configuration files or at runtime. + +To configure a plugin via configuration files, use the `plugins` block. For example: + +```nextflow +plugins { + id 'nf-hello@0.5.0' +} +``` + +To configure plugins at runtime, use the `-plugins` option. For example: + +```bash +nextflow run main.nf -plugins nf-hello@0.5.0 +``` + +To configure multiple plugins at runtime, use the `-plugins` option and a comma-separated list. For example: + +```bash +nextflow run main.nf -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +``` + +:::{note} +Plugin declarations in nextflow configuration files are ignored when specifying plugins via the `-plugins` option. +::: + +## Caching + +When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. + +## Offline usage + +Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. + +To use Nextflow plugins in an offline environment: + +1. Install a self-contained version of Nextflow in an environment with an internet connection. See Standalone distribution for more information. +2. Run `nextflow plugin install @` to download the plugins. + + :::{tip} + Running a pipeline in an environment with an internet connection will also download the plugins used by that pipeline. + ::: + +3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. +4. Specify each plugin and its version in Nextflow configuration files or at runtime. See Configuration for more information. + + :::{warning} + Nextflow will attempt to download newer versions of plugins if their versions are not set. See Plugin identifiers for more information. + ::: \ No newline at end of file From a0463318c5f7ef59e104a3b9b9357840c024ccc4 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 28 Apr 2025 09:36:15 +1200 Subject: [PATCH 02/29] Add run command example Signed-off-by: Christopher Hakkaart --- docs/plugins/example-nf-hello.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 0b96fb4d07..689b5f0a69 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -50,3 +50,9 @@ It also includes several classes that demonstrate different plugin functionality - `HelloFactory`: An example of a workflow event with custom behavior. - `HelloObserver`: An example of a workflow event with custom behavior. - `HelloPlugin`: An example of a plugin entry point. + +The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: + +```bash +nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +``` From 989af211b9e8c0752075845ce394f4170ddcf1cd Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 28 Apr 2025 13:40:43 +1200 Subject: [PATCH 03/29] Update links Signed-off-by: Christopher Hakkaart --- docs/developer/plugins.md | 541 ----------------------------- docs/gradle-plugin.md | 24 +- docs/index.md | 2 - docs/migrating-gradle-plugin.md | 6 +- docs/plugins.md | 63 ---- docs/plugins/developing-plugins.md | 37 +- docs/plugins/example-nf-hello.md | 4 +- docs/plugins/plugins.md | 4 +- docs/plugins/using-plugins.md | 16 +- docs/reference/env-vars.md | 2 +- docs/strict-syntax.md | 2 +- 11 files changed, 54 insertions(+), 647 deletions(-) delete mode 100644 docs/developer/plugins.md delete mode 100644 docs/plugins.md diff --git a/docs/developer/plugins.md b/docs/developer/plugins.md deleted file mode 100644 index 7ea5a779e5..0000000000 --- a/docs/developer/plugins.md +++ /dev/null @@ -1,541 +0,0 @@ -(plugins-dev-page)= - -# Plugins - -This page describes how to create, test, and publish third-party plugins. - -## Plugin structure - -The best way to get started with your own plugin is to refer to the [nf-hello](https://github.com/nextflow-io/nf-hello) repository. This repository provides a minimal plugin implementation with several examples of different extension points and instructions for building, testing, and publishing. - -Plugins can be written in Java or Groovy. - -The minimal dependencies are as follows: - -```groovy -dependencies { - compileOnly project(':nextflow') - compileOnly 'org.slf4j:slf4j-api:1.7.10' - compileOnly 'org.pf4j:pf4j:3.4.1' - - testImplementation project(':nextflow') - testImplementation "org.codehaus.groovy:groovy:4.0.24" - testImplementation "org.codehaus.groovy:groovy-nio:4.0.23" -} -``` - -The plugin subproject directory name must begin with the prefix `nf-` and must include a file named `src/resources/META-INF/MANIFEST.MF` which contains the plugin metadata. The manifest content looks like the following: - -``` -Manifest-Version: 1.0 -Plugin-Class: the.plugin.ClassName -Plugin-Id: the-plugin-id -Plugin-Provider: Your Name or Organization -Plugin-Version: 0.0.0 -``` - -## Extension points - -Nextflow's plugin system exposes a variety of extension points for plugins. This section describes how to use these extension points when writing a plugin, as well as how they are used in a pipeline. - -:::{note} -If you would like to implement something in a plugin that isn't covered by any of the following sections, feel free to create an issue on GitHub and describe your use case. In general, any class in the Nextflow codebase that implements `ExtensionPoint` can be extended by a plugin, and existing plugins are a great source of examples when writing new plugins. -::: - -:::{note} -Plugin extension points must be added to `extensions.idx` in the plugin repository to make them discoverable. See the [`nf-hello`](https://github.com/nextflow-io/nf-hello) plugin for an example. -::: - -### Commands - -Plugins can define custom CLI commands that can be executed with the `nextflow plugin` command. - -To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entrypoint (the class that extends `BasePlugin`). Alternatively, you can implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement two methods, `getCommands()` and `exec()`: - -```groovy -import nextflow.cli.PluginAbstractExec -import nextflow.plugin.BasePlugin - -class MyPlugin extends BasePlugin implements PluginAbstractExec { - @Override - List getCommands() { - [ 'hello' ] - } - - @Override - int exec(String cmd, List args) { - if( cmd == 'hello' ) { - println "Hello! You gave me these arguments: ${args.join(' ')}" - return 0 - } - else { - System.err.println "Invalid command: ${cmd}" - return 1 - } - } -} -``` - -You can then execute this command using the `nextflow plugin` command: - -```bash -nextflow plugin my-plugin:hello --foo --bar -``` - -See the {ref}`cli-plugin` CLI command for usage information. - -### Configuration - -Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option in a safe manner -- if the option isn't defined, it will return `null`. A common practice is to define any configuration for your plugin in a custom config scope. - -For example, you can query a config option in a trace observer hook: - -```groovy -import nextflow.Session -import nextflow.trace.TraceObserver - -class MyObserver implements TraceObserver { - - @Override - void onFlowCreate(Session session) { - final message = session.config.navigate('myplugin.createMessage') - println message - } -} -``` - -You can then set this option in your config file: - -```groovy -// dot syntax -myplugin.createMessage = "I'm alive!" - -// block syntax -myplugin { - createMessage = "I'm alive!" -} -``` - -:::{versionadded} 25.02.0-edge -::: - -Plugins can declare their config options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation: - -```groovy -import nextflow.config.schema.ConfigOption -import nextflow.config.schema.ConfigScope -import nextflow.config.schema.ScopeName -import nextflow.script.dsl.Description - -@ScopeName('myplugin') -@Description(''' - The `myplugin` scope allows you to configure the `nf-myplugin` plugin. -''') -class MyPluginConfig implements ConfigScope { - - MyPluginConfig(Map opts) { - this.createMessage = opts.createMessage - } - - @ConfigOption - @Description('Message to print to standard output when a run is initialized.') - String createMessage -} -``` - -While this approach is not required to support plugin config options, it allows Nextflow to recognize plugin definitions when validating the config. - -### Executors - -Plugins can define custom executors that can then be used with the `executor` process directive. - -To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor: - -```groovy -import nextflow.executor.Executor -import nextflow.util.ServiceName -import org.pf4j.ExtensionPoint - -@ServiceName('my-executor') -class MyExecutor extends Executor implements ExtensionPoint { - - // ... - -} -``` - -You can then use this executor in your pipeline: - -```nextflow -process foo { - executor 'my-executor' - - // ... -} -``` - -:::{tip} -Refer to the source code of Nextflow's built-in executors to see how to implement the various components of an executor. You might be able to implement most of your executor by simply reusing existing code. -::: - -### Filesystems - -Plugins can define custom filesystems that can be used by Nextflow to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. - -To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem: - -```groovy -import java.nio.file.spi.FileSystemProvider - -class MyFileSystemProvider extends FileSystemProvider { - - @Override - String getScheme() { - return 'myfs' - } - - // ... -} -``` - -You can then use this filesystem in your pipeline: - -```nextflow -input = file('myfs://path/to/input/file.txt') -``` - -See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information. Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. - -:::{tip} -Custom filesystems are an advanced type of plugin extension. Before creating a new filesystem, make sure that your use case can't already be supported through an existing filesystem such as HTTP or S3. -::: - -### Functions - -:::{versionadded} 22.09.0-edge -::: - -Plugins can define custom functions, which can then be included in Nextflow pipelines. - -To implement a custom function, create a class in your plugin that extends the `PluginExtensionPoint` class, and implement your function with the `Function` annotation: - -```groovy -import nextflow.Session -import nextflow.plugin.extension.Function -import nextflow.plugin.extension.PluginExtensionPoint - -class MyExtension extends PluginExtensionPoint { - - @Override - void init(Session session) {} - - @Function - String reverseString(String origin) { - origin.reverse() - } - -} -``` - -You can then use this function in your pipeline: - -```nextflow -include { reverseString } from 'plugin/my-plugin' - -channel.of( reverseString('hi') ) -``` - -You can also use an alias: - -```nextflow -include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' -``` - -### Operators - -:::{versionadded} 22.04.0 -::: - -Plugins can define custom channel factories and operators, which can then be included into Nextflow pipelines. - -To implement a custom factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class, and implement your function with the `Factory` or `Operator` annotation: - -```groovy -import groovyx.gpars.dataflow.DataflowReadChannel -import groovyx.gpars.dataflow.DataflowWriteChannel -import nextflow.Session -import nextflow.plugin.extension.Factory -import nextflow.plugin.extension.Operator -import nextflow.plugin.extension.PluginExtensionPoint - -class MyExtension extends PluginExtensionPoint { - - @Override - void init(Session session) {} - - @Factory - DataflowWriteChannel fromQuery(Map opts, String query) { - // ... - } - - @Operator - DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) { - // ... - } - -} -``` - -You can then use them in your pipeline: - -```nextflow -include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' - -def sql = 'select * from FOO' -channel - .fromTable(sql, db: 'test', emitColumns: true) - .sqlInsert(into: 'BAR', columns: 'id', db: 'test') -``` - -The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory -is included under the alias `fromTable`. - -### Process directives - -Plugins that implement a [custom executor](#executors) will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task's configuration. As a best practice, custom executors should try to support all process directives that have executor-specific behavior and are relevant to your executor. - -Nextflow does not provide the ability to define custom process directives in a plugin. Instead, you can use the {ref}`process-ext` directive to provide custom process settings to your executor. Try to use specific names that are not likely to conflict with other plugins or existing pipelines. - -For example, a custom executor can use existing process directives as well as a custom setting through the `ext` directive: - -```groovy -class MyExecutor extends Executor { - - @Override - TaskHandler createTaskHandler(TaskRun task) { - final cpus = task.config.cpus - final memory = task.config.memory - final myOption = task.config.ext.myOption - - println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" - - // ... - } - - // ... - -} -``` - -### Trace observers - -A *trace observer* in Nextflow is an entity that can listen and react to workflow events, such as when a workflow starts, a task completes, a file is published, etc. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. - -Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver`, and implement the `create()` method in your observer factory: - -```groovy -// MyObserverFactory.groovy -import nextflow.Session -import nextflow.trace.TraceObserver -import nextflow.trace.TraceObserverFactory - -class MyObserverFactory implements TraceObserverFactory { - - @Override - Collection create(Session session) { - final enabled = session.config.navigate('myplugin.enabled') - return enabled ? [ new MyObserver() ] : [] - } -} - -// MyObserver.groovy -import java.nio.file.Path - -import nextflow.processor.TaskHandler -import nextflow.trace.TraceObserver -import nextflow.trace.TraceRecord - -class MyObserver implements TraceObserver { - - @Override - void onFlowBegin() { - println "Okay, let's begin!" - } - - @Override - void onProcessComplete(TaskHandler handler, TraceRecord trace) { - println "I completed a task! It's name is '${handler.task.name}'" - } - - @Override - void onProcessCached(TaskHandler handler, TraceRecord trace) { - println "I found a task in the cache! It's name is '${handler.task.name}'" - } - - @Override - void onFilePublish(Path destination, Path source) { - println "I published a file! It's located at ${path.toUriString()}" - } - - @Override - void onFlowError(TaskHandler handler, TraceRecord trace) { - println "Uh oh, something went wrong..." - } - - @Override - void onFlowComplete() { - println 'All done!' - } -} -``` - -You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option: - -```groovy -myplugin.enabled = true -``` - -Refer to the `TraceObserver` [source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. - -## Publishing plugins - -Nextflow resolves plugins through a plugin registry, which stores metadata for each plugin version, including the publishing date, checksum, and download URL for the plugin binary. The default registry is located on GitHub at [nextflow-io/plugins](https://github.com/nextflow-io/plugins/), specifically [this file](https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json). - -To publish a plugin, you must create the plugin release, publish it to GitHub, and make a pull request against the main plugin registry with the metadata for your plugin release. - -A plugin release is a ZIP archive containing the compiled plugin classes, the required dependencies, and a JSON file with the plugin metadata. See the [`nf-hello`](https://github.com/nextflow-io/nf-hello) example to see how to create a plugin release. - -Here is an example meta file for a plugin release: - -```json -{ - "version": "0.2.0", - "url": "https://github.com/nextflow-io/nf-amazon/releases/download/0.2.0/nf-amazon-0.2.0.zip", - "date": "2020-10-12T10:05:44.28+02:00", - "sha512sum": "9e9e33695c1a7c051271..." -} -``` - -:::{note} -The plugin version should be a valid [semantic version](https://semver.org/). -::: - -(testing-plugins)= - -## Testing plugins - -Plugins must be declared in the `nextflow.config` file using the `plugins` scope, for example: - -```groovy -plugins { - id 'nf-amazon@0.2.0' -} -``` - -If a plugin is not locally available, Nextflow checks the repository index for the download URL, downloads and extracts the plugin archive, and installs the plugin into the directory specified by `NXF_PLUGINS_DIR` (default: `${NXF_HOME}/plugins`). - -Since each Nextflow run can have a different set of plugins (and versions), each run keeps a local plugins directory called `.nextflow/plr/` which links the exact set of plugins required for the given run. - -Additionally, the "default plugins" (defined in the Nextflow resources file `modules/nextflow/src/main/resources/META-INF/plugins-info.txt`) are always made available for use. To disable the use of default plugins, set the environment variable `NXF_PLUGINS_DEFAULT=false`. - -When running in development mode, the plugin system uses the `DevPluginClasspath` to load plugin classes from each plugin project build path, e.g. `plugins/nf-amazon/build/classes` and `plugins/nf-amazon/build/target/libs` (for deps libraries). - -## Environment variables - -The following environment variables are available when developing and testing plugins: - -`NXF_PLUGINS_MODE` -: The plugin execution mode, either `prod` for production or `dev` for development (see below for details). - -`NXF_PLUGINS_DIR` -: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production, `./plugins` in development). - -`NXF_PLUGINS_DEFAULT` -: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: `true`). - -`NXF_PLUGINS_DEV` -: Comma-separated list of development plugin root directories. - -`NXF_PLUGINS_TEST_REPOSITORY` -: :::{versionadded} 23.04.0 - ::: -: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry. -: The URI should refer to a plugins repository JSON file or a specific plugin JSON meta file. In the latter case it should match the pattern `https://host.name/some/path/-X.Y.Z-meta.json`. - -: For example: - - ```bash - # custom plugin repository at https://github.com/my-org/plugins - export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json" - - # custom plugin release - export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json" - - nextflow run -plugins nf-hello - ``` - -: This variable is useful for testing a plugin release before publishing it to the main registry. - -## Gradle Tasks - -The following build tasks are defined in the `build.gradle` of each core plugin as well as the `nf-hello` example plugin. - -### `makeZip` - -Create the plugin archive and JSON meta file in the subproject `build/libs` directory. - -```console -$ ls -l1 plugins/nf-tower/build/libs/ -nf-tower-0.1.0.jar -nf-tower-0.1.0.json -nf-tower-0.1.0.zip -``` - -### `copyPluginLibs` - -Copy plugin dependencies JAR files into the subproject `build/target/libs` directory. This is needed only when launching the plugin in development mode. - -### `copyPluginZip` - -Copy the plugin archive to the root project build directory, i.e. `build/plugins`. - -### `uploadPlugin` - -Upload the plugin archive and meta file to the corresponding GitHub repository. Options: - -`release` -: The plugin version, e.g. `1.0.1`. - -`repo` -: The GitHub repository name, e.g. `nf-amazon`. - -`owner` -: The GitHub owning organization, e.g. `nextflow-io`. - -`skipExisting` -: Do not upload a file if it already exists, i.e. checksum is the same (default: `true`). - -`dryRun` -: Execute the tasks without uploading file (default: `false`). - -`overwrite` -: Prevent to overwrite a remote file already existing (default: `false`). - -`userName` -: The user name used to authenticate GitHub API requests. - -`authToken` -: The personal token used to authenticate GitHub API requests. - -### `upload` - -Upload the plugin archive and meta file. - -### `publishIndex` - -Upload the plugins index to the repository hosted at [nextflow-io/plugins](https://github.com/nextflow-io/plugins), which makes them publicly accessible at [this URL](https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json). - -## Additional Resources - -* [PF4J](https://pf4j.org/) -* [Understanding Gradle: The Build Lifecycle](https://proandroiddev.com/understanding-gradle-the-build-lifecycle-5118c1da613f) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 31f341f0ee..2eb45aa150 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -1,3 +1,5 @@ +(gradle-plugin-page)= + # Using the Gradle plugin The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. @@ -6,14 +8,15 @@ The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: +(gradle-plugin-create)= + ## Creating a plugin -The nf-hello plugin uses the Gradle plugin and is a valuable starting point for developers. +The [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin uses the Gradle plugin and is a valuable starting point for developers. To create a Nextflow plugin with the Gradle plugin: -1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. - * See Example: nf-hello for more information about the nf-hello plugin. +1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. See {ref}`nf-hello-page` for more information. 2. Rename the forked `nf-hello` directory with your plugin name. 3. Replace the contents of `settings.gradle` with the following: @@ -72,10 +75,10 @@ To create a Nextflow plugin with the Gradle plugin: - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). -5. Develop your plugin extension points: - - See Extension points for descriptions and examples. +5. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. 6. In the plugin root directory, run `make assemble`. +(gradle-plugin-install)= ## Installing a plugin @@ -89,26 +92,27 @@ To install a plugin locally: Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. ::: -2. Configure your plugin. - * See Using plugins for more information. +2. Configure your plugin. See {ref}`using-plugins-page` for more information. 3. Run your pipeline: ```bash nextflow run main.nf ``` +(gradle-plugin-unit-test)= + ## Unit testing a plugin Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. To run unit tests: -1. Develop your unit tests. - * See HelloDslTest.groovy in the nf-hello plugin for unit test examples. +1. Develop your unit tests. See [HelloDslTest.groovy](https://github.com/nextflow-io/nf-hello/blob/gradle-plugin-example/src/test/groovy/nextflow/hello/HelloDslTest.groovy) in the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin for unit test examples. 2. In the plugin root directory, run `make test`. +(gradle-plugin-package)= -### Packaging, uploading, and publishing a plugin +## Packaging, uploading, and publishing a plugin The Gradle plugin for Nextflow plugins simplifies publishing your plugin. diff --git a/docs/index.md b/docs/index.md index 7943bd88b7..ae443012c5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -67,7 +67,6 @@ config executor cache-and-resume reports -plugins ``` ```{toctree} @@ -146,7 +145,6 @@ migrations/index developer/index developer/diagram developer/packages -developer/plugins ``` ```{toctree} diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index b424ff4233..c19cba3ef4 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -46,7 +46,7 @@ To migrate an existing Nextflow plugin: 3. Replace the contents of `settings.gradle` with the following: - ``` + ```groovy rootProject.name = '' ``` @@ -54,7 +54,7 @@ To migrate an existing Nextflow plugin: 4. In the project root, create a new `build.gradle` file with the following configuration: - ``` + ```groovy // Plugins plugins { id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' @@ -131,4 +131,4 @@ To migrate an existing Nextflow plugin: 6. Update `README.md` with information about the structure of your plugin. 7. In the plugin root directory, run `make assemble`. -The Gradle plugin for Nextflow plugins also supports publishing plugins. See Packaging, uploading, and publishing for more information. +The Gradle plugin for Nextflow plugins also supports publishing plugins. See {ref}`gradle-plugin-package` for more information. diff --git a/docs/plugins.md b/docs/plugins.md deleted file mode 100644 index 68d33151c1..0000000000 --- a/docs/plugins.md +++ /dev/null @@ -1,63 +0,0 @@ -(plugins-page)= - -# Plugins - -Nextflow has a plugin system that allows the use of extensible components that are downloaded and installed at runtime. - -(plugins-core)= - -## Core plugins - -The following functionalities are provided via plugin components, and they make part of the Nextflow *core* plugins: - -- `nf-amazon`: Support for Amazon Web Services. -- `nf-azure`: Support for Microsoft Azure. -- `nf-cloudcache`: Support for the cloud cache (see `NXF_CLOUDCACHE_PATH` under {ref}`config-env-vars`). -- `nf-console`: Implement Nextflow [REPL console](https://www.nextflow.io/blog/2015/introducing-nextflow-console.html). -- `nf-k8s`: Support for Kubernetes. -- `nf-google`: Support for Google Cloud. -- `nf-tower`: Support for [Seqera Platform](https://seqera.io) (formerly Tower Cloud). -- `nf-wave`: Support for [Wave containers](https://seqera.io/wave/) service. - -## Using plugins - -The core plugins do not require any configuration. They are automatically installed when the corresponding feature is requested by a Nextflow pipeline. You can still specify them as described below, e.g. if you want to pin the version of a plugin, however if you try to use a plugin version that isn't compatible with your Nextflow version, Nextflow will fail. - -You can enable a plugin by declaring it in your Nextflow configuration: - -```groovy -plugins { - id 'nf-hello@0.1.0' -} -``` - -Or you can use the `-plugins` command line option: - -```bash -nextflow run -plugins nf-hello@0.1.0 -``` - -The plugin identifier consists of the plugin name and plugin version separated by a `@`. Multiple plugins can be specified in the configuration with multiple `id` declarations, or on the command line as a comma-separated list. When specifying plugins via the command line, any plugin declarations in the configuration file are ignored. - -The plugin version is optional. If it is not specified, Nextflow will download the latest plugin version that is compatible with your Nextflow version. In general, it recommended that you not specify the plugin version unless you actually want to stick to that version, such as for [offline usage](#offline-usage). - -The core plugins are documented in this documentation. For all other plugins, please refer to the plugin's code repository for documentation and support. - -:::{versionadded} 25.02.0-edge -::: - -The plugin version can be prefixed with `~` to pin the major and minor version while allowing the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`, which is `2.9.2`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. - -## Offline usage - -To use Nextflow plugins in an offline environment: - -1. {ref}`Install Nextflow ` on a system with an internet connection. - -2. Download any additional plugins by running `nextflow plugin install `. Alternatively, simply run your pipeline once and Nextflow will download all of the plugins that it needs. - -3. Copy the `nextflow` binary and `$HOME/.nextflow` folder to your offline environment. - -4. In your Nextflow configuration file, specify each plugin that you downloaded, both name and version, including default plugins. This will prevent Nextflow from trying to download newer versions of plugins. - -Nextflow caches the plugins that it downloads, so as long as you keep using the same Nextflow version and pin your plugin versions in your config file, Nextflow will use the locally installed plugins and won't try to download them from the Internet. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 07b3c3fe2a..b56c841f10 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -1,4 +1,4 @@ -(developing-plugins-page)= +(dev-plugins-page)= # Developing plugins @@ -8,19 +8,19 @@ The Nextflow plugin framework streamlines plugin development by providing the st Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: -(developing-plugins-framework)= +(dev-plugins-framework)= ## Framework Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. -(developing-plugins-gradle)= +(dev-plugins-gradle)= ## Gradle plugin for Nextflow plugins -[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. +[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. -(developing-plugins-structure)= +(dev-plugins-structure)= ### Structure @@ -61,10 +61,10 @@ This structure contains the following key files and folders: See {ref}`nf-hello-page` for an example of a plugin built using this structure. :::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. See nf-hello for an example of a plugin with the old structure. +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: -(developing-plugins-make)= +(dev-plugins-make)= ### make commands @@ -83,18 +83,18 @@ make assemble The following `make` commands are available: `assemble` -: Compiles the Nextflow plugin code and assembles it into a zip file. See Creating a plugin for more information. +: Compiles the Nextflow plugin code and assembles it into a zip file. See {ref}`gradle-plugin-create` for more information. `test` -: Runs plugin unit tests. See Unit testing for more information. +: Runs plugin unit tests. See {ref}`gradle-plugin-unit-test` for more information. `install` -: Installs the plugin into the local nextflow plugins directory. See Running locally for more information. +: Installs the plugin into the local nextflow plugins directory. See {ref}`gradle-plugin-install` for more information. `release` -: Publishes the plugin. See Packaging, uploading, and publishing for more information. +: Publishes the plugin. See {ref}`gradle-plugin-package` for more information. -(developing-plugins-versioning)= +(dev-plugins-versioning)= ### Versioning @@ -108,7 +108,7 @@ plugins { See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. -(developing-plugins-extension)= +(dev-plugins-extension)= ## Extension points @@ -217,7 +217,7 @@ This approach is not required to support plugin config options. However, it allo Plugins can define custom executors that can be used with the `executor` process directive. -To implement an executor, create a class in your plugin that extends the [Executor](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: +To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy)class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: ```groovy import nextflow.executor.Executor @@ -249,9 +249,9 @@ See the source code of Nextflow's built-in executors for examples of how to impl ### Filesystems -Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see Remote files. +Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. -To implement a custom filesystem, create a class in your plugin that extends [FileSystemProvider](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: +To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: ```groovy import java.nio.file.spi.FileSystemProvider @@ -371,9 +371,9 @@ The above snippet is based on the [nf-sqldb](https://github.com/nextflow-i ### Process directives -Plugins that implement a custom executor will likely need to access process directives that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. +Plugins that implement a custom executor will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. -Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the ext directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. +Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the {ref}`process-ext` directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. For example, a custom executor can use existing process directives and a custom setting through the `ext` directive: @@ -451,6 +451,7 @@ myplugin.enabled = true See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. +(dev-plugins-env-var)= ## Environment variables diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 689b5f0a69..38e0b4d00e 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -2,9 +2,9 @@ # Example: nf-hello -`nf-hello` is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. +[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/master) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. -The nf-hello plugin has the following structure: +The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/master) has the following structure: ``` nf-hello diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 259d1e9e0c..5d229c08f7 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -28,7 +28,7 @@ Core plugins do not require configuration. The latest versions of core plugins a Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. :::{note} -The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See Environment variables for more information. +The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-var` for more information. :::

Third-party plugins

@@ -64,4 +64,4 @@ Version components: * PATCH: Increment for backward-compatible bug fixes. Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. -See Semantic Versioning for more information about the specification. +See {ref}`dev-plugins-versioning` for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index 8aef78fc7e..da319a0849 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -1,7 +1,11 @@ +(using-plugins-page)= + # Using plugins Nextflow core plugins require no additional configuration. When a pipeline uses a core plugin, Nextflow automatically downloads and uses the latest compatible plugin version. In contrast, third-party plugins must be explicitly declared. When a pipeline uses one or more third-party plugins, Nextflow must be configured to download and use the plugin. +(using-plugins-identifiers)= + ## Identifiers A plugin identifier consists of the plugin name and version, separated by an `@` symbol: @@ -13,7 +17,7 @@ nf-hello@0.5.0 The plugin version is optional. If it is not specified, Nextflow will download the latest version of the plugin that meets the minimum Nextflow version requirements specified by the plugin. :::{note} -Plugin versions are required to run plugins in an Offline environment. +Plugin versions are required for {ref}`offline usage `. ::: :::{versionadded} 25.02.0-edge. @@ -25,6 +29,8 @@ The plugin version can be prefixed with `~` to pin the major and minor versions It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. ::: +(using-plugins-config)= + ## Configuration Plugins can be configured via nextflow configuration files or at runtime. @@ -57,13 +63,15 @@ Plugin declarations in nextflow configuration files are ignored when specifying When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. +(using-plugins-offline)= + ## Offline usage Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. To use Nextflow plugins in an offline environment: -1. Install a self-contained version of Nextflow in an environment with an internet connection. See Standalone distribution for more information. +1. Install a self-contained version of Nextflow in an environment with an internet connection. See {ref}`install-standalone` for more information. 2. Run `nextflow plugin install @` to download the plugins. :::{tip} @@ -71,8 +79,8 @@ To use Nextflow plugins in an offline environment: ::: 3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. -4. Specify each plugin and its version in Nextflow configuration files or at runtime. See Configuration for more information. +4. Specify each plugin and its version in Nextflow configuration files or at runtime. See {ref}`using-plugins-config` for more information. :::{warning} - Nextflow will attempt to download newer versions of plugins if their versions are not set. See Plugin identifiers for more information. + Nextflow will attempt to download newer versions of plugins if their versions are not set. See {ref}`using-plugins-identifiers` for more information. ::: \ No newline at end of file diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index abce1e6548..bc507837ea 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -154,7 +154,7 @@ The following environment variables control the configuration of the Nextflow ru `NXF_PLUGINS_TEST_REPOSITORY` : :::{versionadded} 23.04.0 ::: -: Defines a custom plugin registry or plugin release URL for testing plugins outside of the main registry. See {ref}`testing-plugins` for more information. +: Defines a custom plugin registry or plugin release URL for testing plugins outside of the main registry. `NXF_PUBLISH_FAIL_ON_ERROR` : :::{versionadded} 24.04.3 diff --git a/docs/strict-syntax.md b/docs/strict-syntax.md index 19b8b71148..3f31201739 100644 --- a/docs/strict-syntax.md +++ b/docs/strict-syntax.md @@ -595,4 +595,4 @@ There are two ways to preserve Groovy code: Any Groovy code can be moved into the `lib` directory, which supports the full Groovy language. This approach is useful for temporarily preserving some Groovy code until it can be updated later and incorporated into a Nextflow script. See {ref}`lib-directory` documentation for more information. -For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`plugins-dev-page` for more information on how to develop plugins. +For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`dev-plugins-page` for more information on how to develop plugins. From 5471abcc7914cf504e8a3848e0a1b2bd93ac968f Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 29 Apr 2025 01:39:56 +1200 Subject: [PATCH 04/29] Add repo link Signed-off-by: Christopher Hakkaart --- docs/plugins/example-nf-hello.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 38e0b4d00e..b727838163 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -2,9 +2,9 @@ # Example: nf-hello -[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/master) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. +[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. -The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/master) has the following structure: +The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) has the following structure: ``` nf-hello @@ -56,3 +56,5 @@ The `nf-hello` plugin can be configured via nextflow configuration files or at r ```bash nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 ``` + +See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. From 3dfb9458b9365ffb4abf627046ad8c86693359fe Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 29 Apr 2025 09:38:39 +1200 Subject: [PATCH 05/29] Add Trace observers note Signed-off-by: Christopher Hakkaart --- docs/plugins/developing-plugins.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index b56c841f10..761b4182ea 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -398,6 +398,10 @@ class MyExecutor extends Executor { ### Trace observers +:::{versionchanged} 25.04 +The `TraceObserver` interface is now deprecated. Use [TraceObserverV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverV2.groovy) and [TraceObserverFactoryV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy) instead. +::: + A *trace observer* is an entity that can listen and react to workflow events, such as when a workflow starts, a task is completed, or a file is published. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver` and implement the `create()` method in your observer factory. For example: From e254b63611384abf82ca5f28b2f242dd2bc2a175 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 5 May 2025 18:44:38 +1200 Subject: [PATCH 06/29] Revise defined registry Signed-off-by: Christopher Hakkaart --- docs/migrating-gradle-plugin.md | 2 -- docs/plugins/plugins.md | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index c19cba3ef4..384bc7b5d5 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -17,8 +17,6 @@ The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. -Developers can upload plugins to the plugin registry. The Gradle plugin for Nextflow plugins supports this process with Gradle tasks that package, upload, and publish plugins. - ## Impact on users and developers The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 5d229c08f7..59405d71ab 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -51,7 +51,7 @@ See {ref}`using-plugins-page` for more information. ## Nextflow plugin registry -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. Developers can upload plugins to the registry, with built-in support from the Gradle plugin for Nextflow plugins. +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. ## Versioning From 5a6e4d9ac24bd78784f4c2be0907d88ad55c5651 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 6 May 2025 15:36:00 +1200 Subject: [PATCH 07/29] Add plugin create documentation Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 85 +++++----------------- docs/migrating-gradle-plugin.md | 20 ++--- docs/migrations/25-04.md | 6 ++ docs/plugins/developing-plugins.md | 113 ++++++++++++++++------------- docs/plugins/using-plugins.md | 2 +- docs/reference/cli.md | 6 +- 6 files changed, 104 insertions(+), 128 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 2eb45aa150..81f9303e88 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -2,7 +2,7 @@ # Using the Gradle plugin -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development and incorporates the Gradle plugin by default. You can use the `nextflow plugin create` sub-command to create plugins scaffolds with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. :::{note} Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. @@ -12,71 +12,18 @@ Nextflow Plugins can be developed without the Gradle plugin. However, this appro ## Creating a plugin -The [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin uses the Gradle plugin and is a valuable starting point for developers. +:::{versionadded} 25.04.0 +::: To create a Nextflow plugin with the Gradle plugin: -1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. See {ref}`nf-hello-page` for more information. -2. Rename the forked `nf-hello` directory with your plugin name. -3. Replace the contents of `settings.gradle` with the following: - - ```groovy - rootProject.name = '' - ``` - - Replace `PLUGIN_NAME` with your plugin name. - -4. Replace the contents of `build.gradle` with the following: - - ```groovy - // Plugins - plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' - } - - // Dependencies (optional) - dependencies { - - } - - // Plugin version - version = '' - - nextflowPlugin { - // Minimum Nextflow version - nextflowVersion = '' - - // Plugin metadata - provider = '' - className = '' - extensionPoints = [ - '' - ] - - publishing { - github { - repository = '' - userName = project.findProperty('github_username') - authToken = project.findProperty('github_access_token') - email = project.findProperty('github_commit_email') - indexUrl = '' - } - } - } - ``` - - Replace the following: - - - `DEPENDENCY`: (optional) your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - - `PLUGIN_VERSION:` your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: the minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. - - `PROVIDER`: your name or organization—for example, `nextflow`. - - `CLASS_NAME`: your plugin class name—for example, `nextflow.hello.HelloPlugin`. - - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). -5. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. -6. In the plugin root directory, run `make assemble`. +1. Run `nextflow plugin create`. + - When prompted `Enter plugin name:`, enter your plugin name. + - When prompted `Enter organization:`, enter your organization. + - When prompted `Enter project path:`, enter the project path in your local file system. + - When prompted `All good, are you OK to continue [y/N]?`, enter `y`. +2. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. +3. In the plugin root directory, run `make assemble`. (gradle-plugin-install)= @@ -92,13 +39,17 @@ To install a plugin locally: Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. ::: -2. Configure your plugin. See {ref}`using-plugins-page` for more information. -3. Run your pipeline: +2. Run your pipeline: ```bash - nextflow run main.nf + nextflow run main.nf -plugins @ ``` + :::{note} + Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. + ::: + + (gradle-plugin-unit-test)= ## Unit testing a plugin @@ -107,7 +58,7 @@ Unit tests are small, focused tests designed to verify the behavior of individua To run unit tests: -1. Develop your unit tests. See [HelloDslTest.groovy](https://github.com/nextflow-io/nf-hello/blob/gradle-plugin-example/src/test/groovy/nextflow/hello/HelloDslTest.groovy) in the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin for unit test examples. +1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template/tree/main) for unit test examples. 2. In the plugin root directory, run `make test`. (gradle-plugin-package)= diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 384bc7b5d5..3ca6889746 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -1,3 +1,5 @@ +(migrating-plugin-page)= + # Migrating to the Gradle plugin for Nextflow plugins This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. @@ -19,7 +21,7 @@ The Nextflow plugin registry is a centralized repository of assembled plugins. I ## Impact on users and developers -The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. +The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. ### Plugin Users @@ -36,7 +38,7 @@ To migrate an existing Nextflow plugin: - `nextflow.config` - `launch.sh` - `plugins/build.gradle` -2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. \ +2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. :::{note} Plugin sources should be in `src/main/groovy` or `src/main/java`. @@ -55,7 +57,7 @@ To migrate an existing Nextflow plugin: ```groovy // Plugins plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' } // Dependencies (optional) @@ -93,12 +95,12 @@ To migrate an existing Nextflow plugin: - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. - - `PROVIDER`: Your name or organization—for example, `nextflow`. - - `CLASS_NAME`: Your plugin class name—for example, `nextflow.hello.HelloPlugin`. - - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.03.0-edge`. + - `PROVIDER`: Your name or organization—for example, `acme`. + - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. + - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-plugin-template`. + - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`https://github.com/nextflow-io/plugins/blob/main/plugins.json`](https://github.com/nextflow-io/plugins/blob/main/plugins.json). 5. Replace the contents of `Makefile` with the following: diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index 205ed72e09..a1e64d75a9 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -37,6 +37,12 @@ The third preview of workflow outputs introduces the following breaking changes See {ref}`workflow-output-def` to learn more about the workflow output definition. +

Creating plugins

+ +The `nextflow plugin create` sub-command creates a scaffold for a Nextflow plugin using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). + +See {ref}`gradle-plugin-create` for details. + ## Enhancements

Improved inspect command

diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 761b4182ea..1663aed796 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -20,61 +20,15 @@ Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/p [Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. -(dev-plugins-structure)= - -### Structure - -Plugins built with the Gradle plugin for Nextflow plugins follow a standard project layout. This structure includes source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. - -A typical plugin built with the Gradle plugin for Nextflow plugins has the following structure: - -``` -. -├── COPYING -├── Makefile -├── README.md -├── build.gradle -├── gradle -│ └── wrapper -│ └── ... -├── gradlew -├── settings.gradle -└── src - ├── main - │ └── ... - └── test - └── ... -``` - -This structure contains the following key files and folders: - -- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. -- `src/main/`: The main source directory containing the plugin's implementation and resources.​ -- `src/test/`: The main source directory containing the plugin's unit testing. -- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ -- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ -- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ -- `build.gradle`: The primary build script for Gradle. -- `gradlew`: A script for executing the Gradle Wrapper. -- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. - -See {ref}`nf-hello-page` for an example of a plugin built using this structure. - -:::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. -::: - -(dev-plugins-make)= - ### make commands -The Gradle plugin for Nextflow plugins defines tasks that can be executed with `./gradlew`. For example: +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) defines tasks that can be executed with `./gradlew`. For example: ```bash ./gradlew assemble ``` -For convenience, the more important tasks are wrapped in a makefile and can be executed with the `make` command. For example: +For convenience, the most important tasks are wrapped in a `Makefile` and can be executed with the `make` command. For example: ```bash make assemble @@ -98,16 +52,75 @@ The following `make` commands are available: ### Versioning -The Gradle plugin is versioned and published to a [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: ```nextflow plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' } ``` See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. +(dev-plugins-template)= + +## nf-plugin-template + +The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It incorporates the {ref}`dev-plugins-gradle` by default. You can use the `nextflow plugin create` sub-command to create plugins with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. See {ref}`gradle-plugin-create` for more information. + +(dev-plugins-structure)= + +### Structure + +Plugins built with [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) includes the [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle). The template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. + +Plugins built with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) have the following structure: + +``` +. +├── build.gradle +├── COPYING +├── gradle +│ └── wrapper +│ └── ... +├── gradlew +├── Makefile +├── README.md +├── settings.gradle +└── src + ├── main + │ └── groovy + │ └── + │ └── plugin + │ └── ... + └── test + └── groovy + └── + └── plugin + └── ... +``` + +This structure contains the following key files and folders: + +- `build.gradle`: The primary build script for Gradle. +- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ +- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. +- `gradlew`: A script for executing the Gradle Wrapper. +- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ +- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ +- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. +- `src/main/groovy//plugin/`: The main source directory containing the plugin's implementation and resources.​ +- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit testing. + +See {ref}`nf-hello-page` for an example of a plugin built using this structure. + +:::{note} +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +(dev-plugins-make)= + + (dev-plugins-extension)= ## Extension points diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index da319a0849..282bd92b28 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -20,7 +20,7 @@ The plugin version is optional. If it is not specified, Nextflow will download t Plugin versions are required for {ref}`offline usage `. ::: -:::{versionadded} 25.02.0-edge. +:::{versionadded} 25.02.0-edge ::: The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 055ace3030..f92b2af868 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -910,7 +910,11 @@ Manage plugins and run plugin-specific commands. $ nextflow plugin [options] ``` -The `plugin` command provides several subcommands for managing and using plugins: +`create` + +: :::{versionadded} 25.04.0 + ::: +: Create a plugin scaffold using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. `install ` From 9053ca9b0b1e548b2c010e23ea3b299ff8848c9f Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 6 May 2025 12:41:09 -0500 Subject: [PATCH 08/29] Fix plugin config scope example Signed-off-by: Ben Sherman --- docs/plugins/developing-plugins.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 1663aed796..b43a268712 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -214,6 +214,10 @@ import nextflow.script.dsl.Description ''') class MyPluginConfig implements ConfigScope { + // no-arg constructor is required to enable validation of config options + MyPluginConfig() { + } + MyPluginConfig(Map opts) { this.createMessage = opts.createMessage } From ce87361006e0b76cdeac10aefd4189286506ca2b Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Wed, 7 May 2025 14:08:29 +1200 Subject: [PATCH 09/29] Only support registry Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 28 ++++++++++++---------------- docs/migrating-gradle-plugin.md | 21 ++++++++++----------- docs/plugins/developing-plugins.md | 2 +- docs/plugins/example-nf-hello.md | 2 +- 4 files changed, 24 insertions(+), 29 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 81f9303e88..d3412c432b 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -65,25 +65,21 @@ To run unit tests: ## Packaging, uploading, and publishing a plugin -The Gradle plugin for Nextflow plugins simplifies publishing your plugin. +The Gradle plugin for Nextflow plugins simplifies publishing your plugin to the Nextflow Plugin Registry. -To package, upload, and publish your plugin: +:::{note} +The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +::: -1. Fork the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins). -2. In the plugin root directory, open `build.gradle` and ensure that: - * `github.repository` matches the plugin repository. - * `github.indexUrl` matches your fork of the plugins index repository. -3. Create a file named `$HOME/.gradle/gradle.properties` and add the following: +To package, upload, and publish your plugin to the Nextflow Plugin Registry: + +1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. +2. Add the following properties: ```bash - github_username= - github_access_token= - github_commit_email= + pluginRegistry.accessToken= ``` - Replace the following: - * `GITHUB_USERNAME`: your GitHub username granting access to the plugin repository. - * `GITHUB_ACCESS_TOKEN`: your GitHub access token with permission to upload and commit changes to the plugin repository. - * `GITHUB_EMAIL`: your email address associated with your GitHub account. -4. Run `make release`. -5. Create a pull request against the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins) from your fork. \ No newline at end of file + Replace with your plugin registry access token. + +3. Run `make release`. \ No newline at end of file diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 3ca6889746..75e59aaef4 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -15,10 +15,14 @@ The Gradle plugin for Nextflow plugins simplifies and standardizes the developme The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. -### Nextflow plugin registry +### Nextflow Plugin Registry The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. +:::{note} +The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +::: + ## Impact on users and developers The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. @@ -57,7 +61,7 @@ To migrate an existing Nextflow plugin: ```groovy // Plugins plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' } // Dependencies (optional) @@ -79,13 +83,10 @@ To migrate an existing Nextflow plugin: '' ] - publishing { - github { - repository = '' - userName = project.findProperty('github_username') - authToken = project.findProperty('github_access_token') - email = project.findProperty('github_commit_email') - indexUrl = '' + publishing { + registry { + url = 'https://nf-plugins-registry.dev-tower.net/api' + authToken = project.findProperty('pluginRegistry.accessToken') } } } @@ -99,8 +100,6 @@ To migrate an existing Nextflow plugin: - `PROVIDER`: Your name or organization—for example, `acme`. - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. - - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-plugin-template`. - - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`https://github.com/nextflow-io/plugins/blob/main/plugins.json`](https://github.com/nextflow-io/plugins/blob/main/plugins.json). 5. Replace the contents of `Makefile` with the following: diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 1663aed796..5a987dd6f5 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -56,7 +56,7 @@ The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https ```nextflow plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' } ``` diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index b727838163..b643218eb8 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -54,7 +54,7 @@ It also includes several classes that demonstrate different plugin functionality The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: ```bash -nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +nextflow run hello -plugins nf-hello@0.5.0 ``` See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. From 2bb396034e28b70bbc3515372255b5131a2ae5f7 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 7 May 2025 11:18:32 -0500 Subject: [PATCH 10/29] cleanup Signed-off-by: Ben Sherman --- docs/gradle-plugin.md | 47 +++++----- docs/index.md | 18 ++-- docs/migrating-gradle-plugin.md | 14 +-- docs/migrations/25-04.md | 2 +- docs/plugins/developing-plugins.md | 138 +++++++++++++---------------- docs/plugins/example-nf-hello.md | 25 ++---- docs/plugins/plugins.md | 33 +++---- docs/plugins/using-plugins.md | 27 +++--- docs/reference/stdlib-groovy.md | 2 +- 9 files changed, 140 insertions(+), 166 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index d3412c432b..4ee7e08366 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -2,10 +2,10 @@ # Using the Gradle plugin -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development and incorporates the Gradle plugin by default. You can use the `nextflow plugin create` sub-command to create plugins scaffolds with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. :::{note} -Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: (gradle-plugin-create)= @@ -15,21 +15,19 @@ Nextflow Plugins can be developed without the Gradle plugin. However, this appro :::{versionadded} 25.04.0 ::: -To create a Nextflow plugin with the Gradle plugin: +The easiest way to boostrap a Nextflow plugin based on the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). -1. Run `nextflow plugin create`. - - When prompted `Enter plugin name:`, enter your plugin name. - - When prompted `Enter organization:`, enter your organization. - - When prompted `Enter project path:`, enter the project path in your local file system. - - When prompted `All good, are you OK to continue [y/N]?`, enter `y`. -2. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. -3. In the plugin root directory, run `make assemble`. +To create a Nextflow plugin with the Gradle plugin, simply run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. + +See {ref}`dev-plugins-extension` for more information about developing a plugin. -(gradle-plugin-install)= +1. Run `nextflow plugin create`. +2. Develop your plugin extension points. +3. In the plugin root directory, run `make assemble`. -## Installing a plugin +## Building a plugin -Plugins can be installed locally without being packaged, uploaded, and published. +Plugins can be installed locally without being published. To install a plugin locally: @@ -49,37 +47,38 @@ To install a plugin locally: Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. ::: +(gradle-plugin-test)= -(gradle-plugin-unit-test)= - -## Unit testing a plugin +## Testing a plugin Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. To run unit tests: -1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template/tree/main) for unit test examples. +1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template) for unit test examples. + 2. In the plugin root directory, run `make test`. -(gradle-plugin-package)= +(gradle-plugin-publish)= -## Packaging, uploading, and publishing a plugin +## Publishing a plugin -The Gradle plugin for Nextflow plugins simplifies publishing your plugin to the Nextflow Plugin Registry. +The Nextflow Gradle plugin allows you to publish your plugin to the Nextflow plugin registry from the command line. :::{note} -The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. ::: -To package, upload, and publish your plugin to the Nextflow Plugin Registry: +To publish your plugin: 1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. + 2. Add the following properties: ```bash pluginRegistry.accessToken= ``` - Replace with your plugin registry access token. + Replace `` with your plugin registry access token. -3. Run `make release`. \ No newline at end of file +3. Run `make release`. diff --git a/docs/index.md b/docs/index.md index 635d6d6b3a..3d09716c36 100644 --- a/docs/index.md +++ b/docs/index.md @@ -146,23 +146,23 @@ migrations/index ```{toctree} :hidden: -:caption: Contributing +:caption: Plugins :maxdepth: 1 -developer/index -developer/diagram -developer/packages +plugins/plugins +plugins/using-plugins +plugins/developing-plugins +plugins/example-nf-hello ``` ```{toctree} :hidden: -:caption: Plugins +:caption: Contributing :maxdepth: 1 -plugins/plugins -plugins/using-plugins -plugins/developing-plugins -plugins/example-nf-hello +developer/index +developer/diagram +developer/packages ``` ```{toctree} diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 75e59aaef4..87187281f2 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -1,17 +1,17 @@ (migrating-plugin-page)= -# Migrating to the Gradle plugin for Nextflow plugins +# Migrating to the Nextflow Gradle plugin -This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. +This page introduces the Nextflow Gradle plugin, the Nextflow plugin registry, and how to migrate to the new plugin framework. ## Improvements to the plugin framework The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying plugin development, streamlining publishing and discovery, and improving how plugins are loaded into workflows. These improvements make plugins more accessible, maintainable, and interoperable with Nextflow. -### Gradle plugin for Nextflow plugins +### Nextflow Gradle plugin -The Gradle plugin for Nextflow plugins simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. +The Nextflow Gradle plugin simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. @@ -25,7 +25,7 @@ The Nextflow Plugin Registry is currently available as private beta technology. ## Impact on users and developers -The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. +The impact of the Nextflow Gradle plugin differs for plugin users and developers. ### Plugin Users @@ -33,7 +33,7 @@ If you are a plugin user, no immediate actions are required. The plugin configur ### Plugin developers -Developers are encouraged to migrate to the Gradle plugin for Nextflow plugins and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. +Developers are encouraged to migrate to the Nextflow Gradle plugin and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. To migrate an existing Nextflow plugin: @@ -130,4 +130,4 @@ To migrate an existing Nextflow plugin: 6. Update `README.md` with information about the structure of your plugin. 7. In the plugin root directory, run `make assemble`. -The Gradle plugin for Nextflow plugins also supports publishing plugins. See {ref}`gradle-plugin-package` for more information. +The Nextflow Gradle plugin also supports publishing plugins. See {ref}`gradle-plugin-publish` for more information. diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index 329ac5bc9d..0509d97a53 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -49,7 +49,7 @@ You can explore this lineage from the command line using the {ref}`cli-lineage` See the {ref}`cli-lineage` command and {ref}`config-lineage` config scope for details. -

Streamlined plugin development

+

Simplified plugin development

The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the new [plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 394b364919..470983506b 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -2,79 +2,23 @@ # Developing plugins -The Nextflow plugin framework streamlines plugin development by providing the structure and tools needed to extend Nextflow functionality. The Gradle plugin for Nextflow plugins simplifies development by configuring default Nextflow dependencies and incorporates custom Gradle tasks that streamline building and publishing plugins. - -:::{note} -Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. -::: - -(dev-plugins-framework)= - -## Framework - -Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. - -(dev-plugins-gradle)= - -## Gradle plugin for Nextflow plugins - -[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. - -### make commands - -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) defines tasks that can be executed with `./gradlew`. For example: - -```bash -./gradlew assemble -``` - -For convenience, the most important tasks are wrapped in a `Makefile` and can be executed with the `make` command. For example: - -```bash -make assemble -``` - -The following `make` commands are available: - -`assemble` -: Compiles the Nextflow plugin code and assembles it into a zip file. See {ref}`gradle-plugin-create` for more information. - -`test` -: Runs plugin unit tests. See {ref}`gradle-plugin-unit-test` for more information. - -`install` -: Installs the plugin into the local nextflow plugins directory. See {ref}`gradle-plugin-install` for more information. - -`release` -: Publishes the plugin. See {ref}`gradle-plugin-package` for more information. - -(dev-plugins-versioning)= - -### Versioning - -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: - -```nextflow -plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' -} -``` - -See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. +This page describes how to develop plugins for Nextflow. (dev-plugins-template)= -## nf-plugin-template +## Nextflow plugin template + +The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It uses [Gradle](https://gradle.org/), a build automation tool optimized for Java and Groovy projects, as well as the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle). -The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It incorporates the {ref}`dev-plugins-gradle` by default. You can use the `nextflow plugin create` sub-command to create plugins with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. See {ref}`gradle-plugin-create` for more information. +You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. (dev-plugins-structure)= ### Structure -Plugins built with [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) includes the [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle). The template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. +The plugin template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. -Plugins built with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) have the following structure: +Plugins built with the plugin template have the following structure: ``` . @@ -103,29 +47,70 @@ Plugins built with the [`nf-plugin-template`](https://github.com/nextflow-io/nf- This structure contains the following key files and folders: - `build.gradle`: The primary build script for Gradle. + - `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ + - `gradle/wrapper/`: Holds files related to the Gradle Wrapper. -- `gradlew`: A script for executing the Gradle Wrapper. + +- `gradlew`: The Gradle Wrapper script, which allows you to use Gradle without installing it into your environment. + - `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ + - `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ + - `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. + - `src/main/groovy//plugin/`: The main source directory containing the plugin's implementation and resources.​ -- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit testing. -See {ref}`nf-hello-page` for an example of a plugin built using this structure. +- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit tests. + +See {ref}`nf-hello-page` for an example of a plugin built with the plugin template. + +### Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by providing default configuration, Nextflow dependencies, and custom Gradle tasks for building and publishing plugins. + +It is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), and can be declared and managed like any other dependency in the `build.gradle` file: + +```nextflow +plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' +} +``` :::{note} Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: -(dev-plugins-make)= +### Make commands + +The plugin template includes a Makefile which wraps the most important Gradle tasks provided by the Nextflow Gradle plugin. + +These tasks can be executed with [Make](https://www.gnu.org/software/make/). For example: + +```bash +make assemble +``` + +The following `make` commands are available: + +`assemble` +: Compiles the Nextflow plugin code and assembles it into a zip archive. + +`test` +: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. + +`install` +: Installs the plugin into the local Nextflow plugins directory. +`release` +: Publishes the plugin. See {ref}`gradle-plugin-publish` for more information. (dev-plugins-extension)= ## Extension points -Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how they can be used. +Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how to use them. ### Commands @@ -159,7 +144,7 @@ class MyPlugin extends BasePlugin implements PluginAbstractExec { The command can be run using the `nextflow plugin` command: -``` +```bash nextflow plugin my-plugin:hello --foo --bar ``` @@ -234,7 +219,7 @@ This approach is not required to support plugin config options. However, it allo Plugins can define custom executors that can be used with the `executor` process directive. -To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy)class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: +To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: ```groovy import nextflow.executor.Executor @@ -263,7 +248,6 @@ process foo { See the source code of Nextflow's built-in executors for examples of how to implement various executor components. ::: - ### Filesystems Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. @@ -290,7 +274,7 @@ You can then use this filesystem in your pipeline: input = file('myfs://') ``` -See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for examples of custom filesystems. +See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for an example of a custom filesystem. :::{tip} Custom filesystems are an advanced plugin extension. Before creating a new filesystem, check that your use case cannot already be supported by an existing filesystem such as HTTP or S3. @@ -324,7 +308,7 @@ class MyExtension extends PluginExtensionPoint { You can then add this function to your pipeline: -``` +```nextflow include { reverseString } from 'plugin/my-plugin' channel.of( reverseString('hi') ) @@ -383,7 +367,11 @@ channel ``` :::{note} -The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +::: + +:::{tip} +Before creating a custom operator, consider whether the operator can be defined as a [function](#functions) that can be composed with existing operators such as `map` or `subscribe`. Functions are easier to implement and can be used anywhere in your pipeline, not just channel logic. ::: ### Process directives @@ -472,7 +460,7 @@ myplugin.enabled = true See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. -(dev-plugins-env-var)= +(dev-plugins-env-vars)= ## Environment variables diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index b643218eb8..99ca9984fc 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -2,9 +2,9 @@ # Example: nf-hello -[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. +[nf-hello](https://github.com/nextflow-io/nf-hello) is a simple Nextflow plugin that uses the Nextflow Gradle plugin and provides examples for various plugin extension points. -The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) has the following structure: +The `nf-hello` plugin has the following structure: ``` nf-hello @@ -36,20 +36,13 @@ nf-hello └── HelloFactoryTest.groovy ``` -It includes examples of different plugin extensions: +It includes the following extensions: -- A custom trace observer that prints a message when the workflow starts and when the workflow completes. -- A custom channel factory called reverse. -- A custom operator called goodbye. -- A custom function called randomString. - -It also includes several classes that demonstrate different plugin functionality: - -- `HelloConfig`: An example of how to handle options from the Nextflow configuration. -- `HelloExtension`: An example of how to create custom channel factories, operators, and functions that can be included in pipeline scripts. -- `HelloFactory`: An example of a workflow event with custom behavior. -- `HelloObserver`: An example of a workflow event with custom behavior. -- `HelloPlugin`: An example of a plugin entry point. +- A custom `hello` config scope (see `HelloConfig`). +- A custom trace observer that prints a message when the workflow starts and when the workflow completes (see `HelloObserver`). +- A custom channel factory called `reverse` (see `HelloExtension`). +- A custom operator called `goodbye` (see `HelloExtension`). +- A custom function called `randomString` (see `HelloExtension`). The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: @@ -57,4 +50,4 @@ The `nf-hello` plugin can be configured via nextflow configuration files or at r nextflow run hello -plugins nf-hello@0.5.0 ``` -See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. +See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello) for the plugin source code. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 59405d71ab..470b8159a4 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -4,14 +4,9 @@ ## What are plugins -Nextflow plugins are extensions that enhance the functionality of the Nextflow workflow framework. They allow users to add new capabilities and integrate with external services without modifying core Nextflow code. +Nextflow plugins are extensions that enhance the functionality of Nextflow. They allow users to add new capabilities and integrate with external services without bloating their pipeline code or modifying Nextflow itself. -There are two types of Nextflow plugins: - -* Core plugins -* Third-party plugins - -The main features of each plugin type are described below. +There are two types of Nextflow plugins: core plugins and third-party plugins. The main features of each plugin type are described below.

Core plugins

@@ -20,7 +15,7 @@ Core plugins do not require configuration. The latest versions of core plugins a * `nf-amazon`: Support for Amazon Web Services. * `nf-azure`: Support for Microsoft Azure. * `nf-cloudcache`: Support for the cloud cache. -* `nf-console`: Implement Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). +* `nf-console`: Implementation of the Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). * `nf-google`: Support for Google Cloud. * `nf-tower`: Support for [Seqera Platform](https://seqera.io/platform/). * `nf-wave`: Support for [Wave containers service](https://seqera.io/wave/). @@ -28,14 +23,14 @@ Core plugins do not require configuration. The latest versions of core plugins a Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. :::{note} -The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-var` for more information. +The automatic retrieval of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-vars` for more information. :::

Third-party plugins

Third-party plugins must be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: -``` +```groovy plugins { id 'nf-hello@0.5.0' } @@ -43,7 +38,7 @@ plugins { To configure plugins at runtime, use the `-plugins` option. For example: -``` +```bash nextflow run main.nf -plugins nf-hello@0.5.0 ``` @@ -51,17 +46,23 @@ See {ref}`using-plugins-page` for more information. ## Nextflow plugin registry -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. +:::{versionadded} 25.04.0 +::: + +:::{note} +The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. +::: + +The Nextflow plugin registry is a central repository for publishing and discovering Nextflow plugins. Nextflow is able to use the plugin registry to automatically find and download plugins at runtime. ## Versioning -Nextflow plugins follow the Semantic Versioning format: MAJOR.MINOR.PATCH (e.g., 0.5.0). This helps developers communicate the nature of changes in a release. +Nextflow plugins are free to use any versioning convention. Many plugins, including all core plugins, use [Semantic Versioning](https://semver.org/), which helps developers communicate the kind of changes in a release. -Version components: +Semantic versions have the form MAJOR.MINOR.PATCH (e.g., 0.5.0): -* MAJOR: Increment for incompatible changes. +* MAJOR: Increment for backward-incompatible changes. * MINOR: Increment for backward-compatible feature additions. * PATCH: Increment for backward-compatible bug fixes. Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. -See {ref}`dev-plugins-versioning` for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index 282bd92b28..4e21292cc7 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -10,7 +10,7 @@ Nextflow core plugins require no additional configuration. When a pipeline uses A plugin identifier consists of the plugin name and version, separated by an `@` symbol: -```console +``` nf-hello@0.5.0 ``` @@ -26,61 +26,54 @@ Plugin versions are required for {ref}`offline usage `. The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. :::{tip} -It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. +It is recommended to pin the major and minor version of each plugin that you use, in order to minimize the risk of breaking changes while allowing patch updates to be used automatically. ::: (using-plugins-config)= ## Configuration -Plugins can be configured via nextflow configuration files or at runtime. +Plugins can be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: ```nextflow plugins { id 'nf-hello@0.5.0' + id 'nf-amazon@2.9.0' } ``` To configure plugins at runtime, use the `-plugins` option. For example: -```bash -nextflow run main.nf -plugins nf-hello@0.5.0 -``` - -To configure multiple plugins at runtime, use the `-plugins` option and a comma-separated list. For example: - ```bash nextflow run main.nf -plugins nf-hello@0.5.0,nf-amazon@2.9.0 ``` :::{note} -Plugin declarations in nextflow configuration files are ignored when specifying plugins via the `-plugins` option. +Plugin declarations in Nextflow configuration files are ignored when specifying plugins via the `-plugins` option. ::: ## Caching -When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. +When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). (using-plugins-offline)= ## Offline usage -Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. +When running Nextflow in an offline environment, any required plugins must be downloaded and moved into the offline environment prior to any runs. To use Nextflow plugins in an offline environment: 1. Install a self-contained version of Nextflow in an environment with an internet connection. See {ref}`install-standalone` for more information. -2. Run `nextflow plugin install @` to download the plugins. - :::{tip} - Running a pipeline in an environment with an internet connection will also download the plugins used by that pipeline. - ::: +2. Run `nextflow plugin install @` for each required plugin to download it. Alternatively, run the pipeline once, which will automatically download all plugins required by the pipeline. 3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. + 4. Specify each plugin and its version in Nextflow configuration files or at runtime. See {ref}`using-plugins-config` for more information. :::{warning} Nextflow will attempt to download newer versions of plugins if their versions are not set. See {ref}`using-plugins-identifiers` for more information. - ::: \ No newline at end of file + ::: diff --git a/docs/reference/stdlib-groovy.md b/docs/reference/stdlib-groovy.md index 2d0bb83c8a..add202a445 100644 --- a/docs/reference/stdlib-groovy.md +++ b/docs/reference/stdlib-groovy.md @@ -21,5 +21,5 @@ println groovy.json.JsonOutput.toJson(vals) ``` :::{note} -The set of classes in Nextflow's runtime classpath can change between different Nextflow versions. As a best practice, any code that uses classes outside the Nextflow standard library should either be refactored to only use the Nextflow standard library or be refactored as a {ref}`plugin ` with explicit dependencies. +The set of classes in Nextflow's runtime classpath can change between different Nextflow versions. As a best practice, any code that uses classes outside the Nextflow standard library should either be refactored to only use the Nextflow standard library or be refactored as a {ref}`plugin ` with explicit dependencies. ::: From e153d962808f8a5fb5fb91f99d3b91fdb1820007 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 7 May 2025 11:34:22 -0500 Subject: [PATCH 11/29] Remove registry URL Signed-off-by: Ben Sherman --- docs/migrating-gradle-plugin.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 87187281f2..3280d36b62 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -83,9 +83,8 @@ To migrate an existing Nextflow plugin: '' ] - publishing { + publishing { registry { - url = 'https://nf-plugins-registry.dev-tower.net/api' authToken = project.findProperty('pluginRegistry.accessToken') } } From 1be033668dd4f4b9f3557b9b36f1f0b2aad187da Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 7 May 2025 11:43:49 -0500 Subject: [PATCH 12/29] cleanup migration guide Signed-off-by: Ben Sherman --- docs/migrating-gradle-plugin.md | 38 +++++++++++++----------------- docs/plugins/developing-plugins.md | 2 +- docs/reference/cli.md | 2 +- 3 files changed, 18 insertions(+), 24 deletions(-) diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 3280d36b62..ab07a0ce01 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -1,47 +1,40 @@ (migrating-plugin-page)= -# Migrating to the Nextflow Gradle plugin +# Migrating to the Nextflow plugin registry -This page introduces the Nextflow Gradle plugin, the Nextflow plugin registry, and how to migrate to the new plugin framework. +The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. +## Overview -## Improvements to the plugin framework +### Nextflow plugin registry -The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying plugin development, streamlining publishing and discovery, and improving how plugins are loaded into workflows. These improvements make plugins more accessible, maintainable, and interoperable with Nextflow. - -### Nextflow Gradle plugin - -The Nextflow Gradle plugin simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. - -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. - -### Nextflow Plugin Registry - -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. +The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. It is intended as a replacement for the [plugins index](https://github.com/nextflow-io/plugins) hosted on GitHub. :::{note} -The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. ::: -## Impact on users and developers +### Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. -The impact of the Nextflow Gradle plugin differs for plugin users and developers. +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. -### Plugin Users +## Impact on plugin users If you are a plugin user, no immediate actions are required. The plugin configuration has not changed. -### Plugin developers +## Impact on plugin developers -Developers are encouraged to migrate to the Nextflow Gradle plugin and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. +Developers are encouraged to migrate to the Nextflow Gradle plugin in order to take advantage of the simplified development and publishing process. To migrate an existing Nextflow plugin: 1. Remove the following files and folders: - `buildSrc/` - - `nextflow.config` - `launch.sh` - `plugins/build.gradle` + 2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. :::{note} @@ -95,7 +88,7 @@ To migrate an existing Nextflow plugin: - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.03.0-edge`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.04.0`. - `PROVIDER`: Your name or organization—for example, `acme`. - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. @@ -127,6 +120,7 @@ To migrate an existing Nextflow plugin: ``` 6. Update `README.md` with information about the structure of your plugin. + 7. In the plugin root directory, run `make assemble`. The Nextflow Gradle plugin also supports publishing plugins. See {ref}`gradle-plugin-publish` for more information. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 470983506b..ddce06d875 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -68,7 +68,7 @@ See {ref}`nf-hello-page` for an example of a plugin built with the plugin templa ### Nextflow Gradle plugin -The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by providing default configuration, Nextflow dependencies, and custom Gradle tasks for building and publishing plugins. +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. It is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), and can be declared and managed like any other dependency in the `build.gradle` file: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 758f9b20d2..911bb0390b 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -996,7 +996,7 @@ $ nextflow plugin [options] : :::{versionadded} 25.04.0 ::: -: Create a plugin scaffold using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. +: Create a plugin scaffold using the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. `install ` From 7bfc78da9b4659cdd069ba6567616c1b05552016 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 7 May 2025 12:06:01 -0500 Subject: [PATCH 13/29] cleanup gradle plugin guide Signed-off-by: Ben Sherman --- docs/gradle-plugin.md | 22 ++++++++++++---------- docs/plugins/developing-plugins.md | 6 +++--- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 4ee7e08366..90701d7f8d 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -1,6 +1,6 @@ (gradle-plugin-page)= -# Using the Gradle plugin +# Using the Nextflow Gradle plugin The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. @@ -15,21 +15,17 @@ Nextflow plugins can be developed without the Gradle plugin. However, this appro :::{versionadded} 25.04.0 ::: -The easiest way to boostrap a Nextflow plugin based on the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). +The easiest way to get started with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which in turn uses the Gradle plugin. To create a Nextflow plugin with the Gradle plugin, simply run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. -See {ref}`dev-plugins-extension` for more information about developing a plugin. - -1. Run `nextflow plugin create`. -2. Develop your plugin extension points. -3. In the plugin root directory, run `make assemble`. +See {ref}`dev-plugins-extension` for more information about available plugin extension points. ## Building a plugin -Plugins can be installed locally without being published. +To build a plugin, simply run `make assemble`. -To install a plugin locally: +Plugins can also be installed locally without being published. To install a plugin locally: 1. In the plugin root directory, run `make install`. @@ -51,7 +47,9 @@ To install a plugin locally: ## Testing a plugin -Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. +### Unit tests + +Unit tests are small, focused tests designed to verify the behavior of individual plugin components. To run unit tests: @@ -59,6 +57,10 @@ To run unit tests: 2. In the plugin root directory, run `make test`. +### End-to-end tests + +End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. See [nf-hello](https://github.com/nextflow-io/nf-hello) for an example of an end-to-end test for a Nextflow plugin. + (gradle-plugin-publish)= ## Publishing a plugin diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index ddce06d875..ecaa9740be 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -97,15 +97,15 @@ The following `make` commands are available: `assemble` : Compiles the Nextflow plugin code and assembles it into a zip archive. -`test` -: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. - `install` : Installs the plugin into the local Nextflow plugins directory. `release` : Publishes the plugin. See {ref}`gradle-plugin-publish` for more information. +`test` +: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. + (dev-plugins-extension)= ## Extension points From fa2dc0fb2e4da8c9aa4aec31ef7b90a52b0ae4c0 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Thu, 8 May 2025 10:09:45 +1200 Subject: [PATCH 14/29] Unlist h3 headings Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 90701d7f8d..eaef8af07e 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -47,7 +47,7 @@ Plugins can also be installed locally without being published. To install a plug ## Testing a plugin -### Unit tests +

Unit tests

Unit tests are small, focused tests designed to verify the behavior of individual plugin components. @@ -57,7 +57,7 @@ To run unit tests: 2. In the plugin root directory, run `make test`. -### End-to-end tests +

End-to-end tests

End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. See [nf-hello](https://github.com/nextflow-io/nf-hello) for an example of an end-to-end test for a Nextflow plugin. From 13fba9c2e43776dfb40b29278316bd3f3fc78489 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Fri, 9 May 2025 10:50:03 +1200 Subject: [PATCH 15/29] Update example --- docs/plugins/example-nf-hello.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 99ca9984fc..2d9babde42 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -1,6 +1,6 @@ -(nf-hello-page)= +(nf-template-page)= -# Example: nf-hello +# Example: nf-plugin-template [nf-hello](https://github.com/nextflow-io/nf-hello) is a simple Nextflow plugin that uses the Nextflow Gradle plugin and provides examples for various plugin extension points. From fedced636e0d4e1f4abab71bd8b216e7a9f4c72f Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Thu, 24 Apr 2025 19:57:55 +1200 Subject: [PATCH 16/29] Update plugins docs Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 134 ++++++++ docs/index.md | 13 + docs/migrating-gradle-plugin.md | 134 ++++++++ docs/plugins/developing-plugins.md | 489 +++++++++++++++++++++++++++++ docs/plugins/example-nf-hello.md | 52 +++ docs/plugins/plugins.md | 67 ++++ docs/plugins/using-plugins.md | 78 +++++ 7 files changed, 967 insertions(+) create mode 100644 docs/gradle-plugin.md create mode 100644 docs/migrating-gradle-plugin.md create mode 100644 docs/plugins/developing-plugins.md create mode 100644 docs/plugins/example-nf-hello.md create mode 100644 docs/plugins/plugins.md create mode 100644 docs/plugins/using-plugins.md diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md new file mode 100644 index 0000000000..31f341f0ee --- /dev/null +++ b/docs/gradle-plugin.md @@ -0,0 +1,134 @@ +# Using the Gradle plugin + +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. + +:::{note} +Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +## Creating a plugin + +The nf-hello plugin uses the Gradle plugin and is a valuable starting point for developers. + +To create a Nextflow plugin with the Gradle plugin: + +1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. + * See Example: nf-hello for more information about the nf-hello plugin. +2. Rename the forked `nf-hello` directory with your plugin name. +3. Replace the contents of `settings.gradle` with the following: + + ```groovy + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. Replace the contents of `build.gradle` with the following: + + ```groovy + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + github { + repository = '' + userName = project.findProperty('github_username') + authToken = project.findProperty('github_access_token') + email = project.findProperty('github_commit_email') + indexUrl = '' + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (optional) your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` your plugin version—for example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: the minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. + - `PROVIDER`: your name or organization—for example, `nextflow`. + - `CLASS_NAME`: your plugin class name—for example, `nextflow.hello.HelloPlugin`. + - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. + - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. + - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). +5. Develop your plugin extension points: + - See Extension points for descriptions and examples. +6. In the plugin root directory, run `make assemble`. + + +## Installing a plugin + +Plugins can be installed locally without being packaged, uploaded, and published. + +To install a plugin locally: + +1. In the plugin root directory, run `make install`. + + :::{note} + Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. + ::: + +2. Configure your plugin. + * See Using plugins for more information. +3. Run your pipeline: + + ```bash + nextflow run main.nf + ``` + +## Unit testing a plugin + +Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. + +To run unit tests: + +1. Develop your unit tests. + * See HelloDslTest.groovy in the nf-hello plugin for unit test examples. +2. In the plugin root directory, run `make test`. + + +### Packaging, uploading, and publishing a plugin + +The Gradle plugin for Nextflow plugins simplifies publishing your plugin. + +To package, upload, and publish your plugin: + +1. Fork the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins). +2. In the plugin root directory, open `build.gradle` and ensure that: + * `github.repository` matches the plugin repository. + * `github.indexUrl` matches your fork of the plugins index repository. +3. Create a file named `$HOME/.gradle/gradle.properties` and add the following: + + ```bash + github_username= + github_access_token= + github_commit_email= + ``` + + Replace the following: + * `GITHUB_USERNAME`: your GitHub username granting access to the plugin repository. + * `GITHUB_ACCESS_TOKEN`: your GitHub access token with permission to upload and commit changes to the plugin repository. + * `GITHUB_EMAIL`: your email address associated with your GitHub account. +4. Run `make release`. +5. Create a pull request against the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins) from your fork. \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index aabef901a4..4ab2b430c5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -157,6 +157,17 @@ developer/packages developer/plugins ``` +```{toctree} +:hidden: +:caption: Plugins +:maxdepth: 1 + +plugins/plugins +plugins/using-plugins +plugins/developing-plugins +plugins/example-nf-hello +``` + ```{toctree} :hidden: :caption: Tutorials @@ -175,4 +186,6 @@ tutorials/flux guides/aws-java-sdk-v2 guides/updating-spot-retries +gradle-plugin +migrating-gradle-plugin ``` diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md new file mode 100644 index 0000000000..b424ff4233 --- /dev/null +++ b/docs/migrating-gradle-plugin.md @@ -0,0 +1,134 @@ +# Migrating to the Gradle plugin for Nextflow plugins + +This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. + + +## Improvements to the plugin framework + +The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying plugin development, streamlining publishing and discovery, and improving how plugins are loaded into workflows. These improvements make plugins more accessible, maintainable, and interoperable with Nextflow. + +### Gradle plugin for Nextflow plugins + +The Gradle plugin for Nextflow plugins simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. + +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. + +### Nextflow plugin registry + +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. + +Developers can upload plugins to the plugin registry. The Gradle plugin for Nextflow plugins supports this process with Gradle tasks that package, upload, and publish plugins. + +## Impact on users and developers + +The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. + +### Plugin Users + +If you are a plugin user, no immediate actions are required. The plugin configuration has not changed. + +### Plugin developers + +Developers are encouraged to migrate to the Gradle plugin for Nextflow plugins and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. + +To migrate an existing Nextflow plugin: + +1. Remove the following files and folders: + - `buildSrc/` + - `nextflow.config` + - `launch.sh` + - `plugins/build.gradle` +2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. \ + + :::{note} + Plugin sources should be in `src/main/groovy` or `src/main/java`. + ::: + +3. Replace the contents of `settings.gradle` with the following: + + ``` + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. In the project root, create a new `build.gradle` file with the following configuration: + + ``` + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + github { + repository = '' + userName = project.findProperty('github_username') + authToken = project.findProperty('github_access_token') + email = project.findProperty('github_commit_email') + indexUrl = '' + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. + - `PROVIDER`: Your name or organization—for example, `nextflow`. + - `CLASS_NAME`: Your plugin class name—for example, `nextflow.hello.HelloPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. + - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. + - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). + +5. Replace the contents of `Makefile` with the following: + + ``` + # Build the plugin + assemble: + ./gradlew assemble + + clean: + rm -rf .nextflow* + rm -rf work + rm -rf build + ./gradlew clean + + # Run plugin unit tests + test: + ./gradlew test + + # Install the plugin into local nextflow plugins dir + install: + ./gradlew install + + # Publish the plugin + release: + ./gradlew releasePlugin + ``` + +6. Update `README.md` with information about the structure of your plugin. +7. In the plugin root directory, run `make assemble`. + +The Gradle plugin for Nextflow plugins also supports publishing plugins. See Packaging, uploading, and publishing for more information. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md new file mode 100644 index 0000000000..07b3c3fe2a --- /dev/null +++ b/docs/plugins/developing-plugins.md @@ -0,0 +1,489 @@ +(developing-plugins-page)= + +# Developing plugins + +The Nextflow plugin framework streamlines plugin development by providing the structure and tools needed to extend Nextflow functionality. The Gradle plugin for Nextflow plugins simplifies development by configuring default Nextflow dependencies and incorporates custom Gradle tasks that streamline building and publishing plugins. + +:::{note} +Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +(developing-plugins-framework)= + +## Framework + +Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. + +(developing-plugins-gradle)= + +## Gradle plugin for Nextflow plugins + +[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. + +(developing-plugins-structure)= + +### Structure + +Plugins built with the Gradle plugin for Nextflow plugins follow a standard project layout. This structure includes source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. + +A typical plugin built with the Gradle plugin for Nextflow plugins has the following structure: + +``` +. +├── COPYING +├── Makefile +├── README.md +├── build.gradle +├── gradle +│ └── wrapper +│ └── ... +├── gradlew +├── settings.gradle +└── src + ├── main + │ └── ... + └── test + └── ... +``` + +This structure contains the following key files and folders: + +- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. +- `src/main/`: The main source directory containing the plugin's implementation and resources.​ +- `src/test/`: The main source directory containing the plugin's unit testing. +- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ +- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ +- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ +- `build.gradle`: The primary build script for Gradle. +- `gradlew`: A script for executing the Gradle Wrapper. +- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. + +See {ref}`nf-hello-page` for an example of a plugin built using this structure. + +:::{note} +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. See nf-hello for an example of a plugin with the old structure. +::: + +(developing-plugins-make)= + +### make commands + +The Gradle plugin for Nextflow plugins defines tasks that can be executed with `./gradlew`. For example: + +```bash +./gradlew assemble +``` + +For convenience, the more important tasks are wrapped in a makefile and can be executed with the `make` command. For example: + +```bash +make assemble +``` + +The following `make` commands are available: + +`assemble` +: Compiles the Nextflow plugin code and assembles it into a zip file. See Creating a plugin for more information. + +`test` +: Runs plugin unit tests. See Unit testing for more information. + +`install` +: Installs the plugin into the local nextflow plugins directory. See Running locally for more information. + +`release` +: Publishes the plugin. See Packaging, uploading, and publishing for more information. + +(developing-plugins-versioning)= + +### Versioning + +The Gradle plugin is versioned and published to a [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: + +```nextflow +plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' +} +``` + +See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. + +(developing-plugins-extension)= + +## Extension points + +Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how they can be used. + +### Commands + +Plugins can define custom CLI commands that are executable with the `nextflow plugin` command. + +To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entry point (the class that extends `BasePlugin`). Alternatively, implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement the `getCommands()` and `exec()` methods. For example: + +```groovy +import nextflow.cli.PluginAbstractExec +import nextflow.plugin.BasePlugin + +class MyPlugin extends BasePlugin implements PluginAbstractExec { + @Override + List getCommands() { + [ 'hello' ] + } + + @Override + int exec(String cmd, List args) { + if( cmd == 'hello' ) { + println "Hello! You gave me these arguments: ${args.join(' ')}" + return 0 + } + else { + System.err.println "Invalid command: ${cmd}" + return 1 + } + } +} +``` + +The command can be run using the `nextflow plugin` command: + +``` +nextflow plugin my-plugin:hello --foo --bar +``` + +See the {ref}`cli-plugin` for usage information. + +### Configuration + +Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option safely. If the option isn’t defined, it will return null. + +A common practice is to use a custom config scope to define any configuration for your plugin. For example: + +```groovy +import nextflow.Session +import nextflow.trace.TraceObserver + +class MyObserver implements TraceObserver { + + @Override + void onFlowCreate(Session session) { + final message = session.config.navigate('myplugin.createMessage') + println message + } +} +``` + +This option can then be set in your configuration file: + +```groovy +// dot syntax +myplugin.createMessage = "I'm alive!" + +// block syntax +myplugin { + createMessage = "I'm alive!" +} +``` + +:::{versionadded} 25.02.0-edge +::: + +Plugins can declare their configuration options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation. For example: + +```groovy +import nextflow.config.schema.ConfigOption +import nextflow.config.schema.ConfigScope +import nextflow.config.schema.ScopeName +import nextflow.script.dsl.Description + +@ScopeName('myplugin') +@Description(''' + The `myplugin` scope allows you to configure the `nf-myplugin` plugin. +''') +class MyPluginConfig implements ConfigScope { + + MyPluginConfig(Map opts) { + this.createMessage = opts.createMessage + } + + @ConfigOption + @Description('Message to print to standard output when a run is initialized.') + String createMessage +} +``` + +This approach is not required to support plugin config options. However, it allows Nextflow to recognize plugin definitions when validating the configuration. + +### Executors + +Plugins can define custom executors that can be used with the `executor` process directive. + +To implement an executor, create a class in your plugin that extends the [Executor](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: + +```groovy +import nextflow.executor.Executor +import nextflow.util.ServiceName +import org.pf4j.ExtensionPoint + +@ServiceName('my-executor') +class MyExecutor extends Executor implements ExtensionPoint { + + // ... + +} +``` + +You can then use this executor in your pipeline: + +```groovy +process foo { + executor 'my-executor' + + // ... +} +``` + +:::{tip} +See the source code of Nextflow's built-in executors for examples of how to implement various executor components. +::: + + +### Filesystems + +Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see Remote files. + +To implement a custom filesystem, create a class in your plugin that extends [FileSystemProvider](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: + +```groovy +import java.nio.file.spi.FileSystemProvider + +class MyFileSystemProvider extends FileSystemProvider { + + @Override + String getScheme() { + return 'myfs' + } + + // ... +} +``` + +You can then use this filesystem in your pipeline: + +```nextflow +input = file('myfs://') +``` + +See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for examples of custom filesystems. + +:::{tip} +Custom filesystems are an advanced plugin extension. Before creating a new filesystem, check that your use case cannot already be supported by an existing filesystem such as HTTP or S3. +::: + +### Functions + +:::{versionadded} 22.09.0-edge +::: + +Plugins can define custom functions that can be included in Nextflow pipelines. + +To implement a custom function, create a plugin class that extends the `PluginExtensionPoint` class and implement your function with the `Function` annotation. For example: + +```groovy +import nextflow.Session +import nextflow.plugin.extension.Function +import nextflow.plugin.extension.PluginExtensionPoint + +class MyExtension extends PluginExtensionPoint { + + @Override + void init(Session session) {} + + @Function + String reverseString(String origin) { + origin.reverse() + } +} +``` + +You can then add this function to your pipeline: + +``` +include { reverseString } from 'plugin/my-plugin' + +channel.of( reverseString('hi') ) +``` + +Alternatively, you can use an alias: + +```nextflow +include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' +``` + +### Operators + +:::{versionadded} 22.04.0 +::: + +Plugins can define custom channel factories and operators that can then be included in pipelines. + +To implement a custom channel factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class and implement your function with the `Factory` or `Operator` annotation. For example: + +```groovy +import groovyx.gpars.dataflow.DataflowReadChannel +import groovyx.gpars.dataflow.DataflowWriteChannel +import nextflow.Session +import nextflow.plugin.extension.Factory +import nextflow.plugin.extension.Operator +import nextflow.plugin.extension.PluginExtensionPoint + +class MyExtension extends PluginExtensionPoint { + + @Override + void init(Session session) {} + + @Factory + DataflowWriteChannel fromQuery(Map opts, String query) { + // ... + } + + @Operator + DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) { + // ... + } + +} +``` + +You can then use the custom channel factories or operators in your pipeline: + +```nextflow +include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' + +def sql = 'select * from FOO' +channel + .fromTable(sql, db: 'test', emitColumns: true) + .sqlInsert(into: 'BAR', columns: 'id', db: 'test') +``` + +:::{note} +The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +::: + +### Process directives + +Plugins that implement a custom executor will likely need to access process directives that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. + +Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the ext directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. + +For example, a custom executor can use existing process directives and a custom setting through the `ext` directive: + +```groovy +class MyExecutor extends Executor { + + @Override + TaskHandler createTaskHandler(TaskRun task) { + final cpus = task.config.cpus + final memory = task.config.memory + final myOption = task.config.ext.myOption + + println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" + + // ... + } + + // ... + +} +``` + +### Trace observers + +A *trace observer* is an entity that can listen and react to workflow events, such as when a workflow starts, a task is completed, or a file is published. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. + +Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver` and implement the `create()` method in your observer factory. For example: + +```groovy +import java.nio.file.Path + +import nextflow.processor.TaskHandler +import nextflow.trace.TraceObserver +import nextflow.trace.TraceRecord + +class MyObserver implements TraceObserver { + + @Override + void onFlowBegin() { + println "Okay, let's begin!" + } + + @Override + void onProcessComplete(TaskHandler handler, TraceRecord trace) { + println "I completed a task! It's name is '${handler.task.name}'" + } + + @Override + void onProcessCached(TaskHandler handler, TraceRecord trace) { + println "I found a task in the cache! It's name is '${handler.task.name}'" + } + + @Override + void onFilePublish(Path destination, Path source) { + println "I published a file! It's located at ${path.toUriString()}" + } + + @Override + void onFlowError(TaskHandler handler, TraceRecord trace) { + println "Uh oh, something went wrong..." + } + + @Override + void onFlowComplete() { + println 'All done!' + } +} +``` + +You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option: + +```nextflow +myplugin.enabled = true +``` + +See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. + + +## Environment variables + +The following environment variables are available when developing and testing plugins: + +`NXF_PLUGINS_MODE` +: The plugin execution mode. Either `prod` for production or `dev` for development. + +`NXF_PLUGINS_DIR` +: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production and `./plugins` in development). + +`NXF_PLUGINS_DEFAULT` +: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: true). + + +`NXF_PLUGINS_DEV` +: Comma-separated list of development plugin root directories. + +`NXF_PLUGINS_TEST_REPOSITORY` +: :::{versionadded} 23.04.0. + ::: +: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry. + +: The URI should refer to a plugin repository JSON file or a specific plugin JSON meta file. In the latter case, it should match the pattern `https://host.name/some/path/-X.Y.Z-meta.json`. For example: + + ```bash + # custom plugin repository at https://github.com/my-org/plugins + export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json" + + # custom plugin release + export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json" + + nextflow run main.nf -plugins nf-hello + ``` + +: This variable is useful for testing a plugin release before publishing it to the main registry. diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md new file mode 100644 index 0000000000..0b96fb4d07 --- /dev/null +++ b/docs/plugins/example-nf-hello.md @@ -0,0 +1,52 @@ +(nf-hello-page)= + +# Example: nf-hello + +`nf-hello` is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. + +The nf-hello plugin has the following structure: + +``` +nf-hello +├── COPYING +├── Makefile +├── README.md +├── build.gradle +├── gradle +│ └── wrapper +│ ├── gradle-wrapper.jar +│ └── gradle-wrapper.properties +├── gradlew +├── settings.gradle +└── src + ├── main + │ └── groovy + │ └── nextflow + │ └── hello + │ ├── HelloConfig.groovy + │ ├── HelloExtension.groovy + │ ├── HelloFactory.groovy + │ ├── HelloObserver.groovy + │ └── HelloPlugin.groovy + └── test + └── groovy + └── nextflow + └── hello + ├── HelloDslTest.groovy + └── HelloFactoryTest.groovy +``` + +It includes examples of different plugin extensions: + +- A custom trace observer that prints a message when the workflow starts and when the workflow completes. +- A custom channel factory called reverse. +- A custom operator called goodbye. +- A custom function called randomString. + +It also includes several classes that demonstrate different plugin functionality: + +- `HelloConfig`: An example of how to handle options from the Nextflow configuration. +- `HelloExtension`: An example of how to create custom channel factories, operators, and functions that can be included in pipeline scripts. +- `HelloFactory`: An example of a workflow event with custom behavior. +- `HelloObserver`: An example of a workflow event with custom behavior. +- `HelloPlugin`: An example of a plugin entry point. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md new file mode 100644 index 0000000000..259d1e9e0c --- /dev/null +++ b/docs/plugins/plugins.md @@ -0,0 +1,67 @@ +(plugins-page)= + +# Overview + +## What are plugins + +Nextflow plugins are extensions that enhance the functionality of the Nextflow workflow framework. They allow users to add new capabilities and integrate with external services without modifying core Nextflow code. + +There are two types of Nextflow plugins: + +* Core plugins +* Third-party plugins + +The main features of each plugin type are described below. + +

Core plugins

+ +Core plugins do not require configuration. The latest versions of core plugins are automatically installed when a Nextflow pipeline requests them. Core plugins include: + +* `nf-amazon`: Support for Amazon Web Services. +* `nf-azure`: Support for Microsoft Azure. +* `nf-cloudcache`: Support for the cloud cache. +* `nf-console`: Implement Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). +* `nf-google`: Support for Google Cloud. +* `nf-tower`: Support for [Seqera Platform](https://seqera.io/platform/). +* `nf-wave`: Support for [Wave containers service](https://seqera.io/wave/). + +Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. + +:::{note} +The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See Environment variables for more information. +::: + +

Third-party plugins

+ +Third-party plugins must be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: + +``` +plugins { + id 'nf-hello@0.5.0' +} +``` + +To configure plugins at runtime, use the `-plugins` option. For example: + +``` +nextflow run main.nf -plugins nf-hello@0.5.0 +``` + +See {ref}`using-plugins-page` for more information. + +## Nextflow plugin registry + +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. Developers can upload plugins to the registry, with built-in support from the Gradle plugin for Nextflow plugins. + +## Versioning + +Nextflow plugins follow the Semantic Versioning format: MAJOR.MINOR.PATCH (e.g., 0.5.0). This helps developers communicate the nature of changes in a release. + +Version components: + +* MAJOR: Increment for incompatible changes. +* MINOR: Increment for backward-compatible feature additions. +* PATCH: Increment for backward-compatible bug fixes. + +Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. +See Semantic Versioning for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md new file mode 100644 index 0000000000..8aef78fc7e --- /dev/null +++ b/docs/plugins/using-plugins.md @@ -0,0 +1,78 @@ +# Using plugins + +Nextflow core plugins require no additional configuration. When a pipeline uses a core plugin, Nextflow automatically downloads and uses the latest compatible plugin version. In contrast, third-party plugins must be explicitly declared. When a pipeline uses one or more third-party plugins, Nextflow must be configured to download and use the plugin. + +## Identifiers + +A plugin identifier consists of the plugin name and version, separated by an `@` symbol: + +```console +nf-hello@0.5.0 +``` + +The plugin version is optional. If it is not specified, Nextflow will download the latest version of the plugin that meets the minimum Nextflow version requirements specified by the plugin. + +:::{note} +Plugin versions are required to run plugins in an Offline environment. +::: + +:::{versionadded} 25.02.0-edge. +::: + +The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. + +:::{tip} +It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. +::: + +## Configuration + +Plugins can be configured via nextflow configuration files or at runtime. + +To configure a plugin via configuration files, use the `plugins` block. For example: + +```nextflow +plugins { + id 'nf-hello@0.5.0' +} +``` + +To configure plugins at runtime, use the `-plugins` option. For example: + +```bash +nextflow run main.nf -plugins nf-hello@0.5.0 +``` + +To configure multiple plugins at runtime, use the `-plugins` option and a comma-separated list. For example: + +```bash +nextflow run main.nf -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +``` + +:::{note} +Plugin declarations in nextflow configuration files are ignored when specifying plugins via the `-plugins` option. +::: + +## Caching + +When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. + +## Offline usage + +Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. + +To use Nextflow plugins in an offline environment: + +1. Install a self-contained version of Nextflow in an environment with an internet connection. See Standalone distribution for more information. +2. Run `nextflow plugin install @` to download the plugins. + + :::{tip} + Running a pipeline in an environment with an internet connection will also download the plugins used by that pipeline. + ::: + +3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. +4. Specify each plugin and its version in Nextflow configuration files or at runtime. See Configuration for more information. + + :::{warning} + Nextflow will attempt to download newer versions of plugins if their versions are not set. See Plugin identifiers for more information. + ::: \ No newline at end of file From 6a931849cfb87f1d1980783fc9a226acc2d682dd Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 28 Apr 2025 09:36:15 +1200 Subject: [PATCH 17/29] Add run command example Signed-off-by: Christopher Hakkaart --- docs/plugins/example-nf-hello.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 0b96fb4d07..689b5f0a69 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -50,3 +50,9 @@ It also includes several classes that demonstrate different plugin functionality - `HelloFactory`: An example of a workflow event with custom behavior. - `HelloObserver`: An example of a workflow event with custom behavior. - `HelloPlugin`: An example of a plugin entry point. + +The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: + +```bash +nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +``` From 23d641b5b69f2391be97281922341f12821eb7d4 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 28 Apr 2025 13:40:43 +1200 Subject: [PATCH 18/29] Update links Signed-off-by: Christopher Hakkaart --- docs/developer/plugins.md | 550 ----------------------------- docs/gradle-plugin.md | 24 +- docs/index.md | 2 - docs/migrating-gradle-plugin.md | 6 +- docs/plugins.md | 63 ---- docs/plugins/developing-plugins.md | 54 +-- docs/plugins/example-nf-hello.md | 4 +- docs/plugins/plugins.md | 4 +- docs/plugins/using-plugins.md | 16 +- docs/reference/env-vars.md | 2 +- docs/strict-syntax.md | 2 +- 11 files changed, 67 insertions(+), 660 deletions(-) delete mode 100644 docs/developer/plugins.md delete mode 100644 docs/plugins.md diff --git a/docs/developer/plugins.md b/docs/developer/plugins.md deleted file mode 100644 index 24363bb75c..0000000000 --- a/docs/developer/plugins.md +++ /dev/null @@ -1,550 +0,0 @@ -(plugins-dev-page)= - -# Plugins - -This page describes how to create, test, and publish third-party plugins. - -## Plugin structure - -The best way to get started with your own plugin is to refer to the [nf-hello](https://github.com/nextflow-io/nf-hello) repository. This repository provides a minimal plugin implementation with several examples of different extension points and instructions for building, testing, and publishing. - -Plugins can be written in Java or Groovy. - -The minimal dependencies are as follows: - -```groovy -dependencies { - compileOnly project(':nextflow') - compileOnly 'org.slf4j:slf4j-api:1.7.10' - compileOnly 'org.pf4j:pf4j:3.4.1' - - testImplementation project(':nextflow') - testImplementation "org.codehaus.groovy:groovy:4.0.24" - testImplementation "org.codehaus.groovy:groovy-nio:4.0.23" -} -``` - -The plugin subproject directory name must begin with the prefix `nf-` and must include a file named `src/resources/META-INF/MANIFEST.MF` which contains the plugin metadata. The manifest content looks like the following: - -``` -Manifest-Version: 1.0 -Plugin-Class: the.plugin.ClassName -Plugin-Id: the-plugin-id -Plugin-Provider: Your Name or Organization -Plugin-Version: 0.0.0 -``` - -## Extension points - -Nextflow's plugin system exposes a variety of extension points for plugins. This section describes how to use these extension points when writing a plugin, as well as how they are used in a pipeline. - -:::{note} -If you would like to implement something in a plugin that isn't covered by any of the following sections, feel free to create an issue on GitHub and describe your use case. In general, any class in the Nextflow codebase that implements `ExtensionPoint` can be extended by a plugin, and existing plugins are a great source of examples when writing new plugins. -::: - -:::{note} -Plugin extension points must be added to `extensions.idx` in the plugin repository to make them discoverable. See the [`nf-hello`](https://github.com/nextflow-io/nf-hello) plugin for an example. -::: - -### Commands - -Plugins can define custom CLI commands that can be executed with the `nextflow plugin` command. - -To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entrypoint (the class that extends `BasePlugin`). Alternatively, you can implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement two methods, `getCommands()` and `exec()`: - -```groovy -import nextflow.cli.PluginAbstractExec -import nextflow.plugin.BasePlugin - -class MyPlugin extends BasePlugin implements PluginAbstractExec { - @Override - List getCommands() { - [ 'hello' ] - } - - @Override - int exec(String cmd, List args) { - if( cmd == 'hello' ) { - println "Hello! You gave me these arguments: ${args.join(' ')}" - return 0 - } - else { - System.err.println "Invalid command: ${cmd}" - return 1 - } - } -} -``` - -You can then execute this command using the `nextflow plugin` command: - -```bash -nextflow plugin my-plugin:hello --alpha --beta -``` - -See the {ref}`cli-plugin` CLI command for usage information. - -### Configuration - -Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option in a safe manner -- if the option isn't defined, it will return `null`. A common practice is to define any configuration for your plugin in a custom config scope. - -For example, you can query a config option in a trace observer hook: - -```groovy -import nextflow.Session -import nextflow.trace.TraceObserver - -class MyObserver implements TraceObserver { - - @Override - void onFlowCreate(Session session) { - final message = session.config.navigate('myplugin.createMessage') - println message - } -} -``` - -You can then set this option in your config file: - -```groovy -// dot syntax -myplugin.createMessage = "I'm alive!" - -// block syntax -myplugin { - createMessage = "I'm alive!" -} -``` - -:::{versionadded} 25.02.0-edge -::: - -Plugins can declare their config options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation: - -```groovy -import nextflow.config.schema.ConfigOption -import nextflow.config.schema.ConfigScope -import nextflow.config.schema.ScopeName -import nextflow.script.dsl.Description - -@ScopeName('myplugin') -@Description(''' - The `myplugin` scope allows you to configure the `nf-myplugin` plugin. -''') -class MyPluginConfig implements ConfigScope { - - /* required by extension point -- do not remove */ - MyPluginConfig() {} - - MyPluginConfig(Map opts) { - this.createMessage = opts.createMessage - } - - @ConfigOption - @Description('Message to print to standard output when a run is initialized.') - String createMessage -} -``` - -While this approach is not required to support plugin config options, it allows Nextflow to recognize plugin definitions when validating the config. See {ref}`config-scopes-page` for more information. - -### Executors - -Plugins can define custom executors that can then be used with the `executor` process directive. - -To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor: - -```groovy -import nextflow.executor.Executor -import nextflow.util.ServiceName -import org.pf4j.ExtensionPoint - -@ServiceName('my-executor') -class MyExecutor extends Executor implements ExtensionPoint { - - // ... - -} -``` - -You can then use this executor in your pipeline: - -```nextflow -process hello { - executor 'my-executor' - - // ... -} -``` - -:::{tip} -Refer to the source code of Nextflow's built-in executors to see how to implement the various components of an executor. You might be able to implement most of your executor by simply reusing existing code. -::: - -### Filesystems - -Plugins can define custom filesystems that can be used by Nextflow to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. - -To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem: - -```groovy -import java.nio.file.spi.FileSystemProvider - -class MyFileSystemProvider extends FileSystemProvider { - - @Override - String getScheme() { - return 'myfs' - } - - // ... -} -``` - -You can then use this filesystem in your pipeline: - -```nextflow -input = file('myfs://path/to/input/file.txt') -``` - -See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information. Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. - -:::{tip} -Custom filesystems are an advanced type of plugin extension. Before creating a new filesystem, make sure that your use case can't already be supported through an existing filesystem such as HTTP or S3. -::: - -### Functions - -:::{versionadded} 22.09.0-edge -::: - -Plugins can define custom functions, which can then be included in Nextflow pipelines. - -To implement a custom function, create a class in your plugin that extends the `PluginExtensionPoint` class, and implement your function with the `Function` annotation: - -```groovy -import nextflow.Session -import nextflow.plugin.extension.Function -import nextflow.plugin.extension.PluginExtensionPoint - -class MyExtension extends PluginExtensionPoint { - - @Override - void init(Session session) {} - - @Function - String reverseString(String origin) { - origin.reverse() - } - -} -``` - -You can then use this function in your pipeline: - -```nextflow -include { reverseString } from 'plugin/my-plugin' - -channel.of( reverseString('hi') ) -``` - -You can also use an alias: - -```nextflow -include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' -``` - -### Operators - -:::{versionadded} 22.04.0 -::: - -Plugins can define custom channel factories and operators, which can then be included into Nextflow pipelines. - -To implement a custom factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class, and implement your function with the `Factory` or `Operator` annotation: - -```groovy -import groovyx.gpars.dataflow.DataflowReadChannel -import groovyx.gpars.dataflow.DataflowWriteChannel -import nextflow.Session -import nextflow.plugin.extension.Factory -import nextflow.plugin.extension.Operator -import nextflow.plugin.extension.PluginExtensionPoint - -class MyExtension extends PluginExtensionPoint { - - @Override - void init(Session session) {} - - @Factory - DataflowWriteChannel fromQuery(Map opts, String query) { - // ... - } - - @Operator - DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) { - // ... - } - -} -``` - -You can then use them in your pipeline: - -```nextflow -include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' - -def sql = 'select * from FOO' -channel - .fromTable(sql, db: 'test', emitColumns: true) - .sqlInsert(into: 'BAR', columns: 'id', db: 'test') -``` - -The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory -is included under the alias `fromTable`. - -### Process directives - -Plugins that implement a [custom executor](#executors) will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task's configuration. As a best practice, custom executors should try to support all process directives that have executor-specific behavior and are relevant to your executor. - -Nextflow does not provide the ability to define custom process directives in a plugin. Instead, you can use the {ref}`process-ext` directive to provide custom process settings to your executor. Try to use specific names that are not likely to conflict with other plugins or existing pipelines. - -For example, a custom executor can use existing process directives as well as a custom setting through the `ext` directive: - -```groovy -class MyExecutor extends Executor { - - @Override - TaskHandler createTaskHandler(TaskRun task) { - final cpus = task.config.cpus - final memory = task.config.memory - final myOption = task.config.ext.myOption - - println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" - - // ... - } - - // ... - -} -``` - -(plugins-trace-observers)= - -### Trace observers - -:::{versionchanged} 25.04 -The `TraceObserver` interface is now deprecated. Use [TraceObserverV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverV2.groovy) and [TraceObserverFactoryV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy) instead. -::: - -A *trace observer* in Nextflow is an entity that can listen and react to workflow events, such as when a workflow starts, a task completes, a file is published, etc. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. - -Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver`, and implement the `create()` method in your observer factory: - -```groovy -// MyObserverFactory.groovy -import nextflow.Session -import nextflow.trace.TraceObserver -import nextflow.trace.TraceObserverFactory - -class MyObserverFactory implements TraceObserverFactory { - - @Override - Collection create(Session session) { - final enabled = session.config.navigate('myplugin.enabled') - return enabled ? [ new MyObserver() ] : [] - } -} - -// MyObserver.groovy -import java.nio.file.Path - -import nextflow.processor.TaskHandler -import nextflow.trace.TraceObserver -import nextflow.trace.TraceRecord - -class MyObserver implements TraceObserver { - - @Override - void onFlowBegin() { - println "Okay, let's begin!" - } - - @Override - void onProcessComplete(TaskHandler handler, TraceRecord trace) { - println "I completed a task! It's name is '${handler.task.name}'" - } - - @Override - void onProcessCached(TaskHandler handler, TraceRecord trace) { - println "I found a task in the cache! It's name is '${handler.task.name}'" - } - - @Override - void onFilePublish(Path destination, Path source) { - println "I published a file! It's located at ${path.toUriString()}" - } - - @Override - void onFlowError(TaskHandler handler, TraceRecord trace) { - println "Uh oh, something went wrong..." - } - - @Override - void onFlowComplete() { - println 'All done!' - } -} -``` - -You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option: - -```groovy -myplugin.enabled = true -``` - -Refer to the `TraceObserver` [source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. - -## Publishing plugins - -Nextflow resolves plugins through a plugin registry, which stores metadata for each plugin version, including the publishing date, checksum, and download URL for the plugin binary. The default registry is located on GitHub at [nextflow-io/plugins](https://github.com/nextflow-io/plugins/), specifically [this file](https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json). - -To publish a plugin, you must create the plugin release, publish it to GitHub, and make a pull request against the main plugin registry with the metadata for your plugin release. - -A plugin release is a ZIP archive containing the compiled plugin classes, the required dependencies, and a JSON file with the plugin metadata. See the [`nf-hello`](https://github.com/nextflow-io/nf-hello) example to see how to create a plugin release. - -Here is an example meta file for a plugin release: - -```json -{ - "version": "0.2.0", - "url": "https://github.com/nextflow-io/nf-amazon/releases/download/0.2.0/nf-amazon-0.2.0.zip", - "date": "2020-10-12T10:05:44.28+02:00", - "sha512sum": "9e9e33695c1a7c051271..." -} -``` - -:::{note} -The plugin version should be a valid [semantic version](https://semver.org/). -::: - -(testing-plugins)= - -## Testing plugins - -Plugins must be declared in the `nextflow.config` file using the `plugins` scope, for example: - -```groovy -plugins { - id 'nf-amazon@0.2.0' -} -``` - -If a plugin is not locally available, Nextflow checks the repository index for the download URL, downloads and extracts the plugin archive, and installs the plugin into the directory specified by `NXF_PLUGINS_DIR` (default: `${NXF_HOME}/plugins`). - -Since each Nextflow run can have a different set of plugins (and versions), each run keeps a local plugins directory called `.nextflow/plr/` which links the exact set of plugins required for the given run. - -Additionally, the "default plugins" (defined in the Nextflow resources file `modules/nextflow/src/main/resources/META-INF/plugins-info.txt`) are always made available for use. To disable the use of default plugins, set the environment variable `NXF_PLUGINS_DEFAULT=false`. - -When running in development mode, the plugin system uses the `DevPluginClasspath` to load plugin classes from each plugin project build path, e.g. `plugins/nf-amazon/build/classes` and `plugins/nf-amazon/build/target/libs` (for deps libraries). - -## Environment variables - -The following environment variables are available when developing and testing plugins: - -`NXF_PLUGINS_MODE` -: The plugin execution mode, either `prod` for production or `dev` for development (see below for details). - -`NXF_PLUGINS_DIR` -: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production, `./plugins` in development). - -`NXF_PLUGINS_DEFAULT` -: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: `true`). - -`NXF_PLUGINS_DEV` -: Comma-separated list of development plugin root directories. - -`NXF_PLUGINS_TEST_REPOSITORY` -: :::{versionadded} 23.04.0 - ::: -: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry. -: The URI should refer to a plugins repository JSON file or a specific plugin JSON meta file. In the latter case it should match the pattern `https://host.name/some/path/-X.Y.Z-meta.json`. - -: For example: - - ```bash - # custom plugin repository at https://github.com/my-org/plugins - export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json" - - # custom plugin release - export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json" - - nextflow run -plugins nf-hello - ``` - -: This variable is useful for testing a plugin release before publishing it to the main registry. - -## Gradle Tasks - -The following build tasks are defined in the `build.gradle` of each core plugin as well as the `nf-hello` example plugin. - -### `makeZip` - -Create the plugin archive and JSON meta file in the subproject `build/libs` directory. - -```console -$ ls -l1 plugins/nf-tower/build/libs/ -nf-tower-0.1.0.jar -nf-tower-0.1.0.json -nf-tower-0.1.0.zip -``` - -### `copyPluginLibs` - -Copy plugin dependencies JAR files into the subproject `build/target/libs` directory. This is needed only when launching the plugin in development mode. - -### `copyPluginZip` - -Copy the plugin archive to the root project build directory, i.e. `build/plugins`. - -### `uploadPlugin` - -Upload the plugin archive and meta file to the corresponding GitHub repository. Options: - -`release` -: The plugin version, e.g. `1.0.1`. - -`repo` -: The GitHub repository name, e.g. `nf-amazon`. - -`owner` -: The GitHub owning organization, e.g. `nextflow-io`. - -`skipExisting` -: Do not upload a file if it already exists, i.e. checksum is the same (default: `true`). - -`dryRun` -: Execute the tasks without uploading file (default: `false`). - -`overwrite` -: Prevent to overwrite a remote file already existing (default: `false`). - -`userName` -: The user name used to authenticate GitHub API requests. - -`authToken` -: The personal token used to authenticate GitHub API requests. - -### `upload` - -Upload the plugin archive and meta file. - -### `publishIndex` - -Upload the plugins index to the repository hosted at [nextflow-io/plugins](https://github.com/nextflow-io/plugins), which makes them publicly accessible at [this URL](https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json). - -## Additional Resources - -* [PF4J](https://pf4j.org/) -* [Understanding Gradle: The Build Lifecycle](https://proandroiddev.com/understanding-gradle-the-build-lifecycle-5118c1da613f) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 31f341f0ee..2eb45aa150 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -1,3 +1,5 @@ +(gradle-plugin-page)= + # Using the Gradle plugin The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. @@ -6,14 +8,15 @@ The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: +(gradle-plugin-create)= + ## Creating a plugin -The nf-hello plugin uses the Gradle plugin and is a valuable starting point for developers. +The [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin uses the Gradle plugin and is a valuable starting point for developers. To create a Nextflow plugin with the Gradle plugin: -1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. - * See Example: nf-hello for more information about the nf-hello plugin. +1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. See {ref}`nf-hello-page` for more information. 2. Rename the forked `nf-hello` directory with your plugin name. 3. Replace the contents of `settings.gradle` with the following: @@ -72,10 +75,10 @@ To create a Nextflow plugin with the Gradle plugin: - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). -5. Develop your plugin extension points: - - See Extension points for descriptions and examples. +5. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. 6. In the plugin root directory, run `make assemble`. +(gradle-plugin-install)= ## Installing a plugin @@ -89,26 +92,27 @@ To install a plugin locally: Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. ::: -2. Configure your plugin. - * See Using plugins for more information. +2. Configure your plugin. See {ref}`using-plugins-page` for more information. 3. Run your pipeline: ```bash nextflow run main.nf ``` +(gradle-plugin-unit-test)= + ## Unit testing a plugin Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. To run unit tests: -1. Develop your unit tests. - * See HelloDslTest.groovy in the nf-hello plugin for unit test examples. +1. Develop your unit tests. See [HelloDslTest.groovy](https://github.com/nextflow-io/nf-hello/blob/gradle-plugin-example/src/test/groovy/nextflow/hello/HelloDslTest.groovy) in the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin for unit test examples. 2. In the plugin root directory, run `make test`. +(gradle-plugin-package)= -### Packaging, uploading, and publishing a plugin +## Packaging, uploading, and publishing a plugin The Gradle plugin for Nextflow plugins simplifies publishing your plugin. diff --git a/docs/index.md b/docs/index.md index 4ab2b430c5..b97338ccdc 100644 --- a/docs/index.md +++ b/docs/index.md @@ -67,7 +67,6 @@ config executor cache-and-resume reports -plugins ``` ```{toctree} @@ -154,7 +153,6 @@ developer/index developer/diagram developer/config-scopes developer/packages -developer/plugins ``` ```{toctree} diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index b424ff4233..c19cba3ef4 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -46,7 +46,7 @@ To migrate an existing Nextflow plugin: 3. Replace the contents of `settings.gradle` with the following: - ``` + ```groovy rootProject.name = '' ``` @@ -54,7 +54,7 @@ To migrate an existing Nextflow plugin: 4. In the project root, create a new `build.gradle` file with the following configuration: - ``` + ```groovy // Plugins plugins { id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' @@ -131,4 +131,4 @@ To migrate an existing Nextflow plugin: 6. Update `README.md` with information about the structure of your plugin. 7. In the plugin root directory, run `make assemble`. -The Gradle plugin for Nextflow plugins also supports publishing plugins. See Packaging, uploading, and publishing for more information. +The Gradle plugin for Nextflow plugins also supports publishing plugins. See {ref}`gradle-plugin-package` for more information. diff --git a/docs/plugins.md b/docs/plugins.md deleted file mode 100644 index 68d33151c1..0000000000 --- a/docs/plugins.md +++ /dev/null @@ -1,63 +0,0 @@ -(plugins-page)= - -# Plugins - -Nextflow has a plugin system that allows the use of extensible components that are downloaded and installed at runtime. - -(plugins-core)= - -## Core plugins - -The following functionalities are provided via plugin components, and they make part of the Nextflow *core* plugins: - -- `nf-amazon`: Support for Amazon Web Services. -- `nf-azure`: Support for Microsoft Azure. -- `nf-cloudcache`: Support for the cloud cache (see `NXF_CLOUDCACHE_PATH` under {ref}`config-env-vars`). -- `nf-console`: Implement Nextflow [REPL console](https://www.nextflow.io/blog/2015/introducing-nextflow-console.html). -- `nf-k8s`: Support for Kubernetes. -- `nf-google`: Support for Google Cloud. -- `nf-tower`: Support for [Seqera Platform](https://seqera.io) (formerly Tower Cloud). -- `nf-wave`: Support for [Wave containers](https://seqera.io/wave/) service. - -## Using plugins - -The core plugins do not require any configuration. They are automatically installed when the corresponding feature is requested by a Nextflow pipeline. You can still specify them as described below, e.g. if you want to pin the version of a plugin, however if you try to use a plugin version that isn't compatible with your Nextflow version, Nextflow will fail. - -You can enable a plugin by declaring it in your Nextflow configuration: - -```groovy -plugins { - id 'nf-hello@0.1.0' -} -``` - -Or you can use the `-plugins` command line option: - -```bash -nextflow run -plugins nf-hello@0.1.0 -``` - -The plugin identifier consists of the plugin name and plugin version separated by a `@`. Multiple plugins can be specified in the configuration with multiple `id` declarations, or on the command line as a comma-separated list. When specifying plugins via the command line, any plugin declarations in the configuration file are ignored. - -The plugin version is optional. If it is not specified, Nextflow will download the latest plugin version that is compatible with your Nextflow version. In general, it recommended that you not specify the plugin version unless you actually want to stick to that version, such as for [offline usage](#offline-usage). - -The core plugins are documented in this documentation. For all other plugins, please refer to the plugin's code repository for documentation and support. - -:::{versionadded} 25.02.0-edge -::: - -The plugin version can be prefixed with `~` to pin the major and minor version while allowing the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`, which is `2.9.2`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. - -## Offline usage - -To use Nextflow plugins in an offline environment: - -1. {ref}`Install Nextflow ` on a system with an internet connection. - -2. Download any additional plugins by running `nextflow plugin install `. Alternatively, simply run your pipeline once and Nextflow will download all of the plugins that it needs. - -3. Copy the `nextflow` binary and `$HOME/.nextflow` folder to your offline environment. - -4. In your Nextflow configuration file, specify each plugin that you downloaded, both name and version, including default plugins. This will prevent Nextflow from trying to download newer versions of plugins. - -Nextflow caches the plugins that it downloads, so as long as you keep using the same Nextflow version and pin your plugin versions in your config file, Nextflow will use the locally installed plugins and won't try to download them from the Internet. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 07b3c3fe2a..6f9372950a 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -1,4 +1,4 @@ -(developing-plugins-page)= +(dev-plugins-page)= # Developing plugins @@ -8,19 +8,19 @@ The Nextflow plugin framework streamlines plugin development by providing the st Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: -(developing-plugins-framework)= +(dev-plugins-framework)= ## Framework Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. -(developing-plugins-gradle)= +(dev-plugins-gradle)= ## Gradle plugin for Nextflow plugins -[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. +[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. -(developing-plugins-structure)= +(dev-plugins-structure)= ### Structure @@ -61,10 +61,10 @@ This structure contains the following key files and folders: See {ref}`nf-hello-page` for an example of a plugin built using this structure. :::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. See nf-hello for an example of a plugin with the old structure. +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: -(developing-plugins-make)= +(dev-plugins-make)= ### make commands @@ -83,18 +83,18 @@ make assemble The following `make` commands are available: `assemble` -: Compiles the Nextflow plugin code and assembles it into a zip file. See Creating a plugin for more information. +: Compiles the Nextflow plugin code and assembles it into a zip file. See {ref}`gradle-plugin-create` for more information. `test` -: Runs plugin unit tests. See Unit testing for more information. +: Runs plugin unit tests. See {ref}`gradle-plugin-unit-test` for more information. `install` -: Installs the plugin into the local nextflow plugins directory. See Running locally for more information. +: Installs the plugin into the local nextflow plugins directory. See {ref}`gradle-plugin-install` for more information. `release` -: Publishes the plugin. See Packaging, uploading, and publishing for more information. +: Publishes the plugin. See {ref}`gradle-plugin-package` for more information. -(developing-plugins-versioning)= +(dev-plugins-versioning)= ### Versioning @@ -108,7 +108,7 @@ plugins { See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. -(developing-plugins-extension)= +(dev-plugins-extension)= ## Extension points @@ -146,8 +146,8 @@ class MyPlugin extends BasePlugin implements PluginAbstractExec { The command can be run using the `nextflow plugin` command: -``` -nextflow plugin my-plugin:hello --foo --bar +```bash +nextflow plugin my-plugin:hello --alpha --beta ``` See the {ref}`cli-plugin` for usage information. @@ -201,6 +201,9 @@ import nextflow.script.dsl.Description ''') class MyPluginConfig implements ConfigScope { + /* required by extension point -- do not remove */ + MyPluginConfig() {} + MyPluginConfig(Map opts) { this.createMessage = opts.createMessage } @@ -211,13 +214,13 @@ class MyPluginConfig implements ConfigScope { } ``` -This approach is not required to support plugin config options. However, it allows Nextflow to recognize plugin definitions when validating the configuration. +This approach is not required to support plugin config options. However, it allows Nextflow to recognize plugin definitions when validating the configuration. See {ref}`config-scopes-page` for more information. ### Executors Plugins can define custom executors that can be used with the `executor` process directive. -To implement an executor, create a class in your plugin that extends the [Executor](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: +To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy)class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: ```groovy import nextflow.executor.Executor @@ -235,7 +238,7 @@ class MyExecutor extends Executor implements ExtensionPoint { You can then use this executor in your pipeline: ```groovy -process foo { +process hello { executor 'my-executor' // ... @@ -249,9 +252,9 @@ See the source code of Nextflow's built-in executors for examples of how to impl ### Filesystems -Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see Remote files. +Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. -To implement a custom filesystem, create a class in your plugin that extends [FileSystemProvider](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: +To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: ```groovy import java.nio.file.spi.FileSystemProvider @@ -371,9 +374,9 @@ The above snippet is based on the [nf-sqldb](https://github.com/nextflow-i ### Process directives -Plugins that implement a custom executor will likely need to access process directives that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. +Plugins that implement a custom executor will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. -Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the ext directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. +Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the {ref}`process-ext` directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. For example, a custom executor can use existing process directives and a custom setting through the `ext` directive: @@ -396,8 +399,14 @@ class MyExecutor extends Executor { } ``` +(plugins-trace-observers)= + ### Trace observers +:::{versionchanged} 25.04 +The `TraceObserver` interface is now deprecated. Use [TraceObserverV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverV2.groovy) and [TraceObserverFactoryV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy) instead. +::: + A *trace observer* is an entity that can listen and react to workflow events, such as when a workflow starts, a task is completed, or a file is published. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver` and implement the `create()` method in your observer factory. For example: @@ -451,6 +460,7 @@ myplugin.enabled = true See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. +(dev-plugins-env-var)= ## Environment variables diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 689b5f0a69..38e0b4d00e 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -2,9 +2,9 @@ # Example: nf-hello -`nf-hello` is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. +[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/master) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. -The nf-hello plugin has the following structure: +The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/master) has the following structure: ``` nf-hello diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 259d1e9e0c..5d229c08f7 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -28,7 +28,7 @@ Core plugins do not require configuration. The latest versions of core plugins a Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. :::{note} -The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See Environment variables for more information. +The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-var` for more information. :::

Third-party plugins

@@ -64,4 +64,4 @@ Version components: * PATCH: Increment for backward-compatible bug fixes. Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. -See Semantic Versioning for more information about the specification. +See {ref}`dev-plugins-versioning` for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index 8aef78fc7e..da319a0849 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -1,7 +1,11 @@ +(using-plugins-page)= + # Using plugins Nextflow core plugins require no additional configuration. When a pipeline uses a core plugin, Nextflow automatically downloads and uses the latest compatible plugin version. In contrast, third-party plugins must be explicitly declared. When a pipeline uses one or more third-party plugins, Nextflow must be configured to download and use the plugin. +(using-plugins-identifiers)= + ## Identifiers A plugin identifier consists of the plugin name and version, separated by an `@` symbol: @@ -13,7 +17,7 @@ nf-hello@0.5.0 The plugin version is optional. If it is not specified, Nextflow will download the latest version of the plugin that meets the minimum Nextflow version requirements specified by the plugin. :::{note} -Plugin versions are required to run plugins in an Offline environment. +Plugin versions are required for {ref}`offline usage `. ::: :::{versionadded} 25.02.0-edge. @@ -25,6 +29,8 @@ The plugin version can be prefixed with `~` to pin the major and minor versions It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. ::: +(using-plugins-config)= + ## Configuration Plugins can be configured via nextflow configuration files or at runtime. @@ -57,13 +63,15 @@ Plugin declarations in nextflow configuration files are ignored when specifying When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. +(using-plugins-offline)= + ## Offline usage Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. To use Nextflow plugins in an offline environment: -1. Install a self-contained version of Nextflow in an environment with an internet connection. See Standalone distribution for more information. +1. Install a self-contained version of Nextflow in an environment with an internet connection. See {ref}`install-standalone` for more information. 2. Run `nextflow plugin install @` to download the plugins. :::{tip} @@ -71,8 +79,8 @@ To use Nextflow plugins in an offline environment: ::: 3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. -4. Specify each plugin and its version in Nextflow configuration files or at runtime. See Configuration for more information. +4. Specify each plugin and its version in Nextflow configuration files or at runtime. See {ref}`using-plugins-config` for more information. :::{warning} - Nextflow will attempt to download newer versions of plugins if their versions are not set. See Plugin identifiers for more information. + Nextflow will attempt to download newer versions of plugins if their versions are not set. See {ref}`using-plugins-identifiers` for more information. ::: \ No newline at end of file diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index 2c6210effa..b3105c6097 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -164,7 +164,7 @@ The following environment variables control the configuration of the Nextflow ru `NXF_PLUGINS_TEST_REPOSITORY` : :::{versionadded} 23.04.0 ::: -: Defines a custom plugin registry or plugin release URL for testing plugins outside of the main registry. See {ref}`testing-plugins` for more information. +: Defines a custom plugin registry or plugin release URL for testing plugins outside of the main registry. `NXF_PUBLISH_FAIL_ON_ERROR` : :::{versionadded} 24.04.3 diff --git a/docs/strict-syntax.md b/docs/strict-syntax.md index 2c208f77be..e345bc9df7 100644 --- a/docs/strict-syntax.md +++ b/docs/strict-syntax.md @@ -587,4 +587,4 @@ There are two ways to preserve Groovy code: Any Groovy code can be moved into the `lib` directory, which supports the full Groovy language. This approach is useful for temporarily preserving some Groovy code until it can be updated later and incorporated into a Nextflow script. See {ref}`lib-directory` documentation for more information. -For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`plugins-dev-page` for more information on how to develop plugins. +For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`dev-plugins-page` for more information on how to develop plugins. From 0da062e3e4fd0079ff93bb268085750d94f71413 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 29 Apr 2025 01:39:56 +1200 Subject: [PATCH 19/29] Add repo link Signed-off-by: Christopher Hakkaart --- docs/plugins/example-nf-hello.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 38e0b4d00e..b727838163 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -2,9 +2,9 @@ # Example: nf-hello -[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/master) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. +[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. -The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/master) has the following structure: +The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) has the following structure: ``` nf-hello @@ -56,3 +56,5 @@ The `nf-hello` plugin can be configured via nextflow configuration files or at r ```bash nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 ``` + +See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. From cd64ec0ba96fffb21cc309639f715879866930f4 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 5 May 2025 18:44:38 +1200 Subject: [PATCH 20/29] Revise defined registry Signed-off-by: Christopher Hakkaart --- docs/migrating-gradle-plugin.md | 2 -- docs/plugins/plugins.md | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index c19cba3ef4..384bc7b5d5 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -17,8 +17,6 @@ The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. -Developers can upload plugins to the plugin registry. The Gradle plugin for Nextflow plugins supports this process with Gradle tasks that package, upload, and publish plugins. - ## Impact on users and developers The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 5d229c08f7..59405d71ab 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -51,7 +51,7 @@ See {ref}`using-plugins-page` for more information. ## Nextflow plugin registry -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. Developers can upload plugins to the registry, with built-in support from the Gradle plugin for Nextflow plugins. +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. ## Versioning From a14e237102793d8ff9362423715bf7ed88e2e022 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 6 May 2025 15:36:00 +1200 Subject: [PATCH 21/29] Add plugin create documentation Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 85 +++++----------------- docs/migrating-gradle-plugin.md | 20 ++--- docs/migrations/25-04.md | 6 ++ docs/plugins/developing-plugins.md | 113 ++++++++++++++++------------- docs/plugins/using-plugins.md | 2 +- docs/reference/cli.md | 6 +- 6 files changed, 104 insertions(+), 128 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 2eb45aa150..81f9303e88 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -2,7 +2,7 @@ # Using the Gradle plugin -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development and incorporates the Gradle plugin by default. You can use the `nextflow plugin create` sub-command to create plugins scaffolds with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. :::{note} Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. @@ -12,71 +12,18 @@ Nextflow Plugins can be developed without the Gradle plugin. However, this appro ## Creating a plugin -The [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin uses the Gradle plugin and is a valuable starting point for developers. +:::{versionadded} 25.04.0 +::: To create a Nextflow plugin with the Gradle plugin: -1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. See {ref}`nf-hello-page` for more information. -2. Rename the forked `nf-hello` directory with your plugin name. -3. Replace the contents of `settings.gradle` with the following: - - ```groovy - rootProject.name = '' - ``` - - Replace `PLUGIN_NAME` with your plugin name. - -4. Replace the contents of `build.gradle` with the following: - - ```groovy - // Plugins - plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' - } - - // Dependencies (optional) - dependencies { - - } - - // Plugin version - version = '' - - nextflowPlugin { - // Minimum Nextflow version - nextflowVersion = '' - - // Plugin metadata - provider = '' - className = '' - extensionPoints = [ - '' - ] - - publishing { - github { - repository = '' - userName = project.findProperty('github_username') - authToken = project.findProperty('github_access_token') - email = project.findProperty('github_commit_email') - indexUrl = '' - } - } - } - ``` - - Replace the following: - - - `DEPENDENCY`: (optional) your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - - `PLUGIN_VERSION:` your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: the minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. - - `PROVIDER`: your name or organization—for example, `nextflow`. - - `CLASS_NAME`: your plugin class name—for example, `nextflow.hello.HelloPlugin`. - - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). -5. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. -6. In the plugin root directory, run `make assemble`. +1. Run `nextflow plugin create`. + - When prompted `Enter plugin name:`, enter your plugin name. + - When prompted `Enter organization:`, enter your organization. + - When prompted `Enter project path:`, enter the project path in your local file system. + - When prompted `All good, are you OK to continue [y/N]?`, enter `y`. +2. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. +3. In the plugin root directory, run `make assemble`. (gradle-plugin-install)= @@ -92,13 +39,17 @@ To install a plugin locally: Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. ::: -2. Configure your plugin. See {ref}`using-plugins-page` for more information. -3. Run your pipeline: +2. Run your pipeline: ```bash - nextflow run main.nf + nextflow run main.nf -plugins @ ``` + :::{note} + Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. + ::: + + (gradle-plugin-unit-test)= ## Unit testing a plugin @@ -107,7 +58,7 @@ Unit tests are small, focused tests designed to verify the behavior of individua To run unit tests: -1. Develop your unit tests. See [HelloDslTest.groovy](https://github.com/nextflow-io/nf-hello/blob/gradle-plugin-example/src/test/groovy/nextflow/hello/HelloDslTest.groovy) in the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin for unit test examples. +1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template/tree/main) for unit test examples. 2. In the plugin root directory, run `make test`. (gradle-plugin-package)= diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 384bc7b5d5..3ca6889746 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -1,3 +1,5 @@ +(migrating-plugin-page)= + # Migrating to the Gradle plugin for Nextflow plugins This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. @@ -19,7 +21,7 @@ The Nextflow plugin registry is a centralized repository of assembled plugins. I ## Impact on users and developers -The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. +The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. ### Plugin Users @@ -36,7 +38,7 @@ To migrate an existing Nextflow plugin: - `nextflow.config` - `launch.sh` - `plugins/build.gradle` -2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. \ +2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. :::{note} Plugin sources should be in `src/main/groovy` or `src/main/java`. @@ -55,7 +57,7 @@ To migrate an existing Nextflow plugin: ```groovy // Plugins plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' } // Dependencies (optional) @@ -93,12 +95,12 @@ To migrate an existing Nextflow plugin: - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. - - `PROVIDER`: Your name or organization—for example, `nextflow`. - - `CLASS_NAME`: Your plugin class name—for example, `nextflow.hello.HelloPlugin`. - - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.03.0-edge`. + - `PROVIDER`: Your name or organization—for example, `acme`. + - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. + - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-plugin-template`. + - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`https://github.com/nextflow-io/plugins/blob/main/plugins.json`](https://github.com/nextflow-io/plugins/blob/main/plugins.json). 5. Replace the contents of `Makefile` with the following: diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index c3bc9545fb..8e503b5be2 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -46,6 +46,12 @@ You can explore this lineage from the command line using the {ref}`cli-lineage` See {ref}`data-lineage-page` to get started. +

Creating plugins

+ +The `nextflow plugin create` sub-command creates a scaffold for a Nextflow plugin using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). + +See {ref}`gradle-plugin-create` for details. + ## Enhancements

Improved inspect command

diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 6f9372950a..fb541867ac 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -20,61 +20,15 @@ Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/p [Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. -(dev-plugins-structure)= - -### Structure - -Plugins built with the Gradle plugin for Nextflow plugins follow a standard project layout. This structure includes source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. - -A typical plugin built with the Gradle plugin for Nextflow plugins has the following structure: - -``` -. -├── COPYING -├── Makefile -├── README.md -├── build.gradle -├── gradle -│ └── wrapper -│ └── ... -├── gradlew -├── settings.gradle -└── src - ├── main - │ └── ... - └── test - └── ... -``` - -This structure contains the following key files and folders: - -- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. -- `src/main/`: The main source directory containing the plugin's implementation and resources.​ -- `src/test/`: The main source directory containing the plugin's unit testing. -- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ -- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ -- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ -- `build.gradle`: The primary build script for Gradle. -- `gradlew`: A script for executing the Gradle Wrapper. -- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. - -See {ref}`nf-hello-page` for an example of a plugin built using this structure. - -:::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. -::: - -(dev-plugins-make)= - ### make commands -The Gradle plugin for Nextflow plugins defines tasks that can be executed with `./gradlew`. For example: +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) defines tasks that can be executed with `./gradlew`. For example: ```bash ./gradlew assemble ``` -For convenience, the more important tasks are wrapped in a makefile and can be executed with the `make` command. For example: +For convenience, the most important tasks are wrapped in a `Makefile` and can be executed with the `make` command. For example: ```bash make assemble @@ -98,16 +52,75 @@ The following `make` commands are available: ### Versioning -The Gradle plugin is versioned and published to a [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: ```nextflow plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' } ``` See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. +(dev-plugins-template)= + +## nf-plugin-template + +The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It incorporates the {ref}`dev-plugins-gradle` by default. You can use the `nextflow plugin create` sub-command to create plugins with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. See {ref}`gradle-plugin-create` for more information. + +(dev-plugins-structure)= + +### Structure + +Plugins built with [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) includes the [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle). The template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. + +Plugins built with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) have the following structure: + +``` +. +├── build.gradle +├── COPYING +├── gradle +│ └── wrapper +│ └── ... +├── gradlew +├── Makefile +├── README.md +├── settings.gradle +└── src + ├── main + │ └── groovy + │ └── + │ └── plugin + │ └── ... + └── test + └── groovy + └── + └── plugin + └── ... +``` + +This structure contains the following key files and folders: + +- `build.gradle`: The primary build script for Gradle. +- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ +- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. +- `gradlew`: A script for executing the Gradle Wrapper. +- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ +- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ +- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. +- `src/main/groovy//plugin/`: The main source directory containing the plugin's implementation and resources.​ +- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit testing. + +See {ref}`nf-hello-page` for an example of a plugin built using this structure. + +:::{note} +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +(dev-plugins-make)= + + (dev-plugins-extension)= ## Extension points diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index da319a0849..282bd92b28 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -20,7 +20,7 @@ The plugin version is optional. If it is not specified, Nextflow will download t Plugin versions are required for {ref}`offline usage `. ::: -:::{versionadded} 25.02.0-edge. +:::{versionadded} 25.02.0-edge ::: The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index eca5b65e1d..82e397aeab 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -960,7 +960,11 @@ Manage plugins and run plugin-specific commands. $ nextflow plugin [options] ``` -The `plugin` command provides several subcommands for managing and using plugins: +`create` + +: :::{versionadded} 25.04.0 + ::: +: Create a plugin scaffold using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. `install ` From a71ffcccd100189746c8ae19c5a8e0d5578bfe13 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Wed, 7 May 2025 14:08:29 +1200 Subject: [PATCH 22/29] Only support registry Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 28 ++++++++++++---------------- docs/migrating-gradle-plugin.md | 21 ++++++++++----------- docs/plugins/developing-plugins.md | 2 +- docs/plugins/example-nf-hello.md | 2 +- 4 files changed, 24 insertions(+), 29 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 81f9303e88..d3412c432b 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -65,25 +65,21 @@ To run unit tests: ## Packaging, uploading, and publishing a plugin -The Gradle plugin for Nextflow plugins simplifies publishing your plugin. +The Gradle plugin for Nextflow plugins simplifies publishing your plugin to the Nextflow Plugin Registry. -To package, upload, and publish your plugin: +:::{note} +The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +::: -1. Fork the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins). -2. In the plugin root directory, open `build.gradle` and ensure that: - * `github.repository` matches the plugin repository. - * `github.indexUrl` matches your fork of the plugins index repository. -3. Create a file named `$HOME/.gradle/gradle.properties` and add the following: +To package, upload, and publish your plugin to the Nextflow Plugin Registry: + +1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. +2. Add the following properties: ```bash - github_username= - github_access_token= - github_commit_email= + pluginRegistry.accessToken= ``` - Replace the following: - * `GITHUB_USERNAME`: your GitHub username granting access to the plugin repository. - * `GITHUB_ACCESS_TOKEN`: your GitHub access token with permission to upload and commit changes to the plugin repository. - * `GITHUB_EMAIL`: your email address associated with your GitHub account. -4. Run `make release`. -5. Create a pull request against the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins) from your fork. \ No newline at end of file + Replace with your plugin registry access token. + +3. Run `make release`. \ No newline at end of file diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 3ca6889746..75e59aaef4 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -15,10 +15,14 @@ The Gradle plugin for Nextflow plugins simplifies and standardizes the developme The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. -### Nextflow plugin registry +### Nextflow Plugin Registry The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. +:::{note} +The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +::: + ## Impact on users and developers The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. @@ -57,7 +61,7 @@ To migrate an existing Nextflow plugin: ```groovy // Plugins plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' } // Dependencies (optional) @@ -79,13 +83,10 @@ To migrate an existing Nextflow plugin: '' ] - publishing { - github { - repository = '' - userName = project.findProperty('github_username') - authToken = project.findProperty('github_access_token') - email = project.findProperty('github_commit_email') - indexUrl = '' + publishing { + registry { + url = 'https://nf-plugins-registry.dev-tower.net/api' + authToken = project.findProperty('pluginRegistry.accessToken') } } } @@ -99,8 +100,6 @@ To migrate an existing Nextflow plugin: - `PROVIDER`: Your name or organization—for example, `acme`. - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. - - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-plugin-template`. - - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`https://github.com/nextflow-io/plugins/blob/main/plugins.json`](https://github.com/nextflow-io/plugins/blob/main/plugins.json). 5. Replace the contents of `Makefile` with the following: diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index fb541867ac..cbf62748a7 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -56,7 +56,7 @@ The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https ```nextflow plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' } ``` diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index b727838163..b643218eb8 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -54,7 +54,7 @@ It also includes several classes that demonstrate different plugin functionality The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: ```bash -nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +nextflow run hello -plugins nf-hello@0.5.0 ``` See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. From 60632f220a35d4a3c8cb3d34f10096aa8f9da452 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 8 Aug 2025 12:58:12 -0500 Subject: [PATCH 23/29] cleanup Signed-off-by: Ben Sherman --- docs/gradle-plugin.md | 85 ------------- docs/guides/gradle-plugin.md | 82 +++++++++++++ docs/guides/plugin-registry.md | 160 +++++++++++++++++++++++++ docs/index.md | 5 +- docs/migrating-gradle-plugin.md | 133 --------------------- docs/migrations/25-04.md | 10 +- docs/plugins/developing-plugins.md | 184 +++++++++++++++-------------- docs/plugins/example-nf-hello.md | 60 ---------- docs/plugins/plugins.md | 33 +++--- docs/plugins/using-plugins.md | 27 ++--- docs/reference/cli.md | 2 +- docs/reference/env-vars.md | 3 + docs/reference/stdlib-groovy.md | 2 +- 13 files changed, 378 insertions(+), 408 deletions(-) delete mode 100644 docs/gradle-plugin.md create mode 100644 docs/guides/gradle-plugin.md create mode 100644 docs/guides/plugin-registry.md delete mode 100644 docs/migrating-gradle-plugin.md delete mode 100644 docs/plugins/example-nf-hello.md diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md deleted file mode 100644 index d3412c432b..0000000000 --- a/docs/gradle-plugin.md +++ /dev/null @@ -1,85 +0,0 @@ -(gradle-plugin-page)= - -# Using the Gradle plugin - -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development and incorporates the Gradle plugin by default. You can use the `nextflow plugin create` sub-command to create plugins scaffolds with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. - -:::{note} -Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. -::: - -(gradle-plugin-create)= - -## Creating a plugin - -:::{versionadded} 25.04.0 -::: - -To create a Nextflow plugin with the Gradle plugin: - -1. Run `nextflow plugin create`. - - When prompted `Enter plugin name:`, enter your plugin name. - - When prompted `Enter organization:`, enter your organization. - - When prompted `Enter project path:`, enter the project path in your local file system. - - When prompted `All good, are you OK to continue [y/N]?`, enter `y`. -2. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. -3. In the plugin root directory, run `make assemble`. - -(gradle-plugin-install)= - -## Installing a plugin - -Plugins can be installed locally without being packaged, uploaded, and published. - -To install a plugin locally: - -1. In the plugin root directory, run `make install`. - - :::{note} - Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. - ::: - -2. Run your pipeline: - - ```bash - nextflow run main.nf -plugins @ - ``` - - :::{note} - Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. - ::: - - -(gradle-plugin-unit-test)= - -## Unit testing a plugin - -Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. - -To run unit tests: - -1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template/tree/main) for unit test examples. -2. In the plugin root directory, run `make test`. - -(gradle-plugin-package)= - -## Packaging, uploading, and publishing a plugin - -The Gradle plugin for Nextflow plugins simplifies publishing your plugin to the Nextflow Plugin Registry. - -:::{note} -The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. -::: - -To package, upload, and publish your plugin to the Nextflow Plugin Registry: - -1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. -2. Add the following properties: - - ```bash - pluginRegistry.accessToken= - ``` - - Replace with your plugin registry access token. - -3. Run `make release`. \ No newline at end of file diff --git a/docs/guides/gradle-plugin.md b/docs/guides/gradle-plugin.md new file mode 100644 index 0000000000..ab825053eb --- /dev/null +++ b/docs/guides/gradle-plugin.md @@ -0,0 +1,82 @@ +(gradle-plugin-page)= + +# Using the Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. + +:::{note} +The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. +::: + +(gradle-plugin-create)= + +## Creating a plugin + +:::{versionadded} 25.04.0 +::: + +The easiest way to get started with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which in turn uses the Gradle plugin. + +To create a Nextflow plugin with the Gradle plugin, run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. + +See {ref}`dev-plugins-template` for more information about the Nextflow plugin template. See {ref}`dev-plugins-extension-points` for more information about using plugin extension points. + +## Building a plugin + +To build a plugin, run `make assemble`. + +Plugins can also be installed locally without being published. To install a plugin locally: + +1. In the plugin root directory, run `make install`. + + :::{note} + Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. + ::: + +2. Run your pipeline: + + ```bash + nextflow run main.nf -plugins @ + ``` + + :::{note} + Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. + ::: + +(gradle-plugin-test)= + +## Testing a plugin + +

Unit tests

+ +Unit tests are small, focused tests designed to verify the behavior of individual plugin components. + +To run unit tests: + +1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example unit test. + +2. In the plugin root directory, run `make test`. + +

End-to-end tests

+ +End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. + +(gradle-plugin-publish)= + +## Publishing a plugin + +The Nextflow Gradle plugin allows you to publish your plugin to the Nextflow plugin registry from the command line. + +To publish your plugin: + +1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. + +2. Add the following properties: + + ``` + pluginRegistry.accessToken= + ``` + + Replace `` with your plugin registry access token. + +3. Run `make release`. diff --git a/docs/guides/plugin-registry.md b/docs/guides/plugin-registry.md new file mode 100644 index 0000000000..b106454472 --- /dev/null +++ b/docs/guides/plugin-registry.md @@ -0,0 +1,160 @@ +(plugin-registry-page)= + +# Migrating to the Nextflow plugin registry + +The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. + +:::{note} +The Nextflow plugin registry and Gradle plugin are currently available as a private beta. Plugin developers are encouraged to contact [info@nextflow.io](mailto:info@nextflow.io) for more information about accessing the registry. +::: + +## Overview + +### Nextflow plugin registry + +The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. Nextflow 25.04 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. + +### Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. + +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. + +## Timeline + +The [legacy plugin index](https://github.com/nextflow-io/plugins) will be deprecated in favor of the Nextflow plugin registry. + +:::{note} +The following timeline is tentative and subject to modification. +::: + +

Nextflow 25.04

+ +The Nextflow plugin registry is available as a private beta. Nextflow 25.04 can use the Nextflow plugin registry as an opt-in feature. The Nextflow plugin registry will be automatically kept up-to-date with the [legacy plugin index](https://github.com/nextflow-io/plugins). + +During this time, plugin developers are encouraged to experiment with the Gradle plugin and plugin registry. + +

Nextflow 25.10

+ +The Nextflow plugin registry will be generally available. Nextflow 25.10 will use the plugin registry by default. The legacy plugin index will be **closed to new pull requests**. + +Developers will be required to publish to the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry. + +

Nextflow 26.04

+ +Nextflow 26.04 will only be able to use the Nextflow plugin registry. + +At some point in the future, the legacy plugin index will be **frozen** -- it will no longer receives updates from the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will remain available indefinitely. + +## Impact on plugin users + +No immediate actions are required for plugin users. The plugin configuration has not changed. + +## Impact on plugin developers + +Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The easiest way to do this is to migrate to the Nextflow Gradle plugin, which simplifies the development process and supports publishing to the plugin registry from the command line. + +### Migrating to the Nextflow Gradle plugin + +To migrate an existing Nextflow plugin: + +1. Remove the following files and folders: + - `buildSrc/` + - `launch.sh` + - `plugins/build.gradle` + +2. If your plugin has a `plugins` directory, move the `src` directory to the project root. + + :::{note} + Plugin sources should be in `src/main/groovy` or `src/main/java`. + ::: + +3. Replace the contents of `settings.gradle` with the following: + + ```groovy + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. In the project root, create a new `build.gradle` file with the following configuration: + + ```groovy + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + registry { + authToken = project.findProperty('pluginRegistry.accessToken') + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.04.0`. + - `PROVIDER`: Your name or organization—for example, `acme`. + - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. + +5. Replace the contents of `Makefile` with the following: + + ```Makefile + # Build the plugin + assemble: + ./gradlew assemble + + clean: + rm -rf .nextflow* + rm -rf work + rm -rf build + ./gradlew clean + + # Run plugin unit tests + test: + ./gradlew test + + # Install the plugin into local nextflow plugins dir + install: + ./gradlew install + + # Publish the plugin + release: + ./gradlew releasePlugin + ``` + +6. Update `README.md` with information about the structure of your plugin. + +7. In the plugin root directory, run `make assemble`. + +Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template and add your existing plugin code. See {ref}`dev-plugins-template` for more information about the plugin template. + +### Publishing to the Nextflow plugin registry + +The Nextflow Gradle plugin supports publishing plugins from the command line. See {ref}`gradle-plugin-publish` for more information. + +Once you migrate to the Gradle plugin, you will no longer be able to publish to the legacy plugin index. See the [transition timeline](#timeline) for more information. diff --git a/docs/index.md b/docs/index.md index b97338ccdc..563857b1fb 100644 --- a/docs/index.md +++ b/docs/index.md @@ -163,7 +163,6 @@ developer/packages plugins/plugins plugins/using-plugins plugins/developing-plugins -plugins/example-nf-hello ``` ```{toctree} @@ -183,7 +182,7 @@ tutorials/flux :maxdepth: 1 guides/aws-java-sdk-v2 +guides/gradle-plugin +guides/plugin-registry guides/updating-spot-retries -gradle-plugin -migrating-gradle-plugin ``` diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md deleted file mode 100644 index 75e59aaef4..0000000000 --- a/docs/migrating-gradle-plugin.md +++ /dev/null @@ -1,133 +0,0 @@ -(migrating-plugin-page)= - -# Migrating to the Gradle plugin for Nextflow plugins - -This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. - - -## Improvements to the plugin framework - -The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying plugin development, streamlining publishing and discovery, and improving how plugins are loaded into workflows. These improvements make plugins more accessible, maintainable, and interoperable with Nextflow. - -### Gradle plugin for Nextflow plugins - -The Gradle plugin for Nextflow plugins simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. - -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. - -### Nextflow Plugin Registry - -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. - -:::{note} -The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. -::: - -## Impact on users and developers - -The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. - -### Plugin Users - -If you are a plugin user, no immediate actions are required. The plugin configuration has not changed. - -### Plugin developers - -Developers are encouraged to migrate to the Gradle plugin for Nextflow plugins and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. - -To migrate an existing Nextflow plugin: - -1. Remove the following files and folders: - - `buildSrc/` - - `nextflow.config` - - `launch.sh` - - `plugins/build.gradle` -2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. - - :::{note} - Plugin sources should be in `src/main/groovy` or `src/main/java`. - ::: - -3. Replace the contents of `settings.gradle` with the following: - - ```groovy - rootProject.name = '' - ``` - - Replace `PLUGIN_NAME` with your plugin name. - -4. In the project root, create a new `build.gradle` file with the following configuration: - - ```groovy - // Plugins - plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' - } - - // Dependencies (optional) - dependencies { - - } - - // Plugin version - version = '' - - nextflowPlugin { - // Minimum Nextflow version - nextflowVersion = '' - - // Plugin metadata - provider = '' - className = '' - extensionPoints = [ - '' - ] - - publishing { - registry { - url = 'https://nf-plugins-registry.dev-tower.net/api' - authToken = project.findProperty('pluginRegistry.accessToken') - } - } - } - ``` - - Replace the following: - - - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.03.0-edge`. - - `PROVIDER`: Your name or organization—for example, `acme`. - - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. - -5. Replace the contents of `Makefile` with the following: - - ``` - # Build the plugin - assemble: - ./gradlew assemble - - clean: - rm -rf .nextflow* - rm -rf work - rm -rf build - ./gradlew clean - - # Run plugin unit tests - test: - ./gradlew test - - # Install the plugin into local nextflow plugins dir - install: - ./gradlew install - - # Publish the plugin - release: - ./gradlew releasePlugin - ``` - -6. Update `README.md` with information about the structure of your plugin. -7. In the plugin root directory, run `make assemble`. - -The Gradle plugin for Nextflow plugins also supports publishing plugins. See {ref}`gradle-plugin-package` for more information. diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index 8e503b5be2..a5d11de7e1 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -46,11 +46,15 @@ You can explore this lineage from the command line using the {ref}`cli-lineage` See {ref}`data-lineage-page` to get started. -

Creating plugins

+

Simplified plugin development

-The `nextflow plugin create` sub-command creates a scaffold for a Nextflow plugin using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). +The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the official [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. -See {ref}`gradle-plugin-create` for details. +See the {ref}`gradle-plugin-page` guide for details. + +:::{note} +The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. +::: ## Enhancements diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index cbf62748a7..f7caaffba6 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -2,57 +2,92 @@ # Developing plugins -The Nextflow plugin framework streamlines plugin development by providing the structure and tools needed to extend Nextflow functionality. The Gradle plugin for Nextflow plugins simplifies development by configuring default Nextflow dependencies and incorporates custom Gradle tasks that streamline building and publishing plugins. +This page describes how to develop plugins for Nextflow. + +(dev-plugins-template)= + +## Nextflow plugin template + +The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It uses [Gradle](https://gradle.org/), a build automation tool optimized for Java and Groovy projects, as well as the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle). + +You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. :::{note} -Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +The Nextflow Gradle plugin is currently available as a private beta. See the {ref}`migration guide ` for more information. ::: -(dev-plugins-framework)= +### Structure -## Framework +The plugin template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. -Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. +For example, a plugin created from the plugin template with the name `nf-hello` and organization `nextflow` will have the following structure: -(dev-plugins-gradle)= +```console +nf-hello +├── COPYING +├── Makefile +├── README.md +├── build.gradle +├── gradle +│ └── wrapper +│ ├── gradle-wrapper.jar +│ └── gradle-wrapper.properties +├── gradlew +├── settings.gradle +├── src +│ ├── main +│ │ └── groovy +│ │ └── nextflow +│ │ └── hello +│ │ ├── HelloExtension.groovy +│ │ ├── HelloFactory.groovy +│ │ ├── HelloObserver.groovy +│ │ └── HelloPlugin.groovy +│ └── test +│ └── groovy +│ └── nextflow +│ └── hello +│ └── HelloObserverTest.groovy +└── validation + └── main.nf + └── nextflow.config +``` -## Gradle plugin for Nextflow plugins +This structure contains the following key files and folders: -[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. +- `.github/workflows`: GitHub Action which implements continuous integration for the plugin. -### make commands +- `build.gradle`: The Gradle build script. -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) defines tasks that can be executed with `./gradlew`. For example: +- `COPYING`: The project license, detailing the terms under which the code can be used and distributed.​ -```bash -./gradlew assemble -``` +- `gradle/wrapper/`: Helper files for the Gradle Wrapper. -For convenience, the most important tasks are wrapped in a `Makefile` and can be executed with the `make` command. For example: +- `gradlew`: The Gradle Wrapper script, which allows you to use Gradle without installing it into your environment. -```bash -make assemble -``` +- `Makefile`: Defines common tasks for building, testing, and publishing the plugin with Make.​ -The following `make` commands are available: +- `README.md`: The project README, which provides an overview of the project, including its purpose, features, and instructions for usage and development.​ -`assemble` -: Compiles the Nextflow plugin code and assembles it into a zip file. See {ref}`gradle-plugin-create` for more information. +- `settings.gradle`: The Gradle project configuration, which specifies project-specific settings such as the project name and included modules. -`test` -: Runs plugin unit tests. See {ref}`gradle-plugin-unit-test` for more information. +- `src/main/groovy///`: The main source directory, which contains the plugin source code and resources.​ -`install` -: Installs the plugin into the local nextflow plugins directory. See {ref}`gradle-plugin-install` for more information. +- `src/test/groovy///`: The test source directory, which contains the plugin unit tests. -`release` -: Publishes the plugin. See {ref}`gradle-plugin-package` for more information. +- `validation`: A small Nextflow pipeline which serves as an end-to-end test for the plugin. + +The plugin template also implements the following example features: + +- A custom trace observer that prints a message when the workflow starts and when the workflow completes (see `HelloObserver`). + +- A custom function called `sayHello` (see `HelloExtension`). -(dev-plugins-versioning)= +### Nextflow Gradle plugin -### Versioning +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: +It is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), and can be declared and managed like any other dependency in the `build.gradle` file: ```nextflow plugins { @@ -60,72 +95,39 @@ plugins { } ``` -See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. - -(dev-plugins-template)= - -## nf-plugin-template - -The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It incorporates the {ref}`dev-plugins-gradle` by default. You can use the `nextflow plugin create` sub-command to create plugins with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. See {ref}`gradle-plugin-create` for more information. - -(dev-plugins-structure)= +:::{note} +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: -### Structure +### Make commands -Plugins built with [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) includes the [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle). The template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. +The plugin template includes a Makefile which wraps the most important Gradle tasks provided by the Nextflow Gradle plugin. -Plugins built with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) have the following structure: +These tasks can be executed with [Make](https://www.gnu.org/software/make/). For example: -``` -. -├── build.gradle -├── COPYING -├── gradle -│ └── wrapper -│ └── ... -├── gradlew -├── Makefile -├── README.md -├── settings.gradle -└── src - ├── main - │ └── groovy - │ └── - │ └── plugin - │ └── ... - └── test - └── groovy - └── - └── plugin - └── ... +```bash +make assemble ``` -This structure contains the following key files and folders: - -- `build.gradle`: The primary build script for Gradle. -- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ -- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. -- `gradlew`: A script for executing the Gradle Wrapper. -- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ -- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ -- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. -- `src/main/groovy//plugin/`: The main source directory containing the plugin's implementation and resources.​ -- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit testing. +The following `make` commands are available: -See {ref}`nf-hello-page` for an example of a plugin built using this structure. +`assemble` +: Compiles the Nextflow plugin code and assembles it into a zip archive. -:::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. -::: +`install` +: Installs the plugin into the local Nextflow plugins directory. -(dev-plugins-make)= +`release` +: Publishes the plugin. See {ref}`gradle-plugin-publish` for more information. +`test` +: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. -(dev-plugins-extension)= +(dev-plugins-extension-points)= ## Extension points -Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how they can be used. +Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how to use them. ### Commands @@ -214,8 +216,9 @@ import nextflow.script.dsl.Description ''') class MyPluginConfig implements ConfigScope { - /* required by extension point -- do not remove */ - MyPluginConfig() {} + // no-arg constructor is required to enable validation of config options + MyPluginConfig() { + } MyPluginConfig(Map opts) { this.createMessage = opts.createMessage @@ -233,7 +236,7 @@ This approach is not required to support plugin config options. However, it allo Plugins can define custom executors that can be used with the `executor` process directive. -To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy)class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: +To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: ```groovy import nextflow.executor.Executor @@ -262,7 +265,6 @@ process hello { See the source code of Nextflow's built-in executors for examples of how to implement various executor components. ::: - ### Filesystems Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. @@ -289,7 +291,7 @@ You can then use this filesystem in your pipeline: input = file('myfs://') ``` -See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for examples of custom filesystems. +See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for an example of a custom filesystem. :::{tip} Custom filesystems are an advanced plugin extension. Before creating a new filesystem, check that your use case cannot already be supported by an existing filesystem such as HTTP or S3. @@ -323,7 +325,7 @@ class MyExtension extends PluginExtensionPoint { You can then add this function to your pipeline: -``` +```nextflow include { reverseString } from 'plugin/my-plugin' channel.of( reverseString('hi') ) @@ -382,7 +384,11 @@ channel ``` :::{note} -The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +::: + +:::{tip} +Before creating a custom operator, consider whether the operator can be defined as a [function](#functions) that can be composed with existing operators such as `map` or `subscribe`. Functions are easier to implement and can be used anywhere in your pipeline, not just channel logic. ::: ### Process directives @@ -473,7 +479,7 @@ myplugin.enabled = true See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. -(dev-plugins-env-var)= +(dev-plugins-env-vars)= ## Environment variables diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md deleted file mode 100644 index b643218eb8..0000000000 --- a/docs/plugins/example-nf-hello.md +++ /dev/null @@ -1,60 +0,0 @@ -(nf-hello-page)= - -# Example: nf-hello - -[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. - -The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) has the following structure: - -``` -nf-hello -├── COPYING -├── Makefile -├── README.md -├── build.gradle -├── gradle -│ └── wrapper -│ ├── gradle-wrapper.jar -│ └── gradle-wrapper.properties -├── gradlew -├── settings.gradle -└── src - ├── main - │ └── groovy - │ └── nextflow - │ └── hello - │ ├── HelloConfig.groovy - │ ├── HelloExtension.groovy - │ ├── HelloFactory.groovy - │ ├── HelloObserver.groovy - │ └── HelloPlugin.groovy - └── test - └── groovy - └── nextflow - └── hello - ├── HelloDslTest.groovy - └── HelloFactoryTest.groovy -``` - -It includes examples of different plugin extensions: - -- A custom trace observer that prints a message when the workflow starts and when the workflow completes. -- A custom channel factory called reverse. -- A custom operator called goodbye. -- A custom function called randomString. - -It also includes several classes that demonstrate different plugin functionality: - -- `HelloConfig`: An example of how to handle options from the Nextflow configuration. -- `HelloExtension`: An example of how to create custom channel factories, operators, and functions that can be included in pipeline scripts. -- `HelloFactory`: An example of a workflow event with custom behavior. -- `HelloObserver`: An example of a workflow event with custom behavior. -- `HelloPlugin`: An example of a plugin entry point. - -The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: - -```bash -nextflow run hello -plugins nf-hello@0.5.0 -``` - -See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 59405d71ab..470b8159a4 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -4,14 +4,9 @@ ## What are plugins -Nextflow plugins are extensions that enhance the functionality of the Nextflow workflow framework. They allow users to add new capabilities and integrate with external services without modifying core Nextflow code. +Nextflow plugins are extensions that enhance the functionality of Nextflow. They allow users to add new capabilities and integrate with external services without bloating their pipeline code or modifying Nextflow itself. -There are two types of Nextflow plugins: - -* Core plugins -* Third-party plugins - -The main features of each plugin type are described below. +There are two types of Nextflow plugins: core plugins and third-party plugins. The main features of each plugin type are described below.

Core plugins

@@ -20,7 +15,7 @@ Core plugins do not require configuration. The latest versions of core plugins a * `nf-amazon`: Support for Amazon Web Services. * `nf-azure`: Support for Microsoft Azure. * `nf-cloudcache`: Support for the cloud cache. -* `nf-console`: Implement Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). +* `nf-console`: Implementation of the Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). * `nf-google`: Support for Google Cloud. * `nf-tower`: Support for [Seqera Platform](https://seqera.io/platform/). * `nf-wave`: Support for [Wave containers service](https://seqera.io/wave/). @@ -28,14 +23,14 @@ Core plugins do not require configuration. The latest versions of core plugins a Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. :::{note} -The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-var` for more information. +The automatic retrieval of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-vars` for more information. :::

Third-party plugins

Third-party plugins must be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: -``` +```groovy plugins { id 'nf-hello@0.5.0' } @@ -43,7 +38,7 @@ plugins { To configure plugins at runtime, use the `-plugins` option. For example: -``` +```bash nextflow run main.nf -plugins nf-hello@0.5.0 ``` @@ -51,17 +46,23 @@ See {ref}`using-plugins-page` for more information. ## Nextflow plugin registry -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. +:::{versionadded} 25.04.0 +::: + +:::{note} +The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. +::: + +The Nextflow plugin registry is a central repository for publishing and discovering Nextflow plugins. Nextflow is able to use the plugin registry to automatically find and download plugins at runtime. ## Versioning -Nextflow plugins follow the Semantic Versioning format: MAJOR.MINOR.PATCH (e.g., 0.5.0). This helps developers communicate the nature of changes in a release. +Nextflow plugins are free to use any versioning convention. Many plugins, including all core plugins, use [Semantic Versioning](https://semver.org/), which helps developers communicate the kind of changes in a release. -Version components: +Semantic versions have the form MAJOR.MINOR.PATCH (e.g., 0.5.0): -* MAJOR: Increment for incompatible changes. +* MAJOR: Increment for backward-incompatible changes. * MINOR: Increment for backward-compatible feature additions. * PATCH: Increment for backward-compatible bug fixes. Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. -See {ref}`dev-plugins-versioning` for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index 282bd92b28..4e21292cc7 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -10,7 +10,7 @@ Nextflow core plugins require no additional configuration. When a pipeline uses A plugin identifier consists of the plugin name and version, separated by an `@` symbol: -```console +``` nf-hello@0.5.0 ``` @@ -26,61 +26,54 @@ Plugin versions are required for {ref}`offline usage `. The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. :::{tip} -It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. +It is recommended to pin the major and minor version of each plugin that you use, in order to minimize the risk of breaking changes while allowing patch updates to be used automatically. ::: (using-plugins-config)= ## Configuration -Plugins can be configured via nextflow configuration files or at runtime. +Plugins can be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: ```nextflow plugins { id 'nf-hello@0.5.0' + id 'nf-amazon@2.9.0' } ``` To configure plugins at runtime, use the `-plugins` option. For example: -```bash -nextflow run main.nf -plugins nf-hello@0.5.0 -``` - -To configure multiple plugins at runtime, use the `-plugins` option and a comma-separated list. For example: - ```bash nextflow run main.nf -plugins nf-hello@0.5.0,nf-amazon@2.9.0 ``` :::{note} -Plugin declarations in nextflow configuration files are ignored when specifying plugins via the `-plugins` option. +Plugin declarations in Nextflow configuration files are ignored when specifying plugins via the `-plugins` option. ::: ## Caching -When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. +When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). (using-plugins-offline)= ## Offline usage -Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. +When running Nextflow in an offline environment, any required plugins must be downloaded and moved into the offline environment prior to any runs. To use Nextflow plugins in an offline environment: 1. Install a self-contained version of Nextflow in an environment with an internet connection. See {ref}`install-standalone` for more information. -2. Run `nextflow plugin install @` to download the plugins. - :::{tip} - Running a pipeline in an environment with an internet connection will also download the plugins used by that pipeline. - ::: +2. Run `nextflow plugin install @` for each required plugin to download it. Alternatively, run the pipeline once, which will automatically download all plugins required by the pipeline. 3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. + 4. Specify each plugin and its version in Nextflow configuration files or at runtime. See {ref}`using-plugins-config` for more information. :::{warning} Nextflow will attempt to download newer versions of plugins if their versions are not set. See {ref}`using-plugins-identifiers` for more information. - ::: \ No newline at end of file + ::: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 82e397aeab..580499efef 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -964,7 +964,7 @@ $ nextflow plugin [options] : :::{versionadded} 25.04.0 ::: -: Create a plugin scaffold using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. +: Create a plugin scaffold using the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. `install ` diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index b3105c6097..445e66c2a9 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -161,6 +161,9 @@ The following environment variables control the configuration of the Nextflow ru `NXF_PLUGINS_DIR` : The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins`). +`NXF_PLUGINS_INDEX_URL` +: The plugin registry URL from which to download plugins (default: `https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json`). + `NXF_PLUGINS_TEST_REPOSITORY` : :::{versionadded} 23.04.0 ::: diff --git a/docs/reference/stdlib-groovy.md b/docs/reference/stdlib-groovy.md index 2d0bb83c8a..add202a445 100644 --- a/docs/reference/stdlib-groovy.md +++ b/docs/reference/stdlib-groovy.md @@ -21,5 +21,5 @@ println groovy.json.JsonOutput.toJson(vals) ``` :::{note} -The set of classes in Nextflow's runtime classpath can change between different Nextflow versions. As a best practice, any code that uses classes outside the Nextflow standard library should either be refactored to only use the Nextflow standard library or be refactored as a {ref}`plugin ` with explicit dependencies. +The set of classes in Nextflow's runtime classpath can change between different Nextflow versions. As a best practice, any code that uses classes outside the Nextflow standard library should either be refactored to only use the Nextflow standard library or be refactored as a {ref}`plugin ` with explicit dependencies. ::: From 7e209d757255119a613afe933992bbbcbce45557 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 11 Aug 2025 14:13:30 +1200 Subject: [PATCH 24/29] Revise language and sections Signed-off-by: Christopher Hakkaart --- docs/guides/gradle-plugin.md | 28 ++++++++++++++++++---------- docs/guides/plugin-registry.md | 14 +++++++------- docs/plugins/developing-plugins.md | 10 +++++----- docs/plugins/using-plugins.md | 2 +- 4 files changed, 31 insertions(+), 23 deletions(-) diff --git a/docs/guides/gradle-plugin.md b/docs/guides/gradle-plugin.md index ab825053eb..8fdfcc201f 100644 --- a/docs/guides/gradle-plugin.md +++ b/docs/guides/gradle-plugin.md @@ -5,7 +5,7 @@ The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. :::{note} -The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. +The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`Migrating to the Nextflow plugin registry ` for more information. ::: (gradle-plugin-create)= @@ -15,17 +15,23 @@ The Nextflow Gradle plugin and plugin registry are currently available as a priv :::{versionadded} 25.04.0 ::: -The easiest way to get started with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which in turn uses the Gradle plugin. +The best way to create a plugin with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command and create a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`dev-plugins-template` for more information about the Nextflow plugin template. -To create a Nextflow plugin with the Gradle plugin, run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. +To create Nextflow plugins with the Gradle plugin: -See {ref}`dev-plugins-template` for more information about the Nextflow plugin template. See {ref}`dev-plugins-extension-points` for more information about using plugin extension points. +1. Run `nextflow plugin create`. -## Building a plugin +2. Follow the prompts to add your plugin name, organization name, and project path. -To build a plugin, run `make assemble`. +3. Develop your plugin extension points. See {ref}`dev-plugins-extension-points` for more information. -Plugins can also be installed locally without being published. To install a plugin locally: +4. In the plugin root directory, run `make assemble`. + +## Installing a plugin + +Plugins can be installed locally without being published. + +To install plugins locally: 1. In the plugin root directory, run `make install`. @@ -47,6 +53,8 @@ Plugins can also be installed locally without being published. To install a plug ## Testing a plugin +Testing your Nextflow plugin involves two complementary approaches: unit tests and end-to-end tests. +

Unit tests

Unit tests are small, focused tests designed to verify the behavior of individual plugin components. @@ -59,15 +67,15 @@ To run unit tests:

End-to-end tests

-End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. +End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in Nextflow pipelines. End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. (gradle-plugin-publish)= ## Publishing a plugin -The Nextflow Gradle plugin allows you to publish your plugin to the Nextflow plugin registry from the command line. +The Nextflow Gradle plugin allows you to publish plugins to the Nextflow plugin registry from the command line. -To publish your plugin: +To publish plugins: 1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. diff --git a/docs/guides/plugin-registry.md b/docs/guides/plugin-registry.md index b106454472..043afc4d48 100644 --- a/docs/guides/plugin-registry.md +++ b/docs/guides/plugin-registry.md @@ -52,7 +52,7 @@ No immediate actions are required for plugin users. The plugin configuration has ## Impact on plugin developers -Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The easiest way to do this is to migrate to the Nextflow Gradle plugin, which simplifies the development process and supports publishing to the plugin registry from the command line. +Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The best way to do this is to migrate to the Nextflow Gradle plugin, which simplifies the development process and supports publishing to the plugin registry from the command line. ### Migrating to the Nextflow Gradle plugin @@ -114,12 +114,12 @@ To migrate an existing Nextflow plugin: Replace the following: - - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.04.0`. - - `PROVIDER`: Your name or organization—for example, `acme`. - - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. + - `DEPENDENCY`: (Optional) Your plugins dependency libraries. For example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` Your plugin version. For example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin. For example, `25.04.0`. + - `PROVIDER`: Your name or organization. For example, `acme`. + - `CLASS_NAME`: Your plugin class name. For example, `acme.plugin.MyPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose. For example, `acme.plugin.MyFactory`. 5. Replace the contents of `Makefile` with the following: diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 45c72a678a..ba5b5a55e1 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -13,7 +13,7 @@ The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. :::{note} -The Nextflow Gradle plugin is currently available as a private beta. See the {ref}`migration guide ` for more information. +The Nextflow Gradle plugin is currently available as a private beta. See {ref}`Migrating to the Nextflow plugin registry ` for more information. ::: (dev-plugins-structure)= @@ -73,9 +73,9 @@ This structure contains the following key files and folders: - `settings.gradle`: The Gradle project configuration, which specifies project-specific settings such as the project name and included modules. -- `src/main/groovy///`: The main source directory, which contains the plugin source code and resources.​ +- `src/main/groovy//`: The main source directory, which contains the plugin source code and resources.​ -- `src/test/groovy///`: The test source directory, which contains the plugin unit tests. +- `src/test/groovy//`: The test source directory, which contains the plugin unit tests. - `validation`: A small Nextflow pipeline which serves as an end-to-end test for the plugin. @@ -98,7 +98,7 @@ plugins { ``` :::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +You can develop Nextflow plugins without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: ### Make commands @@ -485,7 +485,7 @@ See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/bl ## Environment variables -The following environment variables are available when developing and testing plugins: +The following environment variables are available to develop and test plugins: `NXF_PLUGINS_MODE` : The plugin execution mode. Either `prod` for production or `dev` for development. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index 4e21292cc7..5e23640622 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -26,7 +26,7 @@ Plugin versions are required for {ref}`offline usage `. The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. :::{tip} -It is recommended to pin the major and minor version of each plugin that you use, in order to minimize the risk of breaking changes while allowing patch updates to be used automatically. +It is recommended to pin the major and minor version of each plugin to minimize the risk of breaking changes while allowing patch updates to be used automatically. ::: (using-plugins-config)= From df5be4a8fcd2d506c79205db00ba936c19782f09 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 11 Aug 2025 14:32:33 +1200 Subject: [PATCH 25/29] Remove repeated text Signed-off-by: Christopher Hakkaart --- docs/migrations/25-04.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index 0727825c5a..b9fef2f550 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -48,7 +48,7 @@ See {ref}`data-lineage-page` to get started.

Simplified plugin development

-The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the official [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. +The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the official [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). It is simpler and easier to maintain than the previous plugin boilerplate. See the {ref}`gradle-plugin-page` guide for details. @@ -56,12 +56,6 @@ See the {ref}`gradle-plugin-page` guide for details. The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. ::: -

Simplified plugin development

- -The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the new [plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. - -See {ref}`gradle-plugin-create` for details. - ## Enhancements

Improved inspect command

From 0180c8634ef1dcb8fd3bed0f43125137b29e855f Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 15 Aug 2025 22:50:42 +0200 Subject: [PATCH 26/29] Update docs [ci skip] Signed-off-by: Paolo Di Tommaso --- docs/guides/gradle-plugin.md | 6 ++-- docs/guides/plugin-registry.md | 55 ++++++++++++++++++++++++++++-- docs/plugins/developing-plugins.md | 4 +-- 3 files changed, 57 insertions(+), 8 deletions(-) diff --git a/docs/guides/gradle-plugin.md b/docs/guides/gradle-plugin.md index 8fdfcc201f..d7f1204623 100644 --- a/docs/guides/gradle-plugin.md +++ b/docs/guides/gradle-plugin.md @@ -5,7 +5,7 @@ The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. :::{note} -The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`Migrating to the Nextflow plugin registry ` for more information. +The Nextflow Gradle plugin and plugin registry are currently available as a public preview. See the {ref}`Migrating to the Nextflow plugin registry ` for more information. ::: (gradle-plugin-create)= @@ -82,9 +82,9 @@ To publish plugins: 2. Add the following properties: ``` - pluginRegistry.accessToken= + npr.apiKey= ``` - Replace `` with your plugin registry access token. + Replace `` with your plugin registry API key. For additional configuration options, see the [Nextflow Gradle plugin documentation](https://github.com/nextflow-io/nextflow-plugin-gradle). 3. Run `make release`. diff --git a/docs/guides/plugin-registry.md b/docs/guides/plugin-registry.md index 043afc4d48..ada3b973e0 100644 --- a/docs/guides/plugin-registry.md +++ b/docs/guides/plugin-registry.md @@ -5,7 +5,7 @@ The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. :::{note} -The Nextflow plugin registry and Gradle plugin are currently available as a private beta. Plugin developers are encouraged to contact [info@nextflow.io](mailto:info@nextflow.io) for more information about accessing the registry. +The Nextflow plugin registry and Gradle plugin are currently available as a public preview. Plugin developers can access the registry by contacting [info@nextflow.io](mailto:info@nextflow.io) to obtain registry access tokens. ::: ## Overview @@ -82,7 +82,7 @@ To migrate an existing Nextflow plugin: ```groovy // Plugins plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' + id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.6' } // Dependencies (optional) @@ -106,7 +106,7 @@ To migrate an existing Nextflow plugin: publishing { registry { - authToken = project.findProperty('pluginRegistry.accessToken') + authToken = project.findProperty('npr.apiKey') } } } @@ -158,3 +158,52 @@ Alternatively, use the `nextflow plugin create` command to re-create your plugin The Nextflow Gradle plugin supports publishing plugins from the command line. See {ref}`gradle-plugin-publish` for more information. Once you migrate to the Gradle plugin, you will no longer be able to publish to the legacy plugin index. See the [transition timeline](#timeline) for more information. + +## Plugin Creation and Publishing Workflow + +### Components + +* **Nextflow** - https://github.com/nextflow-io/nextflow/ +* **Plugin template** - https://github.com/nextflow-io/nf-plugin-template +* **Gradle plugin for Nextflow** - https://github.com/nextflow-io/nextflow-plugin-gradle +* **Plugin registry** - https://registry.nextflow.io/ + +### Workflow Steps + +#### 1. Plugin Name Registration +- User claims a unique plugin name via the Plugin registry (e.g., `nf-foo`) +- Registry admin approves the plugin name claim (one-time process) + +#### 2. Plugin Creation +- User creates a new plugin using: `nextflow plugin create ` +- Nextflow generates plugin scaffold using the Plugin template +- Interactive mode available: `nextflow plugin create` (prompts for inputs) + +#### 3. Plugin Development +- User completes plugin implementation based on the generated scaffold +- Plugin structure follows standardized template with: + - Gradle build configuration using Nextflow Gradle plugin + - Extension points for custom functionality + - Unit and end-to-end tests + - Documentation templates + +#### 4. Local Testing +- Install plugin locally: `make install` +- Test plugin: `make test` +- Build plugin: `make assemble` + +#### 5. Plugin Publishing +- Configure registry API key in `$HOME/.gradle/gradle.properties`: + ``` + npr.apiKey= + ``` +- Release plugin: `make release` +- The Gradle plugin handles upload to the Plugin registry +- Plugin becomes accessible for public distribution +- For additional configuration options, see the [Nextflow Gradle plugin documentation](https://github.com/nextflow-io/nextflow-plugin-gradle) + +### Key Features +- **Registry-based distribution**: Centralized plugin discovery and versioning +- **Standardized structure**: Consistent development experience across plugins +- **Automated publishing**: Command-line publishing with proper validation +- **Backward compatibility**: Gradual migration from legacy GitHub-based system diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index ba5b5a55e1..3b866e313e 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -13,7 +13,7 @@ The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. :::{note} -The Nextflow Gradle plugin is currently available as a private beta. See {ref}`Migrating to the Nextflow plugin registry ` for more information. +The Nextflow Gradle plugin is currently available as a public preview. See {ref}`Migrating to the Nextflow plugin registry ` for more information. ::: (dev-plugins-structure)= @@ -93,7 +93,7 @@ It is versioned and published to the [Gradle Plugin Portal](https://plugins.grad ```nextflow plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' + id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.6' } ``` From fa47aaded4215567c38637248303a8f7bff81b86 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 19 Aug 2025 10:45:31 +1200 Subject: [PATCH 27/29] Add registry guide Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 86 ------------ docs/guides/gradle-plugin.md | 30 ++-- docs/guides/migrate-plugin.md | 146 ++++++++++++++++++++ docs/guides/plugin-registry.md | 215 +++++------------------------ docs/index.md | 1 + docs/migrating-gradle-plugin.md | 126 ----------------- docs/migrations/index.md | 1 + docs/plugins/developing-plugins.md | 3 +- docs/plugins/example-nf-hello.md | 53 ------- 9 files changed, 200 insertions(+), 461 deletions(-) delete mode 100644 docs/gradle-plugin.md create mode 100644 docs/guides/migrate-plugin.md delete mode 100644 docs/migrating-gradle-plugin.md delete mode 100644 docs/plugins/example-nf-hello.md diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md deleted file mode 100644 index eaef8af07e..0000000000 --- a/docs/gradle-plugin.md +++ /dev/null @@ -1,86 +0,0 @@ -(gradle-plugin-page)= - -# Using the Nextflow Gradle plugin - -The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. - -:::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. -::: - -(gradle-plugin-create)= - -## Creating a plugin - -:::{versionadded} 25.04.0 -::: - -The easiest way to get started with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which in turn uses the Gradle plugin. - -To create a Nextflow plugin with the Gradle plugin, simply run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. - -See {ref}`dev-plugins-extension` for more information about available plugin extension points. - -## Building a plugin - -To build a plugin, simply run `make assemble`. - -Plugins can also be installed locally without being published. To install a plugin locally: - -1. In the plugin root directory, run `make install`. - - :::{note} - Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. - ::: - -2. Run your pipeline: - - ```bash - nextflow run main.nf -plugins @ - ``` - - :::{note} - Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. - ::: - -(gradle-plugin-test)= - -## Testing a plugin - -

Unit tests

- -Unit tests are small, focused tests designed to verify the behavior of individual plugin components. - -To run unit tests: - -1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template) for unit test examples. - -2. In the plugin root directory, run `make test`. - -

End-to-end tests

- -End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. See [nf-hello](https://github.com/nextflow-io/nf-hello) for an example of an end-to-end test for a Nextflow plugin. - -(gradle-plugin-publish)= - -## Publishing a plugin - -The Nextflow Gradle plugin allows you to publish your plugin to the Nextflow plugin registry from the command line. - -:::{note} -The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. -::: - -To publish your plugin: - -1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. - -2. Add the following properties: - - ```bash - pluginRegistry.accessToken= - ``` - - Replace `` with your plugin registry access token. - -3. Run `make release`. diff --git a/docs/guides/gradle-plugin.md b/docs/guides/gradle-plugin.md index d7f1204623..5f5162b2fb 100644 --- a/docs/guides/gradle-plugin.md +++ b/docs/guides/gradle-plugin.md @@ -2,7 +2,7 @@ # Using the Nextflow Gradle plugin -The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of improvements. This page introduces [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) and how to use it. :::{note} The Nextflow Gradle plugin and plugin registry are currently available as a public preview. See the {ref}`Migrating to the Nextflow plugin registry ` for more information. @@ -25,7 +25,9 @@ To create Nextflow plugins with the Gradle plugin: 3. Develop your plugin extension points. See {ref}`dev-plugins-extension-points` for more information. -4. In the plugin root directory, run `make assemble`. +4. Develop your tests. See {ref}`gradle-plugin-test` for more information. + +5. In the plugin root directory, run `make assemble`. ## Installing a plugin @@ -45,15 +47,17 @@ To install plugins locally: nextflow run main.nf -plugins @ ``` + Replace `@` with your plugin name and version. + :::{note} - Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. + Plugins can also be configured via Nextflow configuration files. See {ref}`using-plugins-page` for more information. ::: (gradle-plugin-test)= ## Testing a plugin -Testing your Nextflow plugin involves two complementary approaches: unit tests and end-to-end tests. +Testing your Nextflow plugin requires unit tests and end-to-end tests.

Unit tests

@@ -67,24 +71,30 @@ To run unit tests:

End-to-end tests

-End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in Nextflow pipelines. End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. +End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in Nextflow pipelines. + +End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. (gradle-plugin-publish)= ## Publishing a plugin -The Nextflow Gradle plugin allows you to publish plugins to the Nextflow plugin registry from the command line. +The Nextflow Gradle plugin allows you to publish plugins to the [Nextflow plugin registry](https://registry.nextflow.io/) from the command line. -To publish plugins: +To publish plugins to the [Nextflow plugin registry](https://registry.nextflow.io/): 1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. -2. Add the following properties: +2. Add the following property: ``` - npr.apiKey= + npr.apiKey= ``` - Replace `` with your plugin registry API key. For additional configuration options, see the [Nextflow Gradle plugin documentation](https://github.com/nextflow-io/nextflow-plugin-gradle). + Replace `` with your plugin registry API key. See {ref}`` for more information about creating an API key. 3. Run `make release`. + +## Additional resources + +For additional Nextflow Gradle plugin configuration options, see the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) repository. diff --git a/docs/guides/migrate-plugin.md b/docs/guides/migrate-plugin.md new file mode 100644 index 0000000000..c673d56b45 --- /dev/null +++ b/docs/guides/migrate-plugin.md @@ -0,0 +1,146 @@ +(migrate-plugin-page)= + +# Migrating to the Nextflow plugin registry + +The [Nextflow plugin registry](https://registry.nextflow.io/) is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The [legacy plugin index](https://github.com/nextflow-io/plugins) will be deprecated in favor of the Nextflow plugin registry. + +Starting with Nextflow 25.04, the plugin registry can be used as a direct replacement for the legacy plugin index. This page describes the migration timeline, it's impact on users and developers, and how to migrate existing plugins. + +(migrate-plugin-timeline)= + +## Migration timeline + +The migration timeline is tentative and subject to modification. + +

Nextflow 25.04

+ +The Nextflow plugin registry is available as a private beta. Nextflow 25.04 can use the Nextflow plugin registry as an opt-in feature. The Nextflow plugin registry will be automatically kept up-to-date with the [legacy plugin index](https://github.com/nextflow-io/plugins). + +During this time, plugin developers are encouraged to experiment with the Gradle plugin and plugin registry. + +

Nextflow 25.10

+ +The Nextflow plugin registry will be generally available. Nextflow 25.10 will use the plugin registry by default. The legacy plugin index will be **closed to new pull requests**. + +Developers will be required to publish to the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry. + +

Nextflow 26.04

+ +Nextflow 26.04 will only be able to use the Nextflow plugin registry. + +At some point in the future, the legacy plugin index will be **frozen** -- it will no longer receives updates from the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will remain available indefinitely. + +## Migration impact on plugin users + +No immediate actions are required for plugin users. The plugin configuration has not changed. + +## Impact on plugin developers + +Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The best way to do this is to migrate to the {ref}`Nextflow Gradle plugin`, which simplifies the development process and supports publishing to the plugin registry from the command line. + +## Migrating to the Nextflow Gradle plugin + +To migrate an existing Nextflow plugin to the {ref}`Nextflow Gradle plugin`: + +1. Remove the following files and folders: + - `buildSrc/` + - `launch.sh` + - `plugins/build.gradle` + +2. If your plugin has a `plugins` directory, move the `src` directory to the project root. + + :::{note} + Plugin sources should be in `src/main/groovy` or `src/main/java`. + ::: + +3. Replace the contents of `settings.gradle` with the following: + + ```groovy + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. In the project root, create a new `build.gradle` file with the following configuration: + + ```groovy + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.6' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + registry { + authToken = project.findProperty('npr.apiKey') + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (Optional) Your plugins dependency libraries. For example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` Your plugin version. For example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin. For example, `25.04.0`. + - `PROVIDER`: Your name or organization. For example, `acme`. + - `CLASS_NAME`: Your plugin class name. For example, `acme.plugin.MyPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose. For example, `acme.plugin.MyFactory`. + +5. Replace the contents of `Makefile` with the following: + + ```Makefile + # Build the plugin + assemble: + ./gradlew assemble + + clean: + rm -rf .nextflow* + rm -rf work + rm -rf build + ./gradlew clean + + # Run plugin unit tests + test: + ./gradlew test + + # Install the plugin into local nextflow plugins dir + install: + ./gradlew install + + # Publish the plugin + release: + ./gradlew releasePlugin + ``` + +6. Update `README.md` with information about the structure of your plugin. + +7. In the plugin root directory, run `make assemble`. + +Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template and add your existing plugin code. See {ref}`dev-plugins-template` for more information about the plugin template. + +## Publishing to the Nextflow plugin registry + +The Nextflow Gradle plugin supports publishing plugins from the command line. See {ref}`gradle-plugin-publish` for more information. + +:::{warning} +Once you migrate to the Gradle plugin, you will no longer be able to publish to the legacy plugin index. See the [Migration timeline](#migration-timeline) for more information. +::: diff --git a/docs/guides/plugin-registry.md b/docs/guides/plugin-registry.md index ada3b973e0..0bcc68cc38 100644 --- a/docs/guides/plugin-registry.md +++ b/docs/guides/plugin-registry.md @@ -1,209 +1,56 @@ (plugin-registry-page)= -# Migrating to the Nextflow plugin registry +# The Nextflow plugin registry -The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. +The [Nextflow plugin registry](https://registry.nextflow.io/) is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. -:::{note} -The Nextflow plugin registry and Gradle plugin are currently available as a public preview. Plugin developers can access the registry by contacting [info@nextflow.io](mailto:info@nextflow.io) to obtain registry access tokens. -::: - -## Overview - -### Nextflow plugin registry - -The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. Nextflow 25.04 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. - -### Nextflow Gradle plugin - -The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. - -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. - -## Timeline - -The [legacy plugin index](https://github.com/nextflow-io/plugins) will be deprecated in favor of the Nextflow plugin registry. +Nextflow 25.04 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. See {ref}`migrate-plugin-page` for more information about migrating to the Nextflow plugin registry. :::{note} -The following timeline is tentative and subject to modification. +The Nextflow plugin registry is currently available as a public preview. Plugin developers can access the registry by contacting [info@nextflow.io](mailto:info@nextflow.io) to obtain registry access tokens. ::: -

Nextflow 25.04

- -The Nextflow plugin registry is available as a private beta. Nextflow 25.04 can use the Nextflow plugin registry as an opt-in feature. The Nextflow plugin registry will be automatically kept up-to-date with the [legacy plugin index](https://github.com/nextflow-io/plugins). - -During this time, plugin developers are encouraged to experiment with the Gradle plugin and plugin registry. - -

Nextflow 25.10

- -The Nextflow plugin registry will be generally available. Nextflow 25.10 will use the plugin registry by default. The legacy plugin index will be **closed to new pull requests**. - -Developers will be required to publish to the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry. - -

Nextflow 26.04

- -Nextflow 26.04 will only be able to use the Nextflow plugin registry. - -At some point in the future, the legacy plugin index will be **frozen** -- it will no longer receives updates from the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will remain available indefinitely. - -## Impact on plugin users - -No immediate actions are required for plugin users. The plugin configuration has not changed. - -## Impact on plugin developers - -Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The best way to do this is to migrate to the Nextflow Gradle plugin, which simplifies the development process and supports publishing to the plugin registry from the command line. +## Claiming a plugin -### Migrating to the Nextflow Gradle plugin +Ownership of a plugin is required to publish plugins to the Nextflow plugin registry. -To migrate an existing Nextflow plugin: +To claim ownership of a plugin: -1. Remove the following files and folders: - - `buildSrc/` - - `launch.sh` - - `plugins/build.gradle` - -2. If your plugin has a `plugins` directory, move the `src` directory to the project root. +1. Open the [Nextflow plugin registry](https://registry.nextflow.io/) in a browser. +2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address. :::{note} - Plugin sources should be in `src/main/groovy` or `src/main/java`. + If you are logging in for the first time, Seqera will send an authentication link to your email address to complete the login process. ::: -3. Replace the contents of `settings.gradle` with the following: - - ```groovy - rootProject.name = '' - ``` - - Replace `PLUGIN_NAME` with your plugin name. - -4. In the project root, create a new `build.gradle` file with the following configuration: - - ```groovy - // Plugins - plugins { - id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.6' - } - - // Dependencies (optional) - dependencies { - - } - - // Plugin version - version = '' - - nextflowPlugin { - // Minimum Nextflow version - nextflowVersion = '' - - // Plugin metadata - provider = '' - className = '' - extensionPoints = [ - '' - ] - - publishing { - registry { - authToken = project.findProperty('npr.apiKey') - } - } - } - ``` - - Replace the following: +3. Go to the **My plugins** page and select **Claim a plugin**. +4. Enter your unique plugin name or select the plugin you wish to claim in the **Plugin name** field. +5. Enter your organization name in the **Provider** field. - - `DEPENDENCY`: (Optional) Your plugins dependency libraries. For example, `commons-io:commons-io:2.18.0`. - - `PLUGIN_VERSION:` Your plugin version. For example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin. For example, `25.04.0`. - - `PROVIDER`: Your name or organization. For example, `acme`. - - `CLASS_NAME`: Your plugin class name. For example, `acme.plugin.MyPlugin`. - - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose. For example, `acme.plugin.MyFactory`. - -5. Replace the contents of `Makefile` with the following: - - ```Makefile - # Build the plugin - assemble: - ./gradlew assemble - - clean: - rm -rf .nextflow* - rm -rf work - rm -rf build - ./gradlew clean - - # Run plugin unit tests - test: - ./gradlew test - - # Install the plugin into local nextflow plugins dir - install: - ./gradlew install - - # Publish the plugin - release: - ./gradlew releasePlugin - ``` - -6. Update `README.md` with information about the structure of your plugin. + :::{note} + This organization must match with the organization specified when publishing your plugin. + ::: -7. In the plugin root directory, run `make assemble`. +6. Select **Submit Request**. -Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template and add your existing plugin code. See {ref}`dev-plugins-template` for more information about the plugin template. +The plugin will show as **PENDING REVIEW** under **Pending Ownership Requests** until an admin approves the claim. Admin approval is required only once. -### Publishing to the Nextflow plugin registry +## Creating an access token -The Nextflow Gradle plugin supports publishing plugins from the command line. See {ref}`gradle-plugin-publish` for more information. +An API access token is required to publish plugins to the Nextflow plugin registry. -Once you migrate to the Gradle plugin, you will no longer be able to publish to the legacy plugin index. See the [transition timeline](#timeline) for more information. +To create an API access token: -## Plugin Creation and Publishing Workflow +1. Open the [Nextflow plugin registry](https://registry.nextflow.io/) in a browser. +2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address. -### Components + :::{note} + If you are logging in for the first time, Seqera will send an authentication link to your email address to complete the login process. + ::: -* **Nextflow** - https://github.com/nextflow-io/nextflow/ -* **Plugin template** - https://github.com/nextflow-io/nf-plugin-template -* **Gradle plugin for Nextflow** - https://github.com/nextflow-io/nextflow-plugin-gradle -* **Plugin registry** - https://registry.nextflow.io/ +3. Go to the **Access tokens** page. +4. Under **Create New Access Token**, enter a descriptive name for the **Token name** and select the token duration from the **Expiry** drop down. +5. Select **Generate token**. +6. Copy and past token somewhere safe, you won't be able to see it again. -### Workflow Steps - -#### 1. Plugin Name Registration -- User claims a unique plugin name via the Plugin registry (e.g., `nf-foo`) -- Registry admin approves the plugin name claim (one-time process) - -#### 2. Plugin Creation -- User creates a new plugin using: `nextflow plugin create ` -- Nextflow generates plugin scaffold using the Plugin template -- Interactive mode available: `nextflow plugin create` (prompts for inputs) - -#### 3. Plugin Development -- User completes plugin implementation based on the generated scaffold -- Plugin structure follows standardized template with: - - Gradle build configuration using Nextflow Gradle plugin - - Extension points for custom functionality - - Unit and end-to-end tests - - Documentation templates - -#### 4. Local Testing -- Install plugin locally: `make install` -- Test plugin: `make test` -- Build plugin: `make assemble` - -#### 5. Plugin Publishing -- Configure registry API key in `$HOME/.gradle/gradle.properties`: - ``` - npr.apiKey= - ``` -- Release plugin: `make release` -- The Gradle plugin handles upload to the Plugin registry -- Plugin becomes accessible for public distribution -- For additional configuration options, see the [Nextflow Gradle plugin documentation](https://github.com/nextflow-io/nextflow-plugin-gradle) - -### Key Features -- **Registry-based distribution**: Centralized plugin discovery and versioning -- **Standardized structure**: Consistent development experience across plugins -- **Automated publishing**: Command-line publishing with proper validation -- **Backward compatibility**: Gradual migration from legacy GitHub-based system +Once you have your token, see {ref}`gradle-plugin-publish` for instructions on how to use it. diff --git a/docs/index.md b/docs/index.md index 563857b1fb..5af061b1a0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -184,5 +184,6 @@ tutorials/flux guides/aws-java-sdk-v2 guides/gradle-plugin guides/plugin-registry +guides/migrate-plugin guides/updating-spot-retries ``` diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md deleted file mode 100644 index ab07a0ce01..0000000000 --- a/docs/migrating-gradle-plugin.md +++ /dev/null @@ -1,126 +0,0 @@ -(migrating-plugin-page)= - -# Migrating to the Nextflow plugin registry - -The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. - -## Overview - -### Nextflow plugin registry - -The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. It is intended as a replacement for the [plugins index](https://github.com/nextflow-io/plugins) hosted on GitHub. - -:::{note} -The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. -::: - -### Nextflow Gradle plugin - -The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. - -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. - -## Impact on plugin users - -If you are a plugin user, no immediate actions are required. The plugin configuration has not changed. - -## Impact on plugin developers - -Developers are encouraged to migrate to the Nextflow Gradle plugin in order to take advantage of the simplified development and publishing process. - -To migrate an existing Nextflow plugin: - -1. Remove the following files and folders: - - `buildSrc/` - - `launch.sh` - - `plugins/build.gradle` - -2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. - - :::{note} - Plugin sources should be in `src/main/groovy` or `src/main/java`. - ::: - -3. Replace the contents of `settings.gradle` with the following: - - ```groovy - rootProject.name = '' - ``` - - Replace `PLUGIN_NAME` with your plugin name. - -4. In the project root, create a new `build.gradle` file with the following configuration: - - ```groovy - // Plugins - plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' - } - - // Dependencies (optional) - dependencies { - - } - - // Plugin version - version = '' - - nextflowPlugin { - // Minimum Nextflow version - nextflowVersion = '' - - // Plugin metadata - provider = '' - className = '' - extensionPoints = [ - '' - ] - - publishing { - registry { - authToken = project.findProperty('pluginRegistry.accessToken') - } - } - } - ``` - - Replace the following: - - - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.04.0`. - - `PROVIDER`: Your name or organization—for example, `acme`. - - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. - -5. Replace the contents of `Makefile` with the following: - - ``` - # Build the plugin - assemble: - ./gradlew assemble - - clean: - rm -rf .nextflow* - rm -rf work - rm -rf build - ./gradlew clean - - # Run plugin unit tests - test: - ./gradlew test - - # Install the plugin into local nextflow plugins dir - install: - ./gradlew install - - # Publish the plugin - release: - ./gradlew releasePlugin - ``` - -6. Update `README.md` with information about the structure of your plugin. - -7. In the plugin root directory, run `make assemble`. - -The Nextflow Gradle plugin also supports publishing plugins. See {ref}`gradle-plugin-publish` for more information. diff --git a/docs/migrations/index.md b/docs/migrations/index.md index acd1f0fb76..8ac108b5ac 100644 --- a/docs/migrations/index.md +++ b/docs/migrations/index.md @@ -7,6 +7,7 @@ This section covers important information for migrating between Nextflow version ```{toctree} :maxdepth: 1 +25-10 25-04 24-10 24-04 diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 3b866e313e..0327c53d9d 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -10,7 +10,7 @@ This page describes how to develop plugins for Nextflow. The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It uses [Gradle](https://gradle.org/), a build automation tool optimized for Java and Groovy projects, as well as the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle). -You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. +You can use the `nextflow plugin create` sub-command to create plugins using the plugin template. See {ref}`gradle-plugin-create` for more information. :::{note} The Nextflow Gradle plugin is currently available as a public preview. See {ref}`Migrating to the Nextflow plugin registry ` for more information. @@ -496,7 +496,6 @@ The following environment variables are available to develop and test plugins: `NXF_PLUGINS_DEFAULT` : Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: true). - `NXF_PLUGINS_DEV` : Comma-separated list of development plugin root directories. diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md deleted file mode 100644 index 2d9babde42..0000000000 --- a/docs/plugins/example-nf-hello.md +++ /dev/null @@ -1,53 +0,0 @@ -(nf-template-page)= - -# Example: nf-plugin-template - -[nf-hello](https://github.com/nextflow-io/nf-hello) is a simple Nextflow plugin that uses the Nextflow Gradle plugin and provides examples for various plugin extension points. - -The `nf-hello` plugin has the following structure: - -``` -nf-hello -├── COPYING -├── Makefile -├── README.md -├── build.gradle -├── gradle -│ └── wrapper -│ ├── gradle-wrapper.jar -│ └── gradle-wrapper.properties -├── gradlew -├── settings.gradle -└── src - ├── main - │ └── groovy - │ └── nextflow - │ └── hello - │ ├── HelloConfig.groovy - │ ├── HelloExtension.groovy - │ ├── HelloFactory.groovy - │ ├── HelloObserver.groovy - │ └── HelloPlugin.groovy - └── test - └── groovy - └── nextflow - └── hello - ├── HelloDslTest.groovy - └── HelloFactoryTest.groovy -``` - -It includes the following extensions: - -- A custom `hello` config scope (see `HelloConfig`). -- A custom trace observer that prints a message when the workflow starts and when the workflow completes (see `HelloObserver`). -- A custom channel factory called `reverse` (see `HelloExtension`). -- A custom operator called `goodbye` (see `HelloExtension`). -- A custom function called `randomString` (see `HelloExtension`). - -The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: - -```bash -nextflow run hello -plugins nf-hello@0.5.0 -``` - -See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello) for the plugin source code. From 822e178bed07f6b3c22953707a949af4afe1aba2 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 19 Aug 2025 10:47:49 +1200 Subject: [PATCH 28/29] Add missing link Signed-off-by: Christopher Hakkaart --- docs/guides/gradle-plugin.md | 2 +- docs/guides/plugin-registry.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/guides/gradle-plugin.md b/docs/guides/gradle-plugin.md index 5f5162b2fb..1777283d76 100644 --- a/docs/guides/gradle-plugin.md +++ b/docs/guides/gradle-plugin.md @@ -91,7 +91,7 @@ To publish plugins to the [Nextflow plugin registry](https://registry.nextflow.i npr.apiKey= ``` - Replace `` with your plugin registry API key. See {ref}`` for more information about creating an API key. + Replace `` with your plugin registry API key. See {ref}`plugin-registry-create` for more information about creating an API key. 3. Run `make release`. diff --git a/docs/guides/plugin-registry.md b/docs/guides/plugin-registry.md index 0bcc68cc38..e6aa8a7eac 100644 --- a/docs/guides/plugin-registry.md +++ b/docs/guides/plugin-registry.md @@ -35,6 +35,8 @@ To claim ownership of a plugin: The plugin will show as **PENDING REVIEW** under **Pending Ownership Requests** until an admin approves the claim. Admin approval is required only once. +(plugin-registry-create)= + ## Creating an access token An API access token is required to publish plugins to the Nextflow plugin registry. From e5206135a920b614e7142c34904110aae19b0474 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 22 Aug 2025 15:12:16 -0500 Subject: [PATCH 29/29] cleanup Signed-off-by: Ben Sherman --- docs/azure.md | 14 +++--- docs/git.md | 12 +++-- docs/guides/gradle-plugin.md | 48 +++++++++++++------- docs/guides/plugin-registry.md | 72 ++++++++++-------------------- docs/index.md | 21 ++++----- docs/migrations/25-04.md | 6 +-- docs/migrations/25-10.md | 37 +++++++++++++++ docs/migrations/index.md | 1 + docs/plugins/developing-plugins.md | 36 ++++++++------- docs/plugins/plugin-registry.md | 65 +++++++++++++++++++++++++++ docs/plugins/plugins.md | 11 ----- docs/plugins/using-plugins.md | 4 +- docs/reference/env-vars.md | 16 ++++++- 13 files changed, 225 insertions(+), 118 deletions(-) create mode 100644 docs/plugins/plugin-registry.md diff --git a/docs/azure.md b/docs/azure.md index 9342f07631..aabad96d44 100644 --- a/docs/azure.md +++ b/docs/azure.md @@ -497,23 +497,27 @@ By default, Nextflow creates SAS tokens for specific containers and passes them :::{versionadded} 25.05.0-edge ::: +:::{warning} +Only available when using [Fusion filesystem](fusion.md)! +::: + You can also authenticate to Azure Storage using a managed identity when using Fusion. To do this: 1. Create a user-assigned managed identity with the Azure Storage Blob Data Contributor role for your storage account. 2. Attach the user-assigned managed identity to the node pool manually. -3. Set `azure.managedIdentity.clientId` to this identity in your configuration. +3. Set `azure.batch.poolIdentityClientId` to this identity in your configuration. ```groovy azure { - managedIdentity { - clientId = '' + batch { + poolIdentityClientId = '' } } ``` -Each task authenticates as the managed identity, and downloads and uploads files to Azure Storage using these credentials. It is possible to attach more than one managed identity to a pool. Fusion uses the identity specified by `azure.managedIdentity.clientId`. +Each task authenticates as the managed identity, and downloads and uploads files to Azure Storage using these credentials. It is possible to attach more than one managed identity to a pool. Fusion uses the identity specified by `azure.batch.poolIdentityClientId`. ### Task packing @@ -535,7 +539,7 @@ Azure virtual machines come with fixed storage disks that are not expandable. Ta Every process in your pipeline must specify a container in order to be executed on Azure Batch. -Nextflow supports both public and private container images. You can pull public images from repositories such as https://community.wave.seqera.io/. You must authenticate for any private images that you use. Ensure the Batch pool has access when using a private registry like Azure Container Registry (ACR). Provide ACR credentials in the `azure.registry` configuration scope: +Nextflow supports both public and private container images. You can pull public images from repositories such as . You must authenticate for any private images that you use. Ensure the Batch pool has access when using a private registry like Azure Container Registry (ACR). Provide ACR credentials in the `azure.registry` configuration scope: ```groovy azure { diff --git a/docs/git.md b/docs/git.md index cddfcae4c7..0162a480b8 100644 --- a/docs/git.md +++ b/docs/git.md @@ -32,7 +32,7 @@ The following configuration properties are supported for each provider configura : User password required to access private repositories on the SCM server. `providers..token` -: *Required only for private Gitlab servers* +: *Required only for private Gitlab servers and supported for BitBucket* : Private API access token. `providers..platform` @@ -51,19 +51,23 @@ The following configuration properties are supported for each provider configura ### BitBucket -Create a `bitbucket` entry in the [SCM configuration file](#git-configuration) specifying your user name and app password, as shown below: +Create a `bitbucket` entry in the [SCM configuration file](#git-configuration) specifying your user name and either an API token or app password, as shown below: ```groovy providers { bitbucket { user = 'me' - password = 'my-secret' + token = 'my-api-token' } } ``` +:::{versionadded} 25.07.0-edge +API tokens are supported for BitBucket authentication. When both `token` and `password` are provided, the API token takes priority over the app password. +::: + :::{note} -App passwords are substitute passwords for a user account which you can use for scripts and integrating tools in order to avoid putting your real password into configuration files. Learn more at [this link](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/). +API tokens are substitute passwords for a user account which you can use for scripts and integrating tools in order to avoid putting your real password into configuration files. Learn more about [app passwords](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/) and [API tokens](https://support.atlassian.com/bitbucket-cloud/docs/using-api-tokens/). ::: ### BitBucket Server diff --git a/docs/guides/gradle-plugin.md b/docs/guides/gradle-plugin.md index ab825053eb..1777283d76 100644 --- a/docs/guides/gradle-plugin.md +++ b/docs/guides/gradle-plugin.md @@ -2,10 +2,10 @@ # Using the Nextflow Gradle plugin -The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of improvements. This page introduces [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) and how to use it. :::{note} -The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. +The Nextflow Gradle plugin and plugin registry are currently available as a public preview. See the {ref}`Migrating to the Nextflow plugin registry ` for more information. ::: (gradle-plugin-create)= @@ -15,17 +15,25 @@ The Nextflow Gradle plugin and plugin registry are currently available as a priv :::{versionadded} 25.04.0 ::: -The easiest way to get started with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which in turn uses the Gradle plugin. +The best way to create a plugin with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command and create a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`dev-plugins-template` for more information about the Nextflow plugin template. -To create a Nextflow plugin with the Gradle plugin, run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. +To create Nextflow plugins with the Gradle plugin: -See {ref}`dev-plugins-template` for more information about the Nextflow plugin template. See {ref}`dev-plugins-extension-points` for more information about using plugin extension points. +1. Run `nextflow plugin create`. -## Building a plugin +2. Follow the prompts to add your plugin name, organization name, and project path. -To build a plugin, run `make assemble`. +3. Develop your plugin extension points. See {ref}`dev-plugins-extension-points` for more information. -Plugins can also be installed locally without being published. To install a plugin locally: +4. Develop your tests. See {ref}`gradle-plugin-test` for more information. + +5. In the plugin root directory, run `make assemble`. + +## Installing a plugin + +Plugins can be installed locally without being published. + +To install plugins locally: 1. In the plugin root directory, run `make install`. @@ -39,14 +47,18 @@ Plugins can also be installed locally without being published. To install a plug nextflow run main.nf -plugins @ ``` + Replace `@` with your plugin name and version. + :::{note} - Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. + Plugins can also be configured via Nextflow configuration files. See {ref}`using-plugins-page` for more information. ::: (gradle-plugin-test)= ## Testing a plugin +Testing your Nextflow plugin requires unit tests and end-to-end tests. +

Unit tests

Unit tests are small, focused tests designed to verify the behavior of individual plugin components. @@ -59,24 +71,30 @@ To run unit tests:

End-to-end tests

-End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. +End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in Nextflow pipelines. + +End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. (gradle-plugin-publish)= ## Publishing a plugin -The Nextflow Gradle plugin allows you to publish your plugin to the Nextflow plugin registry from the command line. +The Nextflow Gradle plugin allows you to publish plugins to the [Nextflow plugin registry](https://registry.nextflow.io/) from the command line. -To publish your plugin: +To publish plugins to the [Nextflow plugin registry](https://registry.nextflow.io/): 1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. -2. Add the following properties: +2. Add the following property: ``` - pluginRegistry.accessToken= + npr.apiKey= ``` - Replace `` with your plugin registry access token. + Replace `` with your plugin registry API key. See {ref}`plugin-registry-create` for more information about creating an API key. 3. Run `make release`. + +## Additional resources + +For additional Nextflow Gradle plugin configuration options, see the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) repository. diff --git a/docs/guides/plugin-registry.md b/docs/guides/plugin-registry.md index b106454472..7d7dc40029 100644 --- a/docs/guides/plugin-registry.md +++ b/docs/guides/plugin-registry.md @@ -1,43 +1,27 @@ -(plugin-registry-page)= +(migrate-plugin-registry-page)= # Migrating to the Nextflow plugin registry -The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. +The [Nextflow plugin registry](https://registry.nextflow.io/) is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The [legacy plugin index](https://github.com/nextflow-io/plugins) will be deprecated in favor of the Nextflow plugin registry. -:::{note} -The Nextflow plugin registry and Gradle plugin are currently available as a private beta. Plugin developers are encouraged to contact [info@nextflow.io](mailto:info@nextflow.io) for more information about accessing the registry. -::: +Starting with Nextflow 25.10, the plugin registry can be used as a direct replacement for the legacy plugin index. This page describes the migration timeline, it's impact on users and developers, and how to migrate existing plugins. -## Overview +(migrate-plugin-timeline)= -### Nextflow plugin registry +## Migration timeline -The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. Nextflow 25.04 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. - -### Nextflow Gradle plugin - -The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. - -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. - -## Timeline - -The [legacy plugin index](https://github.com/nextflow-io/plugins) will be deprecated in favor of the Nextflow plugin registry. - -:::{note} -The following timeline is tentative and subject to modification. -::: - -

Nextflow 25.04

- -The Nextflow plugin registry is available as a private beta. Nextflow 25.04 can use the Nextflow plugin registry as an opt-in feature. The Nextflow plugin registry will be automatically kept up-to-date with the [legacy plugin index](https://github.com/nextflow-io/plugins). - -During this time, plugin developers are encouraged to experiment with the Gradle plugin and plugin registry. +The migration timeline is tentative and subject to modification.

Nextflow 25.10

The Nextflow plugin registry will be generally available. Nextflow 25.10 will use the plugin registry by default. The legacy plugin index will be **closed to new pull requests**. +The legacy plugin index can still be used by setting the `NXF_PLUGINS_REGISTRY_URL` environment variable: + +```bash +export NXF_PLUGINS_REGISTRY_URL="https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json" +``` + Developers will be required to publish to the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry.

Nextflow 26.04

@@ -46,17 +30,17 @@ Nextflow 26.04 will only be able to use the Nextflow plugin registry. At some point in the future, the legacy plugin index will be **frozen** -- it will no longer receives updates from the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will remain available indefinitely. -## Impact on plugin users +## Migration impact on plugin users No immediate actions are required for plugin users. The plugin configuration has not changed. ## Impact on plugin developers -Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The easiest way to do this is to migrate to the Nextflow Gradle plugin, which simplifies the development process and supports publishing to the plugin registry from the command line. +Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The best way to do this is to migrate to the {ref}`Nextflow Gradle plugin `, which simplifies the development process and supports publishing to the plugin registry from the command line. -### Migrating to the Nextflow Gradle plugin +## Migrating to the Nextflow Gradle plugin -To migrate an existing Nextflow plugin: +To migrate an existing Nextflow plugin to the {ref}`Nextflow Gradle plugin `: 1. Remove the following files and folders: - `buildSrc/` @@ -82,7 +66,7 @@ To migrate an existing Nextflow plugin: ```groovy // Plugins plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' + id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.6' } // Dependencies (optional) @@ -103,23 +87,17 @@ To migrate an existing Nextflow plugin: extensionPoints = [ '' ] - - publishing { - registry { - authToken = project.findProperty('pluginRegistry.accessToken') - } - } } ``` Replace the following: - - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.04.0`. - - `PROVIDER`: Your name or organization—for example, `acme`. - - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. + - `DEPENDENCY`: (Optional) Your plugins dependency libraries. For example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` Your plugin version. For example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin. For example, `25.04.0`. + - `PROVIDER`: Your name or organization. For example, `acme`. + - `CLASS_NAME`: Your plugin class name. For example, `acme.plugin.MyPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose. For example, `acme.plugin.MyFactory`. 5. Replace the contents of `Makefile` with the following: @@ -153,8 +131,6 @@ To migrate an existing Nextflow plugin: Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template and add your existing plugin code. See {ref}`dev-plugins-template` for more information about the plugin template. -### Publishing to the Nextflow plugin registry +## Publishing to the Nextflow plugin registry The Nextflow Gradle plugin supports publishing plugins from the command line. See {ref}`gradle-plugin-publish` for more information. - -Once you migrate to the Gradle plugin, you will no longer be able to publish to the legacy plugin index. See the [transition timeline](#timeline) for more information. diff --git a/docs/index.md b/docs/index.md index 563857b1fb..46729bbc8a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -111,6 +111,17 @@ kubernetes fusion ``` +```{toctree} +:hidden: +:caption: Plugins +:maxdepth: 1 + +plugins/plugins +plugins/using-plugins +plugins/developing-plugins +plugins/plugin-registry +``` + ```{toctree} :hidden: :caption: Language Reference @@ -155,16 +166,6 @@ developer/config-scopes developer/packages ``` -```{toctree} -:hidden: -:caption: Plugins -:maxdepth: 1 - -plugins/plugins -plugins/using-plugins -plugins/developing-plugins -``` - ```{toctree} :hidden: :caption: Tutorials diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index a5d11de7e1..720ca77011 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -48,14 +48,10 @@ See {ref}`data-lineage-page` to get started.

Simplified plugin development

-The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the official [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. +The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the official [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). It is simpler and easier to maintain than the previous plugin boilerplate. See the {ref}`gradle-plugin-page` guide for details. -:::{note} -The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. -::: - ## Enhancements

Improved inspect command

diff --git a/docs/migrations/25-10.md b/docs/migrations/25-10.md index 95c82b7c2d..d2d1d124ee 100644 --- a/docs/migrations/25-10.md +++ b/docs/migrations/25-10.md @@ -10,6 +10,20 @@ This page is a work in progress and will be updated as features are finalized. I ## Enhancements +

Nextflow plugin registry

+ +Nextflow now uses the [Nextflow plugin registry](https://registry.nextflow.io/) to download plugins in a more efficient and scalable manner. + +The legacy plugin index can still be used by setting the `NXF_PLUGINS_REGISTRY_URL` environment variable: + +```bash +export NXF_PLUGINS_REGISTRY_URL="https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json" +``` + +:::{note} +Plugin developers will not be able to submit PRs to the legacy plugin index once the plugin registry is generally available. Plugins should be updated to publish to the Nextflow plugin registry using the {ref}`Nextflow Gradle plugin ` instead. See {ref}`migrate-plugin-registry-page` for details. +::: +

New syntax for workflow handlers

The workflow `onComplete` and `onError` handlers were previously defined by calling `workflow.onComplete` and `workflow.onError` in the pipeline script. You can now define handlers as `onComplete` and `onError` sections in an entry workflow: @@ -29,6 +43,29 @@ workflow { This syntax is simpler and easier to use with the {ref}`strict syntax `. See {ref}`workflow-handlers` for details. +

Configurable date and time formatting

+ +:::{versionadded} 25.07.0-edge +::: + +You can now customize the date and time format used in notifications, reports, and console output using the `NXF_DATE_FORMAT` environment variable. This addresses previous inconsistencies in timestamp representations: + +```bash +# Use ISO format with timezone +export NXF_DATE_FORMAT=iso +nextflow run workflow.nf +# Output: 2016-08-11T09:40:20+02:00 + +# Use custom format +export NXF_DATE_FORMAT="yyyy-MM-dd HH:mm" +nextflow run workflow.nf +# Output: 2016-08-11 09:40 + +# Default format includes seconds +nextflow run workflow.nf +# Output: 11-Aug-2016 09:40:20 +``` + ## Breaking changes - The AWS Java SDK used by Nextflow was upgraded from v1 to v2, which introduced some breaking changes to the `aws.client` config options. See {ref}`the guide ` for details. diff --git a/docs/migrations/index.md b/docs/migrations/index.md index acd1f0fb76..8ac108b5ac 100644 --- a/docs/migrations/index.md +++ b/docs/migrations/index.md @@ -7,6 +7,7 @@ This section covers important information for migrating between Nextflow version ```{toctree} :maxdepth: 1 +25-10 25-04 24-10 24-04 diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index f7caaffba6..6419d5f988 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -10,12 +10,14 @@ This page describes how to develop plugins for Nextflow. The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It uses [Gradle](https://gradle.org/), a build automation tool optimized for Java and Groovy projects, as well as the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle). -You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. +You can use the `nextflow plugin create` sub-command to create plugins using the plugin template. See {ref}`gradle-plugin-create` for more information. :::{note} -The Nextflow Gradle plugin is currently available as a private beta. See the {ref}`migration guide ` for more information. +The Nextflow Gradle plugin is currently available as a public preview. See {ref}`Migrating to the Nextflow plugin registry ` for more information. ::: +(dev-plugins-structure)= + ### Structure The plugin template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. @@ -24,6 +26,7 @@ For example, a plugin created from the plugin template with the name `nf-hello` ```console nf-hello +├── .github/workflows/build.yml ├── COPYING ├── Makefile ├── README.md @@ -55,7 +58,7 @@ nf-hello This structure contains the following key files and folders: -- `.github/workflows`: GitHub Action which implements continuous integration for the plugin. +- `.github/workflows/build.yml`: GitHub Action which implements continuous integration for the plugin. - `build.gradle`: The Gradle build script. @@ -71,9 +74,9 @@ This structure contains the following key files and folders: - `settings.gradle`: The Gradle project configuration, which specifies project-specific settings such as the project name and included modules. -- `src/main/groovy///`: The main source directory, which contains the plugin source code and resources.​ +- `src/main/groovy//`: The main source directory, which contains the plugin source code and resources.​ -- `src/test/groovy///`: The test source directory, which contains the plugin unit tests. +- `src/test/groovy//`: The test source directory, which contains the plugin unit tests. - `validation`: A small Nextflow pipeline which serves as an end-to-end test for the plugin. @@ -91,12 +94,12 @@ It is versioned and published to the [Gradle Plugin Portal](https://plugins.grad ```nextflow plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' + id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.6' } ``` :::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +You can develop Nextflow plugins without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: ### Make commands @@ -199,7 +202,7 @@ myplugin { } ``` -:::{versionadded} 25.02.0-edge +:::{versionadded} 25.04.0 ::: Plugins can declare their configuration options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation. For example: @@ -216,6 +219,12 @@ import nextflow.script.dsl.Description ''') class MyPluginConfig implements ConfigScope { + @ConfigOption + @Description(''' + Message to print to standard output when a run is initialized. + ''') + final String createMessage + // no-arg constructor is required to enable validation of config options MyPluginConfig() { } @@ -223,10 +232,6 @@ class MyPluginConfig implements ConfigScope { MyPluginConfig(Map opts) { this.createMessage = opts.createMessage } - - @ConfigOption - @Description('Message to print to standard output when a run is initialized.') - String createMessage } ``` @@ -299,7 +304,7 @@ Custom filesystems are an advanced plugin extension. Before creating a new files ### Functions -:::{versionadded} 22.09.0-edge +:::{versionadded} 22.10.0 ::: Plugins can define custom functions that can be included in Nextflow pipelines. @@ -422,7 +427,7 @@ class MyExecutor extends Executor { ### Trace observers -:::{versionchanged} 25.04 +:::{versionchanged} 25.04.0 The `TraceObserver` interface is now deprecated. Use [TraceObserverV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverV2.groovy) and [TraceObserverFactoryV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy) instead. ::: @@ -483,7 +488,7 @@ See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/bl ## Environment variables -The following environment variables are available when developing and testing plugins: +The following environment variables are available to develop and test plugins: `NXF_PLUGINS_MODE` : The plugin execution mode. Either `prod` for production or `dev` for development. @@ -494,7 +499,6 @@ The following environment variables are available when developing and testing pl `NXF_PLUGINS_DEFAULT` : Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: true). - `NXF_PLUGINS_DEV` : Comma-separated list of development plugin root directories. diff --git a/docs/plugins/plugin-registry.md b/docs/plugins/plugin-registry.md new file mode 100644 index 0000000000..a3506d8c6f --- /dev/null +++ b/docs/plugins/plugin-registry.md @@ -0,0 +1,65 @@ +(plugin-registry-page)= + +# Nextflow plugin registry + +The [Nextflow plugin registry](https://registry.nextflow.io/) is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. + +Nextflow 25.10 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. See {ref}`migrate-plugin-registry-page` for more information about migrating to the Nextflow plugin registry. + +:::{note} +The Nextflow plugin registry is currently available as a public preview. Plugin developers can access the registry by contacting [info@nextflow.io](mailto:info@nextflow.io) to obtain access to the registry. +::: + +## Claiming a plugin + +Ownership of a plugin is required to publish plugins to the Nextflow plugin registry. + +To claim ownership of a plugin: + +1. Open the [Nextflow plugin registry](https://registry.nextflow.io/) in a browser. + +2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address. + + :::{note} + If you are logging in for the first time, Seqera will send an authentication link to your email address to complete the login process. + ::: + +3. Go to the **My plugins** page and select **Claim a plugin**. + +4. Enter your unique plugin name or select the plugin you wish to claim in the **Plugin name** field. + +5. Enter your organization name in the **Provider** field. + + :::{note} + This organization must match with the organization specified when publishing your plugin. + ::: + +6. Select **Submit Request**. + +The plugin will show as **PENDING REVIEW** under **Pending Ownership Requests** until an admin approves the claim. Admin approval is required only once. + +(plugin-registry-create)= + +## Creating an access token + +An API access token is required to publish plugins to the Nextflow plugin registry. + +To create an API access token: + +1. Open the [Nextflow plugin registry](https://registry.nextflow.io/) in a browser. + +2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address. + + :::{note} + If you are logging in for the first time, Seqera will send an authentication link to your email address to complete the login process. + ::: + +3. Go to the **Access tokens** page. + +4. Under **Create New Access Token**, enter a descriptive name for the **Token name** and select the token duration from the **Expiry** drop down. + +5. Select **Generate token**. + +6. Copy and past token somewhere safe, you won't be able to see it again. + +Once you have your token, see {ref}`gradle-plugin-publish` for instructions on how to use it. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 470b8159a4..5507005c30 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -44,17 +44,6 @@ nextflow run main.nf -plugins nf-hello@0.5.0 See {ref}`using-plugins-page` for more information. -## Nextflow plugin registry - -:::{versionadded} 25.04.0 -::: - -:::{note} -The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. -::: - -The Nextflow plugin registry is a central repository for publishing and discovering Nextflow plugins. Nextflow is able to use the plugin registry to automatically find and download plugins at runtime. - ## Versioning Nextflow plugins are free to use any versioning convention. Many plugins, including all core plugins, use [Semantic Versioning](https://semver.org/), which helps developers communicate the kind of changes in a release. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index 4e21292cc7..405789c0d5 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -20,13 +20,13 @@ The plugin version is optional. If it is not specified, Nextflow will download t Plugin versions are required for {ref}`offline usage `. ::: -:::{versionadded} 25.02.0-edge +:::{versionadded} 25.04.0 ::: The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. :::{tip} -It is recommended to pin the major and minor version of each plugin that you use, in order to minimize the risk of breaking changes while allowing patch updates to be used automatically. +It is recommended to pin the major and minor version of each plugin to minimize the risk of breaking changes while allowing patch updates to be used automatically. ::: (using-plugins-config)= diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index 445e66c2a9..03deed2461 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -54,6 +54,11 @@ The following environment variables control the configuration of the Nextflow ru ::: : When `true`, override the container entrypoint with `/bin/bash` (default: `false`). +`NXF_DATE_FORMAT` +: :::{versionadded} 25.07.0-edge + ::: +: Defines the format for date and time representations in notifications and reports. Supports custom formats (e.g., `yyyy-MM-dd HH:mm:ss`) or `iso` for ISO 8601 format with timezone (default: `dd-MMM-yyyy HH:mm:ss`). + `NXF_DEFAULT_DSL` : :::{versionadded} 22.03.0-edge ::: @@ -161,8 +166,10 @@ The following environment variables control the configuration of the Nextflow ru `NXF_PLUGINS_DIR` : The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins`). -`NXF_PLUGINS_INDEX_URL` -: The plugin registry URL from which to download plugins (default: `https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json`). +`NXF_PLUGINS_REGISTRY_URL` +: :::{versionadded} 25.08.0-edge + ::: +: Specifies the URL of the plugin registry used to download and resolve plugins. This allows using custom or private plugin registries instead of the default public registry. `NXF_PLUGINS_TEST_REPOSITORY` : :::{versionadded} 23.04.0 @@ -194,6 +201,11 @@ The following environment variables control the configuration of the Nextflow ru ::: : Max delay used for HTTP retryable operations (default: `90s`). +`NXF_RETRY_POLICY_MULTIPLIER` +: :::{versionadded} 25.08.0-edge + ::: +: Delay multiplier used for HTTP retryable operations (default: `2.0`). + `NXF_SCM_FILE` : :::{versionadded} 20.10.0 :::