Merge pull request #27 from seqeralabs/poll-files-tweaks

Poll files node: Add deleted files output and configurable 'all results' output

Author: ewels

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## [1.5.0] - 2025-12-18
+
+-   💥 Breaking change: Poll files node
+    -   Added `pollIntervalSeconds` field to output payload alongside `nextPoll`
+    -   "Every poll" output port is now hidden by default, with a checkbox option to enable it
+    -   Added new "Deleted results" output that fires when files are removed from the Data Link
+
+**Warning**: The change in the available ports on this node could affect your workflows.
+Please update with care - make sure that the correct output ports are still connected!
+
 ## [1.4.1] - 2025-11-16
 
 -   Bugfix: Don't use absolute paths for internal API calls for config setup

docs/img/poll_files_node_edit.png

@@ -1,8 +1,8 @@
 # Poll files
 
-**Periodically list a Seqera Data Explorer Data Link and emit messages when new objects appear.**
+**Periodically list a Seqera Data Explorer Data Link and emit messages when objects are added or deleted.**
 
-This node automatically monitors a Data Link for _changes_, making it perfect for event-driven workflows that trigger when new files are uploaded.
+This node automatically monitors a Data Link for _changes_, making it perfect for event-driven workflows that trigger when files are uploaded or removed.
 
 !!! note
 
@@ -15,52 +15,69 @@ This node automatically monitors a Data Link for _changes_, making it perfect fo
 
 ## Configuration
 
-!!! info
+This node works much like the [list files node](list_files.md), but instead of triggering when recieving a message input, it polls repeatedly and outputs events when it detects a change.
+
+!!! warning
+
+    File addition / deletion is only detected across poll events.
+    This means that if you delete a file and create a new one with the same name
+    _inbetween_ two poll events, the change **will not be detected**.
 
-    This node works much like the [list files node](list_files.md), but instead of triggering when recieving a message input, it polls repeatedly and outputs events when it detects a change.
+    If this kind of activity is likely, make sure that you set the polling frequency
+    to be fast enough that the node will check the file listing between events
+    _(poll -> delete file -> poll -> add file -> poll)_.
 
 -   **Seqera config**: Reference to the seqera-config node containing API credentials and default workspace settings.
 -   **Node name**: Optional custom name for the node in the editor.
 -   **Data Link name** (required): Display name of the Data Link. Supports autocomplete.
+-   **Return type** (default **files**): `files`, `folders` or `all`.
+-   **Poll frequency** (default **15 min**): Interval between polls.
+-   **Every poll / Show output port** (default **off**): When enabled, adds an extra output that emits all files on every poll.
 -   **Base path**: Path within the Data Link to start from.
 -   **Prefix**: Prefix filter applied to both files and folders.
 -   **Pattern**: Regular-expression filter applied to files after the prefix filter.
--   **Return type** (default **files**): `files`, `folders` or `all`.
 -   **Max results** (default **100**): Maximum number of objects to return per poll.
 -   **Depth** (default **0**): Folder recursion depth.
--   **Poll frequency** (default **15 min**): Interval between polls.
 -   **Workspace ID**: Override the workspace ID from the Config node.
 
 All properties work the same as the [list files](list_files.md) node, plus automatic polling.
 
-## Outputs (two)
+## Outputs
+
+By default, the node has two outputs:
+
+1. **New results** – Emitted only when one or more _new_ objects are detected since the last poll.
+2. **Deleted results** – Emitted only when one or more objects are _deleted_ since the last poll.
 
-The node has two outputs that fire at different times:
+When **Every poll / Show output port** is enabled, the node has three outputs:
 
 1. **All results** – Emitted every poll with the full, filtered list of files.
 2. **New results** – Emitted only when one or more _new_ objects are detected since the last poll.
+3. **Deleted results** – Emitted only when one or more objects are _deleted_ since the last poll.
 
-Both messages include the same properties:
+All outputs include the same properties:
 
 -   `msg.payload.files` – Array of file objects from the API.
 -   `msg.payload.resourceType`, `msg.payload.resourceRef`, `msg.payload.provider` – Data Link metadata.
 -   `msg.files` – Convenience array of fully-qualified object names (strings).
 -   `msg.payload.nextPoll` (only on **All results** output) – ISO timestamp of the next scheduled poll.
+-   `msg.payload.pollIntervalSeconds` (only on **All results** output) – Poll interval duration in seconds.
 
-## How new files are detected
+## How changes are detected
 
 The node tracks seen files in its context storage. On each poll:
 
 1. Fetch the current list of files from the Data Link
 2. Compare against the list from the previous poll
-3. If new files are found, emit them on output 2
-4. Update the stored list for the next comparison
+3. If new files are found, emit them on the "New results" output
+4. If files are missing (deleted), emit them on the "Deleted results" output
+5. Update the stored list for the next comparison
 
-The comparison is based on the full file path. Files that are deleted and re-uploaded will be detected as "new".
+The comparison is based on the full file path. Files that are deleted and re-uploaded will be detected as "deleted" then "new" on subsequent polls.
 
 !!! info
 
-    The very first poll after the node is created sees everything as new and is handled as a special case. It does not output new results.
+    The very first poll after the node is created sees everything as new and is handled as a special case. It does not output new or deleted results.
 
 ## Required permissions
 
@@ -74,7 +91,7 @@ See the [configuration documentation](configuration.md#required-token-permission
 
 1. Add a **poll-files** node and configure the Data Link
 2. Set **pollFrequency** to your desired interval (e.g., `5:00` for 5 minutes)
-3. Connect output 2 (New results) to a **workflow-launch** node
+3. Connect the output (New results) to a **workflow-launch** node
 4. Configure the launch node to use the file paths from `msg.files`
 5. Deploy
 
@@ -83,12 +100,12 @@ Now every time a new file appears in the Data Link, a workflow will automaticall
 ### Trigger only on specific file types
 
 1. Set **pattern**: `.*\.bam$` to only detect BAM files
-2. Connect output 2 to your processing logic
+2. Connect the output to your processing logic
 3. The node will only emit when new BAM files appear
 
 ## Notes
 
--   The first poll after deployment/restart does **not** emit to the "New results" output (it initializes the tracking state)
+-   The first poll after deployment/restart does **not** emit to the "New results" or "Deleted results" outputs (it initializes the tracking state)
 -   The tracking is reset on each Node-RED restart or flow redeployment
 -   Very frequent polling (< 30 seconds) may impact API rate limits
 -   Custom message properties are preserved in outputs (e.g., `msg._context`)

docs/seqera_nodes/poll_files.md

@@ -31,7 +31,7 @@
     "returnType": "files",
     "x": 100,
     "y": 100,
-    "wires": [[], ["2b7682159053d63c"]]
+    "wires": [["2b7682159053d63c"], []]
   },
   {
     "id": "2b7682159053d63c",
@@ -127,5 +127,13 @@
     "x": 570,
     "y": 200,
     "wires": [[], [], []]
+  },
+  {
+    "id": "2801ba6dc68b2335",
+    "type": "global-config",
+    "env": [],
+    "modules": {
+      "@seqera/node-red-seqera": "1.4.1"
+    }
   }
 ]

examples/02 - Launch on file upload.json

@@ -9,6 +9,20 @@
     <input type="text" id="node-input-name" />
   </div>
 
+  <!-- Data link parameters (same as list node) -->
+  <div class="form-row">
+    <label for="node-input-dataLinkName"><i class="fa fa-link"></i> Data link</label>
+    <input type="text" id="node-input-dataLinkName" />
+  </div>
+  <div class="form-row">
+    <label for="node-input-returnType"><i class="fa fa-filter"></i> Return type</label>
+    <select id="node-input-returnType">
+      <option value="files">Files only</option>
+      <option value="folders">Folders only</option>
+      <option value="all">Everything</option>
+    </select>
+  </div>
+
   <!-- Polling specific -->
   <div class="form-row">
     <label for="node-input-pollFrequency"><i class="fa fa-clock-o"></i> Poll frequency</label>
@@ -20,20 +34,23 @@
       <option value="days">Days</option>
     </select>
   </div>
-
-  <!-- Data link parameters (same as list node) -->
   <div class="form-row">
-    <label for="node-input-dataLinkName"><i class="fa fa-link"></i> Data link name</label>
-    <input type="text" id="node-input-dataLinkName" />
-  </div>
-  <div class="form-row">
-    <label for="node-input-returnType"><i class="fa fa-filter"></i> Return type</label>
-    <select id="node-input-returnType">
-      <option value="files">Files only</option>
-      <option value="folders">Folders only</option>
-      <option value="all">Everything</option>
-    </select>
+    <label for="node-input-outputAllPolls">
+      <i class="fa fa-refresh" aria-hidden="true"></i>
+      Every poll
+    </label>
+    <label for="node-input-outputAllPolls" style="width:70%">
+      <input
+        type="checkbox"
+        id="node-input-outputAllPolls"
+        style="display:inline-block; width:22px; vertical-align:top;"
+        autocomplete="off"
+      />
+      Show output port
+    </label>
   </div>
+
+  <!-- More specifics -->
   <div class="form-row">
     <label for="node-input-basePath"><i class="fa fa-folder-open-o"></i> Base path</label>
     <input type="text" id="node-input-basePath" />
@@ -66,30 +83,37 @@
 
 ### Inputs
 
-: pollFrequency (string) : Poll frequency (default `15 minutes`). Can be configured in seconds, minutes, hours, or days.
 : dataLinkName (string) : The name of the data explorer link.
+: returnType (string) : Select whether to return files, folders or everything.
+: pollFrequency (string) : Poll frequency (default `15 minutes`). Can be configured in seconds, minutes, hours, or days.
+: outputAllPolls (boolean) : Show the "every poll" output port. Emits an output file listing on every poll event, irrespective of changes (disabled by default).
 : basePath (string) : Path within the data link to start browsing. Leave blank for the root.
 : prefix (string) : Optional prefix filter for results (applies to folders and files)
 : pattern (string) : Optional regex pattern filter for results (applies to files only)
-: returnType (string) : Select whether to return files, folders or everything.
 : maxResults (number) : Maximum number of results to return (default 100).
+: depth (number) : Folder recursion depth (default 0).
 : workspaceId (string) : Override the workspace ID from the config node.
 
-All inputs support msg._, flow._, global.\*, env, or JSONata expressions via the **typedInput**.
+All inputs support `msg`, `flow`, `global`, `env`, or JSONata expressions via the **typedInput**.
 
 ### Outputs
 
-The node has two outputs:
+The node has two or three outputs depending on configuration:
 
-1. All results on every poll.
-2. New objects since the previous poll (nothing sent if no new objects).
+1. **All results** (optional, disabled by default) - Emitted every poll with the full list of files.
+2. **New results** - Emitted only when new objects are detected since the previous poll.
+3. **Deleted results** - Emitted only when objects are deleted since the previous poll.
 
-Both outputs have the following properties:
+All outputs have the following properties:
 
-: payload (array) : Fle information aggregated from the API (array of objects).
+: payload (array) : File information aggregated from the API (array of objects).
 : files (array) : File names (array of strings).
 
-All typed-input fields are identical to the _List files_ node with the addition of **poll frequency**.
+The "All results" output also includes:
+
+: payload.nextPoll (string) : ISO timestamp of the next scheduled poll.
+: payload.pollIntervalSeconds (number) : Poll interval duration in seconds.
+
 </script>
 
 <script type="text/javascript">
@@ -104,7 +128,14 @@
     label: function () {
       return this.name || "Poll files";
     },
-    outputLabels: ["All objects", "Only new objects"],
+    outputLabels: function (index) {
+      if (this.outputAllPolls) {
+        // 3 outputs: All, New, Deleted
+        return ["All objects", "New objects", "Deleted objects"][index];
+      }
+      // 2 outputs: New, Deleted
+      return ["New objects", "Deleted objects"][index];
+    },
     defaults: {
       name: { value: "" },
       seqera: { value: "", type: "seqera-config" },
@@ -128,6 +159,7 @@
       pollFrequency: { value: "15" },
       pollUnits: { value: "minutes" },
       returnType: { value: "files" },
+      outputAllPolls: { value: false },
     },
     oneditprepare: function () {
       function ti(id, val, type, def = "str") {
@@ -149,6 +181,9 @@
       $("#node-input-pollFrequency").val(this.pollFrequency || "15");
       $("#node-input-pollUnits").val(this.pollUnits || "minutes");
 
+      // Output all polls checkbox
+      $("#node-input-outputAllPolls").prop("checked", this.outputAllPolls || false);
+
       $("#node-input-returnType").val(this.returnType || "files");
 
       // Add auto-complete for datalink name when type is "str"
@@ -260,6 +295,10 @@
       this.pollFrequency = $("#node-input-pollFrequency").val();
       this.pollUnits = $("#node-input-pollUnits").val();
 
+      // Save output all polls checkbox and update outputs count
+      this.outputAllPolls = $("#node-input-outputAllPolls").prop("checked");
+      this.outputs = this.outputAllPolls ? 3 : 2;
+
       this.returnType = $("#node-input-returnType").val();
     },
   });

nodes/datalink-poll.html

@@ -30,6 +30,7 @@ module.exports = function (RED) {
     node.depthProp = config.depth;
     node.depthPropType = config.depthType;
     node.returnType = config.returnType || "files"; // files|folders|all
+    node.outputAllPolls = config.outputAllPolls || false;
 
     // Poll frequency configuration
     const unitMultipliers = {
@@ -74,11 +75,15 @@ module.exports = function (RED) {
             resourceRef: result.resourceRef,
             provider: result.provider,
             nextPoll: new Date(Date.now() + node.pollFrequencySec * 1000).toISOString(),
+            pollIntervalSeconds: node.pollFrequencySec,
           },
           files: result.files.map((it) => `${result.resourceRef}/${it}`),
         };
 
-        // Second output: only new items since previous poll
+        // Build set of current names for comparison
+        const currentNamesSet = new Set(result.items.map((it) => it.name));
+
+        // New items since previous poll
         let msgNew = null;
         if (previousNamesSet) {
           const newItems = result.items.filter((it) => !previousNamesSet.has(it.name));
@@ -95,11 +100,36 @@ module.exports = function (RED) {
           }
         }
 
+        // Deleted items since previous poll
+        let msgDeleted = null;
+        if (previousNamesSet) {
+          const deletedNames = [...previousNamesSet].filter((name) => !currentNamesSet.has(name));
+          if (deletedNames.length) {
+            msgDeleted = {
+              payload: {
+                files: deletedNames.map((name) => ({ name })),
+                resourceType: result.resourceType,
+                resourceRef: result.resourceRef,
+                provider: result.provider,
+              },
+              files: deletedNames.map((name) => `${result.resourceRef}/${name}`),
+            };
+          }
+        }
+
         // Update cache
-        previousNamesSet = new Set(result.items.map((it) => it.name));
+        previousNamesSet = currentNamesSet;
 
         node.status({ fill: "green", shape: "dot", text: `${result.items.length} items: ${formatDateTime()}` });
-        node.send([msgAll, msgNew]);
+
+        // Send to outputs based on configuration
+        if (node.outputAllPolls) {
+          // Three outputs: [All results, New results, Deleted results]
+          node.send([msgAll, msgNew, msgDeleted]);
+        } else {
+          // Two outputs: [New results, Deleted results]
+          node.send([msgNew, msgDeleted]);
+        }
       } catch (err) {
         node.error(`Seqera datalink poll failed: ${err.message}`);
         node.status({ fill: "red", shape: "dot", text: `error: ${formatDateTime()}` });

nodes/datalink-poll.js

@@ -1,6 +1,6 @@
 {
   "name": "@seqera/node-red-seqera",
-  "version": "1.4.1",
+  "version": "1.5.0",
   "description": "Node-RED nodes for interacting with the Seqera Platform API",
   "author": "Phil Ewels <phil.ewels@seqera.io>",
   "license": "Apache-2.0",