tedge-flows service continues to use the old script logic

[rina23q]

Describe the bug
tedge-flows service detects the file changes automatically and reloads the scripts.
However, I found a corner case, where tedge-flows still uses an old script path instead of the new one after specific steps.

To Reproduce

Preparation:

a flow test.toml

input.mqtt.topics = ["test/in"]
steps = [
    { script = "test-old.js" },
]

and test-old.js

export function onMessage(message, context) {
    return [{
        topic: "test/out",
        payload: `{"old":"I am from test-old.js"}`
    }]
}

Step to reproduce

1. Change the script path to test-new.js, which doesn't exist yet.

steps = [
    { script = "test-new.js" },
]

2. You can confirm the script doesn't exist by tedge flow test test/in ''

$ tedge flows test test/in ''
2026-02-11T13:05:43.574928405Z ERROR flows: Failed to compile flow /etc/tedge/mappers/flows/flows/test.toml: Cannot read file /etc/tedge/mappers/flows/flows/test-new.js: No such file or directory (os error 2)

3. tedge-flows service still converts the message based on the old test-old.js definition.

[test/in] 
[test/out] {"old":"I am from test-old.js"}

4. Create test-new.js with this contents.

export function onMessage(message, context) {
    return [{
        topic: "test/out",
        payload: `{"new":"I am from test-new.js"}`
    }]
}

5. tedge flows test command uses the new script.

$ tedge flows test test/in ''
[test/out] {"new":"I am from test-new.js"}

6. However, tedge-flows service still uses the old definition.

[test/in] 
[test/out] {"old":"I am from test-old.js"}

7. To recover this, restart the tedge-flows service. After restart,

[test/in] 
[test/out] {"new":"I am from test-new.js"}

Expected behavior
tedge-flows service uses the new script contents without restarting it.

Screenshots

Environment (please complete the following information):

• OS [incl. version]
• Hardware [incl. revision]
• System-Architecture Linux 2c8819180ed5 6.8.0-47-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 22:03:50 UTC 2024 aarch64 GNU/Linux
• thin-edge.io version: tedge 1.7.2~143+g747f913

Additional context
Restarting tedge-flows service resolves the issue.

[didier-wenzek]

I understand the motivations for fixing that behavior, which will happen in a dev context, but we need to be clear on the expectations.

1. When the on-disk version of a flow is broken, the working version is kept unchanged in-memory and stay active. This is what is observed step 2 and 3. I think we need to keep this behavior.
2. When a script is created or updated, the flow engine only reloads the flow definitions referencing that script. The request is to have something smarter and to also reload any flow definition that could be fixed by the script being updated. However, we need to avoid reloading everything.
3. Technically it makes sense that tedge flows test doesn't have the same behavior as the mapper: the flows are always loaded from disk and there is no attempt to reload files that have been touched. However, I agree this is a source of confusion. As a user, I expect tedge flows test to reflect what is loaded by the mapper.

I propose to fix the issue along the following lines.

1. The flow engine maintains a list of flow definitions (the .toml files) that cannot be loaded.
2. When the on-disk version of a flow is broken, the working version (if any) stays active, and the path to the broken definition is added to the list of broken files.
3. When a script is changed on disk, the flow engine reloads all running flows with a dependency on that script (as today), but also any flows for which there is a broken version on disk (this is the fix).