Skip to content

fix: Unload flow when removing script instead of removing#4036

Closed
Bravo555 wants to merge 3 commits intothin-edge:mainfrom
Bravo555:fix/unload-flow-when-removing-script
Closed

fix: Unload flow when removing script instead of removing#4036
Bravo555 wants to merge 3 commits intothin-edge:mainfrom
Bravo555:fix/unload-flow-when-removing-script

Conversation

@Bravo555
Copy link
Member

Proposed changes

Unload flow when one of its scripts is removed instead of removing. Removing sometimes made flow not start back up when script was restored. Instead, unloading keeps it in an unloaded list that is checked when relevant script is reloaded.

This is an alternative fix to #4031 using the same test suite, but addresses flow not being reloaded without assuming stale, out-of-order FileDeleted messages, which I could not reproduce. Instead my hypothesis is that the flow wasn't being reloaded because reload_script looks at unloaded flows list to restart flows, but the flow wasn't in that list because it was removed instead of just unloaded.

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue


Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
  • I ran just format as mentioned in CODING_GUIDELINES
  • I used just check as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

reubenmiller and others added 2 commits March 11, 2026 16:10
…ning

Signed-off-by: reubenmiller <reuben.d.miller@gmail.com>
Unload flow when one of its scripts is removed instead of removing.
Removing sometimes made flow not start back up when script was restored.
Instead, unloading keeps it in an unloaded list that is checked when
relevant script is reloaded.

Signed-off-by: Marcel Guzik <marcel.guzik@cumulocity.com>
@Bravo555 Bravo555 temporarily deployed to Test Pull Request March 11, 2026 16:15 — with GitHub Actions Inactive
@Bravo555 Bravo555 marked this pull request as draft March 11, 2026 16:16
@codecov
Copy link

codecov bot commented Mar 11, 2026

Codecov Report

❌ Patch coverage is 0% with 19 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/extensions/tedge_flows/src/actor.rs 0.00% 14 Missing ⚠️
crates/extensions/tedge_flows/src/registry.rs 0.00% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 11, 2026

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass % ⏱️ Duration
852 0 3 852 100 2h43m29.694894s

@reubenmiller
Copy link
Contributor

I was testing with the PoC sm-plugin for a flow, and upgrading an existing flow results in the following log entries:

Mar 11 20:17:08 7e289213ec5f tedge-mapper[808]: 2026-03-11T20:17:08.746513095Z  WARN flows: Removing a script used by a flow protobuf-xform: /etc/tedge/mappers/local/flows/protobuf-xform/lib/main.js
Mar 11 20:17:08 7e289213ec5f tedge-mapper[808]: 2026-03-11T20:17:08.757372524Z  INFO flows: Reloading flow script /etc/tedge/mappers/local/flows/protobuf-xform/lib/main.js

The WARN level for the flow removal is overkill (INFO would be more appropriate), however interesting enough, #4031, only yields a single "Reloading flow" log entry on updates (which is preferable from a users perspective given an update for a user should trigger a reload and the "remove" operation is unnecessary (again from the users point of view).

@Bravo555
Copy link
Member Author

The WARN level for the flow removal is overkill

Indeed we were removing the flow first and the script after, in order to not trigger that warning, but it would be better to adjust it to not appear if removing a script for a flow that's unloaded.

Signed-off-by: Marcel Guzik <marcel.guzik@cumulocity.com>
@Bravo555 Bravo555 temporarily deployed to Test Pull Request March 12, 2026 13:26 — with GitHub Actions Inactive
@Bravo555 Bravo555 marked this pull request as ready for review March 12, 2026 13:30
@Bravo555
Copy link
Member Author

Removed the extra warning and added one more test case for moving a script.

@reubenmiller
Copy link
Contributor

reubenmiller commented Mar 12, 2026

Removed the extra warning and added one more test case for moving a script.

I just tested the update version and there is a lot of noise when updating a single flow (via the poc sm-plugin). The following entries show with 1 software update (which replaces the flow.toml and lib/main.js)...6 log entries in total.

Mar 12 15:20:34 7a272ddfd7d7 tedge-mapper[2303]: 2026-03-12T15:20:34.835061855Z  INFO flows: Loading flow /etc/tedge/mappers/local/flows/protobuf-xform/flow.toml
Mar 12 15:20:34 7a272ddfd7d7 tedge-mapper[2303]: 2026-03-12T15:20:34.848035591Z  INFO flows: Removing flow /etc/tedge/mappers/local/flows/protobuf-xform/flow.toml
Mar 12 15:20:34 7a272ddfd7d7 tedge-mapper[2303]: 2026-03-12T15:20:34.848198424Z  INFO flows: Loading flow /etc/tedge/mappers/local/flows/protobuf-xform/flow.toml
Mar 12 15:20:34 7a272ddfd7d7 tedge-mapper[2303]: 2026-03-12T15:20:34.874713772Z  INFO flows: Reloading flow script /etc/tedge/mappers/local/flows/protobuf-xform/lib/main.js
Mar 12 15:20:34 7a272ddfd7d7 tedge-mapper[2303]: 2026-03-12T15:20:34.874764272Z  INFO flows: Unloading flow /etc/tedge/mappers/local/flows/protobuf-xform/flow.toml
Mar 12 15:20:34 7a272ddfd7d7 tedge-mapper[2303]: 2026-03-12T15:20:34.874907939Z  INFO flows: Loading flow /etc/tedge/mappers/local/flows/protobuf-xform/flow.toml

In comparison to the other PR #4031, which produced the following entries (but yes also not perfect).

Mar 12 15:29:40 b92dc435bbf5 tedge-mapper[820]: 2026-03-12T15:29:40.978003119Z  INFO flows: Loading flow /etc/tedge/mappers/local/flows/protobuf-xform/flow.toml
Mar 12 15:29:40 b92dc435bbf5 tedge-mapper[820]: 2026-03-12T15:29:40.992939339Z  INFO flows: Loading flow /etc/tedge/mappers/local/flows/protobuf-xform/flow.toml
Mar 12 15:29:41 b92dc435bbf5 tedge-mapper[820]: 2026-03-12T15:29:41.01615384Z  INFO flows: Reloading flow script /etc/tedge/mappers/local/flows/protobuf-xform/lib/main.js
Mar 12 15:29:41 b92dc435bbf5 tedge-mapper[820]: 2026-03-12T15:29:41.043625023Z  INFO flows: Reloading flow script /etc/tedge/mappers/local/flows/protobuf-xform/lib/main.js

The main point is that the first set of log entries are confusing for the user and show that we're being very inefficient when processing flows as we're removing, loading, unloading, loading...so this just makes the system harder to reason with, and I would assume it would also be consuming additional resources doing all of these things.

@Bravo555
Copy link
Member Author

The main point is that the first set of log entries are confusing for the user and show that we're being very inefficient when processing flows as we're removing, loading, unloading, loading...so this just makes the system harder to reason with, and I would assume it would also be consuming additional resources doing all of these things.

Indeed, in #4031 logs are more concise, because some events are explicitly ignored, with the motivation that I think is not entirely correct (can't observe FileDeleted events), but the fix itself ends up being correct (when replacing flow assets we ignore temporary deletions when we think the files are going to be recreated soon, so there's less noise).

Then I think we can go forward with #4031, but with the addition of Removing script unloads flow and restoring script loads flow again test, added in 153278c, to make sure regular removals of script files cause the flow to be unloaded/removed as expected.

@Bravo555
Copy link
Member Author

Selected alternative #4031 to be merged.

@Bravo555 Bravo555 closed this Mar 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants