Add refresh v3 implementation #241

carlcsaposs-canonical · 2025-04-02T14:13:13Z

Uses charm-refresh Python package: https://github.com/canonical/charm-refresh

Avoid passing refresh object since we know reconcile will be called later (no need to explicitly call it & have it run twice)

carlcsaposs-canonical · 2025-04-10T07:11:38Z

You can test the refresh with these charmhub branches (built on #242):

dpe/edge/test-refresh-v3-8.0.40
dpe/edge/test-refresh-v3-8.0.41
dpe/edge/test-refresh-v3-8.0.41-2 - use to test charm-only refresh from 8.0.41
dpe/edge/test-refresh-v3-incompat - simulates incompatible workload or charm version
dpe/edge/test-refresh-v3-precheckfail - simulates failed pre check
dpe/edge/test-refresh-v3-uncaught - simulates uncaught exception (e.g. bug, incorrect arch, etc.)
dpe/edge/test-refresh-v3-uncaught-before-snap-refresh - simulates uncaught exception before snap is refreshed
dpe/edge/test-refresh-v3-uncaught-except-upgrade-charm-event - simulates uncaught exception, but doesn't raise on upgrade-charm event (juju will get stuck on rollback if exception raised during upgrade charm event [unless --force-units used on refresh])

or modify the branch locally & re-pack the charm

sinclert-canonical · 2025-06-05T14:54:58Z

Seems like the charm-refresh-build-version package is forcing us to have some v<WORKLOAD_VERSION>/<CHARM_VERSION> tags in the GitHub repository, similarly to what PostgreSQL VM operator has (reference).

Is this right @carlcsaposs-canonical?

carlcsaposs-canonical · 2025-06-06T11:51:39Z

Seems like the charm-refresh-build-version package is forcing us to have some v<WORKLOAD_VERSION>/<CHARM_VERSION> tags in the GitHub repository, similarly to what PostgreSQL VM operator has (reference).

Is this right @carlcsaposs-canonical?

yes, see https://docs.google.com/document/d/1Jv1jhWLl8ejK3iJn7Q3VbCIM9GIhp8926bgXpdtx-Sg/edit?tab=t.0 for more info

sinclert-canonical · 2025-06-06T12:30:37Z

yes, see https://docs.google.com/document/d/1Jv1jhWLl8ejK3iJn7Q3VbCIM9GIhp8926bgXpdtx-Sg/edit?tab=t.0

Judging by PostgreSQL 16 tags, as well as the proposed contents in the release.yaml workflow, we are in a chicken-and-egg situation here, where the build CI steps will not pass until this PR is merged, and the first v8.0/1.0.0 tag created.

Should we temporarily comment the new charmcraft.yaml contents to verify that integration tests pass?

carlcsaposs-canonical · 2025-06-06T13:26:47Z

Judging by PostgreSQL 16 tags, as well as the proposed contents in the release.yaml workflow, we are in a chicken-and-egg situation here, where the build CI steps will not pass until this PR is merged, and the first v8.0/1.0.0 tag created.

Should we temporarily comment the new charmcraft.yaml contents to verify that integration tests pass?

yes, temporarily reverting charmcraft.yaml changes & temporarily hard-coding charm version in refresh_versions.toml is okay for testing as long as we revert before merge

sinclert-canonical · 2025-06-09T09:08:03Z

Given this comment on the PostgreSQL refresh v3 implementation, my understanding is that the errors in the last CI run, are due to the refresh-v3 incompatibility to refresh from non refresh-v3 charms. Correct @carlcsaposs-canonical?

Traceback:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/lib/python3.10/site-packages/charm_refresh/_main.py", line 807, in from_app_databag
    workload=databag["original_workload_version"],
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/lib/python3.10/site-packages/charm_json/_main.py", line 211, in __getitem__
    return _load(parent=self, parent_key=key, data=json.loads(self._databag[key]))
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/lib/python3.10/site-packages/charm_/_main.py", line 117, in __getitem__
    raise KeyError(key)
KeyError: 'original_workload_version'

carlcsaposs-canonical · 2025-06-10T07:30:58Z

due to the refresh-v3 incompatibility to refresh from non refresh-v3 charms

Yes, (without a migration) refresh v3 does not support refreshing from non-refresh v3 charms

The full traceback shows

ValueError: Refresh failed. Automatic recovery not possible. Original versions in app databag are missing or invalid

sinclert-canonical · 2025-06-11T08:46:00Z

📢 This PR is now ready for review + merge

DISCLAIMERS:

I do not advice for nor advice against the use of refresh-v3.
I am not the author of this PR nor I indeed to be (god forbid).
Any questions about its design, architecture or support must be redirected to the original author.

I made unit + integration tests pass, by relying on a a non-obvious setup, required by how refresh-v3 is implemented:

How to make unit tests pass?

Move every interaction with charm_refresh and snap_lib within the class / function that use them. This is needed in order to properly mock their behavior, following the approach of PostgreSQL VM operator PR.
Mock the contents of the refresh_versions.toml. To this end, I tried to follow the approach taken in the PostgreSQL VM operator, where a session level fixture replaces the contents of the file during the tests duration (reference). However that does not work with pytest-xdist, as session level fixtures are not guaranteed to be run just once. Alternatives:
- A) Rely on filelock: officially recommended, but impractical due to absurdly high number of tests.
- B) Mock tomli read: effective, but not recommended nor consistent with PostgreSQL VM operator fixture.

See commits:

[Recommended solution] ebce78a
[Mocking alternative] a03211f

How to make integration tests pass?

Adapt every assertion rely on on resume-upgrade language to assert on resume-refresh.
Run the resume-refresh action not in the leader, but on the non-leader unit with the smallest identifier.
Remove the injection of +testupgrade and +testrollback suffixes to the workload version, due to new parsing logic.

See commits:

faf1dda

Temporal changes

Despite the described changes, refresh v3 can only be tested when upgrading from another refresh v3 charm (reference), in a repository which already started publishing charm versions as tags (in the case of MySQL Router: 8.0/X.Y.Z). As this is not the case, we needed to "fake" several behaviors:

The build process will not write the charm version (as there are no tags published yet).
The charm version needs to be hardcoded in the refresh_versions.toml file.
An existing refresh-v3 charm needs to be already available at the channel used for the integration tests. To this end, I published an AMD64 locally pack charm, with the version 8.0/1.0.0 to the dpe/latest/test-refresh-v3 channel.

See commits:

Follow up

Now the whole test suite passes, so this is ready to be merged + a refresh-v3 version of the charm published. WHenever that happens, we must create a follow-up PR in order to revert the [Temp] prefixed commits.

carlcsaposs-canonical

thank you!

(reviewed diff from fb9d59a)

carlcsaposs-canonical · 2025-06-11T11:03:11Z

refresh_versions.toml

+charm = "8.0/1.1.0"
+


Suggested change

charm = "8.0/1.1.0"

this needs to be removed before merging

also, I believe this will affect the integration test—we will need to run force-refresh-start with check-compatibility=false on PR CI (but not after merging) since development builds will be marked as incompatible (so that if a user ever accidentally deploys a development version of the charm, it's marked as incompatible). see postgresql-operator 16/edge for an example test that implements this

carlcsaposs-canonical · 2025-06-11T11:03:54Z

tests/integration/test_upgrade.py


-    logger.info("Running resume-upgrade on the mysql router leader unit")
-    await run_action(mysql_router_leader_unit, "resume-upgrade")
+    logger.info("Running resume-refresh on the mysql router leader unit")


Suggested change

logger.info("Running resume-refresh on the mysql router leader unit")

logger.info("Running resume-refresh")

not necessarily leader

carlcsaposs-canonical · 2025-06-11T11:04:16Z

tests/integration/test_upgrade.py

@@ -39,6 +35,7 @@
 TEST_APP_NAME = APPLICATION_DEFAULT_APP_NAME


+@markers.amd64_only


would it be helpful for me to rebase #242 so that we can remove the amd64 only marker before merge?

carlcsaposs-canonical · 2025-06-11T11:05:49Z

tests/integration/test_upgrade.py

+    # charm needs to refresh snap to be able to avoid no-op when upgrading.
+    # set an old revision of the snap


consider also setting verisons["workload"] to avoid confusion in juju status output of test

carlcsaposs-canonical · 2025-06-11T11:07:50Z

tests/integration/test_upgrade.py

+    old_version = Version(versions["workload"])
+    new_version = Version(f"{old_version.major - 1}.{old_version.minor}.{old_version.micro}")
+    versions["workload"] = str(new_version)
+    versions["charm"] = "8.0/0.0.0"


nit: there's no need to update the workload version, setting charm version will be enough—which means we can get rid of the packaging import/dependency in integration tests

carlcsaposs-canonical · 2025-06-11T11:08:52Z

charmcraft.yaml

+      # Create venv in `..` so that git working tree is not dirty
+      # python3 -m venv ../refresh-version-venv
+      # source ../refresh-version-venv/bin/activate
+      # poetry install --only build-refresh-version
+      # write-charm-version


Suggested change

# Create venv in `..` so that git working tree is not dirty

# python3 -m venv ../refresh-version-venv

# source ../refresh-version-venv/bin/activate

# poetry install --only build-refresh-version

# write-charm-version

# Create venv in `..` so that git working tree is not dirty

python3 -m venv ../refresh-version-venv

source ../refresh-version-venv/bin/activate

poetry install --only build-refresh-version

write-charm-version

this needs to be applied before merge

carlcsaposs-canonical · 2025-06-11T13:12:44Z

tests/unit/conftest.py

+    def _tomli_load(*args, **kwargs) -> dict:
+        return {
+            "charm_major": 1,
+            "workload": "8.0.0",
+            "charm": "v8.0/1.0.0",
+            "snap": {
+                "name": "charmed-mysql",
+                "revisions": {
+                    "x86_64": "1",
+                    "aarch64": "1",
+                },
+            },
+        }


would patching charm_refresh.snap_name work instead? if so, I think we can write our mocks to not depend on charm_refresh's private api

carlcsaposs-canonical · 2025-06-11T13:13:40Z

src/snap.py

+    Assumes snap is installed
+    """
+    snap_name = charm_refresh.snap_name()
+    snap_unit_path = pathlib.Path(


Suggested change

snap_unit_path = pathlib.Path(

installed_by_unit = pathlib.Path(

nit

(same for other instances of snap_unit_path)

paulomach · 2025-06-11T21:32:31Z

@carlcsaposs-canonical , do you mind making the remaining changes? we can streamline the pr execution

Add refresh v3 draft implementation

6abcf4f

github-actions bot added the Libraries: Out of sync label Apr 2, 2025

carlcsaposs-canonical force-pushed the refresh-v3-draft branch 12 times, most recently from 9a114c0 to 9de758b Compare April 3, 2025 15:01

Simplify reconcile

5b72ce3

Avoid passing refresh object since we know reconcile will be called later (no need to explicitly call it & have it run twice)

carlcsaposs-canonical force-pushed the refresh-v3-draft branch 7 times, most recently from 3fddbf4 to d9bc860 Compare April 7, 2025 07:08

Add self._reconcile_allowed instead of exiting

8399f67

carlcsaposs-canonical force-pushed the refresh-v3-draft branch 5 times, most recently from d7d94fc to db4ec41 Compare April 10, 2025 07:04

carlcsaposs-canonical requested a review from paulomach April 10, 2025 08:07

sinclert-canonical force-pushed the refresh-v3-draft branch 2 times, most recently from 697ae63 to ab25c33 Compare June 5, 2025 14:30

Fix unit tests

ebce78a

sinclert-canonical force-pushed the refresh-v3-draft branch from ab25c33 to 3dd00a0 Compare June 6, 2025 11:07

sinclert-canonical force-pushed the refresh-v3-draft branch from 3dd00a0 to 87f9ef4 Compare June 6, 2025 12:58

sinclert-canonical added 3 commits June 6, 2025 15:28

Fix unit tests [alternative]

a03211f

Fix integration tests

faf1dda

[Temp] Disable charmcraft version build

61b16a8

sinclert-canonical force-pushed the refresh-v3-draft branch from 87f9ef4 to 6dd5997 Compare June 6, 2025 13:30

[Temp] Hard-code charm version

76505d4

sinclert-canonical force-pushed the refresh-v3-draft branch from 6dd5997 to cc2af8d Compare June 6, 2025 14:17

sinclert-canonical force-pushed the refresh-v3-draft branch 7 times, most recently from 6503990 to a89e90a Compare June 11, 2025 07:38

[Temp] Point to a refresh v3 channel

86fceca

sinclert-canonical force-pushed the refresh-v3-draft branch from a89e90a to 86fceca Compare June 11, 2025 08:11

sinclert-canonical marked this pull request as ready for review June 11, 2025 08:46

carlcsaposs-canonical commented Jun 11, 2025

View reviewed changes

	logger.info("Running resume-refresh on the mysql router leader unit")
	logger.info("Running resume-refresh")

		@@ -39,6 +35,7 @@
		TEST_APP_NAME = APPLICATION_DEFAULT_APP_NAME


		@markers.amd64_only

		# charm needs to refresh snap to be able to avoid no-op when upgrading.
		# set an old revision of the snap

	snap_unit_path = pathlib.Path(
	installed_by_unit = pathlib.Path(

Add refresh v3 implementation #241

Are you sure you want to change the base?

Add refresh v3 implementation #241

Uh oh!

Conversation

carlcsaposs-canonical commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlcsaposs-canonical commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sinclert-canonical commented Jun 5, 2025

Uh oh!

carlcsaposs-canonical commented Jun 6, 2025

Uh oh!

sinclert-canonical commented Jun 6, 2025

Uh oh!

carlcsaposs-canonical commented Jun 6, 2025

Uh oh!

sinclert-canonical commented Jun 9, 2025

Uh oh!

carlcsaposs-canonical commented Jun 10, 2025

Uh oh!

sinclert-canonical commented Jun 11, 2025

How to make unit tests pass?

How to make integration tests pass?

Temporal changes

Follow up

Uh oh!

carlcsaposs-canonical left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulomach commented Jun 11, 2025

Uh oh!

Uh oh!

carlcsaposs-canonical commented Apr 2, 2025 •

edited

Loading

carlcsaposs-canonical commented Apr 10, 2025 •

edited

Loading