Run ctst locally by SylvainSenechal · Pull Request #2319 · scality/Zenko

SylvainSenechal · 2026-02-11T10:59:12Z

Run ctst inside Codespace :

Update ./run.sh file
Use cli-testing as a library
Some other refactoring

bert-e · 2026-02-11T10:59:15Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

.devcontainer/setup.sh

SylvainSenechal · 2026-02-11T17:38:44Z

I will clean my commits after reviews

SylvainSenechal · 2026-02-11T17:42:20Z

.github/scripts/end2end/run-e2e-ctst.sh

Trying to minimize the utilisation of global env variable that are used only once, or sometimes even useless, there is more work to do on this topic but this will be for other pr

SylvainSenechal · 2026-02-11T17:44:15Z

.github/scripts/end2end/run-e2e-ctst.sh

        --env=AZURE_BLOB_URL=$AZURE_BACKEND_ENDPOINT  \
        --env=AZURE_QUEUE_URL=$AZURE_BACKEND_QUEUE_ENDPOINT \
        --env=VERBOSE=1 \
+        --env=SDK=true \


Cli testing can be run in sdk mode, or CLI mode.

This shouldn't have impact, because actually this env used to be set in the Dockerfile of the E2E_IMAGE.
I put it here instead, because imo putting it in the Dockerfile was kinda hidding this parameter, and we are already defining all the other envs variable here

SylvainSenechal · 2026-02-11T17:45:45Z

.github/scripts/end2end/run-e2e-ctst.sh

    ]
  }
-}' -- ./run "$COMMAND" $WORLD_PARAMETERS --parallel $PARALLEL_RUNS --retry $RETRIES --retry-tag-filter @Flaky --format junit:$JUNIT_REPORT_PATH "$@"
+}' -- yarn cucumber-js \


Imo much better, before it was hard to see, but a lot of things were happening behind the ./run commands, which wasn't even defined in Zenko, but in Cli-testing.

Here, we are now just naturally using cucumber syntax to run cucumber tests

SylvainSenechal · 2026-02-11T17:46:57Z

.github/workflows/end2end.yaml

this is what I was talking about when I said I struggled to deal with cli-testing auth. It only worked when I added the ~/.netrc thing, which kinda looks smelly

which command failed in which context (codespace, ci, inside docker image...) ?

we need to use the GIT_ACCESS_TOKEN to have access to other repos (instead of GitHub usual reds, which are restricted the current repo + public repos)

the git config above is already a trick to inject the GIT_ACCESS_TOKEN in git urls (so we don't need to modify the url in every place it is used: package.json, ...)

netrc is an alternative to configure git auth, which should not be needed if the git config are made (but maybe we miss some, etc...)

.github/workflows/end2end.yaml

SylvainSenechal · 2026-02-11T17:48:29Z

.github/workflows/end2end.yaml

Also nicer, bye bye """"""""""""", and now we can clearly see which tags are triggered (yeah cucumber allows to use multiple tags with and/or syntax)

tests/ctst/cucumber.config.cjs

SylvainSenechal · 2026-02-11T17:53:20Z

tests/ctst/Dockerfile

This is the equivalent of what we had in Cli-testing too : https://github.com/scality/cli-testing/blob/development/1.3/Dockerfile

I kept it mostly the same, but defined it here, so that cli-testing is used as a library, and not something where we store dockerfiles, commands, configs for Zenko.

SylvainSenechal · 2026-02-11T17:54:23Z

tests/ctst/package.json

Many packages here can be updated to a higher version, but let's keep it for another pr

SylvainSenechal · 2026-02-11T17:57:51Z

tests/ctst/run-ctst-locally.sh

+# certain tests based on their @version tag.
+VERSION=$(cat ../../VERSION | grep -Po 'VERSION="\K[^"]*')
+POD_NAME="ctst-end2end"
+WORLD_PARAMETERS="$(jq -c <<EOF


This is where we still have frictions : The current setup is basically enough to run tests locally, but these tests needs a lot of env variables that are passed as "world parameter". Here, I used a few that I defined manually to make one test work.

Later gonna work on using kube client to get secrets directly at the beginning of the tests, this will give us 2 things :

Reduce setup needed to run test locally

Reduce and refactor the number of things happening in .sh setup files. Final objective is also to have variables defined only once at the same place

seems now we are running as a regular node program: so it should work as well without docker/pod?
→ maybe some tests (and world) need to be adapted, but we should try to investigate what changes it would require (which commands to tweak, ...). Esp. we already need special tricks to run kubectl commands, so it may actually simplify stuff...

Yes, we may get there eventually, this should be doable, but probably lots of variables related to endpoint needs to change with port forward

agreed this is not for this PR

port forward

in most case we use (or should use) the ingress : so no port forward...
we may however need to tweak etc/host or some other trick for DNS resolution, though

.github/scripts/end2end/deploy-zenko.sh

.devcontainer/setup.sh

.github/scripts/end2end/deploy-zenko.sh

francoisferrand · 2026-02-12T07:19:03Z

.github/workflows/end2end.yaml

which command failed in which context (codespace, ci, inside docker image...) ?

we need to use the GIT_ACCESS_TOKEN to have access to other repos (instead of GitHub usual reds, which are restricted the current repo + public repos)

the git config above is already a trick to inject the GIT_ACCESS_TOKEN in git urls (so we don't need to modify the url in every place it is used: package.json, ...)

netrc is an alternative to configure git auth, which should not be needed if the git config are made (but maybe we miss some, etc...)

tests/ctst/Dockerfile

francoisferrand · 2026-02-12T07:26:21Z

tests/ctst/README.md

latest does not work since we have multiple branches : so should always be rebuilt (maybe use an image tag which is unique to the branch, but not sure how to build images & tag them so it is practical...)

tests/ctst/README.md

tests/ctst/run-ctst-locally.sh

francoisferrand · 2026-02-12T07:34:16Z

tests/ctst/run-ctst-locally.sh

+# certain tests based on their @version tag.
+VERSION=$(cat ../../VERSION | grep -Po 'VERSION="\K[^"]*')
+POD_NAME="ctst-end2end"
+WORLD_PARAMETERS="$(jq -c <<EOF


seems now we are running as a regular node program: so it should work as well without docker/pod?
→ maybe some tests (and world) need to be adapted, but we should try to investigate what changes it would require (which commands to tweak, ...). Esp. we already need special tricks to run kubectl commands, so it may actually simplify stuff...

francoisferrand · 2026-02-12T09:20:36Z

tests/ctst/run-ctst-locally.sh

instead of passing specifically "tags", we could just forward"all" arguments to yarn, with "$@" : so the script is a "transparent" wrapper, and more flexible (if we need to pass extra cucumber params, for exemple to run a single test or multiple tags...)

may be a bit more tricky with the optional image though, may need to add more (bash) argument parsing, either to support running like this ./run-ctst-locally.sh [image] [-- <extra params>] or to use a flag ./run-ctst-locally.sh [-i <image>] <extra params>...

I tried something but it got a little bit dirty with the parsing of the image indeed.
I did make a change so that we can now use natural Cucumber tags expression though

tests/ctst/run-ctst-locally.sh

francoisferrand · 2026-02-12T09:27:51Z

tests/ctst/run-ctst-locally.sh

just run.sh?
(since it may be used not just locally -we could/should use it in CI, to avoid having multiple code-path- and already in ctst folder)

I strongly agree to avoid several .sh script. Also to make sure everything will continue to work, I think we should find a way to test that in the CI ?

I'm not a big fan of run.sh, it's too short and grepping "run" returns thousands of results. Actually this is the reason i originally got confused, because in run-e2e-ctst, the pods was executing run, and it took me some time to realize it was the cli-testing run, and not this file. I'm not fixed on this specific name, but I think it's nice to have a more identifiable name, the current name, or "run-cucumber-tests.sh" for example leaves no room for ambiguity.

Also, more importantly, there is indeed an objective which would be that local tests and github ci tests are run using the same same file, but at the moment there isn't done, as there are still some difference between the 2 files. I think eventually we will do it, and when it happens we can think again about the naming

My goal was mainly to avoid duplicate logic. I'm fine to iterate later, we should track that in a JIRA and in the epic and add some comments and the beginning of each 🙏

francoisferrand · 2026-02-16T17:58:32Z

tests/ctst/README.md

+
+# 2. Load into kind and reset the pod
+kind load docker-image ctst-local:dev
+kubectl delete pod ctst-end2end


why would it be created already?
is it created automatically on devcontainer startup?

I updated the comment. Basically, only needed if the pod was already created before with a different image

francoisferrand · 2026-02-16T18:01:18Z

tests/ctst/Dockerfile

 ARG DRCTL_TAG=latest

 FROM ghcr.io/scality/sorbet:$SORBET_TAG AS sorbet
 FROM ghcr.io/scality/zenko-drctl:$DRCTL_TAG AS drctl


note for later (more when we want to run CTST tests outside of k8s) : maybe we do not actually need these binaries, we could run them directly in pods...

(typically, esp. for Sorbet, this is often how it is done: kubectl exec into sorbetctl container...)

francoisferrand · 2026-02-16T18:02:20Z

tests/ctst/cucumber.config.cjs

+        format: [
+            'progress-bar',
+            '@cucumber/pretty-formatter',
+            'json:reports/cucumber-report.json',


does this generate a JUnit report? (needed for the overall Zenko workflow, to report failed tests summary)

it is the "stock" reporter, or the one implemented in cli-testing ?

okay I reviewed this last comment : my code changes do not modify the existing logic, in run-e2e-ctst, cucumber is ran with the same previous parameter (-- format junitxxxx) to generate the report.

What I checked though, is if this report was exported to the artifact, and I found it was not.
So I updated one of the action file to export the report to the artifact, and also found that the xml report is not really human readable, so I also exported an html version (while keeping the first format). Also bumped the action action-junit-report from v4 to v6

francoisferrand

LGTM except maybe for the https://github.com/scality/Zenko/pull/2319/changes#r2813543342 question : need to make sure we don't "break" failed tests reporting via JUnit test report

francoisferrand · 2026-02-16T18:08:27Z

tests/ctst/Dockerfile

+COPY package.json ./
+COPY yarn.lock ./
+
+RUN --mount=type=secret,id=GIT_AUTH_TOKEN \


nit: this will work in CI and codespace, but fail when running locally if SSH is used to auth with GitHub.
So (eventually, if we still use this image) we will need to further improve this to support both GIT_AUTH_TOKEN and SSH... e.g. something like adding the 2 mounts (if possible...), and replacing the URL depending on what we get...

bert-e · 2026-02-17T10:52:45Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Issue: ZENKO-5180

…a library Issue: ZENKO-5180

Issue: ZENKO-5180

…ifact Issue: ZENKO-5180

DarkIsDude

For me we are good. Some small comments. If you have a re-review, just retrigger the flow. Thanks for your work !

DarkIsDude · 2026-02-18T14:01:48Z

tests/ctst/README.md

+
+```bash
+# 1. Build the CTST image
+


Suggested change

DarkIsDude · 2026-02-18T14:02:19Z

tests/ctst/README.md

+```bash
+cd tests/ctst
+./run-ctst-locally.sh @getObject
+./run-ctst-locally.sh "@PreMerge and not @Flaky"  # Use quotes for complex expressions


Suggested change

./run-ctst-locally.sh "@PreMerge and not @Flaky" # Use quotes for complex expressions

./run-ctst-locally.sh "@PreMerge and not @Flaky"

You have the same letter. I think it's pretty a standard for bash command and it's not needed to explain it ?

DarkIsDude · 2026-02-18T14:04:20Z

tests/ctst/run-ctst-locally.sh

My goal was mainly to avoid duplicate logic. I'm fine to iterate later, we should track that in a JIRA and in the epic and add some comments and the beginning of each 🙏

SylvainSenechal force-pushed the improvement/ZENKO-5180 branch from 1047fe5 to e3583d0 Compare February 11, 2026 12:15

SylvainSenechal requested review from a team, DarkIsDude, francoisferrand and maeldonn February 11, 2026 17:35

SylvainSenechal commented Feb 11, 2026

View reviewed changes

.devcontainer/setup.sh Outdated Show resolved Hide resolved

SylvainSenechal commented Feb 11, 2026

View reviewed changes

.github/workflows/end2end.yaml Outdated Show resolved Hide resolved

SylvainSenechal commented Feb 11, 2026

View reviewed changes

tests/ctst/cucumber.config.cjs Show resolved Hide resolved

SylvainSenechal commented Feb 11, 2026

View reviewed changes

scality deleted a comment from bert-e Feb 11, 2026

SylvainSenechal commented Feb 11, 2026

View reviewed changes

.github/scripts/end2end/deploy-zenko.sh Outdated Show resolved Hide resolved

SylvainSenechal marked this pull request as ready for review February 11, 2026 18:04

SylvainSenechal force-pushed the improvement/ZENKO-5180 branch 3 times, most recently from 56485d6 to a71ddc4 Compare February 11, 2026 21:24

francoisferrand reviewed Feb 12, 2026

View reviewed changes

tests/ctst/run-ctst-locally.sh Outdated Show resolved Hide resolved

francoisferrand reviewed Feb 12, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5180 branch 2 times, most recently from 497a7db to faea5c8 Compare February 12, 2026 14:26

SylvainSenechal force-pushed the improvement/ZENKO-5180 branch from 8ece2e3 to b408f36 Compare February 16, 2026 14:01

SylvainSenechal requested review from DarkIsDude and francoisferrand February 16, 2026 14:19

francoisferrand reviewed Feb 16, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5180 branch 11 times, most recently from 2271c1f to 25586b6 Compare February 16, 2026 20:36

scality deleted a comment from bert-e Feb 17, 2026

SylvainSenechal force-pushed the improvement/ZENKO-5180 branch from 25586b6 to 59e3b03 Compare February 17, 2026 11:33

SylvainSenechal added 4 commits February 17, 2026 15:21

remove useless fix-zookeeper file

99b75c9

Issue: ZENKO-5180

add script to run cucumber test in codespace, and use cli-testing as …

cdb197d

…a library Issue: ZENKO-5180

fix kafa typo

9376931

Issue: ZENKO-5180

update action to use latest junit report v6, and upload report to art…

90d69c5

…ifact Issue: ZENKO-5180

SylvainSenechal force-pushed the improvement/ZENKO-5180 branch from 59e3b03 to 90d69c5 Compare February 17, 2026 14:23

SylvainSenechal requested a review from francoisferrand February 17, 2026 14:32

DarkIsDude approved these changes Feb 18, 2026

View reviewed changes

	./run-ctst-locally.sh "@PreMerge and not @Flaky" # Use quotes for complex expressions
	./run-ctst-locally.sh "@PreMerge and not @Flaky"

Conversation

SylvainSenechal commented Feb 11, 2026

Uh oh!

bert-e commented Feb 11, 2026

Hello sylvainsenechal,

Uh oh!

Uh oh!

SylvainSenechal commented Feb 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

francoisferrand Feb 12, 2026 •

edited

Loading

francoisferrand Feb 16, 2026 •

edited

Loading