🧑‍🌾 LogCommandAPI Tests Failing with cli11 version 2.6.0

[miguelgonrod]

Environment

• OS Version: Mac
• Source

Description

Test resgressions:

• LogCommandAPI.RecordBadRegex
• LogCommandAPI.PlaybackBadRegex
• LogCommandAPI.PlaybackBadRemap
• LogCommandAPI.RecordFailedToOpen
• LogCommandAPI.PlaybackFailedToOpen

Expected behavior: Tests Don't failing

Actual behavior: Failing with message /Users/jenkins/jenkins-agent/workspace/gz_transport-ci-main-homebrew-amd64/gz-transport/log/src/cmd/LogCommandAPI_TEST.cc:66 Expected: (output.find(expectedOutput)) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615

Steps to reproduce

1. Go to https://build.osrfoundation.org/job/gz_transport-ci-main-homebrew-amd64/
2. Build environment

Output

83: Running command [/usr/local/bin/gz log record --file ':memory:' --pattern '*']
83: /Users/jenkins/jenkins-agent/workspace/gz_transport-ci-main-homebrew-amd64/gz-transport/log/src/cmd/LogCommandAPI_TEST.cc:66: Failure
83: Expected: (output.find(expectedOutput)) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615
83: [  FAILED  ] LogCommandAPI.RecordBadRegex (439 ms)
83: [ RUN      ] LogCommandAPI.PlaybackBadRegex
83: Running command [/usr/local/bin/gz log playback --file ':memory:' --pattern '*' --wait 0]
83: /Users/jenkins/jenkins-agent/workspace/gz_transport-ci-main-homebrew-amd64/gz-transport/log/src/cmd/LogCommandAPI_TEST.cc:80: Failure
83: Expected: (output.find(expectedOutput)) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615
83: [  FAILED  ] LogCommandAPI.PlaybackBadRegex (184 ms)
83: [ RUN      ] LogCommandAPI.PlaybackBadRemap
83: Running command [/usr/local/bin/gz log playback --file ':memory:' --pattern '.*' --wait 0 --remap '/foo' -f]
83: /Users/jenkins/jenkins-agent/workspace/gz_transport-ci-main-homebrew-amd64/gz-transport/log/src/cmd/LogCommandAPI_TEST.cc:97: Failure
83: Expected: (output.find(expectedOutput)) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615
83: Running command [/usr/local/bin/gz log playback --file ':memory:' --pattern '.*' --wait 0 --remap '/foo:=']
83: /Users/jenkins/jenkins-agent/workspace/gz_transport-ci-main-homebrew-amd64/gz-transport/log/src/cmd/LogCommandAPI_TEST.cc:111: Failure
83: Expected: (output.find(expectedOutput)) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615
83: Running command [/usr/local/bin/gz log playback --file ':memory:' --pattern '.*' --wait 0 --remap '/foo:=' -f]
83: /Users/jenkins/jenkins-agent/workspace/gz_transport-ci-main-homebrew-amd64/gz-transport/log/src/cmd/LogCommandAPI_TEST.cc:126: Failure
83: Expected: (output.find(expectedOutput)) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615

Flaky Ratio:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
gz_transport-ci-main-homebrew-arm64	2025-10-22	2025-10-22	4	1	25.0
gz_transport-ci-main-homebrew-amd64	2025-10-21	2025-10-20	6	3	50.0
gz_transport-ci-gz-transport15-homebrew-amd64	2025-10-20	2025-10-20	2	1	50.0

Not sure why the failure, but the logs indicate that they downgraded some packages versions between #197 and #201

- gz-cmake4
- gz-common6
- gz-fuel-tools10
- gz-gui9
- gz-ionic
- gz-launch8
- gz-math8
- gz-msgs11
- gz-physics8
- gz-plugin3
- gz-rendering9
- gz-sensors9
- gz-sim9
- gz-transport14
- gz-utils3
+ gz-common5
+ gz-fuel-tools9
+ gz-gui8
+ gz-math7
+ gz-msgs10
+ gz-physics7
+ gz-plugin2
+ gz-rendering8
+ gz-sensors8
+ gz-transport13
+ gz-utils2

CC: @Crola1702

[scpeters]

Not sure why the failure, but the logs indicate that they downgraded some packages versions between #197 and #201

• gz-cmake4
• gz-common6
• gz-fuel-tools10
• gz-gui9
• gz-ionic
• gz-launch8
• gz-math8
• gz-msgs11
• gz-physics8
• gz-plugin3
• gz-rendering9
• gz-sensors9
• gz-sim9
• gz-transport14
• gz-utils3

• gz-common5
• gz-fuel-tools9
• gz-gui8
• gz-math7
• gz-msgs10
• gz-physics7
• gz-plugin2
• gz-rendering8
• gz-sensors8
• gz-transport13
• gz-utils2

I looked at the console output, and this comes from the cleanup phase of the job, when it removes previously installed packages, which can be different every time. I think this part of the console output can be ignored

[scpeters]

I reproduced this on my laptop by running one of the test commands:

$ gz log record --file ':memory:' --pattern '*'
A subcommand is required
Run with --help for more information.
Error: Version mismatch. Your configuration file version is
            [15.0.0] but /opt/homebrew/Cellar/gz-transport15/15.0.0_6/libexec/gz/transport15/gz-transport-log-main version is [].

I then tried running the standalone executable with --version:

$ /opt/homebrew/Cellar/gz-transport15/15.0.0_6/libexec/gz/transport15/gz-transport-log-main --version
A subcommand is required
Run with --help for more information.

this smells like an issue with the cli11 package that we use to implement argument parsing for the standalone executables

with that in mind, let's check if the cli11 package changed between the last time these tests passed and when they started failing:

• https://build.osrfoundation.org/job/gz_transport-ci-main-homebrew-amd64/197/

==> Installing osrf/simulation/gz-transport15 dependency: cli11
==> Downloading https://ghcr.io/v2/homebrew/core/cli11/manifests/2.5.0
Already downloaded: /Users/jenkins/Library/Caches/Homebrew/downloads/955a35c0830d26042d3f6b215a91694c507804f6dc6407864afff472b368fb9c--cli11-2.5.0.bottle_manifest.json
==> Pouring cli11--2.5.0.all.bottle.tar.gz
🍺  /usr/local/Cellar/cli11/2.5.0: 35 files, 642.6KB

• https://build.osrfoundation.org/job/gz_transport-ci-main-homebrew-amd64/199/

==> Installing osrf/simulation/gz-transport15 dependency: cli11
==> Downloading https://ghcr.io/v2/homebrew/core/cli11/manifests/2.6.0
Already downloaded: /Users/jenkins/Library/Caches/Homebrew/downloads/f4bba6ff8b3d76e44d389e7e78b6a762d4eafe13086e076627e1aefe4ae52670--cli11-2.6.0.bottle_manifest.json
==> Pouring cli11--2.6.0.all.bottle.tar.gz
🍺  /usr/local/Cellar/cli11/2.6.0: 37 files, 681.7KB

• cli11 2.6.0 Homebrew/homebrew-core#250357

I'm guessing something changed between cli11 version 2.5.0 and 2.6.0 to break this

[scpeters]

changelog for cli11 2.6.0: https://github.com/CLIUtils/CLI11/releases/tag/v2.6.0