[system test] Add CustomCaChainST by see-quick · Pull Request #12497 · strimzi/strimzi-kafka-operator

see-quick · 2026-03-06T12:38:22Z

Type of change

Enhancement / new feature
Refactoring
Documentation

Description

This PR adds a new test suite CustomCaChainST covering user-defined listener using a multi-stage custom CA (i.e., #12364) plus CVE-2026-27133 and CVE-2026-27134.

Checklist

Write tests
Make sure all tests pass
Update documentation

Signed-off-by: see-quick <maros.orsak159@gmail.com>

see-quick · 2026-03-06T12:41:47Z

/gha run pipeline=regression

github-actions · 2026-03-06T12:42:19Z

⏳ System test verification started: link

The following 6 job(s) will be executed:

regression-brokers-and-security-amd64 (oracle-vm-8cpu-32gb-x86-64)
regression-operators-amd64 (oracle-vm-8cpu-32gb-x86-64)
regression-operands-amd64 (oracle-vm-8cpu-32gb-x86-64)
regression-brokers-and-security-arm64 (oracle-vm-8cpu-32gb-arm64)
regression-operators-arm64 (oracle-vm-8cpu-32gb-arm64)
regression-operands-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

codecov · 2026-03-06T13:28:23Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.02%. Comparing base (568706e) to head (5a960e2).
⚠️ Report is 24 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12497      +/-   ##
============================================
+ Coverage     74.99%   75.02%   +0.02%     
+ Complexity     6651     6437     -214     
============================================
  Files           373      373              
  Lines         25367    24881     -486     
  Branches       3376     3205     -171     
============================================
- Hits          19025    18666     -359     
+ Misses         4956     4901      -55     
+ Partials       1386     1314      -72

see 47 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-03-06T17:26:08Z

🎉 System test verification passed: link

Signed-off-by: see-quick <maros.orsak159@gmail.com>

katheris

Thanks @see-quick I've had a look through. Generally the tests look good but I've added some comments around the checks being made. Also I think we should be explicitly testing the case where the user provides a CA that is multistage, but Strimzi still issues the certificates, rather than just the cases of a custom listener certificate, since those are different code paths.

systemtest/src/test/java/io/strimzi/systemtest/security/custom/CustomCaChainST.java

katheris · 2026-03-11T15:31:36Z

systemtest/src/test/java/io/strimzi/systemtest/security/custom/CustomCaChainST.java

+            KubeResourceManager.get().kubeClient().getClient().secrets().inNamespace(testStorage.getNamespaceName()).withName(brokerPodName).get(),
+            brokerPodName + ".crt");
+        LOGGER.info("Broker certificate chain contains {} certificates", brokerChain.size());
+        assertThat("Broker certificate chain should contain at least the leaf cert", brokerChain.size(), is(greaterThanOrEqualTo(1)));


This seems to only check that the leaf cert is presented, and not all three

I think it should be 4? RootCA -> ImCA -> LeafCA -> broker cert? I will use definitely stronger condition is(4) and not this weak greaterThanOrEqualTo.

Or do we want RootCA -> ImCA -> broker cert?

systemtest/src/test/java/io/strimzi/systemtest/security/custom/CustomCaChainST.java

…eaks Signed-off-by: see-quick <maros.orsak159@gmail.com>

development-docs/systemtests/io.strimzi.systemtest.security.custom.CustomCaChainST.md

scholzj · 2026-03-12T22:22:54Z

development-docs/systemtests/io.strimzi.systemtest.security.custom.CustomCaChainST.md

+| 1. | Generate a custom CA chain: Root -> Intermediate -> Leaf. | CA chain is generated. |
+| 2. | Generate a separate foreign Root CA. | Foreign CA is generated. |
+| 3. | Generate four user certificates signed by Leaf CA, Intermediate CA, Root CA, and foreign CA respectively. | User certificates are generated. |
+| 4. | Deploy user cert secrets and a CA trust secret containing only the Leaf CA cert. | Secrets are created in the namespace. |


This should be generated by the User Operator, or?

I am not sure I understand, the point of this test case is verify broker side trust on custom listener. If I let UO to generarate these certs, then all will be signed by the same Clients CA and I would not be able to test negative cases.

Like here UO and CO should not have any involvement in this custom trust as I define custom auth listener which would have LeafCa. Like If I can not use UO generated user certs here as UO will sign it with the ClientsCA not with my LeafCA. My custom broker lsitener trust only the LeafCA so UO generated cert (which is signed by ClientsCA) would be rejected. Or am I wrong?

The point I was trying to make is that I do not understand what Deploy user cert secrets means. User Secrets are normally created by the User Operator. Or what user secrets are you talking about here? So perhaps what you do is the right thing, but the description is not clear?

Ah, I see ... by user cert secrets I meant client certificates secrets ... but I can see it's a bit confusing as I get your point.

It think maybe I can improve this Deploy user cert secrets ... wth Deploy external client certificate secrets (each signed by a different CA) ...?

scholzj · 2026-03-12T22:23:26Z

development-docs/systemtests/io.strimzi.systemtest.security.custom.CustomCaChainST.md

+| 2. | Generate a separate foreign Root CA. | Foreign CA is generated. |
+| 3. | Generate four user certificates signed by Leaf CA, Intermediate CA, Root CA, and foreign CA respectively. | User certificates are generated. |
+| 4. | Deploy user cert secrets and a CA trust secret containing only the Leaf CA cert. | Secrets are created in the namespace. |
+| 5. | Deploy Kafka with a custom listener (port 9122) configured with ssl.client.auth=required and PEM truststore pointing to the Leaf CA. | Kafka cluster is ready with custom listener. |


Should this be custom Clients CA instead?

You mean just changing wording from Leaf CA with custom Clients CA? If so, I can do it for sure.

You talk abotu custom listener (port 9122) configured with ssl.client.auth=required and PEM truststore pointing to the Leaf CA, but these seem to be 3 different things and all are confusing:

Custom listener -> You mean type: custom? Or user-configured? Not clear.

ssl.client.auth=required -> that would type: tls authentication? Or are you talking about this: https://strimzi.io/docs/operators/latest/full/configuring.html#configuring_customized_tls_client_authentication?

And so on

Again, maybe you do the right thing (to be honest, I did not get to the code part yet), but the description is not clear enough.

Custom listener -> You mean type: custom? Or user-configured? Not clear.

It's an internal listener with name custom (i.e., user-configured). Maybe I should re-word it a bit better ... ?

ssl.client.auth=required -> that would type: tls authentication? Or are you talking about this: https://strimzi.io/docs/operators/latest/full/configuring.html#configuring_customized_tls_client_authentication?

Yes

scholzj · 2026-03-12T22:24:03Z

development-docs/systemtests/io.strimzi.systemtest.security.custom.CustomCaChainST.md

+| 4. | Deploy user cert secrets and a CA trust secret containing only the Leaf CA cert. | Secrets are created in the namespace. |
+| 5. | Deploy Kafka with a custom listener (port 9122) configured with ssl.client.auth=required and PEM truststore pointing to the Leaf CA. | Kafka cluster is ready with custom listener. |
+| 6. | Verify that the user with a Leaf-CA-signed cert can produce and consume messages. | Messages are transmitted successfully. |
+| 7. | Verify that users with Intermediate-CA, Root-CA, and foreign-CA-signed certs are rejected. | Producer/consumer time out due to TLS handshake failure. |


This is a godo test. I wonder if should try to test the same on the internal ports as well (9091).

This is a godo test. I wonder if should try to test the same on the internal ports as well (9091).

If you think it make sense and it would add value but I am not sure how much code / addiional coverage it would do. I mean for me it would be just adding one test case (when I tried on my machine it tooks like 6 minutes so it should be fine)

I think this has value. There is a separate code path, and the internal listeners are a big part of GHSA-2qwx-rq6j-8r6j. So I think this is worth 6 minutes of test time. You would need to add custom network policy to allow access to 9091 or 9090 from your clients. But otherwise it might be the same idea as for the Clients CA.

I see, but I would also need to use ClusterCA signed client as Port 9091 (i.e., replication listener) trusts only the ClusterCA for client authentiufication.

You would need to add custom network policy to allow access to 9091 or 9090 from your clients.

Yeah, I think I can use something like this:

NetworkPolicyUtils.allowNetworkPolicyAllIngressForMatchingLabel(testStorage.getNamespaceName(), testStorage.getClusterName() + "-internal-port-access", testStorage.getBrokerPoolSelector().getMatchLabels());

development-docs/systemtests/io.strimzi.systemtest.security.custom.CustomCaChainST.md

ppatierno · 2026-03-13T11:01:40Z

development-docs/systemtests/io.strimzi.systemtest.security.custom.CustomCaChainST.md

+| 1. | Generate a custom CA chain: Root -> Intermediate -> Leaf. | CA chain is generated. |
+| 2. | Generate a separate foreign Root CA. | Foreign CA is generated. |
+| 3. | Generate four user certificates signed by Leaf CA, Intermediate CA, Root CA, and foreign CA respectively. | User certificates are generated. |
+| 4. | Deploy user cert secrets and a CA trust secret containing only the Leaf CA cert. | Secrets are created in the namespace. |


Signed-off-by: see-quick <maros.orsak159@gmail.com>

scholzj · 2026-03-16T15:58:28Z

development-docs/systemtests/io.strimzi.systemtest.security.custom.CustomCaChainST.md

+| Step | Action | Result |
+| - | - | - |
+| 1. | Generate a custom CA chain: Root -> Intermediate -> Leaf. | CA chain is generated. |
+| 2. | Deploy the full custom CA chain as Cluster CA and Clients CA secrets. | CA secrets are deployed. |
+| 3. | Deploy Kafka cluster with custom CAs (generateCertificateAuthority: false) so that broker certificates are signed by the Leaf CA. | Kafka cluster is ready. |
+| 4. | Create a NetworkPolicy allowing all ingress to Kafka broker pods so that test clients can reach port 9091. | NetworkPolicy is created. |
+| 5. | Generate a client certificate signed by the Cluster CA Leaf and create a KafkaTopic. | Client certificate and KafkaTopic are created. |
+| 6. | Create trust secrets with different chain levels: Root + Intermediate + Leaf, Root + Intermediate, Root only, Intermediate only, Leaf only. | Trust secrets are created. |
+| 7. | For each trust secret, verify that clients can produce and consume messages on port 9091. | All five trust configurations succeed. |
+| 8. | Create a trust secret with only a foreign Root CA and verify that clients cannot connect on port 9091. | Producer/consumer time out due to trust failure. |


The test here is not necessarily invalid. But its core should be something different ...

Create a client cert signed by Leaf CA

Try to connect -> should work (I guess you have this at step 5 more or less from the description)

Create a client certificate signed by Root CA or Intermediate CA

Try to connect -> has to fail (=> this seems to be missing)

Try to connect -> should work (I guess you have this at step 5 more or less from the description)

correct.

Try to connect -> has to fail (=> this seems to be missing)

Yes, it's missing I can add these two case.

Okay so trust loop with different trust secrets you think it not needed? I can simplify that test to just do:

Generate custom CA chain

Deploy custom CA chain as Cluster CA and Client CA secrets

Deploy Kafka cluster where broker certificates are signed by Leaf CA

Create NetworkPolicy allowing ...

Generate client cert signed by Leaf CA

Verify that it works

Generate client cert signed by Root CA or IM CA

Verify it doesn't work

WDYT?

Yes, correct.

scholzj · 2026-03-16T16:17:25Z

development-docs/systemtests/io.strimzi.systemtest.security.custom.CustomCaChainST.md

+| 2. | Generate a separate foreign Root CA. | Foreign CA is generated. |
+| 3. | Generate four user certificates signed by Leaf CA, Intermediate CA, Root CA, and foreign CA respectively. | User certificates are generated. |
+| 4. | Deploy external client certificate secrets (each signed by a different CA) and a broker-side CA trust secret containing only the Leaf CA cert. | Secrets are created in the namespace. |
+| 5. | Deploy Kafka with an internal TLS listener using Custom TLS client authentication, with the broker truststore containing only the Leaf CA cert. | Kafka cluster is ready with the configured listener. |


I think this is still wrong. We want to test how we handle the Clients CA and how we establish trust from it. That was the CVE we had. So You need to use the CA from step one as a custom Clients CA and authenticate against it. If you use custom settings, you test how you configured it yourself. not how the operator does it.

Okay, so here I would not use manual truststore configuration.

I would basically do the same as https://github.com/strimzi/strimzi-kafka-operator/pull/12497/changes/BASE..5a960e22b8448bd1ea038355b03a2ecbb48e18ac#r2959611701 but instead using internal port I would use the classic one for the clients. Right?

Yes, correct.

[system test] Add CustomCaChainST

c821009

Signed-off-by: see-quick <maros.orsak159@gmail.com>

see-quick added 2 commits March 10, 2026 09:22

add tets doc

6b9a256

Signed-off-by: see-quick <maros.orsak159@gmail.com>

add system-test-doc

40f00e8

Signed-off-by: see-quick <maros.orsak159@gmail.com>

see-quick marked this pull request as ready for review March 10, 2026 09:18

see-quick requested a review from a team March 10, 2026 09:18

see-quick self-assigned this Mar 10, 2026

see-quick added this to the 1.0.0 milestone Mar 10, 2026

see-quick added needs review System tests labels Mar 10, 2026

see-quick linked an issue Mar 10, 2026 that may be closed by this pull request

[ST] Improve coverage of how server certificates are used when custom multistage CA is used #12404

Open

ppatierno requested a review from katheris March 10, 2026 11:13

katheris reviewed Mar 11, 2026

View reviewed changes

Update a bit tests to also include im-only, leaf-only etc. + a few tw…

2af9e8c

…eaks Signed-off-by: see-quick <maros.orsak159@gmail.com>

scholzj reviewed Mar 12, 2026

View reviewed changes

ppatierno reviewed Mar 13, 2026

View reviewed changes

see-quick added 2 commits March 13, 2026 14:34

address reviews

50c6d68

Signed-off-by: see-quick <maros.orsak159@gmail.com>

Improve sys-documentation + Add 9091 test case

5a960e2

Signed-off-by: see-quick <maros.orsak159@gmail.com>

scholzj reviewed Mar 16, 2026

View reviewed changes

Conversation

see-quick commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Description

Checklist

Uh oh!

see-quick commented Mar 6, 2026

Uh oh!

github-actions bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Mar 6, 2026

Uh oh!

katheris left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

see-quick Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

see-quick Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

see-quick Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

see-quick commented Mar 6, 2026 •

edited

Loading

github-actions bot commented Mar 6, 2026 •

edited

Loading

codecov bot commented Mar 6, 2026 •

edited

Loading

see-quick Mar 16, 2026 •

edited

Loading

see-quick Mar 13, 2026 •

edited

Loading

see-quick Mar 16, 2026 •

edited

Loading