Feature Test Summary - Remove weaker ciphers from the HIGH cipher list in Liberty #24601

[jacobwdv]

Test Strategy

Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.

For any feature, be aware that only FAT tests (not unit or BVT) are executed in our cross-platform testing. To ensure cross-platform testing, ensure you have sufficient FAT coverage to verify the feature.

If delivering tests outside of the standard Liberty FAT framework, do the tests push the results into the cognitive testing database (if not, consult with the CSI Team, who can provide advice and verify if results are being received)?

• a new fat bucket called com.ibm.ws.ssl_fat_ciphers
• old fats that have securityLevel defined need to have it removed (there are many since this was a standard ssl attribute before
• com.ibm.ws.ssl unit tests new class for testing Constants.java

List of FAT projects affected

• Many, any fat which defines securityLevel. These won't be substantial changes but they will need to be done
• com.ibm.ws.ssl_fat_ciphers - new fat

Test strategy

• What functionality is new or modified by this feature?
  • Users can now specify filtering criteria in the enabledCIphers attribute.
  • Liberty will no longer filter the JDK provided effective ciphers list
• What are the positive and negative tests for that functionality?
  • Negative tests
    • enabledCiphers has both filter criteria and static entry (not allowed, error thrown and JDK ciphers list will be used without modifications)
    • enabledCIphers has a wildcard specified in a + entry (not allowed, error thrown and JDK ciphers list will be used without modifications)
  • Positive tests
    * Users specify a securityLevel HIGH - ensure log message is logged
    * Users specify a securityLevel MEDIUM or LOW - ensure warning message is logged
    * handshake cipher verification
  • Unit tests
    * - entry with full cipher name
    * - entry with wildcard name
    * mix of - entries that have wildcard and full cipher names
    * + entry with full cipher name
    * + entry with full name, - entry wtih full name
    * + entry with full name, - entry with wildcard
    * - entry with wildcard and + entry that adds a wildcard removal item back into the list ("-TLS_RSA_* +TLS_RSA_WITH_AES_128_CBC_SHA" )
    * FAT test with client (http or liberty) that handshakes
    * trace.log verification of cipher list being modified
    * client handshake failure if cipher suites used by the client are removed by the server
    * client handshake success if the cipher suite used by the client is available on the server
    * client handshake success if the cipher suite used by the client is available on the server after doing a wildcard removal and re-add

Confidence Level

Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.

Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:

0 - No automated testing delivered

1 - We have minimal automated coverage of the feature, including golden paths. There is a relatively high risk that defects or issues could be found in this feature.

2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths

3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.

4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.

5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.

5 - This should be extensive testing of this feature and I believe the feature is small enough to be tested fully.

Based on your answer above, for any answer other than a 4 or 5, please provide details of what drove your answer. Please be aware that it may be perfectly reasonable in some scenarios to deliver with any value above. We may accept that no automated testing is needed for some features, we may be happy with low levels of testing on samples, for instance, so please don't feel the need to drive to a 5. We need your honest assessment as a team and the reasoning for why you believe shipping at that level is valid. What are the gaps, what is the risk etc. Please also provide links to the follow-on work that is needed to close the gaps (should you deem it needed)