Expand the testing area of JEP-237 (FIPS-140 support) #407
Conversation
Expand the testing area for FIPS-140. Detail that no automated checks will find all compliance issues so given we need to do a manual check there is little point in any automated checks for checking core/plugin compliance today.
|
@basil do you want to review this? |
| The https://github.com/jenkinsci/acceptance-test-harness[`acceptance test harness`] would appear to be a suitable place for tests to detect FIPS regressions and issues. | ||
|
|
||
| As the ATH generally interacts with "external" systems (usually by way of a container) to check for regressions the environment should be a FIPS enabled environment, that includes the docker host as well as the system running Jenkins. | ||
| This system can be independent of the host running the test (junit) code. | ||
| The regression/test suite would need to be tagged with which plugins to test (or not test) and any agents launched for builds as part of the test would also need to be FIPS enabled. | ||
|
|
||
| As with the other automated checks this can detect issues and regressions in features, but can not validate that the code is FIPS-140 compliant. | ||
| These tests are generally the most expensive to run and coverage of plugins/features is variable. | ||
| Whilst the value of finding regressions is certainly important, the ATH should be checking features, and if the feature works outside of FIPS (in a run of the current ATH) then the unit testing of the switched can provide this check, so this leaves the value at checking the JVM configuration is not causing regressions. | ||
| As the JVM configuration is out of scope of the Jenkins project as outlined above, testing this configuration is of little value. |
There was a problem hiding this comment.
Configuring a FIPS-compliant JVM for running ATH tests in CI does not seem insurmountable from an infrastructure perspective, assuming that all of the libraries needed to run such a JVM are free and open-source software, and doing so seems like a valuable method of ensuring that FIPS regressions are not inadvertently introduced in the future.
There was a problem hiding this comment.
configuring a JVM alone is not insurmountable, but then I do not see what value we would get from this expensive testing for the cost of configuring and running it.
A FIPS compliant JVM does not necessarily prevent non FIPS compliant code from running, as such you could regress the FIPS compliant plugin code and the ATH may very well not detect it.
For example - anything launching a SSH Agent today[1] for tests in the ATH will work - but SSH Agents use trilead-api and this is so far from being FIPS compliant it is a non starter to make it compliant.
So what it can catch is if some code that was only exercised when Jenkins was started in FIPS mode regressed - but that would be better tested as a RealJenkinsRule or other unit test as HudsonPrivateSecurityRealmFIPSTest.
But we can achieve most of the same just by starting Jenkins with the FIPS flag to catch these styles of regression should there be a valid reason to perform this style of testing over an ATH style test.
Iff we wanted to run ATHs for jenkins core and plugins then in addition to a fips compliant JVM we would also need to:
- tag the ATHs we want to run by analysing them and adding them to a JUnit category or other.
- ensure that the host that runs the docker deamon is also FIPS compliant (the kernel flag is inherited by containers and some OS/Libraries look at that flag to change behaviour at runtime) (requires dedicated k8s
node poolfrom ci.jenkins.io) - ensure any service specific configuration for the docker fixture is set correctly (e.g. sshd would need to be restricted to FIPS compliant algorithms in the SSHd server/OS combination that the service supports)
- other minor tweaks/updates to the ATH framework (e.g.
DumbSlave) - misc updates to any tests as required, e.g. launching java (e.g. https://github.com/jenkinsci/acceptance-test-harness/blob/5bee05f80e9118e2fb4f530d49e9d50a623422b4/src/test/java/plugins/PipelineTest.java#L74)
I attempted to summarise this on line 166 and the preceding lines,
Whilst the value of finding regressions is certainly important, the ATH should be checking features, and if the feature works outside of FIPS (in a run of the current ATH) then the unit testing of the switched can provide this check, so this leaves the value at checking the JVM configuration is not causing regressions.
but it appears I may have not done so effectively?
There is an additional concern that I did not outline, in that it can bring a false sense of security that it could bring ("the ATH passed this must be FIPS compliant").
[1] trilead-api probably needs to make this plugin just throw a BootFailure when run with the FIPS flag...
Expand the testing area for FIPS-140.
Detail that no automated checks will find all compliance issues so given we need to do a manual check there is little point in any automated checks for checking core/plugin compliance today.
fixes #404
Testing done
Submitter checklist