Use env variable (OPENSEARCH_FIPS_MODE) to enable opensearch to run in FIPS enforced mode instead of checking for existence of bcFIPS jars

[cwperks]

Description

This PR updates logic in ./bin/opensearch-env (and /bin/opensearch-env.bat) to change the condition for running in FIPS approved mode. Currently, opensearch-env checks for the existence of bc-fips* jars under the lib/ folder and accordingly adds system props to ensure opensearch enforces FIPS-approved security policy. bc-fips folders are not currently included in the default distribution, and need to be built in from source by creating a distribution with the FIPS build param (-Pcrypto.standard=FIPS-140-3).

When the min distribution and default distribution start building with -Pcrypto.standard=FIPS-140-3 by default, then it would mean that this block is always hit and FIPS approved mode would be enforced. We do not (yet) want that to be default and instead rely on a cluster administrator to explicitly enable that.

This PR introduces an env var (OPENSEARCH_FIPS_MODE) that a cluster admin can use to enable it.

For example

OPENSEARCH_FIPS_MODE=true ./bin/opensearch

Related Issues

Resolves opensearch-project/opensearch-build#5979

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[coderabbitai]

📝 Walkthrough

Walkthrough

The changes replace automatic FIPS mode detection based on BouncyCastle FIPS JAR presence with an explicit environment variable configuration (OPENSEARCH_FIPS_MODE). Startup scripts for both Unix and Windows now check this variable to enable FIPS, and test configurations are updated to set it for FIPS compliance testing.

Changes

Cohort / File(s)	Summary
Documentation CHANGELOG.md	Added changelog entry documenting the new OPENSEARCH_FIPS_MODE environment variable for FIPS mode configuration.
Startup Scripts distribution/src/bin/opensearch-env, distribution/src/bin/opensearch-env.bat	Replaced automatic FIPS detection (checking for bc-fips JAR files) with explicit OPENSEARCH_FIPS_MODE environment variable. When set to "true" (case-insensitive), enables FIPS by configuring Java security properties and BouncyCastle FIPS approved mode. Removed previous JAR detection logic and fixed minor documentation formatting.
Test Configuration qa/fips-compliance/build.gradle	Added OPENSEARCH_FIPS_MODE=true environment variable to javaRestTest cluster configuration to enable FIPS mode during test execution.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 6

✅ Passed checks (6 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title clearly and concisely describes the main change: introducing OPENSEARCH_FIPS_MODE environment variable instead of checking for bcFIPS jars.
Description check	✅ Passed	The pull request description includes all required sections: detailed Description explaining the changes and rationale, Related Issues linking to #5979, and a completed Check List per the template.
Linked Issues check	✅ Passed	The PR meets the requirements from linked issue #5979 by introducing an explicit OPENSEARCH_FIPS_MODE environment variable allowing administrators to control FIPS enforcement, addressing the concern about unintended FIPS enforcement when distributions are built with -Pcrypto.standard=FIPS-140-3 by default.
Out of Scope Changes check	✅ Passed	All changes are directly related to implementing the OPENSEARCH_FIPS_MODE environment variable mechanism and updating related test configurations, with no extraneous modifications outside the linked issue's scope.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Merge Conflict Detection	✅ Passed	✅ No merge conflicts detected when merging into main

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

No actionable comments were generated in the recent review. 🎉

🧹 Recent nitpick comments

distribution/src/bin/opensearch-env (1)

115-120: Remove duplicate default assignment.

The defaulting to false is applied twice before normalization. One is sufficient.

♻️ Proposed simplification

-OPENSEARCH_FIPS_MODE="${OPENSEARCH_FIPS_MODE:-false}"
-
-# Normalize to lowercase for common inputs: TRUE/True/true
-OPENSEARCH_FIPS_MODE="${OPENSEARCH_FIPS_MODE:-false}"
+# Normalize to lowercase for common inputs: TRUE/True/true
+OPENSEARCH_FIPS_MODE="${OPENSEARCH_FIPS_MODE:-false}"

📜 Recent review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3aed19d and 08723c6.

📒 Files selected for processing (4)

• CHANGELOG.md
• distribution/src/bin/opensearch-env
• distribution/src/bin/opensearch-env.bat
• qa/fips-compliance/build.gradle

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (21)

• GitHub Check: gradle-check
• GitHub Check: precommit (25, macos-15)
• GitHub Check: precommit (21, windows-latest)
• GitHub Check: precommit (25, ubuntu-24.04-arm)
• GitHub Check: precommit (21, windows-2025, true)
• GitHub Check: precommit (25, windows-latest)
• GitHub Check: precommit (21, ubuntu-latest)
• GitHub Check: precommit (21, macos-15)
• GitHub Check: precommit (25, ubuntu-latest)
• GitHub Check: precommit (21, ubuntu-24.04-arm)
• GitHub Check: precommit (25, macos-15-intel)
• GitHub Check: precommit (21, macos-15-intel)
• GitHub Check: detect-breaking-change
• GitHub Check: assemble (21, windows-latest)
• GitHub Check: Analyze (java)
• GitHub Check: assemble (21, ubuntu-latest)
• GitHub Check: assemble (25, windows-latest)
• GitHub Check: assemble (25, ubuntu-latest)
• GitHub Check: assemble (21, ubuntu-24.04-arm)
• GitHub Check: assemble (25, ubuntu-24.04-arm)
• GitHub Check: Mend Security Check

🔇 Additional comments (3)

qa/fips-compliance/build.gradle (1)

34-37: FIPS test wiring looks good.

Explicitly setting OPENSEARCH_FIPS_MODE in the FIPS compliance cluster ensures the new runtime gate is exercised during tests.

distribution/src/bin/opensearch-env.bat (1)

33-39: LGTM: explicit FIPS gate for Windows startup.

Defaulting to false and enabling only on "true" is clear and avoids unintended FIPS activation.

CHANGELOG.md (1)

15-17: Changelog entry looks good.

Captures the new explicit FIPS enablement mechanism.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cwperks]

@beanuwave let me know what you think of this change. I think we should change the check to something different and not have it be checking for the existence of bcFips jars under the lib/ folder.

If the change in this PR looks ok, then I will raise a docs PR to accompany this.

[github-actions]

❗ AI-powered Code-Diff-Analyzer found issues on commit 4cb633e.

Path	Line	Severity	Description
distribution/src/bin/opensearch-env	116	low	FIPS mode configuration changed from automatic detection (jar presence) to explicit opt-in via environment variable. This is a documented design change with legitimate use cases, but could allow FIPS mode to be disabled in environments where it was previously auto-enabled. Default value is 'false', requiring explicit enablement. No evidence of malicious intent - appears to be intentional architecture change per PR #20625.

The table above displays the top 10 most important findings.

Total: 1 | Critical: 0 | High: 0 | Medium: 0 | Low: 1

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

❌ Gradle check result for 4cb633e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[beanuwave]

@cwperks I like this approach - it's one step closer to having bcfips.jar included on the class-path. However, I'm missing the trigger part. OPENSEARCH_FIPS_MODE would likly be set by gradle build script or docker's entrypoint.sh. All tests would need to pass when building with the -Pcrypto.standard=FIPS-140-3 parameter.

[github-actions]

❌ Gradle check result for 08723c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[cwperks]

@beanuwave agreed on running the entire gradle check against a distro built with -Pcrypto.standard=FIPS-140-3, but the scope of this PR is more focused on the logic within opensearch-env which then adds the system props to enforce that the OpenSearch process is run with FIPS approved algorithms.

[github-actions]

❌ Gradle check result for 08723c6: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 08723c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

@cwperks I would keep the if exist "%OPENSEARCH_HOME%\lib\bc-fips*.jar" ( check, people do tailor distributions (for whatever reasons), if the bc-fips libraries are not present, the FIPS related settings are noop essentially, we should fail to start