[Bug]: mongodb replicaset key is not fixed, changing on each "helm template", resulting on "AuthenticationFailed: SCRAM authentication failed, storedKey mismatch" after a while

[damienleger]

Is there an existing issue for this?

• I have searched the existing issues

Description

On values.yml file currently, mongodb.auth.replicaSetKey is not fixed (value on bitnami helm chart: https://github.com/bitnami/charts/blob/343ce6df75c858132b0a42cb7691441da56768df/bitnami/mongodb/values.yaml#L198). Hence, on each "helm template" invocation, the value of the "mongodb-replica-set-key" key in appsmith-mongodb Secret changes. Such invocation is done by ArgoCD on regular basis, or I suppose every time a deployment takes place with "vanilla" helm, ultimately causing mismatch of replicaset key betweens the differents mongodb containers and the error "AuthenticationFailed: SCRAM authentication failed, storedKey mismatch" during the lifecycle of mongodb and eventually a mongodb that refuses to start at all.

To have the expected behavior, we want the value of the "mongodb-replica-set-key" key in appsmith-mongodb Secret being the same as the value of echo ${MONGODB_REPLICA_SET_KEY} in appsmith-mongodb-0 + appsmith-mongodb-1 + appsmith-mongodb-arbiter-0. Adding mongodb.auth.replicaSetKey = replicasetkey in the values.yaml does this.

Steps To Reproduce

1. Invoke helm template twice and notice the only diff is on the "mongodb-replica-set-key" key in appsmith-mongodb Secret. Something like diff <(helm template .) <(helm template .) in bash

2. And you can simulate the behavior of ArgoCD by kubectl apply the result of helm template . on a fresh appsmith install. Notice envvar in mongodb container now matches the Secret that is why it's working on fresh install. kubectl apply once or twice the result of helm template . and randomly restart mongodb containers in between. You should see the error "AuthenticationFailed: SCRAM authentication failed, storedKey mismatch" in the logs of mongodb and the mismatch of replicasetkey in Secret/envvar.

Public Sample App

No response

Environment

Production

Severity

Critical (Broken Production apps)

Issue video log

No response

Version

ee chart appsmith-3.4.5