Integrating the Redis Enterprise Operator with Hashicorp Vault

Overview

Hashicorp Vault can be configured as the source of secrets used by the Redis Enterprise K8s operator as an alternative to Kubernetes secrets.

Clarification: when running in Vault mode, all secrets referenced in the Redis Enterprise custom resources are read from Vault instead of from Kubernetes Secrets. This includes credentials to access the cluster and databases, certificates, license, credentials to access backup storage targets, LDAP servers, etc.
For a full list of secrets that can be specified, please refer to the RedisEnterpriseCluster and RedisEnterpriseDatabase API reference pages.

To configure the operator to read secrets from Vault, set .spec.clusterCredentialSecretType: "vault" in the RedisEnterpriseCluster resource. This will be further explained next.

How to use Hashicorp Vault as a source for secrets:

Prerequisites
Deployment
Deploying the operator
Creating the REC
REC secrets
Deploy REDB admission Controller
Creating an REDB
REDB secrets
RERC secrets
REAADB secrets

Note: when using Openshift it might be recommended to use oc instead of kubectl

Prerequisites

Deploy a Hashicorp Vault instance and make sure there is network access to it from the Kubernetes cluster. The solution has been tested with Hashicorp Vault v1.15.2. The Hashicorp Vault instance must be using TLS.
Configure the Hashicorp Vault Kubernetes authentication for the Kubernetes cluster the operator is being deployed. Refer to the Hashicorp Vault documentation for details.
Deploy the Hashicorp Vault agent sidecar controller on the Kubernetes cluster (https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar)
Note that Hashicorp offers a Vault Enterprise product. The Vault Enterprise product supports namespaces. Those namespaces should not be confused with Kubernetes namespaces. This document assumes that the Hashicorp Vault instance used is the Enterprise product, and a Vault namespace is used. The namespace is referred to as the <VAULT_NAMESPACE> below.
Redis Enterprise will use a kv-v2 secret engine. Make sure it is available on the Hashicorp Vault instance (or create one if needed) and take note of the path it is mounted on, since it will be used later.

Deployment

General considerations

Hashicorp Vault and the Redis Enterprise Operator can be deployed in multiple scenarios that might affect the details of the process below. The document assumes the following:

Hashicorp Vault enterprise is used, and Vault namespaces are used. If that is not the case, it is recommended to remove the namespace parameters, environment variables from the relevant directions.
Multiple Redis Enterprise Clusters are configured within the same K8s cluster, configured to authenticate to Hashicorp Vault.
To ensure privacy and avoid duplication, the K8S_NAMESPACE is appended to multiple names of Hashicorp Vault configurations. That might need to be further adjusted in cases multiple K8s clusters are used with the same K8s namespaces.
The minimum TTL of the Vault token under the policy assigned to redis enterprise should be one hour. (see also https://learn.hashicorp.com/tutorials/vault/token-management#configure-the-token-ttl)

Deploying the operator

Configure a Hashicorp Vault policy. The policy will be used to grant the operator access to the secrets.

Run the following command within the Hashicorp Vault interface (use kubectl exec when Vault is deployed on Kubernetes, replace <K8S_NAMESPACE> with the namespace where the operator is deployed into):

vault policy write -namespace=<VAULT_NAMESPACE> redisenterprise-<K8S_NAMESPACE> - <<EOF
path "secret/data/redisenterprise-<K8S_NAMESPACE>/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/metadata/redisenterprise-<K8S_NAMESPACE>/*" {
  capabilities = ["list"]
}
EOF

Configure a Vault role:

vault write -namespace=<VAULT_NAMESPACE> auth/<AUTH_PATH>/role/redis-enterprise-operator-<K8S_NAMESPACE> \
        bound_service_account_names="redis-enterprise-operator"  \
        bound_service_account_namespaces=<K8S_NAMESPACE> \
        policies=redisenterprise-<K8S_NAMESPACE>

Note - replace <AUTH_PATH> with the path kubernetes auth is enabled in Hashicorp Vault. The default is "kubernetes"

Create the operator's configuration in configmap named 'operator-environment-config' with the relevant Vault configuration:
edit and save this content in a file called operator-environment-config.yaml
run kubectl apply -f operator-environment-config.yaml
see notes on the parameters below.
```
apiVersion: v1
kind: ConfigMap
metadata:
   name: operator-environment-config
data:
   CREDENTIAL_TYPE: "vault"
   VAULT_SERVER_FQDN: <Your FQDN>
   VAULT_SERVICE_PORT_HTTPS: "8200"
   VAULT_SECRET_ROOT: "secret"
   VAULT_SECRET_PREFIX: "redisenterprise-<K8S_NAMESPACE>"   
   VAULT_ROLE: "redis-enterprise-operator-<K8S_NAMESPACE>"
   VAULT_AUTH_PATH: <AUTH_PATH>
   VAULT_NAMESPACE: <VAULT_NAMESPACE>
   VAULT_CACHE_SECRET_EXPIRATION_SECONDS: <the_secret_expiration_in_seconds>
```
- VAULT_SERVER_FQDN: Hashicorp Vault server Fully Qualified Domain Name (FQDN). If the Vault server is running with k8s,
  it would typically be <YOUR_VAULT_SERVICE_NAME>.<YOUR_VAULT_SERVICE_NAMESPACE>):
- VAULT_SECRET_ROOT: the path the kv-2 secret engine being used is enabled on.
- VAULT_SECRET_PREFIX: should be unique to the Redis Enterprise Cluster. Here we use redisenterprise-<K8s_NAMESPACE>.
  This value has to be consistent with Hashicorp Vault roles and policies.
- VAULT_ROLE: the Vault role you configured in the previous step, defaults to redis-enterprise-operator.
- VAULT_AUTH_PATH: the path kubernetes auth is enabled in Hashicorp Vault, defaults to kubernetes - use no leading/trailing slashes.
- VAULT_NAMESPACE: supported in Hashicorp Vault enterprise.
The full secret path would be: <VAULT_SECRET_ROOT>/<VAULT_SECRET_PREFIX>/
- VAULT_CACHE_SECRET_EXPIRATION_SECONDS: Defines the expiration duration of secrets that are fetched from Vault. Secrets are cached in the operator for a period of X seconds (2 min by default). Note - the REC credentials will be re-fetched directly from Vault in case of 'unauthorized' error via the RS API.
Deploy the operator by applying the Redis Labs Kubernetes Operator Bundle as explained here - steps 1,2 (steps 1-4 on OpenShift).
The Operator pod would not be ready before you save the admission controller secret to Vault:
1. Generate a json file with key/cert pair to be used by admission:
```
kubectl exec -it $(kubectl get pod -l name=redis-enterprise-operator -o jsonpath='{.items[0].metadata.name}') -c redis-enterprise-operator -- /usr/local/bin/generate-tls -infer | tail -4 > output.json
```
  - the output.json file is needed for additional steps below (deployment of admission controller)
2. Apply the secret to vault - execute the following within the Hashicorp Vault CLI interface (you will need to copy
  the file from the previous step for example by running kubectl cp output.json vault-0:/tmp -n vault):
```
vault kv put <VAULT_SECRET_ROOT>/redisenterprise-<K8S_NAMESPACE>/admission-tls @output.json
```
Once operator is running, proceed to the steps below. Avoid creating the Redis Enterprise Cluster custom resource.
Create a K8s secret containing the Certificate Authority Certificate (CACert) used to create the Hashicorp Vault instance server certificate.
Name the secret vault-ca-cert and the key vault.ca . Save the CA cert to a file before running the following command:
```
kubectl create secret generic vault-ca-cert \
        --namespace <K8S_NAMESPACE> \
        --from-file=vault.ca=<vault instance server CA certificate path>
```
Note - the server certificate of the Hashicorp Vault instance must be signed by the Certificate Authority used within the secret.

Creating the Redis Enterprise Cluster

Choose a random password. Unlike the default deployment, the operator is not creating a default password for the Redis Enterprise Cluster credentials, and those need to be chosen. It is recommended to use a tool to generate a random password at least 8 characters long.
Save the password as a secret within the Hashicorp Vault instance, replace values as needed. Execute the following command within the Hashicorp Vault CLI interface:
```
vault kv put -namespace=<VAULT_NAMESPACE> <VAULT_SECRET_ROOT>/redisenterprise-<K8S_NAMESPACE>/<REC_NAME> username=<YOUR_USERNAME> password=<YOUR_PASSWORD>
```
Note - The username field in the REC spec will be ignored when using vault. The username from the vault secret will be used instead. Note - this example matches configuring the operator with environment variable values: VAULT_SECRET_ROOT=secret, VAULT_SECRET_PREFIX=redisenterprise-<K8s_NAMESPACE> as mentioned above

Create a role in vault for the REC service account:

vault  write -namespace=<VAULT_NAMESPACE> auth/<AUTH_PATH>/role/redis-enterprise-rec-<K8S_NAMESPACE> \
       bound_service_account_names=<REC_NAME>  \
       bound_service_account_namespaces=<K8S_NAMESPACE> \
       policies=redisenterprise-<K8S_NAMESPACE>

Apply the Redis Enterprise Cluster yaml. Example (make sure the clusterCredentialSecretName is consistent with Hashicorp Vault configuration above):

apiVersion: app.redislabs.com/v1
kind: RedisEnterpriseCluster
metadata:
  name: rec
  labels:
    app: redis-enterprise
spec:
  # Add fields here
  nodes: 3
  clusterCredentialSecretName: rec 
  clusterCredentialSecretType: vault
  clusterCredentialSecretRole: redis-enterprise-rec-<K8S_NAMESPACE>
  vaultCASecret: vault-ca-cert
  podAnnotations:
      vault.hashicorp.com/auth-path: auth/<AUTH_PATH>
      vault.hashicorp.com/namespace: <VAULT_NAMESPACE>

Note - the "clusterCredentialSecretName" field as used to query the secret from Hashicorp Vault. See section below for explanation about secret name field values.

Redis Enterprise Cluster secrets

The full and detailed REC fields documentation can be found here

Cluster credentials and license fields:

Cluster Credential Secret: clusterCredentialSecretName
License Secret: licenseSecretName

Certificates:

These are the certificates and their field name in the REC:

API Certificate: apiCertificateSecretName
CM Certificate: cmCertificateSecretName
Metrics Exporter Certificate: metricsExporterCertificateSecretName
Proxy Certificate: proxyCertificateSecretName
Syncer Certificate: syncerCertificateSecretName
LDAP client Certificate: ldapClientCertificateSecretName

You can read more about the different certificates Here

Show REC example

apiVersion: app.redislabs.com/v1
kind: RedisEnterpriseCluster
metadata:
  name: rec
  labels:
    app: redis-enterprise
spec:
  nodes: 3
  
  licenseSecretName: <VAULT SECRET NAME HERE>
  clusterCredentialSecretName: <VAULT SECRET NAME HERE>
  certificates:
    apiCertificateSecretName: <VAULT SECRET NAME HERE>
    cmCertificateSecretName: <VAULT SECRET NAME HERE>
    metricsExporterCertificateSecretName: <VAULT SECRET NAME HERE>
    proxyCertificateSecretName: <VAULT SECRET NAME HERE>
    syncerCertificateSecretName: <VAULT SECRET NAME HERE>
    ldapClientCertificateSecretName: <VAULT SECRET NAME HERE>

  # vault configuration as explained above: 
  clusterCredentialSecretType: vault
  clusterCredentialSecretRole: redis-enterprise-rec-<K8S_NAMESPACE>
  vaultCASecret: vault-ca-cert
  podAnnotations:
      vault.hashicorp.com/auth-path: auth/<AUTH_PATH>
      vault.hashicorp.com/namespace: <VAULT_NAMESPACE>

Edit and apply the rec.yaml or use patch like in this example, which sets API Certificate secret name:

  kubectl patch rec rec --type merge --patch "{\"spec\": \
    {\"certificates\": \
      {\"apiCertificateSecretName\": \"<VAULT SECRET NAME HERE>\" }}}"

Deploy REDB admission controller (for OLM this is not needed)

It is not recommended to use the admission bundle here if you want to avoid creation of K8s secrets. Instead, do a step-by-step installation.

Create the Kubernetes Validating Webhook (for OLM this is not needed) NOTE: One must replace REPLACE_WITH_NAMESPACE in the following command with the namespace the REC was installed into.

# save cert
CERT=`cat  output.json | jq -r ".cert"`
sed 's/NAMESPACE_OF_SERVICE_ACCOUNT/REPLACE_WITH_NAMESPACE/g' ../admission/webhook.yaml | kubectl create -f -

# create patch file
cat > modified-webhook.yaml <<EOF
webhooks:
- name: redisenterprise.admission.redislabs
  clientConfig:
    caBundle: $CERT
  admissionReviewVersions: ["v1beta1"]
EOF
# patch webhook with caBundle
kubectl patch ValidatingWebhookConfiguration redb-admission --patch "$(cat modified-webhook.yaml)"

Note - use the output.json that was created in the steps above

Make sure admission works, see here for steps description

Creating an REDB

Steps to create an REDB:

Create a password in Vault in this path (change according to the specific configuration, see above)
<VAULT_SECRET_ROOT>/<VAULT_SECRET_PREFIX>/redb-<REDB_NAME>:
where VAULT_SECRET_ROOT and VAULT_SECRET_PREFIX are defined in the operator's ConfigMap as explained above (or set to default values).
e.g. vault kv put secret/redisenterprise-<K8S_NAMESPACE>/redb-mydb password=somepassword
Create the REDB custom resource.
Follow the step 6 here. The REC spec indicted you are running with Vault and no further configuration is required.
The other REDB secrets (2 to 4) should be created in this path redisenterprise-<K8S_NAMESPACE>/. The secrets should comply with the REDB secrets schema.

Note - when using the Redis Enterprise Vault plugin it recommended to set defaultUser: false and associate users through ACL bindings to the REDB

REDB secrets

An REDB has several secrets associate with it as detailed here.

The password for the REDB
Replica Source (optional).
Specifically: clientKeySecret and serverCertSecret fields which holds the Vault secret name
Backup Credentials (optional)
These are backup options, they contain (among other fields) a field for a secret used to access the backup.
- S3 Storage: awsSecretName
- sftp storage: sftpSecretName
- Swift Storage: swiftSecretName
- Azure Blob Storage: absSecretName
- Google Storage: gcsSecretName
Client Auth (optional) - The Secrets containing TLS Client Certificate to use for Authentication

The full and detailed REDB fields documentation can be found here

RERC secrets

The secretName field is supported and should be stored in Hashicorp Vault if the Redis Enterprise Cluster uses Hashicorp Vault as a secret source.

REAADB secrets

A REDB specification is built into REAADB (the globalConfigurations field). All secret names specified there are supported and should be stored in Hashicorp Vault if the Redis Enterprise Cluster uses Hashicorp Vault as a secret source.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Integrating the Redis Enterprise Operator with Hashicorp Vault

Overview

Prerequisites

Deployment

General considerations

Deploying the operator

Creating the Redis Enterprise Cluster

Redis Enterprise Cluster secrets

Cluster credentials and license fields:

Certificates:

Deploy REDB admission controller (for OLM this is not needed)

Creating an REDB

REDB secrets

RERC secrets

REAADB secrets

Files

README.md

Latest commit

History

README.md

File metadata and controls

Integrating the Redis Enterprise Operator with Hashicorp Vault

Overview

Prerequisites

Deployment

General considerations

Deploying the operator

Creating the Redis Enterprise Cluster

Redis Enterprise Cluster secrets

Cluster credentials and license fields:

Certificates:

Deploy REDB admission controller (for OLM this is not needed)

Creating an REDB

REDB secrets

RERC secrets

REAADB secrets