Enable verified SSL support for the Elasticsearch/Opensearch cluster

[keithgg]

Shared Elasticsearch was implemented in #13. While SSL is implemented it could not be completed due to overhangio/tutor#791. That PR has now been merged, so this can be tackled.

• Add a Kubernetes Secret to store the CA certificate generated by the helm chart.
• Mount this secret inside each container that will require it under the directory /etc/ssl/certs/elasticsearch-ca.pem.
• For the forum, check out the PR feat: operators can prefix elasticsearch indexes for multi-tenancy cs_comments_service#404

[felipemontoya]

@MoisesGSalas do you know if it possible to use the certbot process to create this certs instead of relying on self-signed certs?

[gabor-boros]

I just figured out that Tutor 16 includes a newer version of dockerize that now supports -cacert flag. Using that flag, we could let dockerize know we want to use the custom cert we created, however that requires modifications in the tutor-forum source code to optionally pass the -cacert flag if the CA check would fail.

One option for that could be:

# ...

WAIT_SEARCH_SERVER="-wait $SEARCH_SERVER"
if [[ -n $(curl --verbose --silent "${SEARCH_SERVER}" 2>&1 | grep "TLS alert, unknown CA") ]]; then
    CA_CERT_PATH="...somehow get the path..."
    WAIT_SEARCH_SERVER="${WAIT_SEARCH_SERVER} -cacert ${CA_CERT_PATH}"
fi

# ...

dockerize $WAIT_MONGODB $WAIT_SEARCH_SERVER -wait-retry-interval 5s -timeout 600s

[gabor-boros]

@felipemontoya, @MoisesGSalas, @bradenmacdonald I checked the certbot approach and was working on that for a while to make it proper. However, I have some findings that I would love to discuss with you before going on.

Indeed, it is possible to use certbot issuing a certificate for the Elasticsearch cluster. However, the Kubernetes internal resources are not running on a proper domain name (.svc.local), so certbot cannot issue a certificate for that using Let's Encrypt. Therefore, we must point a DNS name to the Kubernetes cluster we have.

Doing so will let us issue the certificate and configure it for Elasticsearch. The server will pick up the certificate with ease, though the question raises: How to communicate with the server? The main concern is that we have to a) reach out to the internet to get back to the server, or b) intervene into the Certificate validation process at some point.

Going with the first option would mean we have to create an additional ingress controller to route the traffic to the Elasticsearch cluster. This introduces extra latency that could have been avoided and it feels off to do so.

The second option makes the Let's Encrypt signed certificate pointless. The cert would not contain localhost domain name --for the same reason as mentioned above--, because of this line we still need to use --insecure. The -H Host: <domain name> flag for the underlying curl command wouldn't work either, because an HTTPS server also needs to get the server name provided already in the TLS handshake so that it knows which cert to use. I.e., the Certificate validation happens before the Host header is parsed. The --insecure flag can be avoided by intervening into the validation process by patching the internal cache of Curl. This could be done using curl -u elastic:${ELASTIC_PASSWORD} --resolve <domain>:9200:127.0.0.1 https://<domain>:9200/_cat/indices. However, that would not solve all "trust issues".

The connection parameters of the instances would still have to set "verify_certs": False because the inter-cluster connection.

So, to summarize: We have to use self-signed certificates and attach the CA (and update the containers' CA cache), or we use trusted CA signed certs, but the clients will still need to use "verify_certs": False.

But there is one thing that bugs me: Do we really want to struggle with this? The communication is not leaving the cluster; not even exposed to the internet, hence I'm not even convinced by now that this is necessary at all.

What are your thoughts?

[bradenmacdonald]

Doing so will let us issue the certificate and configure it for Elasticsearch. The server will pick up the certificate with ease, though the question raises: How to communicate with the server? The main concern is that we have to a) reach out to the internet to get back to the server

Can't you use DNS within the cluster to point that domain name to a cluster-local endpoint, so we don't have to reach out over the internet? i.e. LetsEncrypt sees es1.mycluster.example.com as the ingress IP, but within the cluster, es1.mycluster.example.com resolves to the ES service and doesn't get resolved via the internet.

I'm also fine with using self-signed certs or some simpler option though; we have other priorities at the moment.

[gabor-boros]

@bradenmacdonald I was trying to dig into this. In the Kubernetes cluster, the DNS resolution of *.svc.local addresses are handled automatically by CoreDNS and .svc.local addresses are added/removed as needed. I did not find a way though to extend it with custom records. Also, I feel that would be a bit of hacking to "fake" the DNS record we want to get resolved (es1.mycluster.example.com).

I'm fine with self-signed certs as well, though my question still stands: Do we want to really make the effort to let the pods trusting the cluster-local CA? The whole traffic is happening within the cluster.