Merge branch 'cockroachdb:master' into do-main

Author: bsnyder788

.github/workflows/e2e-kind-create.yaml

@@ -37,5 +37,5 @@ jobs:
           path: "~/.cache/bazel"
           key: bazel
 
-      - name: End-to-end (kind)
-        run: make test/e2e/kind-create
+      - name: End-to-end (k3d)
+        run: make test/e2e/k3d-create

.github/workflows/e2e-kind-decommission.yaml

@@ -37,5 +37,5 @@ jobs:
           path: "~/.cache/bazel"
           key: bazel
 
-      - name: End-to-end (kind)
-        run: make test/e2e/kind-decommission
+      - name: End-to-end (k3d)
+        run: make test/e2e/k3d-decommission

.github/workflows/e2e-kind-upgrades.yaml

@@ -37,5 +37,5 @@ jobs:
           path: "~/.cache/bazel"
           key: bazel
 
-      - name: End-to-end (kind)
-        run: make test/e2e/kind-upgrades
+      - name: End-to-end (k3d)
+        run: make test/e2e/k3d-upgrades

.github/workflows/e2e-kind-upgradessha256.yaml

@@ -37,5 +37,5 @@ jobs:
           path: "~/.cache/bazel"
           key: bazel
 
-      - name: End-to-end (kind)
-        run: make test/e2e/kind-upgradessha256
+      - name: End-to-end (k3d)
+        run: make test/e2e/k3d-upgradessha256

.github/workflows/e2e-kind-versionchecker.yaml

@@ -38,4 +38,4 @@ jobs:
           key: bazel
 
       - name: End-to-end (kind)
-        run: make test/e2e/kind-versionchecker
+        run: make test/e2e/k3d-versionchecker

.github/workflows/nightly-smoketest.yaml

@@ -31,7 +31,7 @@ jobs:
     strategy:
       matrix:
         # supported Kubenetes versions
-        NODE_VERSION: [1.18.19, 1.19.11, 1.20.7, 1.21.2, 1.22.1]
+        NODE_VERSION: [1.19.16, 1.20.15, 1.21.9, 1.22.6, 1.23.3]
     steps:
       - uses: actions/checkout@v3
       - name: Mount bazel cache

CHANGELOG.md

@@ -5,7 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres
 to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-# [Unreleased](https://github.com/cockroachdb/cockroach-operator/compare/v2.6.0...master)
+# [Unreleased](https://github.com/cockroachdb/cockroach-operator/compare/v2.8.0...master)
+
+## Fixed
+
+* Install init container certs with 600 permissions
+
+# [v2.8.0](https://github.com/cockroachdb/cockroach-operator/compare/v2.7.0...v2.8.0)
+
+## Added
+
+* `AutomountServiceAccountToken` field for cluster spec to allow mounting the default service account token.
+
+## Fixed
+
+* Delete the CancelLoop function, fixing a cluster status update bug
+* Correctly detect failed version checker Pods
+* retry cluster status updates, reducing test flakes
+
+# [v2.7.0](https://github.com/cockroachdb/cockroach-operator/compare/v2.6.0...v2.7.0)
+
+## Fixed
+
+* Grant operator deletecollection permissions to fix fullcluster restart flow
+* Grant operator list and update permissions on pvcs to fix pvc resize flow
+* Bump TerminationGracePeriodSeconds from 1m to 5m
+* Prefer user added --join flags over default when explicitly passed
 
 # [v2.6.0](https://github.com/cockroachdb/cockroach-operator/compare/v2.5.3...v2.6.0)
 

DEVELOPER.md

@@ -6,14 +6,12 @@ See: https://docs.bazel.build/versions/master/install.html
 
 Take a look at the [Makefile](https://github.com/cockroachdb/cockroach-operator/blob/master/Makefile) for various
 targets you can run. The e2e testing requires a container engine like docker running as it starts a K8s cluster with
-Kind. The Makefile is simply a wrapper for bazel commands and does not have any build functionality in it.
+[k3d]. The Makefile is simply a wrapper for bazel commands and does not have any build functionality in it.
 
 You will also need kubectl and a running Kubernetes cluster to run the k8s targets like `make k8s/apply`.
 
 Bazel has been primarily tested on Linux but "should" run on macOS as well. Windows at this time has not been tested.
 
-TODO: notes on python configuration for the host.
-
 If you are new to Bazel take a look at the Workspace file, as it lays out which rule sets we are using, such as docker
 rules and k8s rules. The Makefile notes below talk about the different target groups that run various bazel commands.
 
@@ -39,25 +37,24 @@ Please run the testing targets locally before you push a PR.
 
 ## Running Locally
 
-We use [kind] to run the operator in a local environment. This can provide a faster feedback cycle while developing
+We use [k3d] to run the operator in a local environment. This can provide a faster feedback cycle while developing
 since there's no need to set up a remote GKE/OpenShift cluster.
 
-[kind]: https://kind.sigs.k8s.io/
+[k3d]: https://k3d.io
 
 **make dev/up**
 
 This command will get everything set up for you to begin testing out the operator. Specifically it will:
 
-* Start a local docker registry (via docker run) and configure it to work with kind/K8s
-* Start a kind cluster named test (context=kind-test)
+* Start a k3d cluster named test (context=k3d-test) with a managed docker registry
 * Install the CRDs
-* Build a docker image and publish it to the local registry
+* Build a docker image and publish it to the k3d registry
 * Deploy the operator and wait for it to be available
 * Ensure your shell is configured to use the new cluster (kube context)
 
 **make dev/down**
 
-Tears down the kind cluster.
+Tears down the k3d cluster.
 
 ## Testing CR Database
 
@@ -108,8 +105,8 @@ When you have removed your example and the persitent volumes you can use the fol
 1. If you do not have a GKE cluster we have a helper script `./hack/create-gke-cluster.sh -c test`.
 1. Locate the latest released tag here  https://github.com/cockroachdb/cockroach-operator/tags
 1. Clone the tag `git clone --depth 1 --branch <tag_name> https://github.com/cockroachdb/cockroach-operator`
-1. Execute `kubectl apply -f config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml`
-1. Execute `kubectl apply -f manifests/operator.yaml`
+1. Execute `kubectl apply -f install/crds.yaml`
+1. Execute `kubectl apply -f install/operator.yaml`
 1. Check that the operator has deployed properly
 
 The examples directory contains various examples, for example you can run `kubectl apply -f examples/example.yaml`.

Makefile

@@ -18,6 +18,11 @@
 
 SHELL:=/usr/bin/env bash -O globstar
 
+# Define path where we install binaries.
+# NOTE(all): We need to use absolute paths because kubetest2 on go 1.19 does
+# not support relative paths.
+BINPATH := $(abspath bazel-bin)
+
 # values used in workspace-status.sh
 CLUSTER_NAME?=bazel-test
 COCKROACH_DATABASE_VERSION=v21.2.3
@@ -60,7 +65,7 @@ test/verify:
 test/lint:
 	bazel run //hack:verify-gofmt
 
-# NODE_VERSION refers the to version of the kindest/node image. E.g. 1.22.1
+# NODE_VERSION refers the to patch version of Kubernetes (e.g. 1.22.6)
 .PHONY: test/smoketest
 test/smoketest:
 	@bazel run //hack/smoketest -- -dir $(PWD) -version $(NODE_VERSION)
@@ -71,10 +76,9 @@ test/smoketest:
 test/e2e-short:
 	bazel test //e2e/... --test_arg=--test.short
 
-#
 # End to end testing targets
 #
-# kind: use make test/e2e/kind
+# k3d: use make test/e2e/k3d
 # gke: use make test/e2e/gke
 #
 # kubetest2 binaries from the kubernetes testing team is used
@@ -83,48 +87,49 @@ test/e2e-short:
 # Once the repo releases binaries we should vendor the tag or
 # download the built binaries.
 
-# This target is used by kubetest2-tester-exec when running a kind test
-# This target exportis the kubeconfig from kind and then runs
-# k8s:k8s -type kind which checks to see if kind is up and running.
+# This target is used by kubetest2-tester-exec when running a k3d test
+# It run k8s:k8s -type k3d which checks to see if k3d is up and running.
 # Then bazel e2e testing is run.
-# An example of calling this is using make test/e2e/testrunner-kind-upgrades
-test/e2e/testrunner-kind-%: PACKAGE=$*
-test/e2e/testrunner-kind-%:
-	bazel-bin/hack/bin/kind export kubeconfig --name $(CLUSTER_NAME)
-	bazel run //hack/k8s:k8s -- -type kind
+# An example of calling this is using make test/e2e/testrunner-k3d-upgrades
+test/e2e/testrunner-k3d-%: PACKAGE=$*
+test/e2e/testrunner-k3d-%:
+	bazel run //hack/k8s:k8s -- -type k3d
 	bazel test --stamp //e2e/$(PACKAGE)/... --test_arg=-test.v --test_arg=-test.parallel=8 --test_arg=parallel=true
 
-# Use this target to run e2e tests using a kind k8s cluster.
-# This target uses kind to start a k8s cluster  and runs the e2e tests
+# Use this target to run e2e tests using a k3d k8s cluster.
+# This target uses k3d to start a k8s cluster  and runs the e2e tests
 # against that cluster.
-# This is the main entrypoint for running the e2e tests on kind.
-# This target runs kubetest2 kind that starts a kind cluster
+#
+# This is the main entrypoint for running the e2e tests on k3d.
+# This target runs kubetest2 k3d that starts a k3d cluster
 # Then kubetest2 tester exec is run which runs the make target
-# test/e2e/testrunner-kind.
+# test/e2e/testrunner-k3d.
 # After the tests run the cluster is deleted.
 # If you need a unique cluster name override CLUSTER_NAME.
-test/e2e/kind-%: PACKAGE=$*
-test/e2e/kind-%:
-	bazel build //hack/bin/...
-	PATH=${PATH}:bazel-bin/hack/bin kubetest2 kind --cluster-name=$(CLUSTER_NAME) \
-		--up --down -v 10 --test=exec -- make test/e2e/testrunner-kind-$(PACKAGE)
+test/e2e/k3d-%: PACKAGE=$*
+test/e2e/k3d-%:
+	bazel build //hack/bin/... //e2e/kubetest2-k3d/...
+	PATH=$(BINPATH)/hack/bin:$(BINPATH)/e2e/kubetest2-k3d/kubetest2-k3d_/:${PATH} \
+		kubetest2 k3d \
+		--cluster-name=$(CLUSTER_NAME) --image rancher/k3s:v1.23.3-k3s1 --servers 3 \
+		--up --down -v 10 --test=exec -- make test/e2e/testrunner-k3d-$(PACKAGE)
 
 # This target is used by kubetest2-eks to run e2e tests.
 .PHONY: test/e2e/testrunner-eks
 test/e2e/testrunner-eks:
-	KUBECONFIG=$(TMPDIR)/$(CLUSTER_NAME)-eks.kubeconfig.yaml bazel-bin/hack/bin/kubectl create -f hack/eks-storageclass.yaml
+	KUBECONFIG=$(TMPDIR)/$(CLUSTER_NAME)-eks.kubeconfig.yaml $(BINPATH)/hack/bin/kubectl create -f hack/eks-storageclass.yaml
 	bazel test --stamp //e2e/upgrades/...  --action_env=KUBECONFIG=$(TMPDIR)/$(CLUSTER_NAME)-eks.kubeconfig.yaml
 	bazel test --stamp //e2e/create/...  --action_env=KUBECONFIG=$(TMPDIR)/$(CLUSTER_NAME)-eks.kubeconfig.yaml
 	bazel test --stamp //e2e/decommission/...  --action_env=KUBECONFIG=$(TMPDIR)/$(CLUSTER_NAME)-eks.kubeconfig.yaml
 
 # Use this target to run e2e tests with a eks cluster.
-# This target uses kind to start a eks k8s cluster  and runs the e2e tests
+# This target uses kubetest2 to start a eks k8s cluster and runs the e2e tests
 # against that cluster.
 .PHONY: test/e2e/eks
 test/e2e/eks:
 	bazel build //hack/bin/... //e2e/kubetest2-eks/...
-	PATH=${PATH}:bazel-bin/hack/bin:bazel-bin/e2e/kubetest2-eks/kubetest2-eks_/ \
-	bazel-bin/hack/bin/kubetest2 eks --cluster-name=$(CLUSTER_NAME)  --up --down -v 10 \
+	PATH=${PATH}:$(BINPATH)/hack/bin:$(BINPATH)/e2e/kubetest2-eks/kubetest2-eks_/ \
+	$(BINPATH)/hack/bin/kubetest2 eks --cluster-name=$(CLUSTER_NAME)  --up --down -v 10 \
 		--test=exec -- make test/e2e/testrunner-eks
 
 # This target is used by kubetest2-tester-exec when running a gke test
@@ -148,9 +153,9 @@ test/e2e/testrunner-gke:
 	bazel test --stamp //e2e/decommission/...
 
 # Use this target to run e2e tests with a gke cluster.
-# This target uses kind to start a gke k8s cluster  and runs the e2e tests
+# This target uses kubetest2 to start a gke k8s cluster and runs the e2e tests
 # against that cluster.
-# This is the main entrypoint for running the e2e tests on gke kind.
+# This is the main entrypoint for running the e2e tests on gke.
 # This target runs kubetest2 gke that starts a gke cluster
 # Then kubetest2 tester exec is run which runs the make target
 # test/e2e/testrunner-gke.
@@ -163,7 +168,7 @@ test/e2e/testrunner-gke:
 .PHONY: test/e2e/gke
 test/e2e/gke:
 	bazel build //hack/bin/...
-	PATH=${PATH}:bazel-bin/hack/bin bazel-bin/hack/bin/kubetest2 gke --cluster-name=$(CLUSTER_NAME) \
+	PATH=${PATH}:$(BINPATH)/hack/bin kubetest2 gke --cluster-name=$(CLUSTER_NAME) \
 		--zone=$(GCP_ZONE) --project=$(GCP_PROJECT) \
 		--version latest --up --down -v 10 --ignore-gcp-ssh-key \
 		--test=exec -- make test/e2e/testrunner-gke
@@ -175,7 +180,7 @@ test/e2e/testrunner-openshift:
 	bazel test --stamp //e2e/decommission/...  --action_env=KUBECONFIG=$(HOME)/openshift-$(CLUSTER_NAME)/auth/kubeconfig
 
 # Use this target to run e2e tests with a openshift cluster.
-# This target uses kind to start a openshift cluster and runs the e2e tests
+# This target uses kubetest2 to start a openshift cluster and runs the e2e tests
 # against that cluster. A full TLD is required to creat an openshift clutser.
 # This target runs kubetest2 openshift that starts a openshift cluster
 # Then kubetest2 tester exec is run which runs the make target
@@ -185,8 +190,8 @@ test/e2e/testrunner-openshift:
 .PHONY: test/e2e/openshift
 test/e2e/openshift:
 	bazel build //hack/bin/... //e2e/kubetest2-openshift/...
-	PATH=${PATH}:bazel-bin/hack/bin:bazel-bin/e2e/kubetest2-openshift/kubetest2-openshift_/ \
-	     bazel-bin/hack/bin/kubetest2 openshift --cluster-name=$(CLUSTER_NAME) \
+	PATH=${PATH}:$(BINPATH)/hack/bin:$(BINPATH)/e2e/kubetest2-openshift/kubetest2-openshift_/ \
+	     kubetest2 openshift --cluster-name=$(CLUSTER_NAME) \
 	     --gcp-project-id=$(GCP_PROJECT) \
 	     --gcp-region=$(GCP_REGION) \
 	     --base-domain=$(BASE_DOMAIN) \
@@ -257,7 +262,7 @@ dev/syncdeps:
 	@make dev/syncbazel
 
 .PHONY: dev/up
-dev/up:
+dev/up: dev/down
 	@hack/dev.sh up
 
 .PHONY: dev/down

apis/v1alpha1/cluster_types.go

@@ -134,9 +134,17 @@ type CrdbClusterSpec struct {
 	// proper channels in the cockroachdb. Logging configuration is available for cockroach version v21.1.0 onwards.
 	// The logging configuration is taken in format of yaml file, you can check the logging configuration here (https://www.cockroachlabs.com/docs/stable/configure-logs.html#default-logging-configuration)
 	// The default logging for cockroach version v20.x or less is stderr, logging API is ignored for older versions.
+	// NOTE: The `data` field of map must contain an entry called `logging.yaml`
+	// that contains config options.
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Cockroach Database Logging configuration config map"
 	// +optional
 	LogConfigMap string `json:"logConfigMap,omitempty"`
+	// (Optional) AutomountServiceAccountToken determines whether or not the stateful set pods should
+	// automount the service account token. This is the default behavior in Kubernetes. For backward
+	// compatibility reasons, this value defaults to `false` here.
+	// Default: false
+	// +optional
+	AutomountServiceAccountToken bool `json:"automountServiceAccountToken,omitempty"`
 }
 
 // +k8s:openapi-gen=true

config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml

@@ -665,6 +665,13 @@ spec:
                         type: array
                     type: object
                 type: object
+              automountServiceAccountToken:
+                description: '(Optional) AutomountServiceAccountToken determines whether
+                  or not the stateful set pods should automount the service account
+                  token. This is the default behavior in Kubernetes. For backward
+                  compatibility reasons, this value defaults to `false` here. Default:
+                  false'
+                type: boolean
               cache:
                 description: '(Optional) The total size for caches (`--cache` command
                   line parameter) Default: "25%"'
@@ -974,13 +981,16 @@ spec:
                     type: object
                 type: object
               logConfigMap:
-                description: (Optional) LogConfigMap define the config map which contains
-                  log configuration used to send the logs through the proper channels
-                  in the cockroachdb. Logging configuration is available for cockroach
-                  version v21.1.0 onwards. The logging configuration is taken in format
-                  of yaml file, you can check the logging configuration here (https://www.cockroachlabs.com/docs/stable/configure-logs.html#default-logging-configuration)
+                description: '(Optional) LogConfigMap define the config map which
+                  contains log configuration used to send the logs through the proper
+                  channels in the cockroachdb. Logging configuration is available
+                  for cockroach version v21.1.0 onwards. The logging configuration
+                  is taken in format of yaml file, you can check the logging configuration
+                  here (https://www.cockroachlabs.com/docs/stable/configure-logs.html#default-logging-configuration)
                   The default logging for cockroach version v20.x or less is stderr,
-                  logging API is ignored for older versions.
+                  logging API is ignored for older versions. NOTE: The `data` field
+                  of map must contain an entry called `logging.yaml` that contains
+                  config options.'
                 type: string
               maxSQLMemory:
                 description: '(Optional) The maximum in-memory storage capacity available