Skip to content
This repository was archived by the owner on Nov 16, 2020. It is now read-only.

Commit 1ce97a6

Browse files
zimengyangzimengyang
authored
Add CI base images pipeline (#551)
* Initial commit for base image pipeline * Add basic python3, nodejs, java tests * Update ci README TODO: * add more tests
1 parent 1cd8b99 commit 1ce97a6

23 files changed

+1299
-3
lines changed

‎ci/README.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,10 @@ Dispatch Concourse server can be accessed via [ci.dispatchframework.io](https://
55

66
## Current pipelines
77

8-
Dispatch CI consists (as of today) of two pipelines:
8+
Dispatch CI consists (as of today) of the following pipelines:
99
* `dispatch-pr` - executed for every Pull Request. runs basic syntax checks, unit tests and coverage.
1010
* `e2e` - executed for open PRs with `run-e2e` label.
11+
* `base-images` - executed for open PRs on every language base images
1112

1213
New pipelines will be added as needed.
1314

@@ -17,7 +18,22 @@ E2E pipeline runs 3 jobs:
1718

1819
* `build-images` - compiles binaries, builds docker images and pushes them to a docker registry with tag based on date and commit id.
1920
* `deploy-dispatch` - Deploys dispatch o one of pre-created k8s clusters. Access to these clusters is managed using [Pool resource](https://github.com/concourse/pool-resource).
20-
* `run-tests` - Executes all tests defined in `e2e/tests`. Tests are written using [Bats](https://github.com/sstephenson/bats).
21+
* `run-tests` - Executes all tests defined in `e2e/tests`. Tests are written using [Bats](https://github.com/sstephenson/bats).
22+
23+
### Base Images pipeline
24+
25+
`base-images` has several jobs:
26+
* `create-gke-cluster` and `delete-gke-cluster`: create and delete gke cluster. Both jobs are triggered manually.
27+
* `install-dispatch`: deploys dispatch running instance, triggered by every Dispatch release and will update the existing dispatch instance automatically.
28+
* `uninstall-dispatch`: uninstalls dispatch, triggered manually.
29+
30+
For each specific language, will have following jobs:
31+
* `build-<LANGUAGE>-base-image`: builds base image based on language pr.
32+
* `test-<LANGUAGE>-base-image`: runs tests on the base image from above pr. And report status back to github pull request.
33+
34+
To add tests for a base image:
35+
* `dispatch/ci/base-images/tests/<LANGUAGE>/task.yml`: concourse ci job yml runs tests. Customize this file to add more language-specific tests preparation.
36+
* `dispatch/ci/base-images/tests/<LANGUAGE>/tests.bats`: bats file contains all tests functions. Tests inside this file will be executed by default.
2137

2238

2339
## Introduction to Concourse
@@ -43,7 +59,7 @@ dispatch-pr no yes
4359
```
4460
$ fly -t dispatch jobs -p dispatch-pr
4561
name paused status next
46-
dispatch-basic no succeeded n/a
62+
dispatch-basic no succeeded n/a
4763
```
4864

4965
Pipelines in Concourse revolve around three main concepts: *tasks*, *jobs* and *resources* (read more about them [here](http://concourse.ci/concepts.html)).

‎ci/base-images/base-image-tag.yml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
platform: linux
3+
4+
image_resource:
5+
type: docker-image
6+
source:
7+
repository: vmware/dispatch-golang-ci
8+
tag: "1.10"
9+
10+
inputs:
11+
- name: base-image
12+
13+
outputs:
14+
- name: build-context
15+
16+
run:
17+
path: /bin/bash
18+
args:
19+
- -c
20+
- |
21+
set -e -x -u
22+
23+
cp -r base-image/* build-context/
24+
25+
pushd base-image
26+
export IMAGE_TAG="$(date +'%y%m%d%H%M%S')-$(git rev-parse --short HEAD)"
27+
popd
28+
29+
echo ${IMAGE_TAG} > build-context/tag
30+
echo "tag=${IMAGE_TAG}" > build-context/keyval.properties

‎ci/base-images/config-dispatch-env.sh

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
3+
cp dispatch-release/dispatch-linux /usr/local/bin/dispatch
4+
chmod +x /usr/local/bin/dispatch
5+
6+
export INSTALL_DISPATCH=0
7+
export CI=true
8+
export TERM=linux
9+
export DISPATCH_ORGANIZATION=ci-org
10+
export DISPATCH_SERVICE_ACCOUNT="ci-org/ci-user"
11+
export DISPATCH_JWT_PRIVATE_KEY=$(pwd)/ci-keys/ci-user.key
12+
13+
mkdir -p ~/.dispatch
14+
15+
export LOADBALANCER_IP=$(kubectl get svc/ingress-nginx-ingress-controller -n kube-system -o json | jq -r '.status.loadBalancer.ingress[0].ip')
16+
export API_GATEWAY_IP=$(kubectl get svc/api-gateway-kongproxy -n kong -o json | jq -r '.status.loadBalancer.ingress[0].ip')
17+
cp dispatch/ci/base-images/configs/dispatch-config.json ~/.dispatch/config.json
18+
sed -i "s/LOADBALANCER_IP/$LOADBALANCER_IP/g" ~/.dispatch/config.json
19+
sed -i "s/CURRENT_CONTEXT/$(echo $LOADBALANCER_IP | tr '.' '-')/g" ~/.dispatch/config.json
20+
21+
export API_GATEWAY_HTTPS_HOST="https://${API_GATEWAY_IP}:443"
22+
export API_GATEWAY_HTTP_HOST="http://${API_GATEWAY_IP}:80"

‎ci/base-images/config-gke-env.sh

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/bin/bash
2+
3+
# config gke environment: account, project and zone
4+
set -e +x -u
5+
6+
: ${GKE_ZONE:="us-west1-c"}
7+
8+
echo "Setting up GKE key"
9+
echo ${GKE_KEY} | base64 --decode --ignore-garbage > ${HOME}/gcloud-service-key.json
10+
echo "Activating GKE account"
11+
gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json >/dev/null 2>&1
12+
echo "Setting GKE project"
13+
gcloud config set project ${GKE_PROJECT_ID} >/dev/null 2>&1
14+
echo "Setting default GKE compute zone"
15+
gcloud config set compute/zone ${GKE_ZONE} >/dev/null 2>&1
16+
17+
set -x

‎ci/base-images/config-k8s-env.sh

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/bin/bash
2+
3+
# config k8s cluster environment: get cluster credentials
4+
set -e +x -u
5+
6+
export cluster_name=$(cat k8s-cluster/keyval.properties | grep "cluster_name" | cut -d'=' -f2)
7+
gcloud container clusters get-credentials ${cluster_name}
8+
9+
helm init -c
10+
helm repo remove local
11+
12+
set -x

‎ci/base-images/configs/dispatch-config.json

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"current": "CURRENT_CONTEXT",
3+
"contexts": {
4+
"CURRENT_CONTEXT": {
5+
"host": "LOADBALANCER_IP",
6+
"port": 443,
7+
"organization": "",
8+
"cookie": "",
9+
"insecure": true,
10+
"json": false,
11+
"namespace": "dispatch"
12+
}
13+
}
14+
}

‎ci/base-images/configs/dispatch-install.yml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ingress:
2+
serviceType: LoadBalancer
3+
apiGateway:
4+
serviceType: LoadBalancer
5+
host: 10.0.0.1
6+
dispatch:
7+
host: 10.0.0.1
8+
port: 443
9+
faas: FAAS
10+
eventTransport: EVENT_TRANSPORT
11+
image:
12+
host: DOCKER_REGISTRY_HOST
13+
tag: IMAGE_TAG
14+
skipAuth: false

‎ci/base-images/gke-cluster-create.yml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
platform: linux
3+
4+
image_resource:
5+
type: docker-image
6+
source:
7+
repository: vmware/dispatch-k8s-ci
8+
tag: v0.0.12
9+
10+
params:
11+
GKE_KEY:
12+
GKE_PROJECT_ID:
13+
K8S_VERSION: 1.9.6-gke.1
14+
GKE_ZONE: us-west1-c
15+
CLUSTER_NAME_SUFFIX: job
16+
17+
inputs:
18+
- name: dispatch
19+
20+
outputs:
21+
- name: k8s-cluster
22+
23+
run:
24+
path: /bin/bash
25+
args:
26+
- -c
27+
- |
28+
set -e -x -u
29+
30+
source dispatch/ci/base-images/config-gke-env.sh
31+
32+
export cluster_name=dispatch-ci-${CLUSTER_NAME_SUFFIX}-$(date +%s)-${RANDOM}
33+
echo "cluster_name=${cluster_name}" > k8s-cluster/name
34+
35+
gcloud container clusters create -m n1-standard-2 --cluster-version ${K8S_VERSION} ${cluster_name}
36+
gcloud container clusters get-credentials ${cluster_name}
37+
kubectl create clusterrolebinding tiller-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
38+
39+
helm init --wait

‎ci/base-images/gke-cluster-delete.yml

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
---
2+
platform: linux
3+
4+
image_resource:
5+
type: docker-image
6+
source:
7+
repository: vmware/dispatch-k8s-ci
8+
tag: v0.0.12
9+
10+
params:
11+
GKE_KEY:
12+
GKE_PROJECT_ID:
13+
MAX_ATTEMPTS: 18
14+
TIMEOUT: 10
15+
16+
inputs:
17+
- name: dispatch
18+
- name: k8s-cluster
19+
20+
run:
21+
path: /bin/bash
22+
args:
23+
- -c
24+
- |
25+
set -e -x -u
26+
27+
source dispatch/ci/base-images/config-gke-env.sh
28+
source dispatch/ci/base-images/config-k8s-env.sh
29+
30+
set +x
31+
32+
# Delete load balancers
33+
while read -r line && [[ -n ${line} ]]; do
34+
namespace=$(echo $line | awk '{print $1;}')
35+
svc_name=$(echo $line | awk '{print $2;}')
36+
attempts=0
37+
38+
kubectl delete -n ${namespace} svc ${svc_name}
39+
40+
# Wait until LB is deleted asynchronously before deleting cluster (Workaround for https://github.com/kubernetes/kubernetes/issues/51701)
41+
until (kubectl get -n ${namespace} events | grep ${svc_name} | grep -q DeletedLoadBalancer); do
42+
sleep ${TIMEOUT}
43+
attempts=$((attempts+1))
44+
45+
# Prevent possible infinite loop
46+
if [[ ${attempts} -ge ${MAX_ATTEMPTS} ]]; then
47+
break
48+
fi
49+
done
50+
done <<< $(kubectl get svc --all-namespaces | grep LoadBalancer)
51+
52+
# Persistant storage info
53+
export PERSISTANT_STORAGE=$(kubectl -n dispatch get pv -o json | jq -r '.items[0].spec.gcePersistentDisk.pdName // empty')
54+
55+
# Delete cluster
56+
gcloud container clusters delete --quiet ${cluster_name}
57+
# Delete persistent storage
58+
if [[ -n ${PERSISTANT_STORAGE} ]]; then
59+
gcloud compute disks delete --quiet ${PERSISTANT_STORAGE}
60+
fi
61+
62+
set -x

‎ci/base-images/install-dispatch.yml

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
platform: linux
3+
4+
image_resource:
5+
type: docker-image
6+
source:
7+
repository: vmware/dispatch-k8s-ci
8+
tag: v0.0.12
9+
10+
params:
11+
GKE_KEY:
12+
GKE_PROJECT_ID:
13+
DOCKER_REGISTRY_HOST:
14+
FAAS: openfaas
15+
EVENT_TRANSPORT: kafka
16+
17+
inputs:
18+
- name: dispatch
19+
- name: dispatch-release
20+
- name: k8s-cluster
21+
22+
23+
outputs:
24+
- name: ci-keys
25+
26+
run:
27+
path: /bin/bash
28+
args:
29+
- -c
30+
- |
31+
set -e -x -u
32+
33+
source dispatch/ci/base-images/config-gke-env.sh
34+
source dispatch/ci/base-images/config-k8s-env.sh
35+
36+
cp dispatch-release/dispatch-linux /usr/local/bin/dispatch
37+
chmod +x /usr/local/bin/dispatch
38+
39+
export IMAGE_TAG="v$(cat dispatch-release/version)"
40+
41+
set +x
42+
cp dispatch/ci/base-images/configs/dispatch-install.yml install.yml
43+
set -x
44+
45+
# setup install config file
46+
sed -i "s/IMAGE_TAG/${IMAGE_TAG}/g" install.yml
47+
sed -i "s#DOCKER_REGISTRY_HOST#${DOCKER_REGISTRY_HOST}#g" install.yml
48+
sed -i "s/FAAS/${FAAS}/g" install.yml
49+
sed -i "s/EVENT_TRANSPORT/${EVENT_TRANSPORT}/g" install.yml
50+
51+
cp -r dispatch/charts ./charts
52+
53+
dispatch install --file install.yml --charts-dir ./charts
54+
55+
# setup Dispatch config
56+
mkdir -p ~/.dispatch
57+
58+
export LOADBALANCER_IP=$(kubectl get svc/ingress-nginx-ingress-controller -n kube-system -o json | jq -r '.status.loadBalancer.ingress[0].ip')
59+
cp dispatch/ci/base-images/configs/dispatch-config.json ~/.dispatch/config.json
60+
sed -i "s/LOADBALANCER_IP/$LOADBALANCER_IP/g" ~/.dispatch/config.json
61+
sed -i "s/CURRENT_CONTEXT/$(echo $LOADBALANCER_IP | tr '.' '-')/g" ~/.dispatch/config.json
62+
63+
# Bootstrap Dispatch with default org, service-accounts
64+
dispatch manage bootstrap --bootstrap-org ci-org
65+
66+
if (dispatch iam get serviceaccount | grep -q ci-user)
67+
then
68+
# delete servcie account first
69+
dispatch iam delete serviceaccount ci-user
70+
fi
71+
72+
if (dispatch iam get policy | grep -q ci-user-admin-policy)
73+
then
74+
# delete svc acct policy
75+
dispatch iam delete policy ci-user-admin-policy
76+
fi
77+
78+
# generate svc-acct keys
79+
openssl genrsa -out ci-keys/ci-user.key 4096
80+
openssl rsa -in ci-keys/ci-user.key -pubout -outform PEM -out ci-keys/ci-user.key.pub
81+
82+
# Create ci-user service account
83+
dispatch iam create serviceaccount \
84+
ci-user \
85+
--public-key ci-keys/ci-user.key.pub
86+
87+
# Create super-admin policy for the service account
88+
dispatch iam create policy \
89+
ci-user-admin-policy \
90+
--subject ci-user --action "*" --resource "*" \
91+
--global

0 commit comments

Comments
 (0)