Skip to content

Commit 47e1c0a

Browse files
anokfireballanokfireball
committed
🛂 switch from L2 to BGP for metallb
Closes #840.
1 parent af1282b commit 47e1c0a

File tree

12 files changed

+119
-42
lines changed

12 files changed

+119
-42
lines changed

‎README.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ At the highest possible level, this repo and HaC workflow consists of three part
5454
| [Ansible](https://ansible.com/) | OS Configuration | |
5555
| [kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/) | k8s _Distribution_ / Install Mechanism | stacked HA controlplanes |
5656
| [containerd](https://containerd.io/) | OCI Runtime | |
57-
| [Calico](https://www.tigera.io/tigera-products/calico/) | CNI | dual-stack nodes and services |
57+
| [Calico](https://www.tigera.io/tigera-products/calico/) | CNI | used in BGP mode |
5858
| [kube-vip](https://kube-vip.io/) | Virtual IP for controlplane Nodes | used in L2/ARP mode |
5959
| [Flux2](https://fluxcd.io) | GitOps Automation inside the Cluster | |
6060
| [SOPS](https://getsops.io/) | Secrets Management | [age](https://age-encryption.org/) rather than pgp, but not any more user-friendly |
@@ -74,7 +74,7 @@ At the highest possible level, this repo and HaC workflow consists of three part
7474
<td><img width="32" src="https://raw.githubusercontent.com/metallb/metallb/refs/heads/main/website/static/images/logo/metallb-blue.svg"></td>
7575
<td><a href="https://metallb.io/">metallb</a></td>
7676
<td>Cloud-Native Service LoadBalancer</td>
77-
<td>used in L2/ARP mode, so only VIP rather than true LB</td>
77+
<td>used in BGP mode</td>
7878
</tr>
7979
<tr>
8080
<td><img width="32" src="https://raw.githubusercontent.com/kubernetes-sigs/external-dns/refs/heads/master/docs/img/external-dns.png"></td>
@@ -363,12 +363,12 @@ At the highest possible level, this repo and HaC workflow consists of three part
363363

364364
While the ultimate goal is to have as self-sufficient of a setup as possible, some external services are still required for proper operation.
365365

366-
| Service | Purpose | Notes |
367-
| -------------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------ |
368-
| [GitHub](https://github.com/) | Git Repository Hosting, GitOps Source | |
369-
| [INWX](https://www.inwx.de/) | Domain Registrar | |
370-
| [Cloudflare](https://www.cloudflare.com/) | Public DNS Auth Hosting | |
371-
| [netcup](https://www.netcup.de/) | Public Reverse-Proxy for Relevant Services | not _yet_ managed here since the number of public services is tiny |
372-
| [BackBlaze](https://www.backblaze.com/) | Cloud Storage for Backups | the "3" in 3-2-1 for the really important data |
373-
| [TailScale](https://tailscale.com/) | Overlay VPN | used for split-horizon and a direct connection back home |
374-
| VPN Provider | VPN Gateway | different external IP for all the Linux ISOs |
366+
| Service | Purpose | Notes |
367+
| ----------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------ |
368+
| [GitHub](https://github.com/) | Git Repository Hosting, GitOps Source | |
369+
| [INWX](https://www.inwx.de/) | Domain Registrar | |
370+
| [Cloudflare](https://www.cloudflare.com/) | Public DNS Auth Hosting | |
371+
| [netcup](https://www.netcup.de/) | Public Reverse-Proxy for Relevant Services | not _yet_ managed here since the number of public services is tiny |
372+
| [BackBlaze](https://www.backblaze.com/) | Cloud Storage for Backups | the "3" in 3-2-1 for the really important data |
373+
| [TailScale](https://tailscale.com/) | Overlay VPN | used for split-horizon and a direct connection back home |
374+
| VPN Provider | VPN Gateway | different external IP for all the Linux ISOs |

‎ansible/cluster.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,5 @@
88
- { role: os_base }
99
- { role: os_power }
1010
- { role: k8s_cluster }
11-
- { role: k8s_flux }
1211
- { role: k8s_longhorn }
12+
- { role: k8s_flux }

‎ansible/roles/k8s_cluster/tasks/main.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,6 @@
7878
- when: "'controlplane' in group_names"
7979
block:
8080
- import_tasks: tasks/kube-vip.yaml
81-
- import_tasks: tasks/oidc.yaml
8281

8382
- when: not k8s_upgrade
8483
block:
@@ -87,6 +86,10 @@
8786

8887
- import_tasks: join.yaml
8988

89+
- when: "'controlplane' in group_names"
90+
block:
91+
- import_tasks: tasks/oidc.yaml
92+
9093
- when: k8s_upgrade
9194
import_tasks: upgrade.yaml
9295

‎ansible/roles/k8s_cluster/tasks/tasks/calico.yaml‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,37 @@
11
---
2+
# CRD bundles too large to use kubectl apply
23
- name: create calico config directory
34
file:
45
path: /etc/calico
56
state: directory
67

8+
- name: download calico operator CRDs
9+
get_url:
10+
url: |
11+
https://raw.githubusercontent.com/projectcalico/calico/{{ calico_version }}/manifests/operator-crds.yaml
12+
dest: /etc/calico/operator-crds.yaml
13+
register: calico_crds_manifest
14+
15+
- name: check if calico operator CRDs already exist
16+
shell: |
17+
set -o pipefail
18+
# figure out if a CRD with name "apiservers.operator.tigera.io" exists
19+
kubectl get crds -o json | jq '.items[] | select(.metadata.name == "apiservers.operator.tigera.io")' | wc -l
20+
args:
21+
executable: /bin/bash
22+
environment:
23+
KUBECONFIG: /etc/kubernetes/admin.conf
24+
register: calico_crds_exists
25+
ignore_errors: true
26+
changed_when: calico_crds_exists.stdout == '0'
27+
28+
- name: apply calico operator CRDs
29+
command: |
30+
kubectl {{ 'create' if calico_crds_exists.stdout == '0' else 'replace' }} -f /etc/calico/operator-crds.yaml
31+
environment:
32+
KUBECONFIG: /etc/kubernetes/admin.conf
33+
changed_when: calico_crds_exists.stdout == '0' or calico_crds_manifest.changed
34+
735
- name: download calico operator manifest
836
get_url:
937
url: |
@@ -24,7 +52,6 @@
2452
changed_when: calico_operator_exists.stdout == '0'
2553

2654
- name: apply calico operator manifest
27-
# CRD bundle too large to use kubectl apply
2855
command: |
2956
kubectl {{ 'create' if calico_operator_exists.stdout == '0' else 'replace' }} -f /etc/calico/tigera-operator.yaml
3057
environment:
@@ -55,3 +82,16 @@
5582
environment:
5683
KUBECONFIG: /etc/kubernetes/admin.conf
5784
changed_when: calico_custom_resource_exists.stdout == '0' or calico_custom_resource_manifest.changed
85+
86+
- name: copy BGP configuration
87+
template:
88+
src: bgp-config.yaml.j2
89+
dest: /etc/calico/bgp-config.yaml
90+
register: calico_bgp_config_manifest
91+
92+
- name: apply calico BGP config manifest
93+
command: |
94+
kubectl apply -f /etc/calico/bgp-config.yaml
95+
environment:
96+
KUBECONFIG: /etc/kubernetes/admin.conf
97+
changed_when: calico_bgp_config_manifest.changed

‎ansible/roles/k8s_cluster/tasks/tasks/containerd.yaml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,12 +35,12 @@
3535

3636
- name: modify containerd config
3737
shell: >
38-
set -o pipefail &&
39-
containerd config default
40-
| tomlq '.plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.SystemdCgroup = true' -t
41-
| tomlq '.plugins."io.containerd.grpc.v1.cri".registry.config_path = "/etc/containerd/certs.d"' -t
42-
| tomlq '.plugins."io.containerd.grpc.v1.cri".containerd.discard_unpacked_layers = false' -t
43-
> /etc/containerd/config.toml
38+
set -o pipefail &&
39+
containerd config default
40+
| tomlq '.plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.SystemdCgroup = true' -t
41+
| tomlq '.plugins."io.containerd.grpc.v1.cri".registry.config_path = "/etc/containerd/certs.d"' -t
42+
| tomlq '.plugins."io.containerd.grpc.v1.cri".containerd.discard_unpacked_layers = false' -t
43+
> /etc/containerd/config.toml
4444
args:
4545
executable: /bin/bash
4646
changed_when: false

‎ansible/roles/k8s_cluster/tasks/tasks/kubelet.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@
2323
apt:
2424
pkg:
2525
- "cri-tools={{ k8s_version_stripped }}.*"
26-
allow_downgrade: true
2726
allow_change_held_packages: true
2827
state: present
2928
notify:

‎ansible/roles/k8s_cluster/tasks/tasks/upgrade-node.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,4 @@
1414
command: "kubectl uncordon {{ node_name }}"
1515
environment:
1616
KUBECONFIG: /etc/kubernetes/admin.conf
17-
delegate_to: "{{ delegate }}"
17+
delegate_to: "{{ delegate }}"

‎ansible/roles/k8s_cluster/templates/bgp-config.yaml.j2‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
apiVersion: projectcalico.org/v3
2+
kind: BGPFilter
3+
metadata:
4+
name: deny-pods
5+
spec:
6+
exportV4:
7+
- action: Reject
8+
matchOperator: In
9+
cidr: {{ pod_network_cidr_v4 }}
10+
---
11+
apiVersion: projectcalico.org/v3
12+
kind: BGPPeer
13+
metadata:
14+
name: default
15+
spec:
16+
peerIP: {{ bgp_router_ip }}
17+
asNumber: {{ bgp_router_as }}
18+
keepOriginalNextHop: true
19+
filters:
20+
- deny-pods
21+
---
22+
apiVersion: projectcalico.org/v3
23+
kind: BGPConfiguration
24+
metadata:
25+
name: default
26+
spec:
27+
asNumber: {{ bgp_cluster_as }}
28+
nodeToNodeMeshEnabled: true
29+
serviceClusterIPs:
30+
- cidr: {{ service_cidr_v4 }}
31+
serviceLoadBalancerIPs:
32+
- cidr: {{ loadbalancer_cidr_v4 }}

‎ansible/roles/k8s_cluster/templates/custom-resources.yaml.j2‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,11 @@
11
# This section includes base Calico installation configuration.
2-
# For more information, see: https://docs.tigera.io/calico/3.29/reference/installation/api#operator.tigera.io/v1.Installation
2+
# For more information, see: https://docs.tigera.io/calico/3.30/reference/installation/api#operator.tigera.io/v1.Installation
33
apiVersion: operator.tigera.io/v1
44
kind: Installation
55
metadata:
66
name: default
77
spec:
88
calicoNetwork:
9-
bgp: Disabled
109
ipPools:
1110
- name: default-ipv4-ippool
1211
blockSize: {{ calico_node_subnet_prefix_length_v4 }}
@@ -20,13 +19,23 @@ spec:
2019
encapsulation: VXLANCrossSubnet
2120
natOutgoing: Enabled
2221
nodeSelector: all()
23-
2422
---
25-
2623
# This section configures the Calico API server.
27-
# For more information, see: https://docs.tigera.io/calico/3.29/reference/installation/api#operator.tigera.io/v1.APIServer
24+
# For more information, see: https://docs.tigera.io/calico/3.30/reference/installation/api#operator.tigera.io/v1.APIServer
2825
apiVersion: operator.tigera.io/v1
2926
kind: APIServer
3027
metadata:
3128
name: default
3229
spec: {}
30+
---
31+
# Configures the Calico Goldmane flow aggregator.
32+
apiVersion: operator.tigera.io/v1
33+
kind: Goldmane
34+
metadata:
35+
name: default
36+
---
37+
# Configures the Calico Whisker observability UI.
38+
apiVersion: operator.tigera.io/v1
39+
kind: Whisker
40+
metadata:
41+
name: default

‎ansible/roles/k8s_cluster/vars/main.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ pod_network_cidr_v4: 172.32.0.0/16
1111
pod_network_cidr_v6: fd08:172:32::/56
1212
service_cidr_v4: 172.16.0.0/24
1313
service_cidr_v6: fd08:172:16::/112
14+
loadbalancer_cidr_v4: 192.168.2.0/24
1415

1516
kube_vip_version: v0.9.1
1617
vip: 192.168.1.210
@@ -22,5 +23,9 @@ calico_node_subnet_prefix_length_v4: 24
2223
# range: [116:128]
2324
calico_node_subnet_prefix_length_v6: 120
2425

26+
bgp_router_ip: 192.168.1.1
27+
bgp_router_as: 64512
28+
bgp_cluster_as: 64513
29+
2530
reset_cluster: false
2631
reset_confirmation: ""

0 commit comments

Comments
 (0)