Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HCIDOCS-188: Scaling UPI cluster with BMO doesn't explain host networking #86660

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+170 −34
Open

HCIDOCS-188: Scaling UPI cluster with BMO doesn't explain host networking #86660

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions ..._metal/upi/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,25 @@ include::_attributes/common-attributes.adoc[]

toc::[]

After deploying a user-provisioned infrastructure cluster, you can use the Bare Metal Operator (BMO) and other metal3 components to scale bare-metal hosts in the cluster. This approach helps you to scale a user-provisioned cluster in a more automated way.
After deploying a user-provisioned infrastructure cluster, you can use the Bare Metal Operator (BMO) and other metal^3^ components to scale bare-metal hosts in the cluster. This approach helps you to scale a user-provisioned cluster in a more automated way.

include::modules/about-scaling-a-user-provisioned-installation-with-the-bare-metal-operator.adoc[leveloffset=+1]
include::modules/upi-prerequisites-for-scaling-a-upi-cluster.adoc[leveloffset=+2]
include::modules/upi-limitations-for-scaling-a-upi-cluster.adoc[leveloffset=+2]
include::modules/configuring-a-provisioning-resource-to-scale-user-provisioned-clusters.adoc[leveloffset=+1]
include::modules/upi-provisioning-new-hosts-in-a-upi-cluster.adoc[leveloffset=+1]
include::modules/upi-managing-existing-hosts-in-a-upi-cluster.adoc[leveloffset=+1]
include::modules/upi-removing-hosts-from-a-upi-cluster.adoc[leveloffset=+1]

[role="_additional-resources"]
.Additional resources

* xref:../../../installing/installing_bare_metal/ipi/ipi-install-expanding-the-cluster.adoc#preparing-the-bare-metal-node_ipi-install-expanding[Preparing the bare-metal node]

* xref:../../../installing/installing_bare_metal/ipi/ipi-install-installation-workflow.adoc#root-device-hints_ipi-install-installation-workflow[Root device hints]

* xref:../../../installing/installing_bare_metal/ipi/ipi-install-expanding-the-cluster.adoc#ipi-install-diagnosing-duplicate-mac-address_ipi-install-expanding[Diagnosing a duplicate MAC address when provisioning a new host in the cluster]

// Managing existing hosts in a user-provisioned cluster by using the BMO
include::modules/upi-managing-existing-hosts-in-a-upi-cluster.adoc[leveloffset=+1]
johnwilkins marked this conversation as resolved.
Show resolved Hide resolved

// Removing hosts from a user-provisioned cluster by using the BMO
include::modules/upi-removing-hosts-from-a-upi-cluster.adoc[leveloffset=+1]
2 changes: 1 addition & 1 deletion ...about-scaling-a-user-provisioned-installation-with-the-bare-metal-operator.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@
[id="about-scaling-a-user-provisioned-cluster-with-the-bare-metal-operator_{context}"]
= About scaling a user-provisioned cluster with the Bare Metal Operator

You can scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO) and other metal3 components. User-provisioned infrastructure installations do not feature the Machine API Operator. The Machine API Operator typically manages the lifecycle of bare-metal hosts in a cluster. However, it is possible to use the BMO and other metal3 components to scale nodes in user-provisioned clusters without requiring the Machine API Operator.
You can scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO) and other metal^3^ components. User-provisioned infrastructure installations do not feature the Machine API Operator. The Machine API Operator typically manages the lifecycle of bare-metal nodes in a cluster. However, it is possible to use the BMO and other metal^3^ components to scale nodes in user-provisioned clusters without requiring the Machine API Operator.
164 changes: 139 additions & 25 deletions modules/upi-provisioning-new-hosts-in-a-upi-cluster.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Module included in the following assemblies:
//
// * installing/installing_bare_metal/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
// * installing/installing_bare_metal/upi/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
:_mod-docs-content-type: PROCEDURE
[id="upi-provisioning-new-hosts-in-a-upi-cluster_{context}"]
= Provisioning new hosts in a user-provisioned cluster by using the BMO
Expand All @@ -9,7 +9,7 @@ You can use the Bare Metal Operator (BMO) to provision bare-metal hosts in a use

[NOTE]
====
To provision bare-metal hosts to the cluster by using the BMO, you must set the `spec.externallyProvisioned` specification in the `BareMetalHost` custom resource to `false`.
Provisioning bare-metal hosts to the cluster using the BMO sets the `spec.externallyProvisioned` specification in the `BareMetalHost` custom resource to `false` by default. Do not set the `spec.externallyProvisioned` specification to `true`, because this setting results in unexpected behavior.
====

.Prerequisites
Expand All @@ -20,45 +20,156 @@ To provision bare-metal hosts to the cluster by using the BMO, you must set the

.Procedure

. Create the `Secret` CR and the `BareMetalHost` CR.
. Create a configuration file for the bare-metal node. Depending if you use either a static configuration or a DHCP server, choose one of the following example `bmh.yaml` files and configure it to your needs by replacing values in the YAML to match your environment:

.. Save the following YAML in the `bmh.yaml` file:
.Static configuration `bmh.yaml`
+
[source,yaml]
----
---
johnwilkins marked this conversation as resolved.
Show resolved Hide resolved
apiVersion: v1
kind: Secret
metadata:
name: worker1-bmc
name: openshift-worker-<num>-network-config-secret <1>
namespace: openshift-machine-api
type: Opaque
stringData:
nmstate: | <2>
interfaces: <3>
- name: <nic1_name> <4>
type: ethernet
state: up
ipv4:
address:
- ip: <ip_address> <5>
prefix-length: 24
enabled: true
dns-resolver:
config:
server:
- <dns_ip_address> <6>
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: <next_hop_ip_address> <7>
next-hop-interface: <next_hop_nic1_name> <8>
---
apiVersion: v1
johnwilkins marked this conversation as resolved.
Show resolved Hide resolved
kind: Secret
metadata:
name: openshift-worker-<num>-bmc-secret
namespace: openshift-machine-api
type: Opaque
data:
username: <base64_of_uid>
username: <base64_of_uid> <9>
password: <base64_of_pwd>
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
name: worker1
name: openshift-worker-<num>
namespace: openshift-machine-api
spec:
online: true
bootMACAddress: <nic1_mac_address> <10>
bmc:
address: <protocol>://<bmc_url> <1>
credentialsName: "worker1-bmc"
bootMACAddress: <nic1_mac_address>
externallyProvisioned: false <2>
address: <protocol>://<bmc_url> <11>
credentialsName: openshift-worker-<num>-bmc-secret
disableCertificateVerification: false
customDeploy:
method: install_coreos
userData:
name: worker-user-data-managed
namespace: openshift-machine-api
rootDeviceHints:
deviceName: <root_device_hint> <12>
preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret
johnwilkins marked this conversation as resolved.
Show resolved Hide resolved
----
+
--
<1> Replace all instances of `<num>` with a unique compute node number for the bare-metal nodes in the `name`, `credentialsName`, and `preprovisioningNetworkDataName` fields.
<2> Add the NMState YAML syntax to configure the host interfaces. To configure the network interface for a newly created node, specify the name of the secret that has the network configuration. Follow the `nmstate` syntax to define the network configuration for your node. See "Preparing the bare-metal node" for details on configuring NMState syntax.
<3> Optional: If you have configured the network interface with `nmstate`, and you want to disable an interface, set `state: up` with the IP addresses set to `enabled: false` as shown in the following example:
+
[source,yaml]
johnwilkins marked this conversation as resolved.
Show resolved Hide resolved
----
---
apiVersion: v1
kind: Secret
metadata:
name: openshift-worker-<num>-network-config-secret
namespace: openshift-machine-api
# ...
interfaces:
- name: <nic_name>
type: ethernet
state: up
ipv4:
enabled: false
ipv6:
enabled: false
# ...
----
+
<4> Replace `<nic1_name>` with the name of the bare-metal node's first network interface controller (NIC).
<5> Replace `<ip_address>` with the IP address of the bare-metal node's NIC.
<6> Replace `<dns_ip_address>` with the IP address of the bare-metal node's DNS resolver.
<7> Replace `<next_hop_ip_address>` with the IP address of the bare-metal node's external gateway.
<8> Replace `<next_hop_nic1_name>` with the name of the bare-metal node's external gateway.
<9> Replace `<base64_of_uid>` and `<base64_of_pwd>` with the base64 string of the user name and password.
<10> Replace `<nic1_mac_address>` with the MAC address of the bare-metal node's first NIC. See the "BMC addressing" section for additional BMC configuration options.
<11> Replace `<protocol>` with the BMC protocol, such as IPMI, Redfish, or others. Replace `<bmc_url>` with the URL of the bare-metal node's baseboard management controller.
<12> Optional: Replace `<root_device_hint>` with a device path when specifying a root device hint. See "Root device hints" for additional details.
--
+
.DHCP configuration `bmh.yaml`:
+
[source,yaml]
----
---
apiVersion: v1
kind: Secret
metadata:
name: openshift-worker-<num>-bmc-secret <1>
namespace: openshift-machine-api
type: Opaque
data:
username: <base64_of_uid> <2>
password: <base64_of_pwd>
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
name: openshift-worker-<num>
namespace: openshift-machine-api
spec:
online: true
bootMACAddress: <nic1_mac_address> <3>
bmc:
address: <protocol>://<bmc_url> <4>
credentialsName: openshift-worker-<num>-bmc
disableCertificateVerification: false
customDeploy:
method: install_coreos
userData:
name: worker-user-data-managed
namespace: openshift-machine-api
rootDeviceHints:
deviceName: <root_device_hint> <5>
----
<1> You can only use bare-metal host drivers that support virtual media networking booting, for example `redfish-virtualmedia` and `idrac-virtualmedia`.
<2> You must set the `spec.externallyProvisioned` specification in the `BareMetalHost` custom resource to `false`. The default value is `false`.
+
<1> Replace `<num>` with a unique compute node number for the bare-metal nodes in the `name` and `credentialsName` fields.
johnwilkins marked this conversation as resolved.
Show resolved Hide resolved
<2> Replace `<base64_of_uid>` and `<base64_of_pwd>` with the base64 string of the user name and password.
<3> Replace `<nic1_mac_address>` with the MAC address of the bare-metal node's first NIC. See the "BMC addressing" section for additional BMC configuration options.
<4> Replace `<protocol>` with the BMC protocol, such as IPMI, Redfish, or others. Replace `<bmc_url>` with the URL of the bare-metal node's baseboard management controller.
<5> Optional: Replace `<root_device_hint>` with a device path when specifying a root device hint. See "Root device hints" for additional details.
+
[IMPORTANT]
====
If the MAC address of an existing bare-metal node matches the MAC address of the bare-metal host that you are attempting to provision, then the installation will fail. If the host enrollment, inspection, cleaning, or other steps fail, Bare Metal Operator retries the installation continuously. See "Diagnosing a duplicate MAC address when provisioning a new host in the cluster" for additional details.
====

. Create the bare-metal host object by running the following command:
. Create the bare-metal node by running the following command:
+
[source,terminal]
----
Expand All @@ -68,27 +179,31 @@ $ oc create -f bmh.yaml
.Example output
[source,terminal]
----
secret/worker1-bmc created
baremetalhost.metal3.io/worker1 created
secret/openshift-worker-<num>-network-config-secret created
johnwilkins marked this conversation as resolved.
Show resolved Hide resolved
secret/openshift-worker-<num>-bmc-secret created
baremetalhost.metal3.io/openshift-worker-<num> created
----

. Approve all certificate signing requests (CSRs).

.. Verify that the provisioning state of the host is `provisioned` by running the following command:
. Inspect the bare-metal node by running the following command:
+
[source,terminal]
----
$ oc get bmh -A
$ oc -n openshift-machine-api get bmh openshift-worker-<num>
----
+
where:
+
<num>:: Specifies the compute node number.
+
.Example output
[source,terminal]
----
NAMESPACE NAME STATE CONSUMER ONLINE ERROR AGE
openshift-machine-api controller1 externally provisioned true 5m25s
openshift-machine-api worker1 provisioned true 4m45s
NAME STATE CONSUMER ONLINE ERROR
openshift-worker-<num> provisioned true
----

. Approve all certificate signing requests (CSRs).

.. Get the list of pending CSRs by running the following command:
+
[source,terminal]
Expand Down Expand Up @@ -132,5 +247,4 @@ $ oc get nodes
NAME STATUS ROLES AGE VERSION
app1 Ready worker 47s v1.24.0+dc5a2fd
controller1 Ready master,worker 2d22h v1.24.0+dc5a2fd
----

----
21 changes: 16 additions & 5 deletions modules/upi-removing-hosts-from-a-upi-cluster.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Module included in the following assemblies:
//
// * installing/installing_bare_metal/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
// * installing/installing_bare_metal/upi/scaling-a-user-provisioned-cluster-with-the-bare-metal-operator.adoc
:_mod-docs-content-type: PROCEDURE
[id="upi-removing-hosts-from-a-upi-cluster_{context}"]
= Removing hosts from a user-provisioned cluster by using the BMO
Expand All @@ -15,7 +15,7 @@ You can use the Bare Metal Operator (BMO) to remove bare-metal hosts from a user

.Procedure

. Cordon and drain the host by running the following command:
. Cordon and drain the node by running the following command:
+
[source,terminal]
----
Expand Down Expand Up @@ -74,6 +74,18 @@ openshift-machine-api controller1 externally provisioned true
openshift-machine-api worker1 deprovisioning true 57m
----

. Delete the host by running the following command when the `BareMetalHost` state changes to `available`:
+
[source,terminal]
----
$ oc delete bmh -n openshift-machine-api <bmh_name>
----
+
[NOTE]
====
You can run this step without having to cordon and drain the node. It might take some time for the `BareMetalHost` state to change from `deprovisioning` to `available`.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous step here was step 2. Delete the customDeploy specification from the BareMetalHost CR. We can run the delete without doing the edit of the baremetalhost. If step 2 is skipped the delete command seems like it is taking forever to complete. In the background the node is changing state from deprovisioning to available to finally being deleted. Cordoning and draining the node is a graceful way to mark a node unschedulable and to remove workloads. It's an important step.

====

. Delete the node by running the following command:
+
[source,terminal]
Expand All @@ -83,7 +95,7 @@ $ oc delete node <node_name>

.Verification

* Verify the node is deleted by running the following command:
* Verify that you deleted the node by running the following command:
+
[source,terminal]
----
Expand All @@ -95,5 +107,4 @@ $ oc get nodes
----
NAME STATUS ROLES AGE VERSION
controller1 Ready master,worker 2d23h v1.24.0+dc5a2fd
----

----