Skip to content

advanced-usage.md

Latest commit

 

History

History
270 lines (215 loc) · 11.8 KB

advanced-usage.md

File metadata and controls

270 lines (215 loc) · 11.8 KB
Raw

Advanced Usage

Using External Server

This enables k8s-netperf to use the IP address provided via the --serverIP option as server address and the client sends requests to this IP address. This allows dataplane testing between ocp internal client pod and external server.

Note: User has to create a server with the provided IP address and run the intented k8s-netperf driver (i.e uperf, iperf or netperf). User has to enable respective ports on this server to allow the traffic from the client

Once the external server is ready to accept the traffic, users can orhestrate k8s-netperf by running:

k8s-netperf --serverIP=44.243.95.221

Running with VMs

Running k8s-netperf against Virtual Machines (OpenShift CNV) requires

  • OpenShift CNV must be deployed and users should be able to define VMIs
  • SSH keys to be present in the home directory (~/.ssh/id_rsa.pub)
  • OpenShift Routes - k8s-netperf uses this to reach the VMs (k8s-netperf will create the route for the user, but we need Routes)

If the two above are in place, users can orhestrate k8s-netperf to launch VMs by running:

k8s-netperf --vm

Workload annotations and labels

Use --annotation KEY=VALUE repeatedly to add the same annotations to every benchmark pod template or VMI. k8s-netperf rejects empty, malformed, duplicate, and tool-managed annotation keys so it can preserve its Istio and network configuration.

Use --label KEY=VALUE repeatedly to add labels to every benchmark pod template or VMI. Label keys and values must be valid Kubernetes labels. Empty, malformed, or duplicate labels are rejected before resources are created. The app and role labels are managed by k8s-netperf for workload and service selection, so they cannot be supplied with --label.

k8s-netperf --label example.com/team=networking --label environment=staging \
  --annotation example.com/policy-test=enabled

Using User Defined Network - UDN (only on OCP 4.18 and above)

To run k8s-netperf using a UDN primary network for the test instead of the default network of OVN-k:

For a layer3 UDN:

$ k8s-netperf --udnl3

For a layer2 UDN:

$ k8s-netperf --udnl2

It works also with VMs:

$ k8s-netperf --udnl2 --vm --udnPluginBinding=l2bridge

Warning! Support of k8s Services with UDN is not fully supported yet, you may faced inconsistent results when using a service in your tests.

Cluster User Defined Network - C-UDN

k8s-netperf is able to deploy a C-UDN and then attach network interfaces to the pods (or VMs) that allow use the C-UDN as a secondary network. The subnet of the C-UDN is 20.0.0.0/16 and the driver will use this secondary interface for the test. C-UDN is automatically cleanup at the end of the execution.

# For a layer2 C-UDN:
$ k8s-netperf --cudn layer2

# For a layer3 C-UDN:
$ k8s-netperf --cudn layer3

# Both of the setup works with VMs too:
$ k8s-netperf --cudn layer2 --vm
# or
$ k8s-netperf --cudn layer3 --vm

SR-IOV Network Testing

To run k8s-netperf over SR-IOV Virtual Functions, the SR-IOV Network Operator must be installed on the cluster. k8s-netperf will automatically create the required SriovNetworkNodePolicy and SriovNetwork CRs, and clean them up after the test.

Pod SR-IOV

Pass the PF (Physical Function) interface name to the --sriov flag:

k8s-netperf --sriov ens2f0np0

On a single-node cluster:

k8s-netperf --sriov ens2f0np0 --local

By default, the SR-IOV policy targets nodes with the node-role.kubernetes.io/worker label. If SR-IOV NICs are only available on nodes with a different role label, use --sriov-node-selector:

k8s-netperf --sriov ens2f0np0 --sriov-node-selector cnf-worker

VM SR-IOV

SR-IOV is also supported with KubeVirt VMs. When --vm is used with --sriov, VFs are created with deviceType: vfio-pci and passed through to the guest via PCI passthrough. The IP address assigned by whereabouts to the virt-launcher pod is configured inside the guest VM using nmcli over virtctl SSH.

k8s-netperf --sriov ens1f0 --vm --use-virtctl --pod=false

Requirements for VM SR-IOV:

  • OpenShift CNV (KubeVirt) must be deployed
  • The PF must use a driver supported by the VM containerdisk (e.g. Intel ice/iavf drivers are included in Fedora 39; Mellanox mlx5_core is not)
  • SSH keys must be present in ~/.ssh/id_rsa.pub
  • virtctl is recommended (--use-virtctl) for SSH access into the guest VMs

Note: Pod and VM SR-IOV tests cannot run in the same invocation because pods require deviceType: netdevice while VMs require deviceType: vfio-pci. Run them separately with --pod=false for VM-only or without --vm for pod-only.

Note: --sriov is mutually exclusive with --bridge, --macvlan, --ib-write-bw, --hostNet and UDN flags (--udnl2, --udnl3, --cudn).

MACVLAN Network Testing

To run k8s-netperf over a MACVLAN interface, pass the master (host) interface name to the --macvlan flag. k8s-netperf will automatically create a MACVLAN NetworkAttachmentDefinition with whereabouts IPAM in the netperf namespace and clean it up after the test.

Prerequisites:

  • Multus CNI must be installed on the cluster
  • The macvlan CNI plugin must be available on the nodes (/opt/cni/bin/macvlan)
  • The whereabouts IPAM plugin must be installed
k8s-netperf --macvlan eth0

On a single-node cluster:

k8s-netperf --macvlan eth0 --local

Note: --macvlan is mutually exclusive with --bridge, --sriov, --ib-write-bw, --hostNet and UDN flags (--udnl2, --udnl3, --cudn).

Using a Linux Bridge Interface

When using --bridge, a NetworkAttachmentDefinition defining a bridge interface is attached to the VMs and is used for the test. It requires the name of the bridge as it is defined in the NetworkNodeConfigurationPolicy, NMstate operator is required.

For example:

apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br0-eth1
spec:
  desiredState:
    interfaces:
      - name: br0
        description: Linux bridge with eno2 as a port
        type: linux-bridge
        state: up
        ipv4:
          dhcp: true
          enabled: true
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: eno2

Then you can launch a test using the bridge interface:

./bin/amd64/k8s-netperf --vm --bridge br0

By default, it will read the bridgeNetwork.json file from the git repository. If the default IP addresses (10.10.10.12/24 and 10.10.10.14/24) are not available for your setup, it is possible to change it by passing a JSON file as a parameter with --bridgeNetwork, like follow:

k8s-netperf --vm --bridge br0 --bridgeNetwork /path/to/my/bridgeConfig.json

Using OVN Localnet

When using --localnet, k8s-netperf creates a Localnet ClusterUserDefinedNetwork (C-UDN) and attaches a secondary interface to the VMs for the test. The value passed to --localnet must match the OVN-K external / physical network name configured in a NodeNetworkConfigurationPolicy bridge mapping. NMState operator is required. Localnet is VM-only (--vm --pod=false).

For example, map the localnet name physnet to the existing OVS bridge br-ex:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: mapping-localnet
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ''
  desiredState:
    ovn:
      bridge-mappings:
      - localnet: physnet
        bridge: br-ex
        state: present

Alternatively, create a dedicated OVS bridge on a NIC (e.g. ens7f0) and map a localnet to it:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ens7f0-ovs-underlay
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  desiredState:
    interfaces:
      - name: br-ens7f0
        description: "OVS Bridge dedicated to ens7f0 for CUDN Localnets"
        type: ovs-bridge
        state: up
        bridge:
          allow-extra-patch-ports: true
          port:
            - name: ens7f0
    ovn:
      bridge-mappings:
        - localnet: physnet-ens7f0
          bridge: br-ens7f0
          state: present

Then you can launch a test using the localnet name from the bridge mapping:

./bin/amd64/k8s-netperf --vm --pod=false --localnet physnet
# or, with the dedicated bridge example above:
./bin/amd64/k8s-netperf --vm --pod=false --localnet physnet-ens7f0

By default, it will read the localnetNetwork.json file from the git repository. If the default IP addresses (192.168.200.10/24 and 192.168.200.11/24) are not available for your setup, it is possible to change it by passing a JSON file as a parameter with --localnet-config, like follow:

k8s-netperf --vm --pod=false --localnet physnet --localnet-config /path/to/my/localnetConfig.json

Note: --localnet is mutually exclusive with --bridge, --sriov, --macvlan, --ib-write-bw, --hostNet and UDN flags (--udnl2, --udnl3, --cudn).

Privileged pods

If your use case requires running pods with privileged security context, use the --privileged flag:

$ k8s-netperf --privileged

Guaranteed QoS pods

By default benchmark pods set no CPU or memory requests or limits, so they run in the BestEffort QoS class and share CPUs with everything else on the node. --cpu and --memory set the same value as both request and limit on every benchmark pod container (clients and servers, pod network and host network), which gives the pods the Guaranteed QoS class:

$ k8s-netperf --cpu 2 --memory 2Gi
  • Both flags are required together, and both must be positive Kubernetes quantities (--cpu in whole millicores, --memory in whole bytes). Invalid values fail the run before anything is created.
  • The values apply per container. A server pod runs one container per requested driver server (netserver, iperf3, uperf, ib_write_bw, plus the uperf histogram server for TCP_STREAM_LAT), so it requests the value times the number of server containers. Pods that do not fit on a node stay Pending.
  • On nodes with the static CPU Manager policy, Guaranteed containers with a whole number of CPUs are eligible for exclusive CPUs. A fractional --cpu such as 500m still makes the pods Guaranteed, with a warning that their CPUs are shared.
  • After the pods are ready, k8s-netperf checks that each benchmark container requests and is limited to exactly these values, and that each benchmark pod reports the Guaranteed QoS class. It fails the run otherwise, for example when a mutating webhook changes the containers' resources or injects a sidecar without equal requests and limits.
  • With --clean=false, an existing workload is reused only when its rollout has finished and both its template and all of its pods have exactly these CPU and memory settings; otherwise the run stops and asks for a rerun with --clean, which recreates it. Without the flags, the template must not set CPU or memory, while defaults that admission (such as a LimitRange) added to the pods are accepted, as before.
  • SR-IOV VF requests from --sriov are kept alongside CPU and memory. The flags do not affect VMs, which use --sockets, --cores and --threads.

RoCEv2 testing

Using the ib_write_bw driver, your hardware should include RDMA devices:

$ k8s-netperf --ib-write-bw nic:gid --privileged --hostNet

RoCEv2 local testing

On Fedora systems:

sudo modprobe rdma_rxe
sudo rdma link add rxe0 type rxe netdev eth0  # Name of your active interface
kind create cluster --config testing/kind-config-rdma.yaml
kubectl label node kind-control-plane node-role.kubernetes.io/worker=""
k8s-netperf --config config.yaml --hostNet --privileged --ib-write-bw rxe0:1 --local

Cleanup:

kind delete cluster
sudo rdma link delete rxe0