Network binding plugin: Support compute container resource overhead #303

orelmisan · 2024-06-16T20:44:36Z

What this PR does / why we need it:
Some network binding plugins require compute resource overhead, for example the passt plugin requires additional memory overhead in the virt-launcher pod's compute container.
Suggest several alternatives to address this issue.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

Design: A design document was considered and is present (link) or not required
PR: The PR description is expressive enough and will help future contributors
Code: Write code that humans can understand and Keep it simple
Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
Upgrade: Impact of this change on upgrade flows was considered and addressed if required
Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
Community: Announcement to kubevirt-dev was considered

Release note:

NONE

kubevirt-bot · 2024-06-16T20:44:38Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

kubevirt-bot · 2024-06-16T20:44:41Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign alonakaplan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

orelmisan · 2024-06-16T20:44:53Z

/cc @EdDev @ormergi

orelmisan · 2024-06-16T20:45:22Z

/uncc @aburdenthehand @jobbler

ormergi

Thanks for the PR, overall looks good, see my inline comments.

Regarding the the PR description and second commit message, I think it should also mention that memory overhead is necessary to avoid pod eviction due to passt VMs consume more memory then expected, and improve passt VMs scheduling results; passt VMs wont be scheduled on nodes that doesnt have enough memory.

ormergi · 2024-06-18T09:04:27Z

design-proposals/network-binding-plugin/network-binding-plugin.md

  containers:
+    - name: compute
+      resources:
+        requests:


I think resource limits should be as well, in case the pod satisfy QoS class Guaranteed the plugin overhead will not become the reason for violating it.

AFAIK, there is logic [1] to automatically add memory limits when needed (there is also an equivalent for CPU).

[1] https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-controller/services/renderresources.go#L189

ormergi · 2024-06-18T09:07:11Z

design-proposals/network-binding-plugin/network-binding-plugin.md

 Both of the options are not perfect, but the `virt-handler.SyncVMI`
 has fewer cons. Therefore, it was chosen.

+#### Additional resource requests for the virt-launcher compute container


I would rephrase this section to be vogue about the concrete plugin or dependency inside virt-launcher that requires it, saying in case a plugin requires memory overhead it should be specified in the CR, and refer to passt example.

Adjusted the wording, please tell me what you think.

design-proposals/network-binding-plugin/network-binding-plugin.md

ormergi · 2024-06-18T09:38:37Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+The passt binary is shipped and executed inside the virt-launcher pod.
+
+### Domain definition & creation


I would simplify this section saying the sidecar adds passt interface to the domain, similar to vDPA example, maybe mention it use libvirt user-space networking settings https://libvirt.org/formatdomain.html#userspace-slirp-or-passt-connection.
You can also refer to slirp example section saying passt sidecar works in similar way.

design-proposals/network-binding-plugin/network-binding-plugin.md

EdDev

First commit review.

Although I placed comments inline on that commit, I think it is not a must example addition.

At the design level, we are not really interested in a specific binding, but on the general concept.
The general concept can be applied on other bindings, just to provide an example on how to use it.

design-proposals/network-binding-plugin/network-binding-plugin.md

EdDev · 2024-06-18T16:38:55Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+  namespace: default
+spec:
+  config: '{
+      "cniVersion": "0.3.1",


Maybe we should use 1.0.X in the example.

This was removed.
Please ack so we can resolve the thread.

EdDev · 2024-06-18T16:43:41Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+metadata:
+  name: virt-launcher-123
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[{"name":"netbindingpasst","namespace":"mynamespace","cni-args":{"logicNetworkName":"default"}}]'


I think the logicNetworkName is supposed to be passtnet.

This was removed.
Please ack so we can resolve the thread.

design-proposals/network-binding-plugin/network-binding-plugin.md

EdDev · 2024-06-18T16:47:30Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+### Configure Pod network namespace
+
+Not required for passt binding


We do configure networking for passt.

This was removed.
Please ack so we can resolve the thread.

EdDev · 2024-06-18T16:48:54Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+### Run services in the virt-launcher pod
+
+The passt binary is shipped and executed inside the virt-launcher pod.


This should say: Not required for passt binding.

This was removed.
Please ack so we can resolve the thread.

EdDev · 2024-06-18T16:52:11Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+    <portForward proto='udp'/>
+    <model type='virtio-non-transitional'/>
+    <backend type='passt' logFile='/var/run/kubevirt/passt.log'/>
+    <alias name='ua-default'/>


This name is not in-sync with the network name used in the VM spec.

This was removed.
Please ack so we can resolve the thread.

EdDev

Second commit:

The commit message hints and explains the passt binding and everything is driven from it. While this is correct in terms of why we do all this, I do not think we should start the story there.

The story should start from the need, mentioning passt as an example.
The fact that there is a binary alongside libvirt is not that important. If it was part of libvirt we would have needed to take that into account as well.

Also, it would be better to leave the details to the design and in the commit just provide the topic/subject. That way we can easily review it and adjusts it per that review.

EdDev · 2024-06-18T17:39:23Z

design-proposals/network-binding-plugin/network-binding-plugin.md

        - domainAttachment (a standard domain definition that exists in
          the core . E.g `tap`, `sriov`).
        - downwardAPI (E.g `device-info`)
+        - resourceOverhead (currently only additional memory overhead request for the compute container in the virt-launcher pod is supported)


I would be specific here and mention the compute-container in the name.
E.g. computeResources.

This also opens up the different questions and options which should be discussed in a proposal:

Should we consider only the compute container? What about the sidecar container?

Ref the compute container, we can use a specific name for the field (e.g. computeResources) or we can make it configurable using a flag under a general resources.

What is the reason for not just doing a computeMemoryRequest needs to convince the reviewers.

Now, we can make it even more general by moving away from the specific network usage of this and look at a general sidecar hook that KubeVirt supports.
E.g.: The Kubvirt CR will have a policy for allocating resources to sidecar containers and possible the compute container. Referencing this policy from the network binding definition or from a sidecar hook. This may be an overkill, but thinking about it and examining the pors/cons of it can be useful.

EdDev · 2024-06-18T17:44:03Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+Some binding plugins may require an additional binary to be shipped inside the virt-launcher compute image.
+This binary requires additional resources that have to be explicitly stated.


I do not think we need to limit it to an additional binary. It could be just more resources from the cgroupv2 of the compute container, consumed by an existing binary (e.g. libvirt).

EdDev · 2024-06-18T17:53:07Z

design-proposals/network-binding-plugin/network-binding-plugin.md

          networkAttachmentDefinition: default/netbindingpasst
          sidecarImage: quay.io/kubevirt/network-passt-binding
+          resourceOverhead:
+            memory: 800Mi


I think you will have to explain why not:

Directly memoryOverhead: 500Mi (BTW, it should be 500 and not 800).

With explicit request and being open for adding limit.

runtimeClasses have a pretty similar concept, maybe worth aligning the API idea
https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/

ie.

overhead: podFixed: memory: "120Mi" cpu: "250m"

runtimeClasses have a pretty similar concept, maybe worth aligning the API idea

Interesting, thanks for sharing this.
I think the API could be in sync with the general resources concept.
This allows to control the resource additions to the container, with the ability to extend it in the future to any type of resource and define both request and limits.

The oddity here, as I see it, is how we express the resources of a different container (i.e. compute) and still not lock the ability to do the same for the sidecar itself.

@EdDev I'm late to review this, but I don't understand why this API needs to add anything to the compute container and not simply advertise the resources needed to run this binding container?!
Same way as it's done for hotplug / other containers?
https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-controller/services/renderresources.go#L611

runtimeClasses have a pretty similar concept, maybe worth aligning the API idea https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/

ie.

overhead: podFixed: memory: "120Mi" cpu: "250m"

@fabiand I doubt it aligns well. This overhead API was invented for kata and the overhead is being added to any pod created with this runtimeclass, I don't think it aligns.

@EdDev I'm late to review this, but I don't understand why this API needs to add anything to the compute container and not simply advertise the resources needed to run this binding container?! Same way as it's done for hotplug / other containers? https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-controller/services/renderresources.go#L611

@EdDev do we expect all network binding to affect the compute container or is it just passt?
passt specifically uses a feature in qemu there for the GetMemoryOverhead functional can identify that passt will be used and add a passt specific overhead - this is instead of a kubevirt CR API.
What do you think?

@vladikr It is unknown at this stage how future network binding plugins will affect the compute container's resource consumption.
Network binding plugins have the ability to configure the domain, thus the virt stack might consume additional compute resources which KubeVirt cannot account for.

In the past year, the passt binding was converted from being a core feature to a plugin, so KubeVirt will not know in advance that is is used.
Also, we should strive for treating all plugins as equal IMO.

@vladikr my comment was only about the API design, not about using the POD api.

@vladikr Hi
I see that according to the network binding plugin design document, one of the integration points that seems legit is the pod definition phase
There were no exceptions regarding resource rendering, and Kubevirt CR was advised as a potential API to extend in order to integrate into that point.

Given that the network binding design was accepted, and also the implementation of this design also got into the codebase, I am not sure how to proceed, I'm in favor of accepting this design.

Can you please advice @vladikr ?

orelmisan · 2024-06-23T10:52:30Z

Change: Removed the passt example, as it is not essential for to the proposal.

Addressed sidecar and compute container resource specification.

orelmisan · 2024-06-23T10:58:04Z

@EdDev @ormergi PTAL.

EdDev

First commit review.

EdDev · 2024-06-23T12:04:26Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+The sidecar container can have a memory leak and may cause node's destabilization.
+
+
+Alternatives:


Please keep here the decided solution and place the alternatives (with a ref from here) to a dedicated appendix.

EdDev · 2024-06-23T12:19:07Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+   - Coarse level of control.
+   - Only supports CPU and memory requests and limits.
+
+2. Additional API for sidecar resource configuration:


This is specific for the binding plugin, not the sidecar. The sentence is not clear about this.
The definition is per the network binding plugin, and applied if the plugin uses a sidecar.

EdDev · 2024-06-23T12:20:45Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+Cons:
+ - Require an API change.
+ - The API will probably evolve as additional plugins will be created.


I do not understand this one.

Discussed this point offline.
It was removed.

EdDev · 2024-06-23T12:21:53Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+Cons:
+ - Require an API change.
+ - The API will probably evolve as additional plugins will be created.
+ - May require cluster admins to adjust plugin resources during the plugin's lifecycle.


The cluster admin is responsible to register the binding plugin in the first place, so I am unclear what this point means.

Discussed this point offline, improved the wording.

EdDev · 2024-06-23T12:25:12Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+   - Only supports CPU and memory requests and limits.
+
+2. Additional API for sidecar resource configuration:
+The network binding plugin API in the KubeVirt CR could receive an additional input field to specify the sidecar resource requirements:


The specified resource is define for each instance of usage or per the sidecar, irrelevant of the usage?
E.g. there may be 1 interface using the plugin or there may be 3 interfaces using the plugins in the same VM.

Addressed it in the text.

EdDev · 2024-06-23T12:29:53Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+For each network binding plugin used, the VMI controller will add a label on the virt-launcher pod with the following format:
+
+`kubevirt.io/network-binding-plugin:<plugin-name>`


How will the admission webhook be able to identify the relevant container?

Addressed it in the text.

design-proposals/network-binding-plugin/network-binding-plugin.md

EdDev

Thank you for the proposal.

The following general points should be considered and added:

Effort cost estimation for each option.

EdDev · 2024-06-24T06:05:47Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+For some plugins, such as passt, there is a need to execute an additional binary in the compute container.
+Since this binary has its own CPU and memory limits, they should be somehow accounted for.


Please do not limit it to running an additional binary, the addition in memory or other resources may come from different reasons (e.g. libvirt itself requiring more memory due to the expected configuration).

EdDev · 2024-06-24T06:07:57Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+Alternatives:
+1. Manually setting the VM's resources:
+The user can override KubeVirt's algorithms and set resource requirements.


The current logic that adds overhead per internal core logic is colliding with adding manually an overhead? Specifically, if one specifies explicitly resource requests, are these being increased by Kubevirt logic or something else is happening?

The way it works is as follows:
A VM could be defined with both guest memory and memory resource specification:

apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: vm-cirros spec: template: spec: domain: memory: guest: 128Mi resources: requests: memory: 640Mi # 128Mi for the guest + 512Mi for the network binding plugin

The virt-launcher pod's compute container will have a memory request which contains the sum of:

Guest VM's memory (128Mi in this example).

Memory overhead for KubeVirt's components (its size is dependent on the VMI's spec)

Arbitrary memory overhead (512Mi in this example)

The domain XML will contains the the guest's memory specification (128Mi in this example).

As a side note, it is also possible to specify a memory request with less memory than the guest requires.

EdDev · 2024-06-24T06:09:02Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+Cons:
+- Error prune
+- The user does not take into account the overhead considerations KubeVirt takes when templating a virt-launcher pod.


I guess this is related to my prev question. Can you please confirm this is indeed the case?
I would be surprise if this is how it works.

This is not true.
I removed this line.

EdDev · 2024-06-24T06:10:51Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+- The API will probably evolve as additional plugins will be created.
+- May require cluster admins to adjust plugin resources during the plugin's lifecycle.


Like with the sidecar previously, these points are not clear.

Discussed this point offline, adjusted the text.

EdDev · 2024-06-24T06:15:54Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+2. Additional API for compute container resource overhead:
+
+The network binding plugin API in the KubeVirt CR could receive an additional input field to specify the resource requirements overhead for the compute container:


There is a need to explicitly specify if this overhead is added per plugin type or per plugin usage count (i.e. per the number of interfaces referencing it from the VM).

It is also important to specify if it is dependent on any other field/setup.
The previous sidecar resource are dependent on the existence of a sidecar, but this one may not have such a dependency.

Signed-off-by: Orel Misan <[email protected]>

orelmisan · 2024-07-09T15:49:26Z

Change: Split the sidecar's resource specification to #309

EdDev

Thank you!

vladikr · 2024-07-25T18:09:24Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+##### Compute Container Resource Overhead
+
+For some plugins, an additional resource consumption can be expected from the virt-launcher pod compute container.
+For example, there could be need to execute an additional binary in the compute container.


Could you please explain what binary can be executed and how it gets there?
How can an external plugin rely on the fact that a specific binary is part of the virt-launcher image?

This example was derived from the passt use case, on which the passt binary is shipped as part of the virt-launcher compute image.

Another example could be that the plugin configures the domain in a way that causes the virt-stack to consume additional compute resources that KubeVirt cannot account for.

vladikr · 2024-07-25T21:57:10Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+For some plugins, an additional resource consumption can be expected from the virt-launcher pod compute container.
+For example, there could be need to execute an additional binary in the compute container.
+Since this binary has its own CPU and memory limits, they should be somehow accounted for.


I think, we should speak of a specific functionality and not about arbitrary binaries.
virt-launcher should be well defined.
If we know what functionality we're enabling that requires additional overhead, kubevirt needs to know the resources it requires. Similar to other aspects accounted for in GetMemoryOverhead.

I would say that if we want an API for this it should be based on a functionality, not a binding.

Each resource consumption of it.

The plugin can perform actions that affect KubeVirt in ways it cannot account for.
Thus the need to externally add resources to the compute container.

orelmisan · 2024-07-30T18:07:03Z

Thank you for reviewing this proposal @vladikr, sorry it took some time to respond.

aburdenthehand · 2024-09-23T12:41:49Z

On behalf of SIG-compute, can one of @jean-edouard @stu-gott @enp0s3 @vladikr please re-review to move this forward. I can merge if I receive an approve or second lgtm from one of you. Thank you.

kubevirt-bot · 2024-09-30T13:34:53Z

Pull requests that are marked with lgtm should receive a review
from an approver within 1 week.

After that period the bot marks them with the label needs-approver-review.

/label needs-approver-review

enp0s3 · 2024-12-11T12:42:15Z

/cc

enp0s3 · 2024-12-16T19:48:00Z

Deferring the discussion for after the end year holidays

kubevirt-bot · 2025-03-16T20:22:25Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

fossedihelm · 2025-03-24T14:04:12Z

/remove-lifecycle stale

kubevirt-bot · 2025-06-22T14:52:42Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

orelmisan · 2025-06-23T07:06:04Z

/remove-lifecycle stale

kubevirt-bot · 2025-09-21T08:04:56Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

orelmisan · 2025-09-25T08:07:24Z

/remove-lifecycle stale

kubevirt-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jun 16, 2024

kubevirt-bot requested review from aburdenthehand and jobbler June 16, 2024 20:44

kubevirt-bot added the size/L label Jun 16, 2024

kubevirt-bot requested review from EdDev and ormergi June 16, 2024 20:44

kubevirt-bot removed request for aburdenthehand and jobbler June 16, 2024 20:45

ormergi reviewed Jun 18, 2024

View reviewed changes

orelmisan marked this pull request as ready for review June 18, 2024 15:04

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 18, 2024

kubevirt-bot requested review from aburdenthehand and jean-edouard June 18, 2024 15:04

orelmisan force-pushed the net-binding-plugin-mem-overhead branch 2 times, most recently from ed97ca9 to 8432599 Compare June 18, 2024 17:08

EdDev reviewed Jun 18, 2024

View reviewed changes

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from 8432599 to 416926c Compare June 23, 2024 10:50

EdDev reviewed Jun 23, 2024

View reviewed changes

design-proposals/network-binding-plugin/network-binding-plugin.md Show resolved Hide resolved

orelmisan changed the title ~~Network binding plugin: Support memory overhead~~ Network binding plugin: Support resource overhead Jun 23, 2024

EdDev reviewed Jun 24, 2024

View reviewed changes

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from 416926c to e0b43e5 Compare June 24, 2024 15:11

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from d4eb851 to c33498b Compare July 9, 2024 15:46

orelmisan changed the title ~~Network binding plugin: Support resource overhead~~ Network binding plugin: Support compute resource overhead Jul 9, 2024

Network binding plugin: Support compute container resource overhead

591c3a4

Signed-off-by: Orel Misan <[email protected]>

orelmisan changed the title ~~Network binding plugin: Support compute resource overhead~~ Network binding plugin: Support compute container resource overhead Jul 9, 2024

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from c33498b to 591c3a4 Compare July 9, 2024 15:49

EdDev approved these changes Jul 10, 2024

View reviewed changes

kubevirt-bot assigned EdDev Jul 10, 2024

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 10, 2024

vladikr reviewed Jul 25, 2024

View reviewed changes

kubevirt-bot added the needs-approver-review Indicates that a PR requires a review from an approver. label Sep 30, 2024

kubevirt-bot requested a review from enp0s3 December 11, 2024 12:42

kubevirt-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 16, 2025

kubevirt-bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 24, 2025

kubevirt-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 22, 2025

kubevirt-bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 23, 2025

kubevirt-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 21, 2025

kubevirt-bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 25, 2025


		The passt binary is shipped and executed inside the virt-launcher pod.

		### Domain definition & creation


		### Configure Pod network namespace

		Not required for passt binding


		### Run services in the virt-launcher pod

		The passt binary is shipped and executed inside the virt-launcher pod.

		Some binding plugins may require an additional binary to be shipped inside the virt-launcher compute image.
		This binary requires additional resources that have to be explicitly stated.

		The sidecar container can have a memory leak and may cause node's destabilization.


		Alternatives:

		For each network binding plugin used, the VMI controller will add a label on the virt-launcher pod with the following format:

		`kubevirt.io/network-binding-plugin:<plugin-name>`

		For some plugins, such as passt, there is a need to execute an additional binary in the compute container.
		Since this binary has its own CPU and memory limits, they should be somehow accounted for.

		- The API will probably evolve as additional plugins will be created.
		- May require cluster admins to adjust plugin resources during the plugin's lifecycle.


		2. Additional API for compute container resource overhead:

		The network binding plugin API in the KubeVirt CR could receive an additional input field to specify the resource requirements overhead for the compute container:

Network binding plugin: Support compute container resource overhead #303

Are you sure you want to change the base?

Network binding plugin: Support compute container resource overhead #303

Uh oh!

Conversation

orelmisan commented Jun 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kubevirt-bot commented Jun 16, 2024

Uh oh!

kubevirt-bot commented Jun 16, 2024

Uh oh!

orelmisan commented Jun 16, 2024

Uh oh!

orelmisan commented Jun 16, 2024

Uh oh!

ormergi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

orelmisan Jun 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

EdDev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EdDev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

orelmisan commented Jun 16, 2024 •

edited

Loading

orelmisan Jun 18, 2024 •

edited

Loading