Skip to content

docs: Add note for nvidia-smi execution in example deployment #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: Add note for nvidia-smi execution in example deployment #106

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,21 @@ resources:

Verify that it has been scheduled on one of the __CPU__ nodes.

You can also test by running the example deployment YAML under the [example](./example) folder
You can also test by running the example deployment YAML under the [example](./example) folder.

**Note:** If you want to execute `nvidia-smi` in the example deployment, you need to add the following snippet to the `deployment.yml` file. Replace `<node-name>` with the node name you labeled during installation:

Copy link

@nayihz nayihz Feb 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does <node-name> represent? what do you mean the node name you labeled during installation?

Copy link
Author

@barrycheng05 barrycheng05 Mar 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean this section
https://github.com/run-ai/fake-gpu-operator/blob/main/README.md?plain=1#L22

Copy link

@nayihz nayihz Mar 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvidia-smi still cannot be executed when add

env:
- name: NODE_NAME
  value: default

error msg:

# k exec -it sleepy-deployment-6bddfbb7f4-s8mc4 -- sh                                                                                                              :( 130 25-03-03 - 6:59:40
/ # nvidia-smi
sh: nvidia-smi: not found

Is my understanding incorrect?

Copy link
Author

@barrycheng05 barrycheng05 Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears to be a different issue. You might want to check if the device plugin is running correctly.

```yaml
env:
- name: NODE_NAME
value: <node-name>
```

## Troubleshooting

[Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/) should be disabled on the gpu-operator namespace

```
```sh
kubectl label ns gpu-operator pod-security.kubernetes.io/enforce=privileged
```

Expand Down