fix: Changes to support `ray job submit` and prefetch venvs #432

hemildesai · 2025-05-21T17:22:21Z

What does this PR do ?

Create a venv per each node for the worker group using a ray task
Set CUDA_VISIBLE_DEVICES based on bundle index
Always reuse external clusters

8b run matches expectations

Issues

List issues that this PR closes (syntax):

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

...

nemo_rl/utils/venvs.py

nemo_rl/distributed/worker_groups.py

gwarmstrong

Tested with NeMo-Run updated and it works well

.gitignore

Signed-off-by: Hemil Desai <[email protected]>

…eMo#432) Signed-off-by: Hemil Desai <[email protected]>

Signed-off-by: Hemil Desai <[email protected]>

terrykong reviewed May 21, 2025

View reviewed changes

nemo_rl/utils/venvs.py Show resolved Hide resolved

nemo_rl/distributed/worker_groups.py Outdated Show resolved Hide resolved

nemo_rl/distributed/worker_groups.py Outdated Show resolved Hide resolved

nemo_rl/distributed/worker_groups.py Outdated Show resolved Hide resolved

hemildesai force-pushed the hemil/k8s-changes branch from b249973 to 8e155f9 Compare May 22, 2025 03:05

gwarmstrong approved these changes May 22, 2025

View reviewed changes

gwarmstrong mentioned this pull request May 22, 2025

NeMo-RL support for SFT NVIDIA-NeMo/Skills#482

Merged

terrykong reviewed May 29, 2025

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

terrykong linked an issue May 29, 2025 that may be closed by this pull request

NeMo-Run Integration #79

Closed

terrykong mentioned this pull request Jun 2, 2025

Running multinode on azure #463

Open

hemildesai force-pushed the hemil/k8s-changes branch from 01286a2 to a9e3f0e Compare June 4, 2025 01:59

github-actions bot added the documentation Improvements or additions to documentation label Jun 4, 2025

hemildesai force-pushed the hemil/k8s-changes branch from f0ce5db to ed09205 Compare June 4, 2025 02:01

hemildesai marked this pull request as ready for review June 4, 2025 02:04

hemildesai added the CI:L1 Run doctests, unit tests, and functional tests label Jun 4, 2025

hemildesai temporarily deployed to nemo-ci June 4, 2025 19:00 — with GitHub Actions Inactive

hemildesai force-pushed the hemil/k8s-changes branch 2 times, most recently from 4e37337 to 2cd3f4d Compare June 5, 2025 18:11

hemildesai added CI:L1 Run doctests, unit tests, and functional tests and removed CI:L1 Run doctests, unit tests, and functional tests labels Jun 5, 2025

hemildesai temporarily deployed to nemo-ci June 5, 2025 18:21 — with GitHub Actions Inactive

hemildesai added the CI:L0 Run doctests and unit tests label Jun 5, 2025

hemildesai temporarily deployed to nemo-ci June 5, 2025 20:10 — with GitHub Actions Inactive

terrykong previously approved these changes Jun 5, 2025

View reviewed changes

hemildesai dismissed terrykong’s stale review via d24b881 June 5, 2025 21:30

hemildesai added CI:L1 Run doctests, unit tests, and functional tests and removed CI:L1 Run doctests, unit tests, and functional tests labels Jun 5, 2025

hemildesai temporarily deployed to nemo-ci June 5, 2025 21:31 — with GitHub Actions Inactive

hemildesai force-pushed the hemil/k8s-changes branch from 3f8de44 to 32ed15a Compare June 6, 2025 01:19

hemildesai added CI:L1 Run doctests, unit tests, and functional tests and removed CI:L1 Run doctests, unit tests, and functional tests labels Jun 6, 2025

hemildesai temporarily deployed to nemo-ci June 6, 2025 01:20 — with GitHub Actions Inactive

hemildesai removed the CI:L1 Run doctests, unit tests, and functional tests label Jun 9, 2025

terrykong added CI:L1 Run doctests, unit tests, and functional tests and removed CI:L0 Run doctests and unit tests CI:L1 Run doctests, unit tests, and functional tests labels Jun 10, 2025

hemildesai changed the title ~~fix: Changes to support ray job submit~~ fix: Changes to support ray job submit, prefetch venvs Jun 10, 2025

hemildesai changed the title ~~fix: Changes to support ray job submit, prefetch venvs~~ fix: Changes to support ray job submit and prefetch venvs Jun 10, 2025

hemildesai added 2 commits June 10, 2025 14:02

fix: Changes to support ray job submit and prefetch venvs

5783de9

Signed-off-by: Hemil Desai <[email protected]>

pr feedback

fd1c47c

Signed-off-by: Hemil Desai <[email protected]>

hemildesai force-pushed the hemil/k8s-changes branch from 7106193 to fd1c47c Compare June 10, 2025 21:04

hemildesai added CI:L1 Run doctests, unit tests, and functional tests and removed CI:L1 Run doctests, unit tests, and functional tests labels Jun 10, 2025

hemildesai temporarily deployed to nemo-ci June 10, 2025 21:05 — with GitHub Actions Inactive

terrykong previously approved these changes Jun 10, 2025

View reviewed changes

terrykong enabled auto-merge June 10, 2025 21:29

hemildesai disabled auto-merge June 10, 2025 21:51

fix

8752aa4

Signed-off-by: Hemil Desai <[email protected]>

hemildesai dismissed terrykong’s stale review via 8752aa4 June 10, 2025 22:37

hemildesai added CI:L1 Run doctests, unit tests, and functional tests and removed CI:L1 Run doctests, unit tests, and functional tests labels Jun 10, 2025

hemildesai temporarily deployed to nemo-ci June 10, 2025 22:38 — with GitHub Actions Inactive

hemildesai force-pushed the hemil/k8s-changes branch from 7ace6da to 7a3d5f7 Compare June 10, 2025 22:47

fix

3aec87b

Signed-off-by: Hemil Desai <[email protected]>

hemildesai force-pushed the hemil/k8s-changes branch from 7a3d5f7 to 3aec87b Compare June 10, 2025 22:49

terrykong approved these changes Jun 10, 2025

View reviewed changes

terrykong added this pull request to the merge queue Jun 10, 2025

Merged via the queue into main with commit 51f8b26 Jun 11, 2025
13 of 14 checks passed

terrykong deleted the hemil/k8s-changes branch June 11, 2025 01:58

terrykong mentioned this pull request Jun 19, 2025

NeMo Run Guides #531

Open

YzjiaoNvd pushed a commit to YzjiaoNvd/NeMo-RL that referenced this pull request Jul 14, 2025

fix: Changes to support ray job submit and prefetch venvs (NVIDIA-N…

48b1190

…eMo#432) Signed-off-by: Hemil Desai <[email protected]>

KiddoZhu pushed a commit that referenced this pull request Jul 28, 2025

fix: Changes to support ray job submit and prefetch venvs (#432)

c98186e

Signed-off-by: Hemil Desai <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Changes to support `ray job submit` and prefetch venvs #432

fix: Changes to support `ray job submit` and prefetch venvs #432

Uh oh!

hemildesai commented May 21, 2025 •

edited by terrykong

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gwarmstrong left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix: Changes to support ray job submit and prefetch venvs #432

fix: Changes to support ray job submit and prefetch venvs #432

Uh oh!

Conversation

hemildesai commented May 21, 2025 • edited by terrykong Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gwarmstrong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix: Changes to support `ray job submit` and prefetch venvs #432

fix: Changes to support `ray job submit` and prefetch venvs #432

hemildesai commented May 21, 2025 •

edited by terrykong

Loading