Skip to content

[DRAFT] feat(core): introduce pre-eval hooks and e2e tests caching logic #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[DRAFT] feat(core): introduce pre-eval hooks and e2e tests caching logic #383

wants to merge 1 commit into from
+3,152 −32

Conversation

@Glorf
Copy link
Contributor

@Glorf Glorf commented Oct 24, 2025

  • Allow pre-eval hooks
  • Flatten hook logic - all hooks should be treated as interceptors
  • Implement logic facilitating e2e testing using pre- and post-eval hooks
  • Facilitate debugging by finding closest cached request while using test mode
  • Implement pre-eval hooks for pypi package installation and custom datasets downloads

@Glorf
Introduce pre-eval hooks
8f12e7d 
Signed-off-by: Michal Bien <[email protected]>
@Glorf Glorf requested review from a team as code owners October 24, 2025 15:03
@copy-pr-bot
Copy link

copy-pr-bot bot commented Oct 24, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions bot added documentation Improvements or additions to documentation nemo-evaluator tests labels Oct 24, 2025
@Glorf Glorf changed the title [DRAFT] Introduce pre-eval hooks and e2e tests caching logic feat(core): introduce pre-eval hooks and e2e tests caching logic Oct 24, 2025
agronskiy
agronskiy reviewed Oct 24, 2025
View reviewed changes
Copy link
Collaborator

@agronskiy agronskiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: is it a correct assumption that hooks ⊆ interceptors?

@Glorf Glorf changed the title feat(core): introduce pre-eval hooks and e2e tests caching logic [DRAFT] feat(core): introduce pre-eval hooks and e2e tests caching logic Oct 28, 2025
@Glorf Glorf marked this pull request as draft October 28, 2025 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@agronskiy agronskiy agronskiy left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation nemo-evaluator tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Glorf @agronskiy