Skip to content

docs: Extend evaluation section #474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
marta-sd merged 21 commits into from
Nov 21, 2025
Merged

docs: Extend evaluation section #474

marta-sd merged 21 commits into from
Nov 21, 2025

Conversation

@marta-sd
Copy link
Contributor

@marta-sd marta-sd commented Nov 19, 2025

No description provided.

@marta-sd
small fixes to existing docs
5cf4c47 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
endpoint compatibility for logprobs
84ba7c1 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
docs on tasks not defined in fdf
3542932 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
docs on logprobs
8b224cf 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
optional eval packages for nemo fw
bb14636 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
add launcher example for logprobs
db8ae54 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
update index and remove obsolete files
31ccc15 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@copy-pr-bot
Copy link

copy-pr-bot bot commented Nov 19, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions bot added documentation Improvements or additions to documentation nemo-evaluator-launcher nemo-evaluator labels Nov 19, 2025
athitten
athitten reviewed Nov 20, 2025
View reviewed changes
athitten
athitten reviewed Nov 20, 2025
View reviewed changes
athitten
athitten reviewed Nov 20, 2025
View reviewed changes
athitten
athitten reviewed Nov 20, 2025
View reviewed changes
athitten
athitten reviewed Nov 20, 2025
View reviewed changes
athitten
athitten reviewed Nov 20, 2025
View reviewed changes
athitten
athitten reviewed Nov 20, 2025
View reviewed changes
athitten
athitten reviewed Nov 20, 2025
View reviewed changes
marta-sd
marta-sd commented Nov 20, 2025
View reviewed changes
marta-sd and others added 4 commits November 20, 2025 10:44
@marta-sd @athitten
Apply suggestions from code review
dda6ce3 
Co-authored-by: Abhishree Thittenamane <[email protected]>
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
change formatting for on-demand packages
c43bf2a 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
better integrate with the rest of the docs, improve formatting and un…
c691597 
…-orphan the tex-gen guide

Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
simplify the config
d9ad5a9 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd marta-sd marked this pull request as ready for review November 20, 2025 14:40
@marta-sd marta-sd requested review from a team as code owners November 20, 2025 14:40
athitten
athitten reviewed Nov 21, 2025
View reviewed changes
athitten
athitten reviewed Nov 21, 2025
View reviewed changes
marta-sd and others added 2 commits November 21, 2025 09:18
@marta-sd @athitten
Apply suggestions from code review
335b51d 
Co-authored-by: Abhishree Thittenamane <[email protected]>
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
revert accidentally commited change (url for polemo2)
3493b68 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
AWarno
AWarno reviewed Nov 21, 2025
View reviewed changes
docs/evaluation/custom-task.md
Comment on lines +26 to +27

or install `nemo-evaluator` and `nvidia-lm-eval` Python package in your environment of choice:
Copy link
Contributor

@AWarno AWarno Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q: Do I need to install both?

Copy link
Contributor Author

@marta-sd marta-sd Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually no, nvidia-lm-eval would pull nemo-evaluator as its dependency, but we list evaluator in all the docs to make it clear for the users which package they are working with.

AWarno
AWarno reviewed Nov 21, 2025
View reviewed changes
packages/nemo-evaluator-launcher/examples/local_vllm_llama_3_1_8b_logprobs.yaml Outdated
Comment on lines 24 to 26
env_vars:
HF_TOKEN: ${oc.env:HF_TOKEN} # needed to access meta-llama/Llama-3.1-8B gated model

Copy link
Contributor

@AWarno AWarno Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think we should unify it? all over the examples we pass it differently e.g.:

Evaluator/packages/nemo-evaluator-launcher/examples/slurm_llama_3_1_8b_instruct_hf.yaml

Line 46 in 75db6f1

HF_TOKEN: "hf_your_token" # Needed to access meta-llama/Llama-3.1-8B-Instruct gated model

Copy link
Contributor Author

@marta-sd marta-sd Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, let me modify my example. Btw I'll be reviewing all examples in a followup PR.

Copy link
Contributor

@AWarno AWarno Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think yours my be better (than the one above)

AWarno
AWarno previously approved these changes Nov 21, 2025
View reviewed changes
@marta-sd
make it clear that endpoint or checkpoint is a prerequisite
329219f 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
unify how HF token is passed in examples
1e11006 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
parametrize model name in snippet
90b8037 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
athitten
athitten reviewed Nov 21, 2025
View reviewed changes
athitten
athitten previously approved these changes Nov 21, 2025
View reviewed changes
Copy link
Contributor

@athitten athitten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @marta-sd approving. There's one suggestion I made, can be merged once thats added.

@marta-sd @athitten
Update docs/evaluation/run-evals/index.md
f37cc65 
Co-authored-by: Abhishree Thittenamane <[email protected]>
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
fix HF_TOKEN passing in logprobs config
2de2b55 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
add note on harness.task syntax for conflicting tasks
0e0b70f 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
@marta-sd
Copy link
Contributor Author

marta-sd commented Nov 21, 2025

/ok to test 0e0b70f

athitten
athitten approved these changes Nov 21, 2025
View reviewed changes
Copy link
Contributor

@athitten athitten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for addressing the comments @marta-sd !

@marta-sd marta-sd enabled auto-merge (squash) November 21, 2025 21:24
@marta-sd marta-sd merged commit 8bf489f into main Nov 21, 2025
45 of 46 checks passed
@marta-sd marta-sd deleted the martas/docs/evaluation branch November 21, 2025 21:24
@marta-sd marta-sd mentioned this pull request Nov 25, 2025
Merged
lbliii pushed a commit that referenced this pull request Nov 25, 2025
@marta-sd @athitten @lbliii
docs: Extend evaluation section (#474)
05f7950 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
Co-authored-by: Abhishree Thittenamane <[email protected]>
lbliii pushed a commit that referenced this pull request Nov 25, 2025
@marta-sd @athitten @lbliii
docs: Extend evaluation section (#474)
599250e 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
Co-authored-by: Abhishree Thittenamane <[email protected]>
Signed-off-by: Lawrence Lane <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AWarno AWarno AWarno left review comments

@athitten athitten athitten approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation nemo-evaluator nemo-evaluator-launcher

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@marta-sd @AWarno @athitten