Merge upstream huggingface/lighteval main into merge_hf_main

Upstream refactor splits src/lighteval/tasks into per-task files under
src/lighteval/tasks/tasks/ and src/lighteval/tasks/multilingual/tasks/,
drops default_tasks.py / default_prompts.py / multilingual/tasks.py, and
removes the suite field from LightevalTaskConfig.

Port our edits to the new structure:
- tasks/gsm_plus.py: generation_size 16384
- tasks/gsm8k.py: generation_size 2048
- tasks/mgsm.py: hf_revision, suffix exact_match + expr_gold_metric,
  language-specific stop sequences for all 11 subsets
- tasks/piqa.py: switch to lighteval/piqa mirror
- tasks/siqa.py: pin hf_revision
- tasks/mmlu_pro.py: fix upstream's hardcoded ABCD letters so the prompt
  uses dynamic letters based on the number of options; add a parallel
  mmlu_pro_raw task exposing the handmade prompt (no inspect_ai)
- tasks/ruler.py: new home for the ruler prompt helper
- tasks/advbench.py: move here from community_tasks/
- multilingual/tasks/mathalea.py: move here from community_tasks/
- multilingual/tasks/french.py: keep jzhang86/fr_ifeval fallback and the
  generative GPQA-fr-diamond variant with prompt_gpqa_fr_instruct

Other conflict resolutions:
- pyproject.toml: take upstream unpinned transformers, vllm>=0.11.0,
  new inspect-ai and openai deps
- vllm_model.py: keep max_seq_len_to_capture fallback, Mistral eos_token
  guard, prefix-cache None-skip in logprob loop, and
  skip_reading_prefix_cache via guarded attribute assignment; adopt
  upstream's build_vllm_token_prompts helper
- llm_as_judge.py: keep max_model_len=65536, adopt upstream's
  api_key/base_url litellm pass-through
- lighteval_task.py: preserve name/data_dir fallback in load_dataset
  while picking up upstream's data_files support; keep partial args
  detail in __str__ for deterministic cache hashing
- cache_management.py: adopt name-only task_to_configs lookup; keep
  regex that strips function memory addresses for hash determinism

Author: Jeronymous

.github/workflows/pr_style_bot.yaml

@@ -0,0 +1,16 @@
+name: PR Style Bot
+
+on:
+  issue_comment:
+    types: [created]
+
+permissions:
+  pull-requests: write
+
+jobs:
+  style:
+    uses: huggingface/huggingface_hub/.github/workflows/style-bot-action.yml@e000c1c89c65aee188041723456ac3a479416d4c  # main
+    with:
+      python_quality_dependencies: "[quality]"
+    secrets:
+      bot_token: ${{ secrets.HF_STYLE_BOT_ACTION }}

.github/workflows/vllm_main_tests.yaml

@@ -0,0 +1,79 @@
+name: vLLM Main Branch Tests
+
+on:
+  schedule:
+    - cron: '0 2 * * 1'  # Every Monday at 2 AM UTC
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  test_vllm_main:
+    name: Test with vLLM main branch
+    runs-on: 'aws-g4dn-2xlarge-use1-public-80'
+    continue-on-error: true
+
+    steps:
+      - name: Install Git LFS
+        run: |
+          if ! command -v git-lfs &> /dev/null; then
+            sudo apt-get update && sudo apt-get install -y git-lfs
+            git lfs install
+          fi
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          lfs: true
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Install the project
+        run: uv sync --extra dev-gpu
+
+      - name: Install Python development headers
+        run: sudo apt-get update && sudo apt-get install -y python3.12-dev
+
+      - name: Cache CUDA Toolkit
+        id: cache-cuda
+        uses: actions/cache@v4
+        with:
+          path: /usr/local/cuda-12.8
+          key: cuda-toolkit-12-8-${{ runner.os }}
+
+      - name: Install CUDA Toolkit
+        if: steps.cache-cuda.outputs.cache-hit != 'true'
+        run: |
+          wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
+          sudo dpkg -i cuda-keyring_1.1-1_all.deb
+          sudo apt-get update
+          sudo apt-get install -y cuda-toolkit-12-8
+
+      - name: Setup CUDA environment
+        run: |
+          echo "CUDA_HOME=/usr/local/cuda-12.8" >> $GITHUB_ENV
+          echo "/usr/local/cuda-12.8/bin" >> $GITHUB_PATH
+
+      - name: Verify CUDA
+        run: |
+          nvidia-smi
+          nvcc --version
+
+      - name: Install vLLM from main branch
+        run: |
+          uv pip uninstall -y vllm || true
+          uv pip install git+https://github.com/vllm-project/vllm.git@main
+
+      - name: Get vLLM version
+        id: vllm-info
+        run: |
+          VERSION=$(uv run python -c "import vllm; print(vllm.__version__)")
+          echo "version=$VERSION" >> $GITHUB_OUTPUT
+          echo "Testing vLLM version: $VERSION"
+
+      - name: Run tests
+        run: uv run pytest --disable-pytest-warnings --runslow -v -s tests/slow_tests/test_vllm_model.py

docs/source/inspect-ai.mdx

@@ -0,0 +1,127 @@
+# Evaluate your model with Inspect-AI
+
+Pick the right benchmarks with our benchmark finder:
+Search by language, task type, dataset name, or keywords.
+
+> [!WARNING]
+> Not all tasks are compatible with inspect-ai's API as of yet, we are working on converting all of them !
+
+
+<iframe
+	src="https://openevals-open-benchmark-index.hf.space"
+	frameborder="0"
+	width="850"
+	height="450"
+></iframe>
+
+Once you've chosen a benchmark, run it with `lighteval eval`. Below are examples for common setups.
+
+### Examples
+
+1. Evaluate a model via Hugging Face Inference Providers.
+
+```bash
+lighteval eval "hf-inference-providers/openai/gpt-oss-20b" gpqa:diamond
+```
+
+2. Run multiple evals at the same time.
+
+```bash
+lighteval eval "hf-inference-providers/openai/gpt-oss-20b" gpqa:diamond,aime25
+```
+
+3. Compare providers for the same model.
+
+```bash
+lighteval eval \
+    hf-inference-providers/openai/gpt-oss-20b:fireworks-ai \
+    hf-inference-providers/openai/gpt-oss-20b:together \
+    hf-inference-providers/openai/gpt-oss-20b:nebius \
+    gpqa:diamond
+```
+
+You can also compare every providers serving one model in one line:
+
+```bash
+    hf-inference-providers/openai/gpt-oss-20b:all \
+    "lighteval|gpqa:diamond|0"
+```
+
+4. Evaluate a vLLM or SGLang model.
+
+```bash
+lighteval eval vllm/HuggingFaceTB/SmolLM-135M-Instruct gpqa:diamond
+```
+
+5. See the impact of few-shot on your model.
+
+```bash
+lighteval eval hf-inference-providers/openai/gpt-oss-20b "gsm8k|0,gsm8k|5"
+```
+
+6. Optimize custom server connections.
+
+```bash
+lighteval eval hf-inference-providers/openai/gpt-oss-20b gsm8k \
+    --max-connections 50 \
+    --timeout 30 \
+    --retry-on-error 1 \
+    --max-retries 1 \
+    --max-samples 10
+```
+
+7. Use multiple epochs for more reliable results.
+
+```bash
+lighteval eval hf-inference-providers/openai/gpt-oss-20b aime25 --epochs 16 --epochs-reducer "pass_at_4"
+```
+
+8. Push to the Hub to share results.
+
+```bash
+lighteval eval hf-inference-providers/openai/gpt-oss-20b hle \
+    --bundle-dir gpt-oss-bundle \
+    --repo-id OpenEvals/evals \
+    --max-samples 100
+```
+
+Resulting Space:
+
+<iframe
+	src="https://openevals-evals.static.hf.space"
+	frameborder="0"
+	width="850"
+	height="450"
+></iframe>
+
+9. Change model behaviour
+
+You can use any argument defined in inspect-ai's API.
+
+```bash
+lighteval eval hf-inference-providers/openai/gpt-oss-20b aime25 --temperature 0.1
+```
+
+10. Use model-args to use any inference provider specific argument.
+
+```bash
+lighteval eval google/gemini-2.5-pro aime25 --model-args location=us-east5
+```
+
+```bash
+lighteval eval openai/gpt-4o gpqa:diamond --model-args service_tier=flex,client_timeout=1200
+```
+
+
+LightEval prints a per-model results table:
+
+```
+Completed all tasks in 'lighteval-logs' successfully
+
+|                 Model                 |gpqa|gpqa:diamond|
+|---------------------------------------|---:|-----------:|
+|vllm/HuggingFaceTB/SmolLM-135M-Instruct|0.01|        0.01|
+
+results saved to lighteval-logs
+run "inspect view --log-dir lighteval-logs" to view the results
+```

docs/source/offline-evaluation.md

@@ -0,0 +1,46 @@
+# Offline evaluation using local data files
+
+If you are prototyping a task based on files that are not yet hosted on the
+Hub, you can take advantage of the `hf_data_files` argument to point lighteval
+at local JSON/CSV resources. This makes it easy to evaluate datasets that live
+in your repo or that are generated on the fly.
+
+Internally, `hf_data_files` is passed directly to the `data_files` parameter of `datasets.load_dataset` ([docs]((https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_dataset))).
+
+See [adding a custom task](adding-a-custom-task) for more information on how to create a custom task.
+
+```python
+from pathlib import Path
+
+from lighteval.metrics import Metrics
+from lighteval.tasks.lighteval_task import LightevalTaskConfig
+from lighteval.tasks.requests import Doc
+
+
+def local_prompt(line: dict, task_name: str) -> Doc:
+    return Doc(
+        task_name=task_name,
+        query=line["question"],
+        choices=line["choices"],
+        gold_index=line["answer"]
+    )
+
+
+local_data = Path(__file__).parent / "samples" / "faq.jsonl"
+
+local_task = LightevalTaskConfig(
+    name="faq_eval",
+    prompt_function=local_prompt,
+    hf_repo="json",  # Built-in streaming loader for json/jsonl files
+    hf_subset="default",
+    hf_data_files=str(local_data),  # Can also be a dict mapping split names to paths
+    evaluation_splits=["train"],
+    metrics=[Metrics.ACCURACY],
+)
+```
+
+Once the config is registered in `TASKS_TABLE`, running the task with
+`--custom-tasks path/to/your_file.py` will automatically load the local data
+files. You can also pass a dictionary to `hf_data_files` (e.g.
+`{"train": "train.jsonl", "validation": "val.jsonl"}`) to expose multiple
+splits.

examples/custom_tasks_templates/custom_yourbench_task_from_files.py

@@ -0,0 +1,105 @@
+# MIT License
+
+# Copyright (c) 2024 The HuggingFace Team
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import json
+import logging
+import tempfile
+from functools import partial
+from pathlib import Path
+
+from custom_yourbench_task_mcq import yourbench_prompt
+from datasets import Dataset, DatasetDict
+
+from lighteval.metrics.metrics import Metrics
+from lighteval.tasks.lighteval_task import LightevalTask, LightevalTaskConfig
+
+
+logger = logging.getLogger(__name__)
+
+save_dir = str(tempfile.mkdtemp())
+
+ds = DatasetDict(
+    {
+        "train": Dataset.from_dict(
+            {
+                "question": ["What is 2+2?", "Capital of France?"],
+                "choices": [["1", "2", "3", "4"], ["Paris", "Berlin", "Rome", "Madrid"]],
+                "gold": [[3], [0]],
+            }
+        )
+    }
+)
+
+
+CustomTaskConfig = partial(
+    LightevalTaskConfig,
+    prompt_function=yourbench_prompt,
+    hf_avail_splits=["train"],
+    evaluation_splits=["train"],
+    few_shots_split=None,
+    few_shots_select=None,
+    generation_size=16,
+    metrics=[Metrics.gpqa_instruct_metric],
+    version=0,
+)
+
+# Example 1: save to disk (huggingface format) ####
+
+ds.save_to_disk(save_dir)
+
+yourbench_mcq = CustomTaskConfig(
+    name="tiny_mcqa_dataset",
+    hf_repo="arrow",
+    hf_subset="default",
+    hf_data_files=f"{save_dir}/**/*.arrow",
+)
+
+task = LightevalTask(yourbench_mcq)
+eval_docs = task.eval_docs()
+
+print("\n>>READING TASK FROM ARROW<<")
+for doc in eval_docs:
+    print(doc)
+
+
+# Example 2: jsonlines format ####
+
+jsonl_path = Path(save_dir) / "train.jsonl"
+with open(jsonl_path, "w") as f:
+    for row in ds["train"]:
+        f.write(json.dumps(row) + "\n")
+
+yourbench_mcq = CustomTaskConfig(
+    name="tiny_mcqa_dataset",
+    hf_repo="json",
+    hf_subset="default",
+    hf_data_files=str(jsonl_path),
+)
+
+task = LightevalTask(yourbench_mcq)
+eval_docs = task.eval_docs()
+
+print("\n>>READING TASK FROM JSONLINES<<")
+for doc in eval_docs:
+    print(doc)
+
+# TASKS_TABLE = [yourbench_mcq]