Skip to content

Commit ac5ad68

Browse files
elronbandelelronbandel
authored
Fix examples (#1907)
* Fix tablebench Signed-off-by: elronbandel <[email protected]> * Update some examples Signed-off-by: elronbandel <[email protected]> * Unify dependencies installation Signed-off-by: elronbandel <[email protected]> * Update tests Signed-off-by: elronbandel <[email protected]> * Update Signed-off-by: elronbandel <[email protected]> * Fix torr Signed-off-by: elronbandel <[email protected]> * Update Signed-off-by: elronbandel <[email protected]> --------- Signed-off-by: elronbandel <[email protected]>
1 parent 076649a commit ac5ad68

25 files changed

+676
-159
lines changed

‎.github/workflows/catalog_preparation.yml‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -37,12 +37,8 @@ jobs:
3737
python-version: '3.9'
3838
cache: 'pip'
3939

40-
- run: echo "blis==0" > constraints.txt
41-
- run: curl -LsSf https://astral.sh/uv/install.sh | sh
42-
- run: uv pip install --upgrade --system torch --index-url https://download.pytorch.org/whl/cpu
43-
- run: uv pip install --system -c constraints.txt -e ".[tests]"
44-
- run: |
45-
pip install --only-binary :all: spacy
40+
- name: Install Dependencies
41+
run: bash utils/install.sh
4642

4743
- name: Hugging Face Login
4844
run: |

‎.github/workflows/examples_tests.yml‎

Lines changed: 2 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -38,19 +38,8 @@ jobs:
3838
with:
3939
python-version: '3.10'
4040

41-
- run: echo "blis==0" > constraints.txt
42-
- run: curl -LsSf https://astral.sh/uv/install.sh | sh
43-
- run: uv pip install --upgrade --system torch --index-url https://download.pytorch.org/whl/cpu
44-
- run: uv pip install --system -c constraints.txt -e ".[tests]"
45-
- run: |
46-
pip install --only-binary :all: spacy
47-
pip install networkx==3.2.1
48-
49-
- name: Install spacy (binary)
50-
run: |
51-
pip install --only-binary :all: spacy
52-
pip install networkx==3.2.1
53-
pip install datasets==3.5.1
41+
- name: Install Dependencies
42+
run: bash utils/install.sh
5443

5544
- name: Hugging Face Login
5645
run: |

‎.github/workflows/library_eager_execution_tests.yml‎

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -34,14 +34,9 @@ jobs:
3434
python-version: '3.9'
3535
cache: 'pip'
3636

37-
- run: echo "blis==0" > constraints.txt
38-
- run: curl -LsSf https://astral.sh/uv/install.sh | sh
39-
- run: uv pip install --upgrade --system torch --index-url https://download.pytorch.org/whl/cpu
40-
- run: uv pip install --system -c constraints.txt -e ".[tests]"
41-
- run: |
42-
pip install --only-binary :all: spacy
43-
pip install networkx==3.2.1
44-
37+
- name: Install Dependencies
38+
run: bash utils/install.sh
39+
4540
- run: pip install coverage[toml]
4641

4742
- name: Run Tests

‎.github/workflows/library_tests.yml‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -33,15 +33,8 @@ jobs:
3333
python-version: '3.9'
3434
cache: 'pip'
3535

36-
- run: echo "blis==0" > constraints.txt
37-
- run: curl -LsSf https://astral.sh/uv/install.sh | sh
38-
- run: uv pip install --upgrade --system torch --index-url https://download.pytorch.org/whl/cpu
39-
- run: uv pip install --system -c constraints.txt -e ".[tests]"
40-
- run: |
41-
pip install --only-binary :all: spacy
42-
pip install networkx==3.2.1
43-
44-
- run: pip install coverage[toml]
36+
- name: Install Dependencies
37+
run: bash utils/install.sh
4538

4639
- name: Run Tests
4740
run: coverage run -m unittest discover -s tests/library -p "test_*.py"

‎.github/workflows/performance.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,13 +30,13 @@ jobs:
3030
with:
3131
python-version: '3.10'
3232

33-
- name: Install Requirements
33+
- name: Install Dependencies
3434
run: |
3535
curl -LsSf https://astral.sh/uv/install.sh | sh
3636
uv pip install --system ".[tests,watsonx,inference-tests]"
3737
uv pip install --system litellm
3838
uv pip install --system diskcache
39-
39+
4040
- name: Hugging Face Login
4141
run: |
4242
for i in {1..5}; do

‎examples/evaluate_benchmark_with_custom_provider.py‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
11
from unitxt import evaluate, load_dataset
22
from unitxt.inference import CrossProviderInferenceEngine
33

4-
data = load_dataset(
5-
"benchmarks.glue[max_samples_per_subset=5, format=formats.chat_api, system_prompt=system_prompts.general.be_concise]",
4+
dataset = load_dataset(
5+
"benchmarks.glue",
6+
format="formats.chat_api",
7+
system_prompt="system_prompts.general.be_concise",
8+
max_samples_per_subset=5,
69
split="test",
710
use_cache=True,
811
)
912

1013
model = CrossProviderInferenceEngine(
11-
model="llama-3-8b-instruct", temperature=0.0, top_p=1.0, provider="watsonx"
14+
model="llama-3-2-3b-instruct", temperature=0.0, top_p=1.0, provider="watsonx"
1215
)
1316
"""
1417
We are using a CrossProviderInferenceEngine inference engine that supply api access to provider such as:
@@ -18,12 +21,12 @@
1821
about the the open ai api arguments the CrossProviderInferenceEngine follows.
1922
"""
2023

21-
predictions = model(data)
24+
predictions = model(dataset)
2225

23-
results = evaluate(predictions=predictions, data=data)
26+
results = evaluate(predictions=predictions, data=dataset)
2427

2528
print("Global Results:")
2629
print(results.global_scores.summary)
2730

28-
print("Instance Results:")
29-
print(results.instance_scores.summary)
31+
print("Subsets Results:")
32+
print(results.subsets_scores.summary)

‎examples/evaluate_llama_vision_benchmark.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@
66
allow_unverified_code=True,
77
):
88
test_dataset = load_dataset(
9-
"benchmarks.llama_vision[format=formats.chat_api,max_samples_per_subset=30]",
9+
"benchmarks.llama_vision",
10+
format="formats.chat_api",
11+
max_samples_per_subset=30,
1012
split="test",
1113
use_cache=False,
1214
)

‎examples/evaluate_vision_default_benchmark.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,9 @@
88
allow_unverified_code=True,
99
):
1010
test_dataset = load_dataset(
11-
"benchmarks.vision_default[format=formats.chat_api,max_samples_per_subset=30]",
11+
"benchmarks.vision_default",
12+
format="formats.chat_api",
13+
max_samples_per_subset=30,
1214
split="test",
1315
)
1416

‎examples/evaluate_vision_full_benchmark.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,9 @@
88
allow_unverified_code=True,
99
):
1010
test_dataset = load_dataset(
11-
"benchmarks.vision_full[format=formats.chat_api,max_samples_per_subset=512]",
11+
"benchmarks.vision_full",
12+
format="formats.chat_api",
13+
max_samples_per_subset=512,
1214
split="test",
1315
)
1416

‎prepare/cards/fin_qa.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@
2222
"train": url.format("train"),
2323
"validation": url.format("dev"),
2424
"test": url.format("test"),
25-
}
25+
},
26+
data_classification_policy=["public"],
2627
),
2728
preprocess_steps=[
2829
Copy(field="qa/question", to_field="question"),

0 commit comments

Comments
 (0)