Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I am getting the error "TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')" and context_recall as 0 #1796

Open
tvsathish opened this issue Dec 26, 2024 · 0 comments
Labels
bug Something isn't working question Further information is requested

Comments

@tvsathish
Copy link

tvsathish commented Dec 26, 2024

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
Why am I getting this error "TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')" and why is my context_recall value showing up as 0.000000

Code Examples

import pprint

import pandas as pd
from langchain_openai import AzureChatOpenAI
from langchain_openai import AzureOpenAIEmbeddings
from ragas import SingleTurnSample, EvaluationDataset
from ragas import evaluate
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness, SemanticSimilarity


def empty_nan_value(cell_value):
    return '' if pd.isna(cell_value) else cell_value


def create_turn_sample(row):
    return SingleTurnSample(
        user_input=row['user_input'],
        retrieved_contexts=[empty_nan_value(row['context1']), empty_nan_value(row['context2']),
                            empty_nan_value(row['context3']),
                            empty_nan_value(row['context4'])],
        response=row['response'],
        reference=row['reference'])


df = pd.read_excel("<RESULTS_EXCEL_FILE>")

eval_dataset = EvaluationDataset([create_turn_sample(row) for index, row in df.iterrows()])

azure_config = {
    "base_url": "<AZURE_CHAT_COMPLETIONS_URL>",
    # your endpoint
    "model_deployment": "<AZURE_DEPLOYMENT>",  # your model deployment name
    "model_name": "gpt-4o"  # your model name
}

evaluator_llm = LangchainLLMWrapper(AzureChatOpenAI(
    openai_api_version="2024-08-01-preview",
    azure_endpoint=azure_config["base_url"],
    azure_deployment=azure_config["model_deployment"],
    model=azure_config["model_name"],
    validate_base_url=False,
))

metrics = [
    LLMContextRecall(llm=evaluator_llm),
    FactualCorrectness(llm=evaluator_llm),
    Faithfulness(llm=evaluator_llm)
]
results = evaluate(dataset=eval_dataset, metrics=metrics)
pprint.pprint(results)

df = results.to_pandas()
df.head()

`
AZURE_OPENAI_KEY is environment variable

Additional context
Output:

Evaluating: 14%|████████████████████████████ | 25/174 [01:55<02:04, 1.19it/s]Exception raised in Job[31]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 23%|████████████████████████████████████████████▊ | 40/174 [02:55<03:08, 1.41s/it]Exception raised in Job[34]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 25%|█████████████████████████████████████████████████▎ | 44/174 [02:58<01:56, 1.12it/s]Exception raised in Job[37]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 30%|██████████████████████████████████████████████████████████▎ | 52/174 [03:59<06:44, 3.31s/it]Exception raised in Job[58]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 34%|██████████████████████████████████████████████████████████████████ | 59/174 [04:17<06:32, 3.41s/it]Exception raised in Job[43]: TimeoutError()
Evaluating: 35%|████████████████████████████████████████████████████████████████████▎ | 61/174 [04:59<19:02, 10.11s/it]Exception raised in Job[46]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 41%|███████████████████████████████████████████████████████████████████████████████▌ | 71/174 [05:02<02:13, 1.29s/it]Exception raised in Job[70]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 41%|████████████████████████████████████████████████████████████████████████████████▋ | 72/174 [05:03<01:46, 1.05s/it]Exception raised in Job[73]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 44%|█████████████████████████████████████████████████████████████████████████████████████▏ | 76/174 [05:44<19:13, 11.77s/it]Exception raised in Job[61]: TimeoutError()
Evaluating: 56%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 98/174 [07:04<05:08, 4.06s/it]Exception raised in Job[91]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 58%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 101/174 [07:05<01:56, 1.59s/it]Exception raised in Job[100]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 67%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 116/174 [08:06<04:46, 4.93s/it]Exception raised in Job[109]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 67%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 117/174 [08:07<03:35, 3.78s/it]Exception raised in Job[115]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 75%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 131/174 [09:10<05:46, 8.06s/it]Exception raised in Job[136]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 79%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 137/174 [09:11<01:04, 1.74s/it]Exception raised in Job[142]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 80%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 140/174 [09:11<00:30, 1.10it/s]Exception raised in Job[139]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 84%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 146/174 [09:27<02:06, 4.53s/it]Exception raised in Job[124]: TimeoutError()
Evaluating: 88%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 153/174 [10:13<00:58, 2.77s/it]Exception raised in Job[148]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 90%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 157/174 [10:14<00:16, 1.01it/s]Exception raised in Job[157]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 91%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 158/174 [10:15<00:16, 1.01s/it]Exception raised in Job[169]: TypeError(ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'')
Evaluating: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 174/174 [12:19<00:00, 4.25s/it]
{'context_recall': 0.0000, 'factual_correctness': 0.1174, 'faithfulness': 0.0996}

@tvsathish tvsathish added the question Further information is requested label Dec 26, 2024
@dosubot dosubot bot added the bug Something isn't working label Dec 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant