- Updated CodeVulnerability and UngroundedAttributes evaluators for RedTeam to use the binary true/false scoring pattern so their results align with service responses.
- Fixed NoneType error when generating usage summary in evaluation results.
- Fixed results for f1_score.
- Added detection and retry handling for network errors wrapped in generic exceptions with "Error sending prompt with conversation ID" message
- Fix results for ungrounded_attributes
- score_mode grader improvements
- fix for Red Team to ensure hate/unfairness evaluation rows populate when OneDP sync evaluators report results under the hate_unfairness metric name.
- TaskAdherenceEvaluator: treat tool definitions as optional so evaluations with only query/response inputs no longer raise “Either 'conversation' or individual inputs must be provided.”
- Handle input data for evaluation result when evaluators.
- Added
scenarioproperty to red team evaluation request to align scores with red team concepts of attack success.
- Added App Insights redaction for agent safety run telemetry so adversarial prompts are not stored in collected logs.
- Improved RedTeam coverage across risk sub-categories to ensure comprehensive security testing
- Made RedTeam's
AttackStrategy.Tenseseed prompts dynamic to allow use of this strategy with additional risk categories - Refactors error handling and result semantics in the RedTeam evaluation system to improve clarity and align with Attack Success Rate (ASR) conventions (passed=False means attack success)
- Fixed RedTeam evaluation error related to context handling for context-dependent risk categories
- Fixed RedTeam prompt application for model targets during Indirect Jailbreak XPIA (Cross-Platform Indirect Attack)
-
Updated
IndirectAttackrisk category for RedTeam toIndirectJailbreakto better reflect its purpose. This change allows users to apply cross-domain prompt injection (XPIA) attack strategies across all risk categories, enabling more comprehensive security testing of AI systems against indirect prompt injection attacks during red teaming. -
Added
TaskAdherence,SensitiveDataLeakage, andProhibitedActionsas cloud-only agent safety risk categories for red teaming. -
Updated all evaluators' output to be of the following schema:
gpt_{evaluator_name},{evaluator_name}: float score,{evaluator_name}_result: pass/fail based on threshold,{evaluator_name}_reason,{evaluator_name}_threshold{evaluator_name}_prompt_tokens,{evaluator_name}_completion_tokens,{evaluator_name}_total_tokens,{evaluator_name}_finish_reason{evaluator_name}_model: model used for evaluation{evaluator_name}_sample_input,{evaluator_name}_sample_output: input and output used for evaluation
This change standardizes the output format across all evaluators and follows OTel convention.
image_tagparameter inAzureOpenAIPythonGraderis now optional.
- **kwargs in an evaluator signature receives input columns that are not otherwise named in the evaluator's signature
- AOAI Graders now accept a "credential" parameter that can be used for authentication with an AzureOpenAIModelConfiguration
- Added
is_reasoning_modelparameter support toCoherenceEvaluator,FluencyEvaluator,SimilarityEvaluator,GroundednessEvaluator,RetrievalEvaluator, andRelevanceEvaluatorto enable reasoning model configuration for o1/o3 models.
- Support for multi-level nesting in OpenAI grader (experimental)
- Pinning duckdb version to 1.3.2 for redteam extra to fix error
TypeError: unhashable type: '_duckdb.typing.DuckDBPyType'
- Added support for user-supplied tags in the
evaluatefunction. Tags are key-value pairs that can be used for experiment tracking, A/B testing, filtering, and organizing evaluation runs. The function accepts atagsparameter. - Added support for user-supplied TokenCredentials with LLM based evaluators.
- Enhanced
GroundednessEvaluatorto support AI agent evaluation with tool calls. The evaluator now accepts agent response data containing tool calls and can extract context fromfile_searchtool results for groundedness assessment. This enables evaluation of AI agents that use tools to retrieve information and generate responses. Note: Agent groundedness evaluation is currently supported only when thefile_searchtool is used. - Added
languageparameter toRedTeamclass for multilingual red team scanning support. The parameter accepts values fromSupportedLanguagesenum including English, Spanish, French, German, Italian, Portuguese, Japanese, Korean, and Simplified Chinese, enabling red team attacks to be generated and conducted in multiple languages. - Added support for IndirectAttack and UngroundedAttributes risk categories in
RedTeamscanning. These new risk categories expand red team capabilities to detect cross-platform indirect attacks and evaluate ungrounded inferences about human attributes including emotional state and protected class information.
- Fixed issue where evaluation results were not properly aligned with input data, leading to incorrect metrics being reported.
- Deprecating
AdversarialSimulatorin favor of the AI Red Teaming Agent.AdversarialSimulatorwill be removed in the next minor release. - Moved retry configuration constants (
MAX_RETRY_ATTEMPTS,MAX_RETRY_WAIT_SECONDS,MIN_RETRY_WAIT_SECONDS) fromRedTeamclass to newRetryManagerclass for better code organization and configurability.
- Added
evaluate_queryparameter to all RAI service evaluators that can be passed as a keyword argument. This parameter controls whether queries are included in evaluation data when evaluating query-response pairs. Previously, queries were always included in evaluations. When set toTrue, both query and response will be evaluated; when set toFalse(default), only the response will be evaluated. This parameter is available across all RAI service evaluators includingContentSafetyEvaluator,ViolenceEvaluator,SexualEvaluator,SelfHarmEvaluator,HateUnfairnessEvaluator,ProtectedMaterialEvaluator,IndirectAttackEvaluator,CodeVulnerabilityEvaluator,UngroundedAttributesEvaluator,GroundednessProEvaluator, andEciEvaluator. Existing code that relies on queries being evaluated will need to explicitly setevaluate_query=Trueto maintain the previous behavior.
- Added support for Azure OpenAI Python grader via
AzureOpenAIPythonGraderclass, which serves as a wrapper around Azure Open AI Python grader configurations. This new grader object can be supplied to the mainevaluatemethod as if it were a normal callable evaluator. - Added
attack_success_thresholdsparameter toRedTeamclass for configuring custom thresholds that determine attack success. This allows users to set specific threshold values for each risk category, with scores greater than the threshold considered successful attacks (i.e. higher threshold means higher tolerance for harmful responses). - Enhanced threshold reporting in RedTeam results to include default threshold values when custom thresholds aren't specified, providing better transparency about the evaluation criteria used.
- Fixed red team scan
output_pathissue where individual evaluation results were overwriting each other instead of being preserved as separate files. Individual evaluations now create unique files while the user'soutput_pathis reserved for final aggregated results. - Significant improvements to TaskAdherence evaluator. New version has less variance, is much faster and consumes fewer tokens.
- Significant improvements to Relevance evaluator. New version has more concrete rubrics and has less variance, is much faster and consumes fewer tokens.
- The default engine for evaluation was changed from
promptflow(PFClient) to an in-SDK batch client (RunSubmitterClient)- Note: We've temporarily kept an escape hatch to fall back to the legacy
promptflowimplementation by setting_use_pf_client=Truewhen invokingevaluate(). This is due to be removed in a future release.
- Note: We've temporarily kept an escape hatch to fall back to the legacy
- Added support for Azure Open AI evaluation via
AzureOpenAIScoreModelGraderclass, which serves as a wrapper around Azure Open AI score model configurations. This new grader object can be supplied to the mainevaluatemethod as if it were a normal callable evaluator. - Added new experimental risk categories ProtectedMaterial and CodeVulnerability for redteam agent scan.
-
Significant improvements to IntentResolution evaluator. New version has less variance, is nearly 2x faster and consumes fewer tokens.
-
Fixes and improvements to ToolCallAccuracy evaluator. New version has less variance. and now works on all tool calls that happen in a turn at once. Previously, it worked on each tool call independently without having context on the other tool calls that happen in the same turn, and then aggregated the results to a score in the range [0-1]. The score range is now [1-5].
-
Fixed MeteorScoreEvaluator and other threshold-based evaluators returning incorrect binary results due to integer conversion of decimal scores. Previously, decimal scores like 0.9375 were incorrectly converted to integers (0) before threshold comparison, causing them to fail even when above the threshold. #41415
-
Added a new enum
ADVERSARIAL_QA_DOCUMENTSwhich moves all the "file_content" type prompts away fromADVERSARIAL_QAto the new enum -
AzureOpenAIScoreModelGraderevaluator now supportspass_thresholdparameter to set the minimum score required for a response to be considered passing. This allows users to define custom thresholds for evaluation results, enhancing flexibility in grading AI model responses.
- Introduces
AttackStrategy.MultiTurnandAttackStrategy.CrescendotoRedTeam. These strategies attack the target of aRedTeamscan over the course of multi-turn conversations.
- AdversarialSimulator in
ADVERSARIAL_CONVERSATIONmode was broken. It is now fixed.
- azure-ai-evaluation failed with module not found #40992
- New
<evaluator>.binary_aggregatefield added to evaluation result metrics. This field contains the aggregated binary evaluation results for each evaluator, providing a summary of the evaluation outcomes. - Added support for Azure Open AI evaluation via 4 new 'grader' classes, which serve as wrappers around Azure Open AI grader configurations. These new grader objects can be supplied to the main
evaluatemethod as if they were normal callable evaluators. The new classes are:- AzureOpenAIGrader (general class for experienced users)
- AzureOpenAILabelGrader
- AzureOpenAIStringCheckGrader
- AzureOpenAITextSimilarityGrader
- In the experimental RedTeam's scan method, the
data_onlyparam has been replaced withskip_evalsand if you do not want data to be uploaded, use theskip_uploadflag.
- Fixed error in
evaluatewhere data fields could not contain numeric characters. Previously, a data file with schema:throws error when passed into"query1": "some query", "response": "some response"evaluator_configas{"evaluator_name": {"column_mapping": {"query": "${data.query1}", "response": "${data.response}"}},}. Now, users may import data containing fields with numeric characters.
- New
RedTeamagent functionality to assess the safety and resilience of AI systems against adversarial prompt attacks
-
Enhanced binary evaluation results with customizable thresholds
- Added threshold support for QA and ContentSafety evaluators
- Evaluation results now include both the score and threshold values
- Configurable threshold parameter allows custom binary classification boundaries
- Default thresholds provided for backward compatibility
- Quality evaluators use "higher is better" scoring (score ≥ threshold is positive)
- Content safety evaluators use "lower is better" scoring (score ≤ threshold is positive)
-
New Built-in evaluator called CodeVulnerabilityEvaluator is added.
- It provides capabilities to identify the following code vulnerabilities.
- path-injection
- sql-injection
- code-injection
- stack-trace-exposure
- incomplete-url-substring-sanitization
- flask-debug
- clear-text-logging-sensitive-data
- incomplete-hostname-regexp
- server-side-unvalidated-url-redirection
- weak-cryptographic-algorithm
- full-ssrf
- bind-socket-all-network-interfaces
- client-side-unvalidated-url-redirection
- likely-bugs
- reflected-xss
- clear-text-storage-sensitive-data
- tarslip
- hardcoded-credentials
- insecure-randomness
- It also supports multiple coding languages such as (Python, Java, C++, C#, Go, Javascript, SQL)
- It provides capabilities to identify the following code vulnerabilities.
-
New Built-in evaluator called UngroundedAttributesEvaluator is added.
-
It evaluates ungrounded inference of human attributes for a given query, response, and context for a single-turn evaluation only,
-
where query represents the user query and response represents the AI system response given the provided context.
-
Ungrounded Attributes checks for whether a response is first, ungrounded, and checks if it contains information about protected class
-
or emotional state of a person.
-
It identifies the following attributes:
- emotional_state
- protected_class
- groundedness
-
-
New Built-in evaluators for Agent Evaluation (Preview)
- IntentResolutionEvaluator - Evaluates the intent resolution of an agent's response to a user query.
- ResponseCompletenessEvaluator - Evaluates the response completeness of an agent's response to a user query.
- TaskAdherenceEvaluator - Evaluates the task adherence of an agent's response to a user query.
- ToolCallAccuracyEvaluator - Evaluates the accuracy of tool calls made by an agent in response to a user query.
- Fixed error in
GroundednessProEvaluatorwhen handling non-numeric values like "n/a" returned from the service. - Uploading local evaluation results from
evaluatewith the same run name will no longer result in each online run sharing (and bashing) result files.
- Multimodal specific evaluators
ContentSafetyMultimodalEvaluator,ViolenceMultimodalEvaluator,SexualMultimodalEvaluator,SelfHarmMultimodalEvaluator,HateUnfairnessMultimodalEvaluatorandProtectedMaterialMultimodalEvaluatorhas been removed. Please useContentSafetyEvaluator,ViolenceEvaluator,SexualEvaluator,SelfHarmEvaluator,HateUnfairnessEvaluatorandProtectedMaterialEvaluatorinstead. - Metric name in ProtectedMaterialEvaluator's output is changed from
protected_material.fictional_characters_labeltoprotected_material.fictional_characters_defect_rate. It's now consistent with other evaluator's metric names (ending with_defect_rate).
- CSV files are now supported as data file inputs with
evaluate()API. The CSV file should have a header row with column names that match thedataandtargetfields in theevaluate()method and the filename should be passed as thedataparameter. Column name 'Conversation' in CSV file is not fully supported yet.
ViolenceMultimodalEvaluator,SexualMultimodalEvaluator,SelfHarmMultimodalEvaluator,HateUnfairnessMultimodalEvaluatorandProtectedMaterialMultimodalEvaluatorwill be removed in next release.
- Removed
[remote]extra. This is no longer needed when tracking results in Azure AI Studio. - Fixed
AttributeError: 'NoneType' object has no attribute 'get'while running simulator with 1000+ results - Fixed the non adversarial simulator to run in task-free mode
- Content safety evaluators (violence, self harm, sexual, hate/unfairness) return the maximum result as the main score when aggregating per-turn evaluations from a conversation into an overall evaluation score. Other conversation-capable evaluators still default to a mean for aggregation.
- Fixed bug in non adversarial simulator sample where
tasksundefined
- Changed minimum required python version to use this package from 3.8 to 3.9
- Stop dependency on the local promptflow service. No promptflow service will automatically start when running evaluation.
- Evaluators internally allow for custom aggregation. However, this causes serialization failures if evaluated while the
environment variable
AI_EVALS_BATCH_USE_ASYNCis set to false.
- Added image support in
ContentSafetyEvaluator,ViolenceEvaluator,SexualEvaluator,SelfHarmEvaluator,HateUnfairnessEvaluatorandProtectedMaterialEvaluator. Provide image URLs or base64 encoded images inconversationinput for image evaluation. See below for an example:
evaluator = ContentSafetyEvaluator(credential=azure_cred, azure_ai_project=project_scope)
conversation = {
"messages": [
{
"role": "system",
"content": [
{"type": "text", "text": "You are an AI assistant that understands images."}
],
},
{
"role": "user",
"content": [
{"type": "text", "text": "Can you describe this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.britannica.com/68/178268-050-5B4E7FB6/Tom-Cruise-2013.jpg"
},
},
],
},
{
"role": "assistant",
"content": [
{
"type": "text",
"text": "The image shows a man with short brown hair smiling, wearing a dark-colored shirt.",
}
],
},
]
}
print("Calling Content Safety Evaluator for multi-modal")
score = evaluator(conversation=conversation)- Please switch to generic evaluators for image evaluations as mentioned above.
ContentSafetyMultimodalEvaluator,ContentSafetyMultimodalEvaluatorBase,ViolenceMultimodalEvaluator,SexualMultimodalEvaluator,SelfHarmMultimodalEvaluator,HateUnfairnessMultimodalEvaluatorandProtectedMaterialMultimodalEvaluatorwill be deprecated in the next release.
- Removed
[remote]extra. This is no longer needed when tracking results in Azure AI Foundry portal. - Fixed
AttributeError: 'NoneType' object has no attribute 'get'while running simulator with 1000+ results
- Removing
azure-ai-inferenceas dependency. - Fixed
AttributeError: 'NoneType' object has no attribute 'get'while running simulator with 1000+ results
- The
parallelparameter has been removed from composite evaluators:QAEvaluator,ContentSafetyChatEvaluator, andContentSafetyMultimodalEvaluator. To control evaluator parallelism, you can now use the_parallelkeyword argument, though please note that this private parameter may change in the future. - Parameters
query_response_generating_prompty_kwargsanduser_simulator_prompty_kwargshave been renamed toquery_response_generating_prompty_optionsanduser_simulator_prompty_optionsin the Simulator's call method.
- Fixed an issue where the
output_pathparameter in theevaluateAPI did not support relative path. - Output of adversarial simulators are of type
JsonLineListand the helper functionto_eval_qr_json_linesnow outputs context from both user and assistant turns along withcategoryif it exists in the conversation - Fixed an issue where during long-running simulations, API token expires causing "Forbidden" error. Instead, users can now set an environment variable
AZURE_TOKEN_REFRESH_INTERVALto refresh the token more frequently to prevent expiration and ensure continuous operation of the simulation. - Fixed an issue with the
ContentSafetyEvaluatorthat caused parallel execution of sub-evaluators to fail. Parallel execution is now enabled by default again, but can still be disabled via the '_parallel' boolean keyword argument during class initialization. - Fix
evaluatefunction not producing aggregated metrics if ANY values to be aggregated were None, NaN, or otherwise difficult to process. Such values are ignored fully, so the aggregated metric of[1, 2, 3, NaN]would be 2, not 1.5.
- Refined error messages for serviced-based evaluators and simulators.
- Tracing has been disabled due to Cosmos DB initialization issue.
- Introduced environment variable
AI_EVALS_DISABLE_EXPERIMENTAL_WARNINGto disable the warning message for experimental features. - Changed the randomization pattern for
AdversarialSimulatorsuch that there is an almost equal number of Adversarial harm categories (e.g. Hate + Unfairness, Self-Harm, Violence, Sex) represented in theAdversarialSimulatoroutputs. Previously, for 200max_simulation_resultsa user might see 140 results belonging to the 'Hate + Unfairness' category and 40 results belonging to the 'Self-Harm' category. Now, user will see 50 results for each of Hate + Unfairness, Self-Harm, Violence, and Sex. - For the
DirectAttackSimulator, the prompt templates used to generate simulated outputs for each Adversarial harm category will no longer be in a randomized order by default. To override this behavior, passrandomize_order=Truewhen you call theDirectAttackSimulator, for example:
adversarial_simulator = DirectAttackSimulator(azure_ai_project=azure_ai_project, credential=DefaultAzureCredential())
outputs = asyncio.run(
adversarial_simulator(
scenario=scenario,
target=callback,
randomize_order=True
)
)- Added
GroundednessProEvaluator, which is a service-based evaluator for determining response groundedness. - Groundedness detection in Non Adversarial Simulator via query/context pairs
import importlib.resources as pkg_resources
package = "azure.ai.evaluation.simulator._data_sources"
resource_name = "grounding.json"
custom_simulator = Simulator(model_config=model_config)
conversation_turns = []
with pkg_resources.path(package, resource_name) as grounding_file:
with open(grounding_file, "r") as file:
data = json.load(file)
for item in data:
conversation_turns.append([item])
outputs = asyncio.run(custom_simulator(
target=callback,
conversation_turns=conversation_turns,
max_conversation_turns=1,
))- Adding evaluator for multimodal use cases
- Renamed environment variable
PF_EVALS_BATCH_USE_ASYNCtoAI_EVALS_BATCH_USE_ASYNC. RetrievalEvaluatornow requires acontextinput in addition toqueryin single-turn evaluation.RelevanceEvaluatorno longer takescontextas an input. It now only takesqueryandresponsein single-turn evaluation.FluencyEvaluatorno longer takesqueryas an input. It now only takesresponsein single-turn evaluation.- AdversarialScenario enum does not include
ADVERSARIAL_INDIRECT_JAILBREAK, invoking IndirectJailbreak or XPIA should be done withIndirectAttackSimulator - Outputs of
SimulatorandAdversarialSimulatorpreviously hadto_eval_qa_json_linesand now hasto_eval_qr_json_lines. Whereto_eval_qa_json_lineshad:
{"question": <user_message>, "answer": <assistant_message>}to_eval_qr_json_lines now has:
{"query": <user_message>, "response": assistant_message}- Non adversarial simulator works with
gpt-4omodels using thejson_schemaresponse format - Fixed an issue where the
evaluateAPI would fail with "[WinError 32] The process cannot access the file because it is being used by another process" when venv folder and target function file are in the same directory. - Fix evaluate API failure when
trace.destinationis set tonone - Non adversarial simulator now accepts context from the callback
-
Improved error messages for the
evaluateAPI by enhancing the validation of input parameters. This update provides more detailed and actionable error descriptions. -
GroundednessEvaluatornow supportsqueryas an optional input in single-turn evaluation. Ifqueryis provided, a different prompt template will be used for the evaluation. -
To align with our support of a diverse set of models, the following evaluators will now have a new key in their result output without the
gpt_prefix. To maintain backwards compatibility, the old key with thegpt_prefix will still be present in the output; however, it is recommended to use the new key moving forward as the old key will be deprecated in the future.CoherenceEvaluatorRelevanceEvaluatorFluencyEvaluatorGroundednessEvaluatorSimilarityEvaluatorRetrievalEvaluator
-
The following evaluators will now have a new key in their result output including LLM reasoning behind the score. The new key will follow the pattern "<metric_name>_reason". The reasoning is the result of a more detailed prompt template being used to generate the LLM response. Note that this requires the maximum number of tokens used to run these evaluators to be increased.
Evaluator New max_tokenfor GenerationCoherenceEvaluator800 RelevanceEvaluator800 FluencyEvaluator800 GroundednessEvaluator800 RetrievalEvaluator1600 -
Improved the error message for storage access permission issues to provide clearer guidance for users.
- Removed
numpydependency. All NaN values returned by the SDK have been changed to fromnumpy.nantomath.nan. credentialis now required to be passed in for all content safety evaluators andProtectedMaterialsEvaluator.DefaultAzureCredentialwill no longer be chosen if a credential is not passed.- Changed package extra name from "pf-azure" to "remote".
- Adversarial Conversation simulations would fail with
Forbidden. Added logic to re-fetch token in the exponential retry logic to retrive RAI Service response. - Fixed an issue where the Evaluate API did not fail due to missing inputs when the target did not return columns required by the evaluators.
- Enhance the error message to provide clearer instruction when required packages for the remote tracking feature are missing.
- Print the per-evaluator run summary at the end of the Evaluate API call to make troubleshooting row-level failures easier.
- Added
typefield toAzureOpenAIModelConfigurationandOpenAIModelConfiguration - The following evaluators now support
conversationas an alternative input to their usual single-turn inputs:ViolenceEvaluatorSexualEvaluatorSelfHarmEvaluatorHateUnfairnessEvaluatorProtectedMaterialEvaluatorIndirectAttackEvaluatorCoherenceEvaluatorRelevanceEvaluatorFluencyEvaluatorGroundednessEvaluator
- Surfaced
RetrievalScoreEvaluator, formally an internal part ofChatEvaluatoras a standalone conversation-only evaluator.
- Removed
ContentSafetyChatEvaluatorandChatEvaluator - The
evaluator_configparameter ofevaluatenow maps in evaluator name to a dictionaryEvaluatorConfig, which is aTypedDict. Thecolumn_mappingbetweendataortargetand evaluator field names should now be specified inside this new dictionary:
Before:
evaluate(
...,
evaluator_config={
"hate_unfairness": {
"query": "${data.question}",
"response": "${data.answer}",
}
},
...
)After
evaluate(
...,
evaluator_config={
"hate_unfairness": {
"column_mapping": {
"query": "${data.question}",
"response": "${data.answer}",
}
}
},
...
)- Simulator now requires a model configuration to call the prompty instead of an Azure AI project scope. This enables the usage of simulator with Entra ID based auth. Before:
azure_ai_project = {
"subscription_id": os.environ.get("AZURE_SUBSCRIPTION_ID"),
"resource_group_name": os.environ.get("RESOURCE_GROUP"),
"project_name": os.environ.get("PROJECT_NAME"),
}
sim = Simulator(azure_ai_project=azure_ai_project, credentails=DefaultAzureCredentials())After:
model_config = {
"azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"),
"azure_deployment": os.environ.get("AZURE_DEPLOYMENT"),
}
sim = Simulator(model_config=model_config)If api_key is not included in the model_config, the prompty runtime in promptflow-core will pick up DefaultAzureCredential.
- Fixed issue where Entra ID authentication was not working with
AzureOpenAIModelConfiguration
dataandevaluatorsare now required keywords inevaluate.
- The
syntheticnamespace has been renamed tosimulator, and sub-namespaces under this module have been removed - The
evaluateandevaluatorsnamespaces have been removed, and everything previously exposed in those modules has been added to the root namespaceazure.ai.evaluation - The parameter name
project_scopein content safety evaluators have been renamed toazure_ai_projectfor consistency with evaluate API and simulators. - Model configurations classes are now of type
TypedDictand are exposed in theazure.ai.evaluationmodule instead of coming frompromptflow.core. - Updated the parameter names for
questionandanswerin built-in evaluators to more generic terms:queryandresponse.
- First preview
- This package is port of
promptflow-evals. New features will be added only to this package moving forward. - Added a
TypedDictforAzureAIProjectthat allows for better intellisense and type checking when passing in project information