Skip to content

Add id and finish_reason to Gemini #1800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

davide-andreoli
Copy link

@davide-andreoli davide-andreoli commented May 21, 2025

Add id and finish_reason to Gemini

Description

Following PR#1761 , this update adds vendor_id and finish_reason to Gemini answers.

Below the details:

  • vendor_id: maps to responseId
  • finish_reason: finishReason for the first candidate of the output. The finish reason is added in vendor_details, following the attribute specification

The handling of finish_reasons aligns with the current approach for other response parts, where only the first candidate's reason is considered.

Testing

The changes were validated using my API key, and the behavior matches expectations.

Impact

This update impacts only Gemini Responses, but it is not a breaking change.

@davide-andreoli
Copy link
Author

Hello @DouweM, as agreed in PR#1761, here is the new PR.

@DouweM DouweM closed this May 21, 2025
@DouweM DouweM reopened this May 21, 2025
@DouweM
Copy link
Contributor

DouweM commented May 21, 2025

@davide-andreoli Thanks! Can you please have a look at the failing tests, and also implement this in the new models/google.py (assuming it makes sense there as well)?

@davide-andreoli
Copy link
Author

@DouweM The test failures are due to the vendor_details field.
Since gemini tests are configured to always send a finish_reason (which makes sense, as it's a non-optional field in the response) the vendor_details field is awlays present in the Gemini ModelResponses.
However, the snapshot comparison fails because vendor_details is marked with repr=False in the ModelResponse class.

I see two potential ways to address this:

  1. Remove repr=False from the vendor_details field in the ModelResponse class. This would be the cleanest solution, but I want to make sure it doesn't go against the original design intent
  2. Update the tests to exclude vendor_details before performing snapshot comparisons. This is a bit less elegant but still effective

Please let me know which approach you’d prefer.

And sure, no problem with adding the fields to models/google.py as well.

@DouweM
Copy link
Contributor

DouweM commented May 22, 2025

@davide-andreoli I had a chat about this with @dmontagu and we decided to make it so that dataclass fields with default values are excluded from reps: #1812. So once that merges (should be soon), you can merge/rebase, drop reps=False from vendor_details and hopefully it won't require too many examples to be updated!

@davide-andreoli davide-andreoli force-pushed the feat/add-id-to-gemini branch from 28dec6a to 273e783 Compare May 23, 2025 07:12
@davide-andreoli
Copy link
Author

Hello @DouweM, work is done and tests are passing.
As agreed I removed repr=False from vendor_details and added the fields in models/google.py as well.


for message in result.all_messages():
if isinstance(message, ModelResponse):
assert message.vendor_details is None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come we're not getting a finish reason here, even though we don't fake a response like in the Gemini test, and the other agent runs in this file do get one? 🤔

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're not getting a finish reason here because there is no finish reason in the cassette file for this test, which is test_googe_no_finish_reason.yaml.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davide-andreoli Just to make sure I understand correctly, did that response come without a finish reason, or was it manually edited out of the cassette? I worry about at some point regenerating the cassette and breaking the test. If it came without a finish reason, why this and not all of the other ones in the file?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DouweM you're right, it does not make sense.

The test coverage is flagging the check for a None finish reason, since the code verifies whether it’s set. However, in practice, it’s difficult to obtain an API response that omits this field, as it’s currently included by default.

I added the check to make the code more resilient, but given the circumstances, I see two options: either remove the check from the implementation or mock the client response in the test to simulate the missing field.

Let me know which approach you prefer.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DouweM you're right, it does not make sense.

The test coverage is flagging the check for a None finish reason, since the code verifies whether it’s set. However, in practice, it’s difficult to obtain an API response that omits this field, as it’s currently included by default.

I added the check to make the code more resilient, but given the circumstances, I see two options: either remove the check from the implementation or mock the client response in the test to simulate the missing field.

Let me know which approach you prefer.


for message in result.all_messages():
if isinstance(message, ModelResponse):
assert message.vendor_details is None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davide-andreoli Just to make sure I understand correctly, did that response come without a finish reason, or was it manually edited out of the cassette? I worry about at some point regenerating the cassette and breaking the test. If it came without a finish reason, why this and not all of the other ones in the file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants