Alt text generation should generate text in the same locale of the Site Language setting

[afercia]

Description

I quickly tested the alt text generation and noticed a few things that need improvements. I will try to create more issues but unfortunately I don't have much bandwidth to dedicate to this right now.

So far, to my understanding, alt text generation always generates text in English. That's less than ideal when the Site Language is other than English. By default, users would want any generated text to be in the language they chose.

As a sensible default, the generated text should be in the Site Language locale (to not be confused with the user locale).

There might be edge cases where the Site Language locale is in a certain language but editors would want to craft a post in a different language, e.g.:

• The whole post content is in another language (because of reasons).
• The post contains a section that is in another language, the section contains an image that might need a different language for the alt text.
• More potential edge cases.

At the very least, the ability instructions should instruct the agent to use the language of the site locale. This would be a reasonable default behavior.

To cover edge cases, users should always have a way to override the default and set a language of their choice.
I'm not sure what the best way to override the default locale would be. I'd think that rather than providing a select to choose the language, a Traslate ability would fit better with the AI abilities paradigm.

Step-by-step reproduction instructions

• In WP Admin > Settings > General, set the Site Language to a language other than English e.g. Italian.
• In WP Admin > Users > Profile, keep the Language to 'Site Default'.
• Create a post and add an image block with an image of a Pizza with pepperoni 🍕
• Use the ability to generate the alt text.
• Observe the generated alt text is in English.

Screenshot to illustrate the site language is in Italian (the user language is 'Site Default') however the generated alt text is in English.

Screenshots, screen recording, code snippet

No response

Environment info

No response

Please confirm that you have searched existing issues in the repo.

• Yes

Please confirm that you have tested with all plugins deactivated except the AI plugin.

• Yes

Please confirm which theme type you used for testing.

• Block
• Classic
• Hybrid (e.g. classic with theme.json)
• Not sure

[afercia]

Quickly testing, I just added two lines to the ability instructions:

- Locale: Generate the alt text in the Site Language locale
- Locale: When the image is within the post content, double check the post language and generate the alt text in that language if it differs from the Site Language

That seems to provide a better default output:

Screenshots to illustrate:

1
Site language in Italian, post content in Italian. The generated text is now in Italian:

2
Site language in Italian, actual post content in French. The generated text is now in French:

There may be better ways to achieve this.
I see some abilities use a method get_post_context() which could be useful.
I couldn't find anything related to the locale. Worth considering some helper funcion / Ability to get the site locale.
The core/get-site-info ability could be handy to retrieve the site locale but it seems a bit overkill and it's currently buggy. I created a Trac tickef for that, see https://core.trac.wordpress.org/ticket/64977
I'm not sure the locale should be forced in the implementation of the callback though because editors wouldn't have a way to override the default locale when needed.

[dkotter]

I've opened #357 to help with this. I pass the site locale into our system instructions and update those instructions to tell the LLM to match the language of the content given and if none is given, match the locale instead. In my testing this seemed to give better results, though noting we can't really control what the LLM returns to us so probably still times where it returns in the wrong language

[afercia]

though noting we can't really control what the LLM returns to us so probably still times where it returns in the wrong language

Yes that may happen sometimes. It's too early to make a decision and more testing on th efield is needed but I'd think users should be provided with a way to translate the suggested generated text anyways. Just mentioning as something to consider for the future.