Bug: input generator instance is reset by runner, losing state

[mattbit]

I have written a custom input generator, similar to a PersonaSimulator. I used the WithGeneratorMixin to provide the LLM generator instance.

The issue is that this generator is reset to default value at runtime:

my_simulator = MySimulator(generator=Generator(model="my-model"))

At runtime in the scenario, the simulator generator is reset to the default value from get_default_generator.

This happens because the instance is serialized and then deserialized at runtime, but the generator field is excluded from the serialized object. So it's lost.

Some questions:

• why is the input generator serialized and deserialized (can't we instead keep the same instance?)
• instead of serializing, shouldn't we use model_copy()
• do we really need to exclude the generator field from the serialized version? It should be serializable

In my case, the only solution was to set the default generator so that it would be set to the right value every time, but this is not practicel.

[linear]

OSS-28

[kevinmessiaen]

Looking at the code, the serialisation round trip seems to occur when outputs is not provided. It is done in order to set the output to the target while not updating the scenario so that running same scenario with multiple targets would work. As you mentioned the serialisation round trip is unnecessary and should be fixed and simplified with a `model_dump``

As for the generator, it should indeed be serialised. I think it was excluded at the beginning due to missing discriminated_base and not added back once the issue was resolved.

[abhigyan631]

Is this issue already fixed or in review? If not, can I please take it up?

[kevinmessiaen]

@abhigyan631 Yes it has been fixed and the PR is in review. Thanks for asking!