Add generators to case functions.

[jenstroeger]

Following issue #229 and the discussion there, adding generators to case function requires discussion and some work:

• What about pytest marks ? marking a single subcase (1 yield) / marking all subcases (all of the yields) ?
• Should we consider the "return" of a generator ?
• How to manage pytest ids one by one ? All at once ?
• How to manage the compliance with the @case decorator ? (ids, filters, tags, etc.). Maybe the best way is to not create a new decorator but to have extra arguments in this decorator ?

Let the conversation begin 🤓

[smarie]

Thanks @jenstroeger ! I am curious to see the thoughts of the community of users here

[jenstroeger]

I think the feature itself would be useful and offer a lot of flexibility to improve automated, larger test sets — for example I’d create a generator which randomly picks a large number of example cases from a Faker provider.¹ How nice would that be 🤓

As for implementing that in your package and answering the above questions, I’m not fluent enough with the details of the pytest implementation… but hopefully the community can help out so this feature can come alive soon 🤞🏼

Footnotes

1. In fact, this feature request is somewhat motivated by the discussion here: HypothesisWorks/hypothesis#3614. ↩

[jenstroeger]

@smarie any more thoughts on this?

As for an example, I think using pytest-repeat combined with a case function that produces random data using Faker would be neat. Something like this:

# The case function returns a generator.
def case_names() -> Generator[str]:
    fake = Faker()
    while True:
        yield fake.name()

# We need to specify where to draw test cases from. The decorator order probably matters.
# Now the `cases` argument is a generator object from which values can be pulled.
@parametrize("name", cases=case_names())
@pytest.mark.repeat(10)
def test_something_faker(name: str) -> None:
    assert something_to_test(name)

So, basically, if the case function is a generator then just next() a value from it (I think even allow StopIteration to raise and fail testing) and pass that value to the test function. This way, I can use the repeat() decorator, or I can use the same case function multiple times and every time it produces a different value.