Skip to content

Hedgehog with convenience attributes for xUnit.net

License

Notifications You must be signed in to change notification settings

hedgehogqa/fsharp-hedgehog-xunit

Folders and files

NameName
Last commit message
Last commit date

Latest commit

fad70bd Β· Jul 10, 2023
Jun 30, 2023
Jun 22, 2023
Jul 4, 2023
Dec 30, 2020
Jul 5, 2023
Jul 3, 2023
Jun 27, 2023
Jun 27, 2023
Jul 5, 2023
Jun 30, 2023
Feb 7, 2021
Jun 28, 2023
Jul 4, 2023
Jul 10, 2023

Repository files navigation

Hedgehog.Xunit

Coverage Status

Hedgehog with convenience attributes for xUnit.net.

Features

  • Test method arguments generated with a custom GenX.auto...
  • ...or with a custom Generator.
  • Property.check called for each test.

Getting Started in C#

This readme is for F#. Go here for C# documentation.

Getting Started in F#

Install the Hedgehog.Xunit package from Visual Studio's Package Manager Console:

PM> Install-Package Hedgehog.Xunit

Suppose you have a test that uses Hedgehog.Experimental and looks similar to the following:

open Xunit
open Hedgehog
[<Fact>]
let ``Reversing a list twice yields the original list`` () =
  property {
    let! xs = GenX.auto<int list>
    return List.rev (List.rev xs) = xs
  } |> Property.check

Then using Hedgehog.Xunit, you can simplify the above test to

open Hedgehog.Xunit
[<Property>]
let ``Reversing a list twice yields the original list, with Hedgehog.Xunit`` (xs: int list) =
  List.rev (List.rev xs) = xs

Documentation

Hedgehog.Xunit provides the following attributes:

πŸ”– [<Property>]

Methods with [<Property>] have their arguments generated by GenX.auto.

type ``class with a test`` (output: Xunit.Abstractions.ITestOutputHelper) =
  [<Property>]
  let ``Can generate an int`` (i: int) =
    output.WriteLine $"Test input: {i}"
	
=== Output ===
Test input: 0
Test input: -1
Test input: 1
...
Test input: 522317518
Test input: 404306656
Test input: 1550509078

Property.check is called.

[<Property>]
let ``This test fails`` (b: bool) =
  b

=== Output ===
Hedgehog.FailedException: *** Failed! Falsifiable (after 2 tests):
(false)

If the test returns Async<_> or Task<_>, then Async.RunSynchronously is called, which blocks the thread. This may have significant performance implications as tests run 100 times by default.

[<Property>]
let ``Async with exception shrinks`` (i: int) = async {
  do! Async.Sleep 100
  if i > 10 then
    failwith "whoops!"
  }

=== Output ===
Hedgehog.FailedException: *** Failed! Falsifiable (after 12 tests):
(11)

A test returning a Result in an Error state will be treated as a failure.

[<Property>]
let ``Result with Error shrinks`` (i: int) =
  if i > 10 then
    Error ()
  else
    Ok ()

=== Output ===
Hedgehog.FailedException: *** Failed! Falsifiable (after 13 tests and 2 shrinks):
[11]

Tests returning Async<Result<_,_>> or Task<Result<_,_>> are run synchronously and are expected to be in the Ok state.

Tests returning a Property<unit> or Property<bool> will have Property.check automatically called:

[<Property>]
let ``returning a failing property<bool> with an external number gen fails and shrinks`` i = property {
  let! _50 = Gen.constant 50
  return i <= _50
}

=== Output ===
System.Exception: *** Failed! Falsifiable (after 23 tests and 5 shrinks):
[51]
50

βš™οΈ [<Property>] Configuration

[<Property>]'s constructor may take several arguments:

The Property attribute extends Xunit.FactAttribute, so it may also take DisplayName, Skip, and Timeout.

🧰 AutoGenConfig and AutoGenConfigArgs

GenX.defaults is the AutoGenConfig used by default.

Here's how to add your own generators:

  1. Create a class with a single static property or method that returns an instance of AutoGenConfig.
  2. Provide the type of this class as an argument to [<Property>]. (This works around the constraint that Attribute parameters must be a constant.)
type AutoGenConfigContainer =
  static member __ =
    GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 13)

[<Property(typeof<AutoGenConfigContainer>)>]
let ``This test passes`` (i: int) =
  i = 13

If the method takes arguments, you must provide them using AutoGenConfigArgs.

type ConfigWithArgs =
  static member __ a b =
    GenX.defaults
    |> AutoGenConfig.addGenerator (Gen.constant a)
    |> AutoGenConfig.addGenerator (Gen.constant b)

[<Property(AutoGenConfig = typeof<ConfigWithArgs>, AutoGenConfigArgs = [|"foo"; 13|])>]
let ``This also passes`` s i =
  s = "foo" && i = 13

🧰 Tests

Specifies the number of tests to be run, though more or less may occur due to shrinking or early failure.

[<Property(3<tests>)>]
let ``This runs 3 times`` () =
  ()

🧰 Shrinks

Specifies the maximal number of shrinks that may run.

[<Property(Shrinks = 0<shrinks>)>]
let ``No shrinks occur`` i =
  if i > 50 then failwith "oops"

🧰 Size

Sets the Size to a value for all runs.

[<Property(Size = 2)>]
let ``"i" mostly ranges between -1 and 1`` i =
  printfn "%i" i

πŸ”– [<Properties>]

This optional attribute can decorate modules or classes. It sets default arguments for AutoGenConfig, AutoGenConfigArgs, Tests, Shrinks, and Size. These will be overridden by any explicit arguments on [<Property>].

type Int13   = static member __ = GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 13)
type Int2718 = static member __ = GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 2718)

[<Properties(typeof<Int13>, 1<tests>)>]
module ``Module with <Properties> tests`` =

  [<Property>]
  let ``this passes and runs once`` (i: int) =
    i = 13

  [<Property(typeof<Int2718>, 2<tests>)>]
  let ``this passes and runs twice`` (i: int) =
    i = 2718

πŸ”– GenAttribute

To assign a generator to a test's parameter, extend GenAttribute and override Generator:

type Int5() =
  inherit GenAttribute<int>()
  override _.Generator = Gen.constant 5

[<Property>]
let ``can set parameter as 5`` ([<Int5>] i) =
  Assert.StrictEqual(5, i)

Here's a more complex example of GenAttribute that takes a parameter and overrides Property's AutoGenConfig:

type AutoGenConfigContainer =
  static member __ =
    GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 1)

type ConstInt(i: int)=
  inherit GenAttribute<int>()
  override _.Generator = Gen.constant i
    
[<Property(typeof<AutoGenConfigContainer>)>]
let ``GenAttribute overrides Property's AutoGenConfig`` (one, [<ConstInt 2>] two) =
  Assert.StrictEqual(1, one)
  Assert.StrictEqual(2, two)

πŸ”– [<Recheck>]

This optional method attribute invokes Property.recheck with the given Size and Seed. It must be used with Property.

[<Property>]
[<Recheck("44_13097736474433561873_6153509253234735533_")>]
let ``this passes`` i =
  i = 12345

Tips

Use named arguments to select the desired constructor overload.

[<Properties(Tests = 13<tests>, AutoGenConfig = typeof<AutoGenConfigContainer>)>]
module __ =
  [<Property(AutoGenConfig = typeof<AutoGenConfigContainer>, Tests = 2718<tests>, Skip = "just because")>]
  let ``Not sure why you'd do this, but okay`` () =
    ()

Consider extending PropertyAttribute or PropertiesAttribute to hardcode commonly used arguments.

type Int5() =
  inherit GenAttribute<int>()
  override _.Generator = Gen.constant 5
  
type Int13 = static member __ = GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 13)

type PropertyInt13Attribute() = inherit PropertyAttribute(typeof<Int13>)
module __ =
  [<PropertyInt13>]
  let ``this passes`` (thirteen: int) ([<Int5>] five: int) =
    thirteen = 13 && five = 5

type PropertiesInt13Attribute() = inherit PropertiesAttribute(typeof<Int13>)
[<PropertiesInt13>]
module ___ =
  [<Property>]
  let ``this also passes`` (thirteen: int) ([<Int5>] five: int) =
    thirteen = 13 && five = 5
Known issue with generating a single tuple.

GenX.autoWith can generate a tuple.

[<Fact>]
let ``This passes`` () =
  Property.check <| property {
      let! a, b =
        GenX.defaults
        |> AutoGenConfig.addGenerator (Gen.constant (1, 2))
        |> GenX.autoWith<int*int>
      Assert.Equal(1, a)
      Assert.Equal(2, b)
  }

However, blindly converting the above test to Hedgehog.Xunit will fail.

type CustomTupleGen = static member __ = GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant (1, 2))
[<Property(typeof<CustomTupleGen>)>]
let ``This fails`` ((a,b) : int*int) =
  Assert.Equal(1, a)
  Assert.Equal(2, b)

This is because F# functions whose only parameter is a tuple will generate IL that un-tuples that parameter, yielding a function whose arity is the number of elements in the tuple. More concretely, this F#

let ``This fails`` ((a,b) : int*int) = ()

yields this IL (in debug mode)

.method public static 
    void 'This fails' (
        valuetype [System.Private.CoreLib]System.Int32 _arg1_0,
        valuetype [System.Private.CoreLib]System.Int32 _arg1_1
    ) cil managed 
{
    .maxstack 8
    IL_0000: ret
}

Due to this behavior Hedgehog.Xunit can't know that the original parameter was a tuple. It will therefore not use the registered tuple generator. A workaround is to pass a second (possibly unused) parameter.

type CustomTupleGen = static member __ = GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant (1, 2))
[<Property(typeof<CustomTupleGen>)>]
let ``This passes`` (((a,b) : int*int), _: bool) =
  Assert.Equal(1, a)
  Assert.Equal(2, b)

The updated F#

let ``This passes`` (((a,b) : int*int), _: bool) = ()

yields this IL

.method public static 
    void 'This passes' (
        class [System.Private.CoreLib]System.Tuple`2<valuetype [System.Private.CoreLib]System.Int32, valuetype [System.Private.CoreLib]System.Int32> _arg1,
        valuetype [System.Private.CoreLib]System.Boolean _arg2
    ) cil managed 
{
    .maxstack 8
    IL_0000: ret
}

Source of IL.