Hedgehog.Xunit

Hedgehog with convenience attributes for xUnit.net.

Features

• Test method arguments generated with a custom GenX.auto...
• ...or with a custom Generator.
• Property.check called for each test.

Getting Started in C#

This readme is for F#. Go here for C# documentation.

Getting Started in F#

Install the Hedgehog.Xunit package from Visual Studio's Package Manager Console:

PM> Install-Package Hedgehog.Xunit

Suppose you have a test that uses Hedgehog.Experimental and looks similar to the following:

open Xunit
open Hedgehog
[<Fact>]
let ``Reversing a list twice yields the original list`` () =
  property {
    let! xs = GenX.auto<int list>
    return List.rev (List.rev xs) = xs
  } |> Property.check

Then using Hedgehog.Xunit, you can simplify the above test to

open Hedgehog.Xunit
[<Property>]
let ``Reversing a list twice yields the original list, with Hedgehog.Xunit`` (xs: int list) =
  List.rev (List.rev xs) = xs

Documentation

Hedgehog.Xunit provides the following attributes:

• [<Property>] Extends xUnit's Fact to call Hedgehog's property.
• [<Properties>] Configures all Property tagged tests in a module or class.
• GenAttribute Extend this attribute to set a parameter's generator.
• [<Recheck>] Run a test with a specific Size and Seed.

🔖 [<Property>]

Methods with [<Property>] have their arguments generated by GenX.auto.

type ``class with a test`` (output: Xunit.Abstractions.ITestOutputHelper) =
  [<Property>]
  let ``Can generate an int`` (i: int) =
    output.WriteLine $"Test input: {i}"
	
=== Output ===
Test input: 0
Test input: -1
Test input: 1
...
Test input: 522317518
Test input: 404306656
Test input: 1550509078

Property.check is called.

[<Property>]
let ``This test fails`` (b: bool) =
  b

=== Output ===
Hedgehog.FailedException: *** Failed! Falsifiable (after 2 tests):
(false)

If the test returns Async<_> or Task<_>, then Async.RunSynchronously is called, which blocks the thread. This may have significant performance implications as tests run 100 times by default.

[<Property>]
let ``Async with exception shrinks`` (i: int) = async {
  do! Async.Sleep 100
  if i > 10 then
    failwith "whoops!"
  }

=== Output ===
Hedgehog.FailedException: *** Failed! Falsifiable (after 12 tests):
(11)

A test returning a Result in an Error state will be treated as a failure.

[<Property>]
let ``Result with Error shrinks`` (i: int) =
  if i > 10 then
    Error ()
  else
    Ok ()

=== Output ===
Hedgehog.FailedException: *** Failed! Falsifiable (after 13 tests and 2 shrinks):
[11]

Tests returning Async<Result<_,_>> or Task<Result<_,_>> are run synchronously and are expected to be in the Ok state.

Tests returning a Property<unit> or Property<bool> will have Property.check automatically called:

[<Property>]
let ``returning a failing property<bool> with an external number gen fails and shrinks`` i = property {
  let! _50 = Gen.constant 50
  return i <= _50
}

=== Output ===
System.Exception: *** Failed! Falsifiable (after 23 tests and 5 shrinks):
[51]
50

⚙️ [<Property>] Configuration

[<Property>]'s constructor may take several arguments:

• AutoGenConfig and AutoGenConfigArgs: Set an AutoGenConfig to use when generating arguments.
• Tests: Specifies the number of tests to be run.
• Shrinks: Specifies the maximal number of shrinks that may run.
• Size: Sets the Size to a value for all runs.

The Property attribute extends Xunit.FactAttribute, so it may also take DisplayName, Skip, and Timeout.

🧰 AutoGenConfig and AutoGenConfigArgs

GenX.defaults is the AutoGenConfig used by default.

Here's how to add your own generators:

1. Create a class with a single static property or method that returns an instance of AutoGenConfig.
2. Provide the type of this class as an argument to [<Property>]. (This works around the constraint that Attribute parameters must be a constant.)

type AutoGenConfigContainer =
  static member __ =
    GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 13)

[<Property(typeof<AutoGenConfigContainer>)>]
let ``This test passes`` (i: int) =
  i = 13

If the method takes arguments, you must provide them using AutoGenConfigArgs.

type ConfigWithArgs =
  static member __ a b =
    GenX.defaults
    |> AutoGenConfig.addGenerator (Gen.constant a)
    |> AutoGenConfig.addGenerator (Gen.constant b)

[<Property(AutoGenConfig = typeof<ConfigWithArgs>, AutoGenConfigArgs = [|"foo"; 13|])>]
let ``This also passes`` s i =
  s = "foo" && i = 13

🧰 Tests

Specifies the number of tests to be run, though more or less may occur due to shrinking or early failure.

[<Property(3<tests>)>]
let ``This runs 3 times`` () =
  ()

🧰 Shrinks

Specifies the maximal number of shrinks that may run.

[<Property(Shrinks = 0<shrinks>)>]
let ``No shrinks occur`` i =
  if i > 50 then failwith "oops"

🧰 Size

Sets the Size to a value for all runs.

[<Property(Size = 2)>]
let ``"i" mostly ranges between -1 and 1`` i =
  printfn "%i" i

🔖 [<Properties>]

This optional attribute can decorate modules or classes. It sets default arguments for AutoGenConfig, AutoGenConfigArgs, Tests, Shrinks, and Size. These will be overridden by any explicit arguments on [<Property>].

type Int13A         = static member __ =          GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 13  ) |> AutoGenConfig.addGenerator (Gen.constant "A") 
type Int2718        = static member __ =          GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 2718) 
type Int2718Minimal = static member __ = AutoGenConfig.defaults |> AutoGenConfig.addGenerator (Gen.constant 2718) 
 
[<Properties(typeof<Int13A>, 1<tests>)>] 
module ``Module with <Properties> tests`` = 
 
  [<Property>] 
  let ``this passes and runs once`` (i: int) = 
    i = 13 
 
  [<Property(typeof<Int2718>, 2<tests>)>] 
  let ``this passes and runs twice`` (i: int) = 
    i = 2718 
 
  [<Property(typeof<Int2718Minimal>)>] 
  let ``the `Int13A` and `Int2718Minimal` configs are merged, so this passes`` (i: int, s: string) = 
    // `<Property>` uses the "minimal" `AutoGenConfig.defaults`. Using `GenX.defaults` would override the `A` in `Int13A`. 
    i = 2718 && s = "A" 

🔖 GenAttribute

To assign a generator to a test's parameter, extend GenAttribute and override Generator:

type Int5() =
  inherit GenAttribute<int>()
  override _.Generator = Gen.constant 5

[<Property>]
let ``can set parameter as 5`` ([<Int5>] i) =
  Assert.StrictEqual(5, i)

Here's a more complex example of GenAttribute that takes a parameter and overrides Property's AutoGenConfig:

type AutoGenConfigContainer =
  static member __ =
    GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 1)

type ConstInt(i: int)=
  inherit GenAttribute<int>()
  override _.Generator = Gen.constant i
    
[<Property(typeof<AutoGenConfigContainer>)>]
let ``GenAttribute overrides Property's AutoGenConfig`` (one, [<ConstInt 2>] two) =
  Assert.StrictEqual(1, one)
  Assert.StrictEqual(2, two)

🔖 [<Recheck>]

This optional method attribute invokes Property.recheck with the given Size and Seed. It must be used with Property.

[<Property>]
[<Recheck("44_13097736474433561873_6153509253234735533_")>]
let ``this passes`` i =
  i = 12345

Tips

Use named arguments to select the desired constructor overload.

[<Properties(Tests = 13<tests>, AutoGenConfig = typeof<AutoGenConfigContainer>)>]
module __ =
  [<Property(AutoGenConfig = typeof<AutoGenConfigContainer>, Tests = 2718<tests>, Skip = "just because")>]
  let ``Not sure why you'd do this, but okay`` () =
    ()

Consider extending PropertyAttribute or PropertiesAttribute to hardcode commonly used arguments.

type Int5() =
  inherit GenAttribute<int>()
  override _.Generator = Gen.constant 5
  
type Int13 = static member __ = GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant 13)

type PropertyInt13Attribute() = inherit PropertyAttribute(typeof<Int13>)
module __ =
  [<PropertyInt13>]
  let ``this passes`` (thirteen: int) ([<Int5>] five: int) =
    thirteen = 13 && five = 5

type PropertiesInt13Attribute() = inherit PropertiesAttribute(typeof<Int13>)
[<PropertiesInt13>]
module ___ =
  [<Property>]
  let ``this also passes`` (thirteen: int) ([<Int5>] five: int) =
    thirteen = 13 && five = 5

Known issue with generating a single tuple.

GenX.autoWith can generate a tuple.

[<Fact>]
let ``This passes`` () =
  Property.check <| property {
      let! a, b =
        GenX.defaults
        |> AutoGenConfig.addGenerator (Gen.constant (1, 2))
        |> GenX.autoWith<int*int>
      Assert.Equal(1, a)
      Assert.Equal(2, b)
  }

However, blindly converting the above test to Hedgehog.Xunit will fail.

type CustomTupleGen = static member __ = GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant (1, 2))
[<Property(typeof<CustomTupleGen>)>]
let ``This fails`` ((a,b) : int*int) =
  Assert.Equal(1, a)
  Assert.Equal(2, b)

This is because F# functions whose only parameter is a tuple will generate IL that un-tuples that parameter, yielding a function whose arity is the number of elements in the tuple. More concretely, this F#

let ``This fails`` ((a,b) : int*int) = ()

yields this IL (in debug mode)

.method public static 
    void 'This fails' (
        valuetype [System.Private.CoreLib]System.Int32 _arg1_0,
        valuetype [System.Private.CoreLib]System.Int32 _arg1_1
    ) cil managed 
{
    .maxstack 8
    IL_0000: ret
}

Due to this behavior Hedgehog.Xunit can't know that the original parameter was a tuple. It will therefore not use the registered tuple generator. A workaround is to pass a second (possibly unused) parameter.

type CustomTupleGen = static member __ = GenX.defaults |> AutoGenConfig.addGenerator (Gen.constant (1, 2))
[<Property(typeof<CustomTupleGen>)>]
let ``This passes`` (((a,b) : int*int), _: bool) =
  Assert.Equal(1, a)
  Assert.Equal(2, b)

The updated F#

let ``This passes`` (((a,b) : int*int), _: bool) = ()

yields this IL

.method public static 
    void 'This passes' (
        class [System.Private.CoreLib]System.Tuple`2<valuetype [System.Private.CoreLib]System.Int32, valuetype [System.Private.CoreLib]System.Int32> _arg1,
        valuetype [System.Private.CoreLib]System.Boolean _arg2
    ) cil managed 
{
    .maxstack 8
    IL_0000: ret
}

Source of IL.