Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Should catch unexpected exceptions and fail the test suite #59

Open
metametadata opened this issue Aug 21, 2018 · 10 comments
Open

Should catch unexpected exceptions and fail the test suite #59

metametadata opened this issue Aug 21, 2018 · 10 comments

Comments

@metametadata
Copy link

Example:

(ns unit.a.b
  (:require 
    [clojure.test.check.generators :as gen]
    [com.gfredericks.test.chuck.clojure-test :as chuck]
    [clojure.test :as ct :refer :all]))

(deftest foo
  (chuck/checking
    "bar"
    10
    [x gen/int]
    (throw (ex-info "oops" {}))))

Expected:

<somehow report the exception with stacktrace>
...
0 failures, 1 errors.
Tests failed.

Actual:

Tests failed, smallest case: [{x 0}] 
Seed 1534810136116

Ran 1 tests containing 0 assertions.
0 failures, 0 errors.

My current workaround is to redefine the reporting method so that it always "fails" the test via (is false):

(defmethod ct/report ::chuck/shrunk
  [m]
  (is false "Property test failed, see logs for more info")
  (println
    ;"\nCase:\n" (with-out-str (pp/pprint (:fail m)))
    "\nShrunk:\n" (with-out-str (pp/pprint (-> m :shrunk :smallest)))
    (str "\n(num-tests: " (:num-tests m)
         ", total-nodes-visited: " (-> m :shrunk :total-nodes-visited)
         ", depth: " (-> m :shrunk :depth)
         ", failing-size: " (:failing-size m)
         ", seed: " (:seed m) ")"))
  (println "Error:" (pr-str (-> m :result-data :clojure.test.check.properties/error))))

Output:

FAIL in (foo) (b.clj:21)
bar
Property test failed, see logs for more info
expected: false
  actual: false

Shrunk:
 [{x 0}]
 
(num-tests: 1, total-nodes-visited: 0, depth: 0, failing-size: 0, seed: 1534810550729)
Error: #error {
 :cause "oops"
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "oops"
   :data {}
   :at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic "core.clj" 4739]
  [clojure.core$ex_info invoke "core.clj" 4739]
  ...]}

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
Tests failed.

Workaround output for the expected failure:

(deftest foo
  (chuck/checking
    "bar"
    1000
    [x gen/int]
    (is (not= 5 x))))
lein test :only unit.a.b/foo

FAIL in (foo) (b.clj:21)
bar
Property test failed, see logs for more info
expected: false
  actual: false

Shrunk:
 [{x 5}]
 
(num-tests: 114, total-nodes-visited: 3, depth: 0, failing-size: 113, seed: 1534810835278)
Error: nil

lein test :only unit.a.b/foo

FAIL in (foo) (b.clj:179)
bar
expected: (not= 5 x)
  actual: (not (not= 5 5))

Ran 1 tests containing 2 assertions.
2 failures, 0 errors.
Tests failed.
@gfredericks
Copy link
Owner

I'm having trouble reproducing this.

E.g.:

$ for file in $(ls); do echo === $file ===; cat $file | sed 's/^/> /'; done && clojure the-test.clj | tail
=== deps.edn ===
> {:deps {com.gfredericks/test.chuck {:mvn/version "0.2.9"}
>         org.clojure/test.check     {:mvn/version "0.9.0"}}}
=== the-test.clj ===
> (ns unit.a.b
>   (:require
>     [clojure.test.check.generators :as gen]
>     [com.gfredericks.test.chuck.clojure-test :as chuck]
>     [clojure.test :as ct :refer :all]))
> 
> (deftest foo
>   (chuck/checking
>     "bar"
>     10
>     [x gen/int]
>     (throw (ex-info "oops" {}))))
> 
> (run-tests)
  [clojure.main$script_opt invokeStatic "main.clj" 338]
  [clojure.main$script_opt invoke "main.clj" 333]
  [clojure.main$main invokeStatic "main.clj" 424]
  [clojure.main$main doInvoke "main.clj" 387]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.lang.Var applyTo "Var.java" 702]
  [clojure.main main "main.java" 37]]}))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.

@metametadata
Copy link
Author

I see, thanks. I used lein test inside the existing project, so maybe something else in lein user profile or project.clj causes the issue. I'll reply when I find the real cause.

@metametadata
Copy link
Author

metametadata commented Aug 21, 2018 •
edited
Loading

Looks like the problem is reproducible because of [org.clojure/test.check "0.10.0-alpha2"]. Exception is caught as expected in 0.9.0 and latest version 0.10.0-alpha3.

Unfortunately, the default reporting is not very helpful: it is very long and noisey (~300 lines in 0.9.0 and ~500 in 0.10.0-alpha3, mostly stacktraces) with smallest shrunk case somewhere in the middle. So I guess I still have to redefine the reporting method.

@metametadata
Copy link
Author

Ah, seems like the redefined ct/report is able to catch unexpected exception only in the "broken" [org.clojure/test.check "0.10.0-alpha2"]. It's not convenient, because sometimes the shrunk case is the reason for the exception, but it's very difficult to find :smallest in the output. Is it possible to customize exception reporting somehow or shall I issue a new feature request?

@gfredericks
Copy link
Owner

I'm confused by these two statements, which seem contradictory:

Exception is caught as expected in 0.9.0 and latest version 0.10.0-alpha3

seems like the redefined ct/report is able to catch unexpected exception only in the "broken" [org.clojure/test.check "0.10.0-alpha2"]

If you could post some code (and deps.edn) that doesn't do what you want and explain what you'd like it to do, I can sort this out quicker.

@metametadata
Copy link
Author

Sure, sorry for confusion.

Exception is caught as expected in 0.9.0 and latest version 0.10.0-alpha3.

The library catches the unexpected exception and reports it. So it is good.

seems like the redefined ct/report is able to catch unexpected exception only in the "broken" [org.clojure/test.check "0.10.0-alpha2"]

I meant that I can't redefine what the library outputs when it reports the unexepected error. Unless the broken "0.10.0-alpha2" is used (in such case my own custom defmethod ct/report ::chuck/shrunk catches the unexpected exception, for some reason).

The example:

; deps.clj
{:deps {com.gfredericks/test.chuck {:mvn/version "0.2.9"}
        org.clojure/test.check     {:mvn/version "0.10.0-alpha3"}}}
; the-test.clj
(ns the-test
  (:require
    [clojure.test.check.generators :as gen]
    [com.gfredericks.test.chuck.clojure-test :as chuck]
    [clojure.test :as ct :refer :all]))

(deftest foo
  (chuck/checking
    "bar"
    10
    [x gen/int]
    (throw (ex-info "oops" {}))))

(run-tests)

This is the output:

ᐅ clojure the-test.clj

Testing the-test

FAIL in (foo) (clojure_test.cljc:30)
bar
{:shrunk {:total-nodes-visited 0, :depth 0, :pass? false, :result #error {
 :cause oops
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message oops
   :data {}
   :at [clojure.core$ex_info invokeStatic core.clj 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic core.clj 4739]
  [clojure.core$ex_info invoke core.clj 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke the-test.clj 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic clojure_test.cljc 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke clojure_test.cljc 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke the-test.clj 8]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 657]
  [clojure.core$apply invoke core.clj 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke properties.cljc 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke properties.cljc 29]
  [clojure.test.check.rose_tree$fmap invokeStatic rose_tree.cljc 77]
  [clojure.test.check.rose_tree$fmap invoke rose_tree.cljc 73]
  [clojure.test.check.generators$fmap$fn__463 invoke generators.cljc 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke generators.cljc 58]
  [clojure.test.check.generators$call_gen invokeStatic generators.cljc 42]
  [clojure.test.check.generators$call_gen invoke generators.cljc 38]
  [clojure.test.check$quick_check invokeStatic check.cljc 126]
  [clojure.test.check$quick_check doInvoke check.cljc 59]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.RestFn applyTo RestFn.java 132]
  [clojure.core$apply invokeStatic core.clj 661]
  [clojure.core$apply invoke core.clj 652]
  [the_test$fn__3748$fn__3749 invoke the-test.clj 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic clojure_test.cljc 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke clojure_test.cljc 100]
  [the_test$fn__3748 invokeStatic the-test.clj 8]
  [the_test$fn__3748 invoke the-test.clj 7]
  [clojure.test$test_var$fn__9209 invoke test.clj 716]
  [clojure.test$test_var invokeStatic test.clj 716]
  [clojure.test$test_var invoke test.clj 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars$fn__9235 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars invokeStatic test.clj 730]
  [clojure.test$test_all_vars invokeStatic test.clj 736]
  [clojure.test$test_ns invokeStatic test.clj 757]
  [clojure.test$test_ns invoke test.clj 742]
  [clojure.core$map$fn__5587 invoke core.clj 2747]
  [clojure.lang.LazySeq sval LazySeq.java 40]
  [clojure.lang.LazySeq seq LazySeq.java 49]
  [clojure.lang.Cons next Cons.java 39]
  [clojure.lang.RT boundedLength RT.java 1785]
  [clojure.lang.RestFn applyTo RestFn.java 130]
  [clojure.core$apply invokeStatic core.clj 659]
  [clojure.test$run_tests invokeStatic test.clj 767]
  [clojure.test$run_tests doInvoke test.clj 767]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.test$run_tests invokeStatic test.clj 772]
  [clojure.test$run_tests invoke test.clj 767]
  [the_test$eval3773 invokeStatic the-test.clj 14]
  [the_test$eval3773 invoke the-test.clj 14]
  [clojure.lang.Compiler eval Compiler.java 7062]
  [clojure.lang.Compiler load Compiler.java 7514]
  [clojure.lang.Compiler loadFile Compiler.java 7452]
  [clojure.main$load_script invokeStatic main.clj 278]
  [clojure.main$script_opt invokeStatic main.clj 338]
  [clojure.main$script_opt invoke main.clj 333]
  [clojure.main$main invokeStatic main.clj 424]
  [clojure.main$main doInvoke main.clj 387]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 702]
  [clojure.main main main.java 37]]}, :result-data #:clojure.test.check.properties{:error #error {
 :cause oops
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message oops
   :data {}
   :at [clojure.core$ex_info invokeStatic core.clj 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic core.clj 4739]
  [clojure.core$ex_info invoke core.clj 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke the-test.clj 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic clojure_test.cljc 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke clojure_test.cljc 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke the-test.clj 8]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 657]
  [clojure.core$apply invoke core.clj 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke properties.cljc 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke properties.cljc 29]
  [clojure.test.check.rose_tree$fmap invokeStatic rose_tree.cljc 77]
  [clojure.test.check.rose_tree$fmap invoke rose_tree.cljc 73]
  [clojure.test.check.generators$fmap$fn__463 invoke generators.cljc 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke generators.cljc 58]
  [clojure.test.check.generators$call_gen invokeStatic generators.cljc 42]
  [clojure.test.check.generators$call_gen invoke generators.cljc 38]
  [clojure.test.check$quick_check invokeStatic check.cljc 126]
  [clojure.test.check$quick_check doInvoke check.cljc 59]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.RestFn applyTo RestFn.java 132]
  [clojure.core$apply invokeStatic core.clj 661]
  [clojure.core$apply invoke core.clj 652]
  [the_test$fn__3748$fn__3749 invoke the-test.clj 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic clojure_test.cljc 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke clojure_test.cljc 100]
  [the_test$fn__3748 invokeStatic the-test.clj 8]
  [the_test$fn__3748 invoke the-test.clj 7]
  [clojure.test$test_var$fn__9209 invoke test.clj 716]
  [clojure.test$test_var invokeStatic test.clj 716]
  [clojure.test$test_var invoke test.clj 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars$fn__9235 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars invokeStatic test.clj 730]
  [clojure.test$test_all_vars invokeStatic test.clj 736]
  [clojure.test$test_ns invokeStatic test.clj 757]
  [clojure.test$test_ns invoke test.clj 742]
  [clojure.core$map$fn__5587 invoke core.clj 2747]
  [clojure.lang.LazySeq sval LazySeq.java 40]
  [clojure.lang.LazySeq seq LazySeq.java 49]
  [clojure.lang.Cons next Cons.java 39]
  [clojure.lang.RT boundedLength RT.java 1785]
  [clojure.lang.RestFn applyTo RestFn.java 130]
  [clojure.core$apply invokeStatic core.clj 659]
  [clojure.test$run_tests invokeStatic test.clj 767]
  [clojure.test$run_tests doInvoke test.clj 767]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.test$run_tests invokeStatic test.clj 772]
  [clojure.test$run_tests invoke test.clj 767]
  [the_test$eval3773 invokeStatic the-test.clj 14]
  [the_test$eval3773 invoke the-test.clj 14]
  [clojure.lang.Compiler eval Compiler.java 7062]
  [clojure.lang.Compiler load Compiler.java 7514]
  [clojure.lang.Compiler loadFile Compiler.java 7452]
  [clojure.main$load_script invokeStatic main.clj 278]
  [clojure.main$script_opt invokeStatic main.clj 338]
  [clojure.main$script_opt invoke main.clj 333]
  [clojure.main$main invokeStatic main.clj 424]
  [clojure.main$main doInvoke main.clj 387]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 702]
  [clojure.main main main.java 37]]}}, :time-shrinking-ms 1, :smallest [{x 0}]}, :failed-after-ms 4, :num-tests 1, :seed 1536598020405, :fail [{x 0}], :result #error {
 :cause oops
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message oops
   :data {}
   :at [clojure.core$ex_info invokeStatic core.clj 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic core.clj 4739]
  [clojure.core$ex_info invoke core.clj 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke the-test.clj 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic clojure_test.cljc 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke clojure_test.cljc 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke the-test.clj 8]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 657]
  [clojure.core$apply invoke core.clj 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke properties.cljc 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke properties.cljc 29]
  [clojure.test.check.rose_tree$fmap invokeStatic rose_tree.cljc 77]
  [clojure.test.check.rose_tree$fmap invoke rose_tree.cljc 73]
  [clojure.test.check.generators$fmap$fn__463 invoke generators.cljc 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke generators.cljc 58]
  [clojure.test.check.generators$call_gen invokeStatic generators.cljc 42]
  [clojure.test.check.generators$call_gen invoke generators.cljc 38]
  [clojure.test.check$quick_check invokeStatic check.cljc 126]
  [clojure.test.check$quick_check doInvoke check.cljc 59]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.RestFn applyTo RestFn.java 132]
  [clojure.core$apply invokeStatic core.clj 661]
  [clojure.core$apply invoke core.clj 652]
  [the_test$fn__3748$fn__3749 invoke the-test.clj 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic clojure_test.cljc 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke clojure_test.cljc 100]
  [the_test$fn__3748 invokeStatic the-test.clj 8]
  [the_test$fn__3748 invoke the-test.clj 7]
  [clojure.test$test_var$fn__9209 invoke test.clj 716]
  [clojure.test$test_var invokeStatic test.clj 716]
  [clojure.test$test_var invoke test.clj 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars$fn__9235 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars invokeStatic test.clj 730]
  [clojure.test$test_all_vars invokeStatic test.clj 736]
  [clojure.test$test_ns invokeStatic test.clj 757]
  [clojure.test$test_ns invoke test.clj 742]
  [clojure.core$map$fn__5587 invoke core.clj 2747]
  [clojure.lang.LazySeq sval LazySeq.java 40]
  [clojure.lang.LazySeq seq LazySeq.java 49]
  [clojure.lang.Cons next Cons.java 39]
  [clojure.lang.RT boundedLength RT.java 1785]
  [clojure.lang.RestFn applyTo RestFn.java 130]
  [clojure.core$apply invokeStatic core.clj 659]
  [clojure.test$run_tests invokeStatic test.clj 767]
  [clojure.test$run_tests doInvoke test.clj 767]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.test$run_tests invokeStatic test.clj 772]
  [clojure.test$run_tests invoke test.clj 767]
  [the_test$eval3773 invokeStatic the-test.clj 14]
  [the_test$eval3773 invoke the-test.clj 14]
  [clojure.lang.Compiler eval Compiler.java 7062]
  [clojure.lang.Compiler load Compiler.java 7514]
  [clojure.lang.Compiler loadFile Compiler.java 7452]
  [clojure.main$load_script invokeStatic main.clj 278]
  [clojure.main$script_opt invokeStatic main.clj 338]
  [clojure.main$script_opt invoke main.clj 333]
  [clojure.main$main invokeStatic main.clj 424]
  [clojure.main$main doInvoke main.clj 387]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 702]
  [clojure.main main main.java 37]]}, :result-data #:clojure.test.check.properties{:error #error {
 :cause oops
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message oops
   :data {}
   :at [clojure.core$ex_info invokeStatic core.clj 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic core.clj 4739]
  [clojure.core$ex_info invoke core.clj 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke the-test.clj 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic clojure_test.cljc 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke clojure_test.cljc 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke the-test.clj 8]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 657]
  [clojure.core$apply invoke core.clj 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke properties.cljc 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke properties.cljc 29]
  [clojure.test.check.rose_tree$fmap invokeStatic rose_tree.cljc 77]
  [clojure.test.check.rose_tree$fmap invoke rose_tree.cljc 73]
  [clojure.test.check.generators$fmap$fn__463 invoke generators.cljc 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke generators.cljc 58]
  [clojure.test.check.generators$call_gen invokeStatic generators.cljc 42]
  [clojure.test.check.generators$call_gen invoke generators.cljc 38]
  [clojure.test.check$quick_check invokeStatic check.cljc 126]
  [clojure.test.check$quick_check doInvoke check.cljc 59]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.RestFn applyTo RestFn.java 132]
  [clojure.core$apply invokeStatic core.clj 661]
  [clojure.core$apply invoke core.clj 652]
  [the_test$fn__3748$fn__3749 invoke the-test.clj 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic clojure_test.cljc 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke clojure_test.cljc 100]
  [the_test$fn__3748 invokeStatic the-test.clj 8]
  [the_test$fn__3748 invoke the-test.clj 7]
  [clojure.test$test_var$fn__9209 invoke test.clj 716]
  [clojure.test$test_var invokeStatic test.clj 716]
  [clojure.test$test_var invoke test.clj 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars$fn__9235 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars invokeStatic test.clj 730]
  [clojure.test$test_all_vars invokeStatic test.clj 736]
  [clojure.test$test_ns invokeStatic test.clj 757]
  [clojure.test$test_ns invoke test.clj 742]
  [clojure.core$map$fn__5587 invoke core.clj 2747]
  [clojure.lang.LazySeq sval LazySeq.java 40]
  [clojure.lang.LazySeq seq LazySeq.java 49]
  [clojure.lang.Cons next Cons.java 39]
  [clojure.lang.RT boundedLength RT.java 1785]
  [clojure.lang.RestFn applyTo RestFn.java 130]
  [clojure.core$apply invokeStatic core.clj 659]
  [clojure.test$run_tests invokeStatic test.clj 767]
  [clojure.test$run_tests doInvoke test.clj 767]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.test$run_tests invokeStatic test.clj 772]
  [clojure.test$run_tests invoke test.clj 767]
  [the_test$eval3773 invokeStatic the-test.clj 14]
  [the_test$eval3773 invoke the-test.clj 14]
  [clojure.lang.Compiler eval Compiler.java 7062]
  [clojure.lang.Compiler load Compiler.java 7514]
  [clojure.lang.Compiler loadFile Compiler.java 7452]
  [clojure.main$load_script invokeStatic main.clj 278]
  [clojure.main$script_opt invokeStatic main.clj 338]
  [clojure.main$script_opt invoke main.clj 333]
  [clojure.main$main invokeStatic main.clj 424]
  [clojure.main$main doInvoke main.clj 387]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 702]
  [clojure.main main main.java 37]]}}, :failing-size 0, :pass? false}
expected: (not-exception? (:result result))
  actual: (not (not-exception? #error {
 :cause "oops"
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "oops"
   :data {}
   :at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic "core.clj" 4739]
  [clojure.core$ex_info invoke "core.clj" 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke "the-test.clj" 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic "clojure_test.cljc" 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke "clojure_test.cljc" 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke "the-test.clj" 8]
  [clojure.lang.AFn applyToHelper "AFn.java" 154]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 657]
  [clojure.core$apply invoke "core.clj" 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke "properties.cljc" 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke "properties.cljc" 29]
  [clojure.test.check.rose_tree$fmap invokeStatic "rose_tree.cljc" 77]
  [clojure.test.check.rose_tree$fmap invoke "rose_tree.cljc" 73]
  [clojure.test.check.generators$fmap$fn__463 invoke "generators.cljc" 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke "generators.cljc" 58]
  [clojure.test.check.generators$call_gen invokeStatic "generators.cljc" 42]
  [clojure.test.check.generators$call_gen invoke "generators.cljc" 38]
  [clojure.test.check$quick_check invokeStatic "check.cljc" 126]
  [clojure.test.check$quick_check doInvoke "check.cljc" 59]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [clojure.lang.AFn applyToHelper "AFn.java" 156]
  [clojure.lang.RestFn applyTo "RestFn.java" 132]
  [clojure.core$apply invokeStatic "core.clj" 661]
  [clojure.core$apply invoke "core.clj" 652]
  [the_test$fn__3748$fn__3749 invoke "the-test.clj" 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic "clojure_test.cljc" 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke "clojure_test.cljc" 100]
  [the_test$fn__3748 invokeStatic "the-test.clj" 8]
  [the_test$fn__3748 invoke "the-test.clj" 7]
  [clojure.test$test_var$fn__9209 invoke "test.clj" 716]
  [clojure.test$test_var invokeStatic "test.clj" 716]
  [clojure.test$test_var invoke "test.clj" 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke "test.clj" 734]
  [clojure.test$default_fixture invokeStatic "test.clj" 686]
  [clojure.test$default_fixture invoke "test.clj" 682]
  [clojure.test$test_vars$fn__9235 invoke "test.clj" 734]
  [clojure.test$default_fixture invokeStatic "test.clj" 686]
  [clojure.test$default_fixture invoke "test.clj" 682]
  [clojure.test$test_vars invokeStatic "test.clj" 730]
  [clojure.test$test_all_vars invokeStatic "test.clj" 736]
  [clojure.test$test_ns invokeStatic "test.clj" 757]
  [clojure.test$test_ns invoke "test.clj" 742]
  [clojure.core$map$fn__5587 invoke "core.clj" 2747]
  [clojure.lang.LazySeq sval "LazySeq.java" 40]
  [clojure.lang.LazySeq seq "LazySeq.java" 49]
  [clojure.lang.Cons next "Cons.java" 39]
  [clojure.lang.RT boundedLength "RT.java" 1785]
  [clojure.lang.RestFn applyTo "RestFn.java" 130]
  [clojure.core$apply invokeStatic "core.clj" 659]
  [clojure.test$run_tests invokeStatic "test.clj" 767]
  [clojure.test$run_tests doInvoke "test.clj" 767]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [clojure.test$run_tests invokeStatic "test.clj" 772]
  [clojure.test$run_tests invoke "test.clj" 767]
  [the_test$eval3773 invokeStatic "the-test.clj" 14]
  [the_test$eval3773 invoke "the-test.clj" 14]
  [clojure.lang.Compiler eval "Compiler.java" 7062]
  [clojure.lang.Compiler load "Compiler.java" 7514]
  [clojure.lang.Compiler loadFile "Compiler.java" 7452]
  [clojure.main$load_script invokeStatic "main.clj" 278]
  [clojure.main$script_opt invokeStatic "main.clj" 338]
  [clojure.main$script_opt invoke "main.clj" 333]
  [clojure.main$main invokeStatic "main.clj" 424]
  [clojure.main$main doInvoke "main.clj" 387]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.lang.Var applyTo "Var.java" 702]
  [clojure.main main "main.java" 37]]}))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.

I find this report to be very difficult to interpret, because it's long (~380 lines in 0.10.0-alpha3, ~230 in 0.9.0) and buries :smallest somewhere in the middle. The solution could be to allow customization of this report (e.g. also via defmethod).

@gfredericks
Copy link
Owner

I think this hinges on this function, which does something different depending on whether the tests threw an exception or not. The else branch of the if delegates to a special ct/report type of ::shrunk, while the then branch just relies on the default reporting of is. I think the then branch could be refactored to use a second special ct/report type (::error?) that includes the generated data in a similar way that the ::shrunk impl does.

I think this would give you what you need, does that sound right?

@gfredericks gfredericks reopened this Sep 23, 2018
@metametadata
Copy link
Author

Yes, sounds good!

@gfredericks
Copy link
Owner

Is this something you can help with with a pull request?

@metametadata
Copy link
Author

metametadata commented Sep 24, 2018 •
edited
Loading

Sure, will do in spare time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gfredericks @metametadata