Make tasty warn if there are tests with the same name

[phadej]

It's not uncommon when people make new tests by copying old ones and forget to update names, resulting in test-suites with duplicate names. This makes -p not being able to differentiate between them.

I think it would be a nice QoL if tasty at least warned about such "mistakes".

I just spent an hour or so looking at the wrong test and not understanding why it doesn't work: surprise, it did; i was looking at the wrong one.

[Bodigrim]

Yeah, I had a similar experience once. I'm in favor emitting such warnings to stderr.

I think a reasonable way to implement would be to scan testActions for duplicates in launchTestTree:

tasty/core/Test/Tasty/Run.hs

Lines 636 to 637 in cfb4b34

	(testActions, fins) <- createTestActions opts tree
	let NumThreads numThreads = lookupOption opts

I'm cautious about potential performance issues: it's not uncommon to have a test suite of 100000+ tests.

[phadej]

it's not uncommon to have a test suite of 100000+ tests.

I don't think that is an issue. In my opinion it would be enough to test for uniqueness shallowly in each test group. I'd argue if you same name sub-TestTrees, it's worth a warning already, no need to do full tree comparison, in other words

testGroup "top"
  [ testGroup "foo" tests1
  , testGroup "foo" tests2
  ]

is IMO already worth a warning even if names in tests1 and tests2 are disjoint.

[Bodigrim]

That's fair enough, I suppose. Still we need to bear in mind that TestTree has "invisible" layers which do not affect test names such as adjustOption or withResource.

I often use tasty as a general-purpose task manager for batch jobs, so I routinely end up with 10000+ tests in a single test group.