Standarize the notation of subtests in TAP

[kinow]

This issue was originally reported in TestAnything/testanything.github.io#20, and the conversation has been moved here since it is related to the TAP specification.

It concerns the subtests in TAP and a formal standard to be followed by TAP consumer and producers. In the latest TAP specification (TAP 13) there is no formal definition of subtests. There were proposals and draft documents on this, some tools like Test::More and tap4j support with limitations.

Please feel free to chime in and suggest enhancements for this standard. A message is being written to the mailing list pointing to this issue too. The mailing list has several users that participated in the IETF draft and have lots of experience with TAP and its implementations.

/cc @Leont @xmojmr

[jonathanKingston]

One thing that did not seem to be mentioned on the TAP mailing list was that the potential for a sub test to look like this:

ok 1 - something
ok 1 -     something
ok 2 - something

or

ok 1 - something
ok 1 -- something
ok 2 - something

I believe most libraries would support a failure just fine for both of those, its simpler that requiring an update to the libraries and forces stricter reasoning of tests which is a bonus.

[Leont]

We really want to have this. I'm not fond of the current mechanism, but given it's out there and there's a lot of stuff outputting it already it may be a passed station.

[Leont]

I believe most libraries would support a failure just fine for both of those, its simpler that requiring an update to the libraries and forces stricter reasoning of tests which is a bonus.

Both parsers I've worked on would consider those outputs an error. Subtests really should be something legacy parsers consider comments, which fortunately still gives a lot of space.

[kinow]

@jonathanKingston I'm happy with the old proposal for subtests

TAP 13
1..1
ok 1 - Subtest 1
    ok 1 - subtest 2a
    ok 2 - subtest 2b

I'd be fine with other formats, but since there are tools in Perl, Java, Node, which output something similar to that, I'd be more keen to simply formalize it in a TAP 14 specification.

@Leont

We really want to have this. I'm not fond of the current mechanism, but given it's out there and there's a lot of stuff outputting it already it may be a passed station.

+1 I believe we have the formal TAP 13 specification, and a meta TAP 13.5, or TAP 13 + proposals/custom code. Considering the number of users in different programming languages outputting subtests, I think a proposal similar to what most tools produce (and that got consensus here and in the mailing list) would be a good approach.

As a side note, there is an open issue in tap4j where the TAP producer emits the subtests first. The current state machine in tap4j is not able to consume a subtest before. Quoting the example in that issue below:

    ok 1 - subtest 1a
    ok 2 - subtest 1b
    1..2
ok 1 - Subtest 1
    ok 1 - subtest 2a
    ok 2 - subtest 2b
    1..2
ok 2 - Subtest 2
1..2
ok

A perl test that will produce this output is as follows:

use Test::More;

subtest 'Subtest 1' => sub {
    ok(1,'subtest 1a');
    ok(1,'subtest 1b');
};
subtest 'Subtest 2' => sub {
    ok(1,'subtest 2a');
    ok(1,'subtest 2b');
};
done_testing();

It'd be nice to have the specification in cases like this to guide the developer :-)

[jonathanKingston]

We should not be afraid to break old implementations and if anything that should be the correct behaviour. Old implementations should not ever ignore sub tests. This is a testing setup, ignoring tests would be a mistake. Due to the non standard nature of all the current use of subtests it is possible subtests could fail and the parent succeed - this should be caught or be a parse error; either for me is fine.
This brings me to another ticket I need to make about how TAP software should respond if it can't be parsed.

I'm a little against looking to satisfy older implementations which we might end up calling 13.5 erroneously. This is a similar situation to what Markdown has been through and where possible I would prefer to go for breaking behaviour and errors will be spotted sooner rather than skipped over.

This was my theory around the formats above which should be treated as safe in most parsers where extra white space would be ignored around the comment.

I would therefore aim for:

• False negative changes only
• Try not to risk being interpreted by any parser as YAML
• Discover the most TAP consumers which can cope with the two points above

I'm actually a fan of the tap4j as it would aid streaming output of the tests before they have completed running.

The other issue with the old proposal was that there is no acknowledgement of sub test count which I think is an issue.

• Do sub tests count as the main count?
• Sub tests counts help consumers fail earlier from a streaming output of tests

One last point, the number of main consumers and producers is still actually small, patching all of them to support a newer version shouldn't be a large piece of work.

[Leont]

We should not be afraid to break old implementations and if anything that should be the correct behaviour. Old implementations should not ever ignore sub tests. This is a testing setup, ignoring tests would be a mistake.

The way it was designed means that even if the subtests are ignored the correct conclusion can still be made. It leads to a data-loss, not not to incorrect operation.

Due to the non standard nature of all the current use of subtests it is possible subtests could fail and the parent succeed - this should be caught or be a parse error; either for me is fine.

In the parser I wrote a discrepancy generates a non-fatal parse error.

I'm a little against looking to satisfy older implementations which we might end up calling 13.5 erroneously. This is a similar situation to what Markdown has been through and where possible I would prefer to go for breaking behaviour and errors will be spotted sooner rather than skipped over.

Do we really have that problem?

This was my theory around the formats above which should be treated as safe in most parsers where extra white space would be ignored around the comment.

I don't follow you. They don't have extra whitespace at the start of the line, so they won't be ignored even on legacy parsers.

[jonathanKingston]

The point I am making is we are expecting the producer to be error free by always returning a not ok parent test. I would prefer if both producer and consumer were more fragile rather than the 'Just cope' attitude the philosophy suggests.

This was my theory around the formats above which should be treated as safe in most parsers where extra white space would be ignored around the comment.

I don't follow you. They don't have extra whitespace at the start of the line, so they won't be ignored even on legacy parsers.

That is my point, why would you want them ignored. I want a testing framework to do one of two things when it doesn't know what to do:

• Fatal error
• Add a parse error as if it were a failed test it encountered

Older parsers may or may not be able to cope with invalid subtests. The following will not fail with some of the parsers I have seen:

ok 1 - test
  not ok 1 - test
ok 2 - test

However both these would:

ok 1 - test
not ok 1 -    test
ok 2 - test

and

ok 1 - test
not ok 1 -- test
ok 2 - test

The way it was designed means that even if the subtests are ignored the correct conclusion can still be made. It leads to a data-loss, not not to incorrect operation.

This is unlikely to be true for all producers but I take your point that the parent test should fail however I would prefer the correct behaviour for parsers is that they fail.

In the parser I wrote a discrepancy generates a non-fatal parse error.

This is unlikely to be true for all consumers, in fact the TAP philosophy (which likely needs addressing) suggests any such issue should be skipped.

---

The only way I would support this with the subtests being before like @kinow mentioned. Even then the format isn't as easy to read.

[Leont]

Older parsers may or may not be able to cope with invalid subtests. The following will not fail with some of the parsers I have seen:

Even as OKs they would fail because the numbering is incorrect.

This is unlikely to be true for all producers but I take your point that the parent test should fail however I would prefer the correct behaviour for parsers is that they fail.

Agreed. Parsing only the parent test should clearly be considered legacy behavior, but legacy has its uses.

This is unlikely to be true for all consumers, in fact the TAP philosophy (which likely needs addressing) suggests any such issue should be skipped.

My take on the philosophy would be: "report errors, but when possibly make them non-fatal", as that way the user can be presented with as much information as possible.

[jonathanKingston]

Very true however subtests could be part of the same numbering potentially.

Is there other useful meta data that subtests have?

Yeah I will raise an issue also with that as I certainly don't think it is clear enough.

[jonathanKingston]

@Ovid I saw some interesting comments on the mailing list of yours. (http://www.ietf.org/mail-archive/web/tap/current/msg00530.html) Would you be able to mention your concerns here?

I believe a way of demarcating subtests is needed and it should satisfy these conditions:

• Subtests be tested by older parsers as if it was a parent or fail in most cases
• YAML doesn't become a nightmare to parse
• Continue to be as human readable as possible
• Allow for test numbering
  • Allow for sub test counts 1..20
  • Allow for sub test numbering 1.1.1 and 1

[Leont]

Allow for sub test numbering 1.1.1 and 1

I'm not sure what you mean with that exactly.