discussion-draft for tests.md

[mjherzog]

This PR covers a rewrite of docs/tests.md to cover key definitions including:

• test groups
• test types
• error handling
• test output messages

[matt-phylum]

Do any PURL implementations support info or warning messages? Do any consumers check those messages? Are there meaningful messages to generate? Implementations would need to either store these messages on the parsed PURL or return a wrapper that includes these messages.

[mjherzog]

This discussion section is not based on what current PURL implementations do. The proposed set of messages are from #614 which is on hold pending the conclusion of this discussion.
I originally used italics to call out open discussion sections, but that does not seem so clear so I switched to using a blockquote to mark the beginning and end of a discussion section.

[matt-phylum]

I don't think it is necessary or beneficial for PURL to specify error message strings to be produced by implementations. It seems potentially reasonable to have an explanation of why the test fails, but specifying the errors produced in this way prohibits or discourages implementations from handling errors in better or more customary ways. Implementations languages that throw exceptions or return typed results/outcomes should return typed errors, ie a syntactically invalid PURL and a PURL that fails package-type-specific validation should result in different types or enum values such that the consuming software doesn't need to analyze a string to determine what's going on.

It's possible to specify that there is a conversion from however the implementation specifies errors into a string, but this seems more useful for the purposes of this test case than in actual usage.

[mjherzog]

The idea of adding these messages was introduced with #614 which is pending the conclusion of this discussion.
The messages are not intended to replace the expected_failure_reason, but to provide additional information about the detected problem.
But the current approach for expected_failure_reason is pretty generic. For example the first set of test cases in tests/spec/specification-test.json have a pattern with a specific description but a generic failure reason, such as;

• description: "a scheme is always required", and expected_failure_reason: "Should fail to parse a PURL from invalid purl input".
  Wouldn't it be more helpful to have a more specific failure reason like: "Should fail to parse a PURL with scheme component missing from the PURL input" ?

[matt-phylum]

What is the difference between a validation test and a parsing test? They both parse a PURL and they both check that the expected values are produced.

[mjherzog]

The output from a parsing test is the set of decoded PURL components.
The output from a validation test (as currently proposed) is a PURL string.

[matt-phylum]

The output from a building test is also a PURL string. PURL implementations generally have two operations: they can turn a PURL string into its components, and they can turn components into PURL strings. I can't think of something that round trip or validation tests could do that parsing or building tests can't already do. There's already some discussion about getting rid of round trip tests, and I wonder if this is another thing that will later be getting removed.

[mjherzog]

@matt-phylum What do you suggest we call the proposed validation test type? Do you recommend 2 sub-types for a parse test where one returns the components and another returns a string?

[matt-phylum]

I don't understand why there would be a parse test that returns a string. If the parse is correct, the components will be correct. Once the input has been parsed, there is no memory of what it was before being parsed, so a test that parsing pkg:pypi/Django_package results in pkg:pypi/django-package is no different from a test that parsing pkg:pypi/Django_package results in {"package_type": "pypi", "name": "django-package"} and that building {"package_type": "pypi", "name": "django-package"}, unsurprisingly, results in pkg:pypi/django-package.

[mjherzog]

The apparent difference is in the expected test output. For the current parse test type the expected output is a set of decoded PURL components. For the proposed validation test type the expected output is a PURL string. I can see how you can get the string output by combining a parse test and a build test - are you recommending that we should not have that combination as a test type and leave the combination processing to the tool/implementation?

[matt-phylum]

Unless there is an intention that these tests are only used by implementations that have a dedicated PURL string -> canonicalized PURL string routine that isn't implemented by parsing and building, I don't think it makes sense to have a test that parse+build for a given input string produces a specific output string. It's easier for implementers to test that both parsing and building work as expected and it avoids bugs where canonicalization is applied differently when parsing or building.

[mjherzog]

@matt-phylum Thank you. Your perspective is clear to me now.
Jan has separately said that: "... the roundtrip exists for a very good reason: test canonicalization of string input."
See #738.
I will put this topic on the agenda for our TG2 meeting this Friday. I will post a new PR (including your input) with consolidated updates because the commentary is a bit hard to follow as PR changes in the markdown file.

[mjherzog]

@matt-phylum Thank you for your comments - see my replies above. I am still learning how to use GH PRs and Issues to manage a discussion document (instead of a gdoc with comments).

[mjherzog]

Closed in favor of

• Rewrite 2 of tests.md #752