From a78ed2fd33adcb42e1b2981cb7539090f663f86e Mon Sep 17 00:00:00 2001 From: Michael Herzog Date: Thu, 13 Nov 2025 16:08:11 -0800 Subject: [PATCH 1/2] discussion-draft for tests.md --- docs/tests.md | 146 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 115 insertions(+), 31 deletions(-) diff --git a/docs/tests.md b/docs/tests.md index 5ad70fbb..9b41d369 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1,33 +1,117 @@ ## Tests -To support the language-neutral testing of `purl` implementations, a test -suite is provided as JSON document named `test-suite-data.json`. This JSON -document contains an array of objects. Each object represents a test with -these key/value pairs some of which may not be normalized: - -- **purl**: a `purl` string. -- **canonical**: the same `purl` string in canonical, normalized form -- **type**: the `type` corresponding to this `purl`. -- **namespace**: the `namespace` corresponding to this `purl`. -- **name**: the `name` corresponding to this `purl`. -- **version**: the `version` corresponding to this `purl`. -- **qualifiers**: the `qualifiers` corresponding to this `purl` as an object - of {key: value} qualifier pairs. -- **subpath**: the `subpath` corresponding to this `purl`. -- **is_invalid**: a boolean flag set to true if the test should report an - error - -To test `purl` parsing and building, a tool can use this test suite and for -every listed test object, run these tests: - -- parsing the test canonical `purl` then re-building a `purl` from these - parsed components should return the test canonical `purl` - -- parsing the test `purl` should return the components parsed from the test - canonical `purl` - -- parsing the test `purl` then re-building a `purl` from these parsed - components should return the test canonical `purl` - -- building a `purl` from the test components should return the test canonical - `purl` +The Package-URL (PURL) specification provides a JSON Schema and test +files to support language-neutral testing of PURL implementations. The +objectives for the PURL schema and test files are to enable tools to conform +to the PURL specification for tool functions such as: +- build a canonical PURL string from a set of PURL component-level data +- parse a canonical PURL string into a set of PURL components +- validate an input PURL string and optionally provide warning or info +messages or correct errors in the input PURL string + +The current JSON schema is available at: `purl-spec/schemas/purl-test.schema-0.1.json`. + +The test files are available at: +- `purl-spec/tests/spec/`: This folder contains JSON test files that are not +for a specific PURL type. + - `specification-test.json` - This file contains an array of test objects + that primarily cover testing the validity of individual PURL components or + separators between a pair of PURL components. + - component-`test.json`: These are test files for a specific PURL component. +- `purl-spec/tests/types/`: This folder contains one JSON test file for each +registered PURL type. These tests primarily cover the validity of a complete +PURL for the corresponding PURL type. + +Two key properties in the PURL test JSON schema are: + +**Test groups** + +There are two PURL test groups: +- **base**: Test group for base conformance tests. Base tests are pass/fail. +- **advanced**: Test group for advanced tests. Advanced tests are more +permissive than base tests. They may provide severity messages or correct +errors. + +*Discussion point:* +- *Is there a better name than advanced for the second test +group? perhaps Permissive? Or name the groups Strict and Permissive?]* + +**Test types** + +There are four PURL test types: +- **build**: A test to build a canonical PURL output string from an input of +decoded PURL components. See also `/docs/how-build.md`. +- **parse**: A test to parse decoded components from a canonical PURL +input string. See also `/docs/how-parse.md`. +- **roundtrip**: A test to parse an input PURL string and then rebuild it as a canonical PURL output string. +- **validation**: A test to validate a PURL input string and report severity +messages - info, warning or error. A validation test may optionally correct +errors in an output PURL string. + +*Discussion points:* +- *The validation test type is currently a proposed PURL test schema change. +See https://github.com/package-url/purl-spec/pull/614.* +- *Do we need both roundtrip and validation test types?* +- *For tests in the advanced test group we may want to distinguish between +validation test types that return only a validation message and a remediation +test type which corrects an error or errors and also returns a message (see +below) that explains the remediation.* + - *For example, the first test in `tests/types/pypi.test.json` is:*\ + "description": "pypi names have special rules and not case sensitive. + Roundtrip an input purl to canonical.",\ + "test_group": "advanced",\ + "test_type": "roundtrip",\ + "input": "pkg:PYPI/Django_package@1.11.1.dev1",\ + "expected_output": "pkg:pypi/django-package@1.11.1.dev1",\ + "expected_failure": false,\ + "expected_failure_reason": null` + + *This test case is arguably an example of an advanced remediation test case + because it corrects two errors. An advanced validation test case should + arguably return two error messages such as: + - *The pkg component must be lowercased.* + - *The name component must be lowercased.* + + *It would be helpful for the remediation test case to also provide similar + warning messages documenting the remediation steps.* + +**Test case basics** + +The standard error-handling behaviour for all test cases is based on two +properties from the PURL test schema: +- `expected_failure` + - description: "true if this test input is expected to fail to be +processed." + - type: boolean + - default: false + +- `expected_failure_reason`: "The reason why this test is is expected to fail +if expected_failure is true." + - default: null + +In the case of a test case failure the `expected_output` is null. + +*Discussion points:* +- *The current test schema does not include any specific properties for error +or other test result messages. Most of the test cases with an expected +failure are in the file `test/spec/specification-test.json`. Based on those +examples a tool could usually derive an error message from the description, +but the message would be indirect. For example from the first test case the +description is: "pypi names have special rules and not case sensitive". A +better message could be:"The name component for a PyPI package is not case +senstive".* +- *The PR to add a validation test type (https://github.com/package-url/purl-spec/pull/614) proposes a new `purl validation message` property with three levels of +validation severity messages:* + - *info: "Informational validation message"* + - *warning: "Warning validation message* + - *error: "Error validation message"* + + *NB - there is no message proposed for a successful test.* + +- *We should consider adding the message levels for all test types, not just +the validation test type.* + + + + + From accf771102d0b9dad5f14a11fb381d617b04c217 Mon Sep 17 00:00:00 2001 From: Michael Herzog Date: Thu, 13 Nov 2025 16:35:58 -0800 Subject: [PATCH 2/2] Many minor changes --- docs/tests.md | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/docs/tests.md b/docs/tests.md index 9b41d369..55792eef 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -18,9 +18,11 @@ for a specific PURL type. that primarily cover testing the validity of individual PURL components or separators between a pair of PURL components. - component-`test.json`: These are test files for a specific PURL component. + See https://github.com/package-url/purl-spec/pull/738 which adds + `tests/spec/qualifiers.json`. - `purl-spec/tests/types/`: This folder contains one JSON test file for each -registered PURL type. These tests primarily cover the validity of a complete -PURL for the corresponding PURL type. +registered PURL type. These tests should be focused on test cases that are +specific to a PURL type, such as those for namespace or qualifiers. Two key properties in the PURL test JSON schema are: @@ -43,7 +45,8 @@ There are four PURL test types: decoded PURL components. See also `/docs/how-build.md`. - **parse**: A test to parse decoded components from a canonical PURL input string. See also `/docs/how-parse.md`. -- **roundtrip**: A test to parse an input PURL string and then rebuild it as a canonical PURL output string. +- **roundtrip**: A test to parse an input PURL string and then rebuild it as a + canonical PURL output string. - **validation**: A test to validate a PURL input string and report severity messages - info, warning or error. A validation test may optionally correct errors in an output PURL string. @@ -52,6 +55,10 @@ errors in an output PURL string. - *The validation test type is currently a proposed PURL test schema change. See https://github.com/package-url/purl-spec/pull/614.* - *Do we need both roundtrip and validation test types?* +- *Should the proposed validation test type be split into two test types for:* + - *validation: message output* + - *remediation: corrects errors and also provides messages documenting the + corrections* - *For tests in the advanced test group we may want to distinguish between validation test types that return only a validation message and a remediation test type which corrects an error or errors and also returns a message (see @@ -64,11 +71,11 @@ below) that explains the remediation.* "input": "pkg:PYPI/Django_package@1.11.1.dev1",\ "expected_output": "pkg:pypi/django-package@1.11.1.dev1",\ "expected_failure": false,\ - "expected_failure_reason": null` + "expected_failure_reason": null *This test case is arguably an example of an advanced remediation test case because it corrects two errors. An advanced validation test case should - arguably return two error messages such as: + arguably return two error messages such as:* - *The pkg component must be lowercased.* - *The name component must be lowercased.* @@ -79,7 +86,7 @@ below) that explains the remediation.* The standard error-handling behaviour for all test cases is based on two properties from the PURL test schema: -- `expected_failure` +- `expected_failure` - description: "true if this test input is expected to fail to be processed." - type: boolean @@ -106,10 +113,13 @@ validation severity messages:* - *warning: "Warning validation message* - *error: "Error validation message"* + *We will need to clearly define the difference between an info message and + a warning message.* + *NB - there is no message proposed for a successful test.* -- *We should consider adding the message levels for all test types, not just -the validation test type.* +- *We should add message levels for all test types, not just the validation +test type.*