From 66b29ac9d94b2f863c29cf9dac8242bfd483c579 Mon Sep 17 00:00:00 2001 From: Neville Brown Date: Fri, 21 Feb 2025 13:49:34 +0000 Subject: [PATCH 1/7] Changes to team charter, plus some de-duplication and correction of related policies --- .../executable-specifications.md | 13 ++++++ quality-assurance/specflow-cucumber.md | 9 ----- quality-assurance/test-strategy.md | 14 +++++-- quality-assurance/ukho-quality-charter.md | 40 +++++++------------ 4 files changed, 37 insertions(+), 39 deletions(-) create mode 100644 quality-assurance/executable-specifications.md delete mode 100644 quality-assurance/specflow-cucumber.md diff --git a/quality-assurance/executable-specifications.md b/quality-assurance/executable-specifications.md new file mode 100644 index 00000000..c675c5a6 --- /dev/null +++ b/quality-assurance/executable-specifications.md @@ -0,0 +1,13 @@ +# Executable Specifications + +## Background + +An executable specification is an expression of business requirements, usually in a language that resembles prose, which can be executed as automated tests. [Reqnroll](https://reqnroll.net/), which is a fork of Specflow is an example. This uses a 'Given - When - Then' syntax, which is the most common form of executable specification. + +## When to use executable specifications + +The downside of executable specifications is that they add an extra technology plus a layer of mapping code to a project. In theory this is offset by the advantage that tests can be created in collaboration with business users as part of a [BDD](./bdd.md) process. It is therefore our policy that an executable specification like Specflow should NOT be used except as part of a BDD process. If a BDD process has not been used, it usually better to express the intent of tests by naming them well and making them easy to read. + +## Technology + +Specflow is no longer a supported technology. We should take the opportunity to remove it from our code where appropriate, either by replacing the specification layer entirely, or migrating it to [Reqnroll](https://reqnroll.net/). \ No newline at end of file diff --git a/quality-assurance/specflow-cucumber.md b/quality-assurance/specflow-cucumber.md deleted file mode 100644 index a5627352..00000000 --- a/quality-assurance/specflow-cucumber.md +++ /dev/null @@ -1,9 +0,0 @@ -# Executable Specifications - Specflow and Cucumber - -## Background - -An executable specification is an expression of business requirements, usually in a language that resembles prose, which can be executed as automated tests. [Specflow](https://specflow.org/) and [Cucumber](https://cucumber.io/) are examples. These both use a 'Given - When - Then' syntax, which the most common form of executable specification. - -## When to use executable specifications - -The downside of Specflow / Cucumber is that they add an extra technology plus a layer of mapping code to a project. In theory this is offset by the advantage that tests can be created in collaboration with business users as part of a [BDD](./bdd.md) process. It is therefore our policy that an executable specification like Specflow should NOT be used except as part of a BDD process. If a BDD process has not been used, it usually better to express the intent of tests by naming them well and making them easy to read. diff --git a/quality-assurance/test-strategy.md b/quality-assurance/test-strategy.md index a69011e0..bfb78965 100644 --- a/quality-assurance/test-strategy.md +++ b/quality-assurance/test-strategy.md @@ -18,19 +18,25 @@ Automation must be used when possible. We suggest that all teams using automation should adhere to these principles: * New functionality must be covered by passing automated tests (unless there is a legitimate reason not to). -* Make use of test design patterns and principles. -* Add tests at the correct level of the test pyramid – lower is better. +* Test design patterns and principles should be used. +* Tests should be used at the optimum level of the test pyramid – lower is better. +* All functional code must have accompanying unit +* All API development must have accompanying API tests +* Testing at the UI level should be limited to a small number tests where possible +* There must be a compelling reason to use a gherkin-based layer ( see [specific guidance](specflow-cucumber.md) ) +* Test code must be treated with the same care and attention as production code (including pairing on writing) +* The team will maintain their automated test suite (test content, relevance and code quality) to ensure it adds maximum value * All tests should be independent of each other. ### Manual Scripted * Manual scripted testing should only be used when automation is not deemed the best approach (e.g. too costly, not feasible, not applicable). * Tests will be created using appropriate test design techniques, e.g. risk based, decision tables, boundary values analysis. -* The team will decide on where these manual tests will be stored and maintained; Azure DevOps or feature files are two examples. +* The team will decide on where these manual tests will be stored and maintained; Azure DevOps is preferred. ### Exploratory -* Exploratory testing should be used to verify quality in addition to regular automated and manual scripted testing. +* Exploratory testing should be used to verify quality in addition to regular automated and manual scripted testing (see [specific guidance](exploratory-testing.md) ). * Sessions should be planned with a charter (including time-box, area to review, personas). --- diff --git a/quality-assurance/ukho-quality-charter.md b/quality-assurance/ukho-quality-charter.md index fc010647..5cd8e961 100644 --- a/quality-assurance/ukho-quality-charter.md +++ b/quality-assurance/ukho-quality-charter.md @@ -5,40 +5,28 @@ This is the UKHO Delivery Quality Charter - a set of objectives that will improv ## The Team * All members of the team are equal – there should be no differences between developers and testers in: - * Permissions they have been assigned (e.g. permissions to access environments) - * Access to hardware or software - * Weight of opinion -* Teams that are multi-disciplinary, cross-functional and equal achieve better results + * Permissions they have been assigned (e.g. permissions to access environments). + * Access to hardware or software. + * Weight of opinion. +* Teams that are multi-disciplinary, cross-functional and equal achieve better results. ## Ways of Working -* Planning, creating and executing tests (including manual testing) is the responsibility of **everyone** in the team -* The role of the tester in the team is to **champion** quality assurance and testing at **all** stages of the development process -* The team will use BDD for user-focused stories in the story refinement process (for more details see our '[BDD](bdd.md)' page) -* To ensure the team is aware of the Test Approach and their role in delivering this, the quality champion will present the approach to the team early in a project +* Planning, creating and executing tests (including manual testing) is the responsibility of **everyone** in the team. +* The role of the tester in the team is to **champion** quality assurance and testing at **all** stages of the development process. +* The tester ensures that a test approach is created and agreed with an appropriate set of stakeholders. The whole delivery team is responsible for the quality of the test approach, and for carrying it out. +* The tester ensures that information about testing is gathered into a Test Summary Report (TSR), and that it is reviewed by an appropriate set of stakeholders. The **whole delivery team** must review the TSR and agree with its conclusions. +* Test approaches should reflect the [UKHO Test Strategy](test-strategy.md), with appropriate levels of automated, manual and non-functional testing. + ## Acceptance Criteria -* Every story will have acceptance criteria added before being worked on -* Acceptance criteria will not be changed without cross-team awareness -* Acceptance criteria will be testable, i.e. capable of being proven true or false +* Every story will have acceptance criteria added before being worked on. +* Acceptance criteria will not be changed without cross-team awareness. +* Acceptance criteria will be testable, i.e. capable of being proven true or false. +* Testers should champion the use of good quality acceptance criteria. For more details see our '[Acceptance Criteria](acceptance-criteria.md)' page. -## Test Automation - -* Write tests at the correct level of the test pyramid – **lower** is **better** - * All functional code must have accompanying unit or component tests - * All API development must have accompanying API tests - * Testing at the UI level is limited to a small number of happy-path tests -* There must be a compelling reason to use SpecFlow (its use must not be a default position) -* Test code must be treated with the same care and attention as production code (including pairing on writing) -* The team will maintain their automated test suite (test content, relevance and code quality) to ensure it adds maximum value - -For more details see our '[Test Strategy](test-strategy.md)' page. -## Manual Testing -* Manual testing should never be a team's default test method and should only be used when the team agrees that full automation of a test is not feasible (e.g. the technical complexity is too high or the time taken to fully automate a test far outweighs the benefit of having that test automated) -* When used, evidence of manual testing (e.g. screenshots) is not required -* Exploratory testing is a valid form of manual testing and should be part of the test approach From 8e356cfdf7bcddbe23db26181041fc481837d3fe Mon Sep 17 00:00:00 2001 From: Neville Brown Date: Fri, 21 Feb 2025 14:11:42 +0000 Subject: [PATCH 2/7] markdown correction --- quality-assurance/executable-specifications.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quality-assurance/executable-specifications.md b/quality-assurance/executable-specifications.md index c675c5a6..8b1ff7f3 100644 --- a/quality-assurance/executable-specifications.md +++ b/quality-assurance/executable-specifications.md @@ -10,4 +10,4 @@ The downside of executable specifications is that they add an extra technology p ## Technology -Specflow is no longer a supported technology. We should take the opportunity to remove it from our code where appropriate, either by replacing the specification layer entirely, or migrating it to [Reqnroll](https://reqnroll.net/). \ No newline at end of file +Specflow is no longer a supported technology. We should take the opportunity to remove it from our code where appropriate, either by replacing the specification layer entirely, or migrating it to [Reqnroll](https://reqnroll.net/). From 1eef19355f0de93bcd0bba7aab2c2e6b3c29ba00 Mon Sep 17 00:00:00 2001 From: Neville Brown Date: Mon, 24 Feb 2025 13:23:27 +0000 Subject: [PATCH 3/7] Update quality-assurance/test-strategy.md Co-authored-by: David Bailey <122549318+datbailey@users.noreply.github.com> --- quality-assurance/test-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quality-assurance/test-strategy.md b/quality-assurance/test-strategy.md index bfb78965..3dd6c46a 100644 --- a/quality-assurance/test-strategy.md +++ b/quality-assurance/test-strategy.md @@ -20,7 +20,7 @@ We suggest that all teams using automation should adhere to these principles: * New functionality must be covered by passing automated tests (unless there is a legitimate reason not to). * Test design patterns and principles should be used. * Tests should be used at the optimum level of the test pyramid – lower is better. -* All functional code must have accompanying unit +* All functional code must have accompanying unit tests. * All API development must have accompanying API tests * Testing at the UI level should be limited to a small number tests where possible * There must be a compelling reason to use a gherkin-based layer ( see [specific guidance](specflow-cucumber.md) ) From 77e3008928db5174f2adcc4787079417a6e211d0 Mon Sep 17 00:00:00 2001 From: Neville Brown Date: Mon, 24 Feb 2025 13:25:45 +0000 Subject: [PATCH 4/7] removed typo spaces --- quality-assurance/test-strategy.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/quality-assurance/test-strategy.md b/quality-assurance/test-strategy.md index bfb78965..1b68c743 100644 --- a/quality-assurance/test-strategy.md +++ b/quality-assurance/test-strategy.md @@ -20,12 +20,12 @@ We suggest that all teams using automation should adhere to these principles: * New functionality must be covered by passing automated tests (unless there is a legitimate reason not to). * Test design patterns and principles should be used. * Tests should be used at the optimum level of the test pyramid – lower is better. -* All functional code must have accompanying unit -* All API development must have accompanying API tests -* Testing at the UI level should be limited to a small number tests where possible -* There must be a compelling reason to use a gherkin-based layer ( see [specific guidance](specflow-cucumber.md) ) -* Test code must be treated with the same care and attention as production code (including pairing on writing) -* The team will maintain their automated test suite (test content, relevance and code quality) to ensure it adds maximum value +* All functional code must have accompanying unit. +* All API development must have accompanying API tests. +* Testing at the UI level should be limited to a small number tests where possible. +* There must be a compelling reason to use a gherkin-based layer ( see [specific guidance](specflow-cucumber.md)). +* Test code must be treated with the same care and attention as production code (including pairing on writing). +* The team will maintain their automated test suite (test content, relevance and code quality) to ensure it adds maximum value. * All tests should be independent of each other. ### Manual Scripted @@ -36,7 +36,7 @@ We suggest that all teams using automation should adhere to these principles: ### Exploratory -* Exploratory testing should be used to verify quality in addition to regular automated and manual scripted testing (see [specific guidance](exploratory-testing.md) ). +* Exploratory testing should be used to verify quality in addition to regular automated and manual scripted testing (see [specific guidance](exploratory-testing.md)). * Sessions should be planned with a charter (including time-box, area to review, personas). --- From cab5bdf02a70b7c025568952c37990657c5507d1 Mon Sep 17 00:00:00 2001 From: Neville Brown Date: Mon, 24 Feb 2025 13:40:00 +0000 Subject: [PATCH 5/7] Fixed typos --- quality-assurance/test-strategy.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/quality-assurance/test-strategy.md b/quality-assurance/test-strategy.md index 1b68c743..b3089f56 100644 --- a/quality-assurance/test-strategy.md +++ b/quality-assurance/test-strategy.md @@ -20,10 +20,10 @@ We suggest that all teams using automation should adhere to these principles: * New functionality must be covered by passing automated tests (unless there is a legitimate reason not to). * Test design patterns and principles should be used. * Tests should be used at the optimum level of the test pyramid – lower is better. -* All functional code must have accompanying unit. +* All functional code must have accompanying unit tests. * All API development must have accompanying API tests. * Testing at the UI level should be limited to a small number tests where possible. -* There must be a compelling reason to use a gherkin-based layer ( see [specific guidance](specflow-cucumber.md)). +* There must be a compelling reason to use a gherkin-based layer (see [specific guidance](specflow-cucumber.md)). * Test code must be treated with the same care and attention as production code (including pairing on writing). * The team will maintain their automated test suite (test content, relevance and code quality) to ensure it adds maximum value. * All tests should be independent of each other. From 14a5d233b8ca235181352e8d639930ffdd6793ac Mon Sep 17 00:00:00 2001 From: Neville Brown Date: Mon, 24 Feb 2025 13:43:03 +0000 Subject: [PATCH 6/7] fixed link --- quality-assurance/test-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quality-assurance/test-strategy.md b/quality-assurance/test-strategy.md index b3089f56..7a9f727d 100644 --- a/quality-assurance/test-strategy.md +++ b/quality-assurance/test-strategy.md @@ -23,7 +23,7 @@ We suggest that all teams using automation should adhere to these principles: * All functional code must have accompanying unit tests. * All API development must have accompanying API tests. * Testing at the UI level should be limited to a small number tests where possible. -* There must be a compelling reason to use a gherkin-based layer (see [specific guidance](specflow-cucumber.md)). +* There must be a compelling reason to use a gherkin-based layer (see [specific guidance](executable-specifications.md)). * Test code must be treated with the same care and attention as production code (including pairing on writing). * The team will maintain their automated test suite (test content, relevance and code quality) to ensure it adds maximum value. * All tests should be independent of each other. From 05c0938d007cd91199ab7667182aafe8f7dcfd5d Mon Sep 17 00:00:00 2001 From: Neville Brown Date: Wed, 26 Feb 2025 16:15:15 +0000 Subject: [PATCH 7/7] reworded the statement on dev team reviewing --- quality-assurance/ukho-quality-charter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quality-assurance/ukho-quality-charter.md b/quality-assurance/ukho-quality-charter.md index 5cd8e961..14fbd1b6 100644 --- a/quality-assurance/ukho-quality-charter.md +++ b/quality-assurance/ukho-quality-charter.md @@ -15,7 +15,7 @@ This is the UKHO Delivery Quality Charter - a set of objectives that will improv * Planning, creating and executing tests (including manual testing) is the responsibility of **everyone** in the team. * The role of the tester in the team is to **champion** quality assurance and testing at **all** stages of the development process. * The tester ensures that a test approach is created and agreed with an appropriate set of stakeholders. The whole delivery team is responsible for the quality of the test approach, and for carrying it out. -* The tester ensures that information about testing is gathered into a Test Summary Report (TSR), and that it is reviewed by an appropriate set of stakeholders. The **whole delivery team** must review the TSR and agree with its conclusions. +* The tester ensures that information about testing is gathered into a Test Summary Report (TSR), and that it is reviewed by an appropriate set of stakeholders. The tester must ensure that all members of the team that have contributed to development understand the testing and agree with the conclusion of the TSR. * Test approaches should reflect the [UKHO Test Strategy](test-strategy.md), with appropriate levels of automated, manual and non-functional testing.