EPIC: Display expectation checks in service

[neilfwar]

Context:

• The current data quality checks on the service look at whether the data provided matches the specifications (i.e. validity), but doesn’t check whether the data is accurate.
• The Data Management team have highlighted a number of ways that we can check the accuracy of the data and have built / are building various data quality checks (internally known as “expectations”).
• They have built two of these already:
  • Out-of-bounds: This checks whether the geospatial data that the LPA has provided is within the LPA’s boundaries. If any of the points or polygons fall outside of the boundary, then that entity will fail the check. See this table for the results of this check.
  • Conservation area count: This checks whether the number of conservation area entities inside the LPA’s boundaries matches the number entities that we’ve gathered from other sources. If the number of entities provided by that LPA don’t match the number we’ve gathered from other sources, the LPA will fail the check. See this table for the results of this check.
• There are two more checks that are on the Data Management team’s backlog for this quarter:
  • Deleted entities: This checks whether an LPA has deleted entities from a resource. Entities should persist and be updated with an ‘end date’ (or whatever other field is appropriate, depending on the specific dataset), rather than being deleted/removed from the file.
  • Referential integrity between area and document datasets: Ensures that references in geospatial datasets (e.g. Article 4 direction area) correctly link to existing records in their respective document datasets (e.g. Article 4 direction).

Goals:

The intention and scope of this initiative is to:

• Explore and test with users how these checks might be integrated into the service.
• Implement the first two of these checks, displaying the results in the service.