diff --git a/building/config.json b/building/config.json index 34a7efa5..e6668d4c 100644 --- a/building/config.json +++ b/building/config.json @@ -710,6 +710,13 @@ "title": "Docker", "blurb": "" }, + { + "uuid": "8b527b08-52f4-4380-9d04-ebd866a09ad4", + "slug": "tooling/analyzers/guidance", + "path": "building/tooling/analyzers/guidance.md", + "title": "Guidance", + "blurb": "" + }, { "uuid": "70960db8-c1cb-4d1b-93bc-749b72fe3ee7", "slug": "tooling/analyzers/interface", diff --git a/building/tooling/analyzers/README.md b/building/tooling/analyzers/README.md index d1775b38..b0060dbf 100644 --- a/building/tooling/analyzers/README.md +++ b/building/tooling/analyzers/README.md @@ -19,3 +19,4 @@ You can use the following documents to learn more about building an analyzer: - [The Analyzer interface.](/docs/building/tooling/analyzers/interface) - [How to build a Docker image with Docker for local testing and deployment](/docs/building/tooling/analyzers/docker) - [Writing Analyzer comments](/docs/building/tooling/analyzers/comments) +- [Guidance for building an Analyzer](/docs/building/tooling/analyzers/guidance) diff --git a/building/tooling/analyzers/guidance.md b/building/tooling/analyzers/guidance.md new file mode 100644 index 00000000..04594ce0 --- /dev/null +++ b/building/tooling/analyzers/guidance.md @@ -0,0 +1,26 @@ +# Analyzer Guidance + +This document provides guidance when implementing an analyzer. + +## General + +- Analyze _all_ submitted files, except tests/invalidator/editor/example/exemplar files +- Don't aim for 100% correctness, which will be near impossible to achieve. + It's okay if the analyzer can't find all issues or is unable to infer certain tags. + +## Compilation + +If your language requires a compilation step, consider these points: + +- Compiling the solution will usually allow for higher fidelity. + For example, the definitive type of any value is only known _after_ compilation. + Do be aware of the trade-offs (e.g. slower to run and more memory usage). +- Consider in-memory compilation (if possible), to improve performance. + +## Testing + +- Have an extensive test suite +- Use golden tests to verify the behavior of the analyzer. + These tests should use the Docker image that will be deployed to verify the analyzer. +- Consider having tests for each approach. + You want the analyzer to work well for these solutions, and it will help assigning tags to the approaches later on. diff --git a/building/tooling/analyzers/interface.md b/building/tooling/analyzers/interface.md index d8643202..4b52e45c 100644 --- a/building/tooling/analyzers/interface.md +++ b/building/tooling/analyzers/interface.md @@ -85,5 +85,10 @@ The contents of `stdout` and `stderr` from each run will be persisted into files You may write an `analysis.out` file that contains debugging information you want to later view. +## Further reading + +Before building an analyzer, please read our [Analyzer Guidance][analyzer-guidance]. + [website-copy-repo]: https://github.com/exercise/website-copy [writing-analyzer-comments]: /docs/building/tooling/analyzers/comments +[analyzer-guidance]: /docs/building/tooling/analyzers/guidance