Real life examples?

[velle]

I love scrut :) I need exactly this kind of tool.

I think the usage of scrut can be divided into roughly three categories:

1. Tests only Dont generate any html. Possible with both .md and .t files.
2. Tests+docs. Writing tests in markdown, and compiling those .md tests into html. It is generally speaking possible to learn about a tools behavior, including edge cases, by studying a set of test cases. But normal test cases are often not something that is easily read. Ideally this solution (using scrut for "tests+docs") would result in test cases that are easy to read, well as easy as it gets, and comparable to studying toy examples in a demos folder.
3. Verify docs Writing regular docs in .md, but verifying that the tool output behaves like the docs says. The focus is on checking that the examples in the docs are correct, not so much on testing that the code is correct.

Do you agree with this summary?

As far as I can see, this repo demonstrates only category 1.

Do you know of any real or toy examples where they use scrut? If so, I would really like to study such examples closer. In particular Im interested in demonstrations of cat 2 and 3.

Sincerely

[ukautz]

@velle Thanks for the flowers :)

The overarching idea for Scrut is to make CLI testing intuitive and easy: If you know how to use your CLI on the command line and know some markdown then you can write integration tests.

Beyond that Markdown was specifically chosen to invite users to not only write tests but also documentation of those tests, so that long-term maintenance is less painful. HTML rendering fits into that, as it further helps maintainers to read the test documents. We don't do render into HTML explicitly, but simply use tooling that does that "on the fly" with markdown documents (e.g. the github Markdown rendererd here). I think that matches your 2nd point.

Your point 3) is pretty much my ideal world, that I would call "living run-books", where instructions are both provided, explained and validated in one. I cannot point you to an external example though. The selftest/commands folder goes a bit in that direction, but it's a synthetic example that doesn't add much actual value.

So yes, I think what you wrote is about the same scope of what we envisioned, but there we don't have any external examples to show-case that. You might find some better examples here on github - the starcm use-case looks close and pyrefly repo looks interesting

[velle]

I appreciate your detailed reply :)

I am making some e2e tests for a cli which is a client for a web service. Most test files need to start with a setup phase (either in a separate file using prepend, or hardcopied into each test file). Currently I have 3 test files all depending on the same setup, but eventually there will be 10+, and some having slightly different setup. The setup makes preparations both locally and on the web service.

I’ve tried a few different approaches, but I haven’t been able to make it as neat as I was hoping. That's why I asked for examples.

• prepend: Using prepend means that the rendered version will in no way show the setup phase, and not even hint that there is a setup phase.
• hardcoded setup in each test file: Redundant code, harder to maintain/update. The reader has to read the setup in each file — they can’t tell whether it’s the same until they’ve actually read it. The most relevant part, the focus of this particular file, comes further down in the file, so the reader has to scroll past redundant setup for each file.

A few unpolished enhancement ideas/dreams:

• The rendered file should contain a section in the top showing what was prepended, with hyperlinks for those files.
• The prepended setup could be rendered in each of the resulting html files, but then it should be clearly marked as setup, and maybe placed in a div that is by default collapsed.
• If hardcoding setup into a test file, then I would like to mark some parts as setup, and have a rendering engine that would tag those setions as setup and collapse them by default.
• When the scrut tool processes a given file, the scrut blocks that are part of a prepended/appended file should not be considered tests. Ie, if the prepended _setup.py contains 6 tests, then that should not be counted as 6 succeeded tests. Ideally, if anything in the prepended file fails, that should result in an error (different from a fail), and evaluation of that test file should abort.
• In addition to the current option of hardcoding environment variables in the front-matter, I would like the option to load environment variables from a .env file. Two steps in my current setup are loading environment variables from files (one secret, one with other config).

Have you considered any changes along these lines, or do you have other plans or ideas that might help improve setups like mine?