META - restructuring the documentation #1993
Description
The user guide and how-to section need to be overhauled in various ways to make it easier for users to understand what is going on.
As we work on the user guide we should try to implement a framework to decide the kind of content that should go in each specific part of the documentation website. I have been reading https://diataxis.fr/, and I think we should implement much of what is suggested there.
Currently the documentation is split across five sections:
- the user guide
- the how-to section
- the API reference
- the examples gallery
- the skrub-materials website
We should think about what goes where. My idea for the content of each section is the following (inspired by the diataxis framework):
User guide
The user guide should be guiding the user through learning how to use the various skrub functionalities. This means having some kind of thread that connects the different sections, and should be explaining the main concepts that are involved in working with skrub.
How-to
The how-to section should contain self-contained articles that explain "here's how you can do X with skrub". Think of these as cooking recipes: there is a specific problem that we want to solve, and this is how we do it with skrub.
A lot of what is currently in the user guide should be moved to this section instead to reduce the clutter and only keep the core concepts in the user guide.
There should also be cross-references between the user guide and the how-to section.
API reference
Overall, the API reference is not in a bad shape, although we may want to re-evaluate the name of each section, and possibly the order/position of some of the objects.
We may also want to move some of the examples that are used in the user guide to the respective objects in the API reference, to once again lighten the user guide.
Examples gallery
The examples gallery should remain pretty much the same, although we may want to move some of the examples from the gallery to the user guide. My reasoning is that some examples fit very well in the narrative that we should go for in the user guide, and having the examples in the user guide marked as tutorials would reduce a lot the amount of friction a user would encounter by having to switch.
I want to clearly label the examples as "tutorials" so that users that are not interested in that can skip over and continue in the next sessions.
Skrub materials
The skrub-materials website deserves more love than it's been getting. The main use reasoning behind the website is storing longer posts that would otherwise take a long time to run or would be hard to manage in some other way if they were added to the CI.
Additional considerations
A general problem of the user guide is that it goes in far too much detail explaining the reasoning behind choices, or very niche/advanced use cases. As a result, some of the pages are way too long and hard to read. We can mitigate this problem by moving some of the content to the how-to guide (for situations where we are clearly explaining a self-contained use case), by using the "admonition" directive of sphinx (https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-admonition) or simply by trimming unnecessary material.
A lot of titles in the user guide need to be improved: in some situations, the title is describing what is inside a section, rather than referring to the problem or concept that is explained in that section. This makes it hard for users who don't know the library to find content, because they won't know the correct keywords that would lead them to the right place. This is a problem throughout the documentation, including the examples.
There should be a section about common errors and pitfalls that users may encounter when they use data ops and other skrub features.