openzim/zim-requests#777 showed that we have a very significant weakness in current logic of selections which are used with a custom homepage (instead of the masonry or list of articles most selections are using).
This is the case for instance of medicine selection (I don't know if we have other cases tbh).
In this selection, every Zimfarm recipe is using a custom ZIM main page.
For instance, in English the main page is currently https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Medicine/Open_Textbook_of_Medicine2
This main page and all articles referenced on this main page are not "naturally" medicine pages (i.e not part of the project). While this is not an issue for the ZIM main page because it is naturally included in the article list by the scraper, it is a problem for the articles referenced on this main page.
We currently maintain this list in the code:
|
echo "Wikipedia:Books/Cancer_care" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Children's health" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Ears nose throat" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Endocrine disease" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Eye diseases" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/General surgery" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:WikiProject_Medicine/Books/Heart_disease" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Infectious disease" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Medications" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Men's health" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Neurology" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Orthopedics" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Mental health" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:Books/Skin diseases" >> "$CUSTOM_DIR/medicine.tsv" |
|
echo "Wikipedia:WikiProject_Women's_Health/Books/Women's_health" >> "$CUSTOM_DIR/medicine.tsv" |
First weakness is that should this landing page change to something else, or should the list of article on it change, we need to be notified and modify the wp1_selection_tool code. Experience showed that we are not notified.
The second weakness is that for all languages except english, the medicine selection is build based on interwiki links for all articles of the en selection. This means that interwiki links of all these articles referenced on the landing page must be fine, or the ZIM will miss them and the homepage will be "broken" (it will have some text in-place of the expected link to the article). Same problem if in one non-en language the landing page is not linking to the same articles we have in our codebase for en.
The third weakness is that this assumes we have only one landing page or that only few articles are needed. This is not really the case currently. For instance fr still uses https://fr.wikipedia.org/wiki/Utilisateur:Doc_James/Open_Textbook_of_Medicine page which is very different. Luckily all pages referenced on this page are part of the medicine wiki project
I see two ways forward.
A first naive approach could be that we ask for all articles referenced on the landing page to be added to the medicine project so that we always scrape them. This is naive because maybe the project team does not want to have these pages under their umbrella and in any case it means that interwiki links must be properly set on all these articles ... and experience showed it is not the case.
The only robust approach I see is taht we modify wp1_selection_tools to automatically detect the articles on the home page. We would reference one or few wikidata ID (not the article title) of the landing pages used across ZIMs. And then we would automatically add the selections the articles found on these landing pages. It means that interwiki links must only be properly set on the landing pages (probably reasonable). For instance currently medicine ZIMs are using either Q115739862 or Q20200278, see https://mdwiki.org/wiki/WikiProjectMed:App/Intro_Pages. A query to https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q115739862|Q20200278&props=sitelinks&format=json gives all the landing pages in all languages. This means we will fetch "a bit more" than strictly required, but we probably don't mind to have two landing pages in the ZIM instead of one (only one will be really used anyway).
openzim/zim-requests#777 showed that we have a very significant weakness in current logic of selections which are used with a custom homepage (instead of the masonry or list of articles most selections are using).
This is the case for instance of medicine selection (I don't know if we have other cases tbh).
In this selection, every Zimfarm recipe is using a custom ZIM main page.
For instance, in English the main page is currently https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Medicine/Open_Textbook_of_Medicine2
This main page and all articles referenced on this main page are not "naturally" medicine pages (i.e not part of the project). While this is not an issue for the ZIM main page because it is naturally included in the article list by the scraper, it is a problem for the articles referenced on this main page.
We currently maintain this list in the code:
wp1_selection_tools/build_custom_selections.sh
Lines 71 to 85 in 3f759b5
First weakness is that should this landing page change to something else, or should the list of article on it change, we need to be notified and modify the wp1_selection_tool code. Experience showed that we are not notified.
The second weakness is that for all languages except english, the medicine selection is build based on interwiki links for all articles of the
enselection. This means that interwiki links of all these articles referenced on the landing page must be fine, or the ZIM will miss them and the homepage will be "broken" (it will have some text in-place of the expected link to the article). Same problem if in one non-en language the landing page is not linking to the same articles we have in our codebase foren.The third weakness is that this assumes we have only one landing page or that only few articles are needed. This is not really the case currently. For instance
frstill uses https://fr.wikipedia.org/wiki/Utilisateur:Doc_James/Open_Textbook_of_Medicine page which is very different. Luckily all pages referenced on this page are part of themedicinewiki projectI see two ways forward.
A first naive approach could be that we ask for all articles referenced on the landing page to be added to the medicine project so that we always scrape them. This is naive because maybe the project team does not want to have these pages under their umbrella and in any case it means that interwiki links must be properly set on all these articles ... and experience showed it is not the case.
The only robust approach I see is taht we modify wp1_selection_tools to automatically detect the articles on the home page. We would reference one or few wikidata ID (not the article title) of the landing pages used across ZIMs. And then we would automatically add the selections the articles found on these landing pages. It means that interwiki links must only be properly set on the landing pages (probably reasonable). For instance currently medicine ZIMs are using either
Q115739862orQ20200278, see https://mdwiki.org/wiki/WikiProjectMed:App/Intro_Pages. A query to https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q115739862|Q20200278&props=sitelinks&format=json gives all the landing pages in all languages. This means we will fetch "a bit more" than strictly required, but we probably don't mind to have two landing pages in the ZIM instead of one (only one will be really used anyway).