Cannot see wikipedia_en_all_maxi_2023-11.zim file on Android 12

[tigran123]

Describe the bug

Android app cannot see the zim file for WikiPedia English "all_maxi_2023_11"

Expected behavior

It should see it, obviously.

Steps to reproduce the behavior:

1. Place the zim file to /Android/media/Kiwix
2. Run Kiwix Android app and go to the Library tab and pull down for refresh
3. Observe that nothing happens and the message "No files here" remains where it was.

I do not think that this is specific to this particular zim file. I just got this new Note 10+ phone and so the first thing to try was to install Kiwix and point it to the latest WikiPedia zim file.

Screenshots

Environment

• Version of Kiwix Android : 3.8.1 Build: 7230801
• Device : Samsung Galaxy Note 10+
• OS version : Android 12

Logs

[kelson42]

@tigran123 Thank you for your bug report. Does it work better with the nightly or with the version 3.9.0 (to be released later today)?

[tigran123]

Tried with the nightly kiwix-universal-2023-12-16.apk -- exactly the same result, no .zim files are seen in /Android/media/Kiwix. Could it be that I should place the .zim files in some other directory?

Also, does the selection of "storage device" in the Settings matter? Oh, I just tried it in the nightly build and it actually does something -- it gives me a directory chooser type of dialog, where I can navigate my external storage, but no Kiwix subdirectory is visible inside /Android/media -- only org.whatsapp and org.telegram.messenger. So, how do I make /Android/media/Kiwix directory visible to Kiwix? Should I rename it to something like org.kiwix and reboot?

[tigran123]

UPDATE: good news, if I select the parent of 'Kiwix' directory, i.e. /Android/media in the directory-chooser in Kiwix Settings/Storage -- then it does manage to find the Wikipedia .zim file! So it works now. And it finds the Russian Wikipedia also. All is well now!

This is with running 2023-12-16 nightly build.

[kelson42]

@tigran123 Thank you for running the investigation. Glad you are able to read the file again.
@MohitMaliFtechiz Could you please share an explanation, this behaviour still looks fishy to me.

[tigran123]

Another small point: that "rotating circle" thing, indicating that the program is searching for more .zim files, keeps turning forever. Hmmm, surely that means it is draining my battery now? Shouldn't it stop eventually? Ok, I do have about 50,000 files under 'Android/media/Books', but this is a very fast 1TB micro sd and just doing something like stat(2) on 50,000 files should not take more than a few seconds...

[MohitMaliFtechiz]

@tigran123 Thanks for you research.

but no Kiwix subdirectory is visible inside /Android/media -- only org.whatsapp and org.telegram.messenger. So, how do I make /Android/media/Kiwix directory visible to Kiwix?

@kelson42 This is decided by the Android system which directory is to show/hide e.g. Android/data, and Android/obb folders are protected and not selectable by folder chooser. Apart from these two directories Android/media is accessible by the Android system, but if you are not seeing the Kiwix folder then it definitely protected by the system. Have you created this folder from the device or you just copy this folder from somewhere else like from your PC? If you copy this folder from your PC please check the folder permissions.

I have the Samsung Galaxy M21 and I have created the kiwix folder inside the Android/media directory on my SD card and kiwix is reading the file, Also, this directory is visible when I use the folder chooser.

[tigran123]

@MohitMaliFtechiz In the latest message I reported that Kiwix managed to see the content of the /Android/media/Kiwix directory when pointed to its parent, i.e. /Android/media. So, nothing is hidden or protected -- the data (*.zim files) is visible even though the subdirectory Kiwix was not listed by the chooser UI.

Yes, I created the /Android/media/Kiwix directory and its content with the microsd card outside the phone, i.e. connected directly to my Linux machine and mounted as exfat filesystem, because this way is much faster than going via USB cable and doing adb push to the phone.

[MohitMaliFtechiz]

@MohitMaliFtechiz In the latest message I reported that Kiwix managed to see the content of the /Android/media/Kiwix directory when pointed to its parent, i.e. /Android/media. So, nothing is hidden or protected -- the data (*.zim files) is visible even though the subdirectory Kiwix was not listed by the chooser UI

@tigran123 By choosing the parent directory android permits us to read the content inside its subdirectories even if they are protected, that's why Kiwix can read the content after choosing the Android/media.

Yes, I created the /Android/media/Kiwix directory and its content with the MicroSD card outside the phone, i.e. connected directly to my Linux machine and mounted as exfat filesystem, because this way is much faster than going via USB cable and doing adb push to the phone.

Ohh, that's why it is not recognized by the chooser UI, can you please check the folder permissions? Probably android does not have permission to directly read that folder.

[tigran123]

I thought exFAT did not support the concept of Unix permissions at all? When I mount it I use uid=1000 option to make everything accessible (read/write/execute) by user tigran (which has uid=1000). But I will check later -- I will extract the microsd from the phone and will check the permissions.

[tigran123]

Yes, I was right -- exFAT does NOT support the concept of Unix permissions, see below:

$ sudo touch new
$ ls -al new
-rwxr-xr-x 1 root root 0 Dec 18 12:41 new*
$ df .
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sdb1      exfat   29G  224K   29G   1% /mnt
$ sudo chmod 0777 new
$ sudo ls -l new
-rwxr-xr-x 1 root root 0 Dec 18 12:41 new

[MohitMaliFtechiz]

@tigran123 are you creating the folder with sudo? By using sudo it will only be accessible for root.

I suspect the Android chooser UI is not showing this folder to select because our application does not have permission to read that folder. After you choose the Android/media, our app has access to read the content inside its subdirectory, so it reads the ZIM files.

Please try the below approach to fix the problem.

• Can you please try to create a folder without sudo and place the ZIM files inside that?
• If you are creating the folder with sudo please change that folder permission with chmod 777 instead of 0777.
• Try to create the folder inside the phone or connect the phone via USB and create the folder inside Android/media and place ZIM files inside it.

[tigran123]

Ha-ha-ha, my friend, are you trying to teach Unix the person who helped to create the Linux kernel? (Google "Tigran Aivazian" is your friend :)

Using "sudo" has nothing whatsoever to do with it. I used it only because I mounted that card without uid=1000 option, so the system emulated uid=0 on the vfs layer and that is what userspace applications saw.

Also, 0 in 0777 means "octal mode" and it is exactly the same as using 777 because chmod(1) command uses octal mode by default. This is basic Unix knowledge.

Anyway, for completeness and for your satisfaction I have repeated the same test without sudo and the result, of course, is exactly the same:

$ rmdir new
$ df .
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sdb1      exfat   29G  192K   29G   1% /mnt
$ mount | grep exfat
/dev/sdb1 on /mnt type exfat (rw,relatime,uid=1000,fmask=0022,dmask=0022,iocharset=utf8,errors=remount-ro)
$ mkdir new
$ ls -ld new
drwxr-xr-x 2 tigran root 32768 Dec 19 10:00 new/
$ chmod 777 new
$ ls -ld new
drwxr-xr-x 2 tigran root 32768 Dec 19 10:00 new/

As I explained -- exFAT does NOT have a concept of permissions. The reason why it "appears" to have such a thing on Android is because Android itself emulates this, e.g. if a directory's name matches the program's name (like com.termux for Termux, for example) and under other conditions.

[tigran123]

Btw, a useful reference for you to read is this:

https://digitalfortress.tech/php/difference-file-mode-0777-vs-777/#:~:text=It%20is%20worthy%20to%20note,all%20numeric%20arguments%20as%20octal.

It explains why it is always better to use 0777 and not 777 -- even though for chmod(1) command they are exactly the same, but for use in a program (in PHP, C, Python etc) they are completely different.

[MohitMaliFtechiz]

@tigran123 Thanks for your reference and explanation i really appreciate it.

Let's back to the topic. I tried all the things you have mentioned, and on my phone, it can read the ZIM files from the Android/media/kiwix folder, it is visible to the chooser UI as well.

Since Android introduced the scoped storage, many restrictions have been introduced while accessing the storage. As Android chooser UI does not recognize the kiwix folder inside the Android/media in your device, it might be restricted by the Android system due to some conditions because it is accessible on my device. It might be treating this folder as a restricted or protected folder as it behaves for Android/data, and Android/obb folders. These folders are not available for the chooser UI.

We should first ensure what was happening on your device.

• Try to rename the folder in your device e.g. org.kiwix would be good.
• Try to move the folder outside the Android/media folder as this folder is specially designed for media files.
• Try to create the folder inside Android/media through the phone.
• Connect the phone via USB, and create the folder inside Android/media and then place ZIM files inside it. See if ZIM files are visible in the app or not.

[tigran123]

The name of the application is not org.kiwix but org.kiwix.kiwixmobile, so if I rename /Android/media/Kiwix to /Android/media/org.kiwix.kiwixmobile then Android automatically changes its uid corresponding to the username u0_a327, i.e. to the uid of Kiwix process on my phone:

d2s:/storage/9C33-6BBD/Android/media $ ls -al
total 2560
drwxrwx--- 14 root    everybody 524288 2023-12-15 19:56 Audio
drwxrwx--- 17 root    everybody 524288 2023-12-15 15:10 Books
drwxrwx---  2 u0_a298 everybody 524288 2023-12-15 13:55 com.whatsapp
drwxrwx---  2 u0_a327 everybody 524288 2023-12-19 11:58 org.kiwix.kiwixmobile
drwxrwx---  2 u0_a297 everybody 524288 2023-12-15 13:55 org.telegram.messenger

But the Kiwix app UI chooser still does not see the org.kiwix.kiwixmobile subdirectory. Moreover, even giving it permission for the parent directory (i.e. /Android/media) does NOT help it to find the .zim files, i.e. now it does not see anything at all (like the old version).

I will try some other combinations and let you know.

Btw, it is very annoying that the app allows the system to put itself to sleep or/and lock the screen -- it should request a wake lock to prevent that and keep the screen on.

[tigran123]

I forgot to mention that the UI chooser does not see the Audio and Books subdirectories of /Android/media either -- only org.telegram.messenger and com.whatsapp. Very strange.

[tigran123]

Ah... I figured it out! Even when you select "External" in the Settings, the UI chooser still by default goes to "Internal" storage, i.e. ignores your choice! This is why we see those two subdirectories org.telegram.messenger and com.whatsapp -- they do also exist in /Android/media on the internal storage (as well as on the external microsd).

So, now if I go to the menu from within the UI chooser and manually select External storage, then it sees the /Android/media/kiwix subdirectory and the .zim files inside it, so everything works now!

[tigran123]

But there is still a tiny bug -- the "rotating circle" is rotating eternally, i.e. never stops, even though the selected directory only contained TWO zim files and both are already displayed in the Library tab. So it looks like the search goes on forever and wastes battery.

[tigran123]

Btw, it is a mystery how could selecting the /Android/media on the Internal storage help Kiwix app to find zim files in /Android/media/Kiwix on the External storage! But as I cannot reproduce this, I am assuming that when I reported above that it found them, the UI must have used the External storage by default. I certainly did not change it manually. But now I can consistently get it to work by changing the storage manually to External in the UI chooser, so it works as it should, except that if possible it should have selected External storage by itself due to my choice of "External" in the little selection box which comes up prior to invoking the UI chooser.

[MohitMaliFtechiz]

Ah... I figured it out! Even when you select "External" in the Settings, the UI chooser still by default goes to "Internal" storage, i.e. ignores your choice! This is why we see those two subdirectories org.telegram.messenger and com.whatsapp -- they do also exist in /Android/media on the internal storage (as well as on the external microsd).
So, now if I go to the menu from within the UI chooser and manually select External storage, then it sees the /Android/media/kiwix subdirectory and the .zim files inside it, so everything works now!

Ohh!! Happy to know that all things are working fine.

Btw, it is a mystery how could selecting the /Android/media on the Internal storage help Kiwix app to find zim files in /Android/media/Kiwix on the External storage! But as I cannot reproduce this, I am assuming that when I reported above that it found them, the UI must have used the External storage by default. I certainly did not change it manually. But now I can consistently get it to work by changing the storage manually to External in the UI chooser, so it works as it should, except that if possible it should have selected External storage by itself due to my choice of "External" in the little selection box which comes up prior to invoking the UI chooser.

We have placed a fix for it in #3619, now it will by default show the SD card to select the folder.

But there is still a tiny bug -- the "rotating circle" is rotating eternally, i.e. never stops, even though the selected directory only contained TWO zim files and both are already displayed in the Library tab. So it looks like the search goes on forever and wastes battery.

@tigran123 We have this kind of issue opened #2445.

Another small point: that "rotating circle" thing, indicating that the program is searching for more .zim files, keeps turning forever. Hmmm, surely that means it is draining my battery now? Shouldn't it stop eventually? Ok, I do have about 50,000 files under 'Android/media/Books', but this is a very fast 1TB micro sd and just doing something like stat(2) on 50,000 files should not take more than a few seconds...

we will place a fix for it. Thanks for providing the additional information about it, it would be very helpful to debug the problem.

[tigran123]

Thank you @MohitMaliFtechiz and @kelson42 for dealing with this issue. All is well now.

[tigran123]

Reopening as the problem is back after upgrade to the official 3.9.1 version from Google Playstore. And a few other serious issues as well.

[tigran123]

So, here are the problems with the latest 3.9.1 version of Kiwix from Google Playstore:

1. No permission is requested automatically after a fresh install (no previous version of Kiwix present). So I had to kill the process via Settings and manually add the Files/media permission and then restart Kiwix.
2. Default storage is Internal, should be External (but this is minor and cosmetic)
3. When External storage is selected no dialog comes up (like in the nightly build) which would allow to point to the exact location.
4. Most importantly and pertinent to the current issue -- no .zim files are found in /Android/media/kiwix (the nightly build of 16 December found them without problems). Note that this happens only on Android 12 on my mobile (Note 10+), but on my tablet (Samsung Galaxy Tab S4) running Android 10 it (official 3.9.1) did find the .zim files, but only after I prompted the app via + button in the bottom right corner and pointed to each individual .zim file. Ah, this brings us to the next item:
5. The + button in the Library tab is absent when the application is executed on the mobile, but present when running on the tablet. I understand, of course, that some bits of the UI may be specifically tuned to a specific screen size -- this is the proper use of this feature. But hiding a key function such as manually adding a .zim file just because we are running on the mobile screen constitutes improper use -- please fix this.

[tigran123]

Oh, I should have read the message which comes up on startup -- it does say that the version of Kiwix from Google Play is useless and points to the URL where one can download the proper version. Thank you! I will try that one and will report the results.

[tigran123]

Ok, I have downloaded this file and tested it:

https://download.kiwix.org/release/kiwix-android/kiwix-3.9.1.apk

Most of the problems described above went away, but the main problem remains -- it still does NOT find the zim files automatically, even after pointing to the directory, containing them, and giving all the permissions etc. However, the + button is present and I can use it to point to the specific .zim file and open it -- this works fine. Then I can bookmark the needed article and access it later via bookmark. However, it would still be nice to see the list of present .zim files in the Library tab.

[kelson42]

@tigran123 Thank you for staying around, reporting the nightmare of file mgmt on Android. That said we can not work properly with one ticket reporting 4 different things. The rule for proper handling is one problem = one issue/ticket.

[tigran123]

Yes, I understand -- so my latest message says that all problems BAR ONE have gone now, by using the full 3.9.1 version from download.kiwix.org. So, there is just one problem remaining and that is pertaining to this specific issue description.

[tigran123]

Yet another UPDATE: After 15 minutes of scanning the /Android/media/kiwix directory which contains just two .zim files (and nothing else) the Library tab was finally populated with the correct files: Wikipedia in English (101.7GB version of 2023-11-30) and Wikipedia in Russian (32.7GB version of 2023-12-08).

So, that means there are no more problems and I can close this issue again. However, I hope you agree that it is very unusual to spend 15 minutes scanning a directory with two files. Ok, presumably it also retrieves some metadata from those files, but even so it should not spend 15 minutes doing that...

[kelson42]

However, I hope you agree that it is very unusual to spend 15 minutes scanning a directory with two files.

Yes, this is worth to make a ticket. We have a least a usability problem here IMHO.

[tigran123]

Ok, I will time it accurately and open a ticket for that.

[tigran123]

My initial timing guess was quite accurate -- the actual time taken was 14 minutes and 5 seconds! :)