Performance improvement question

[megapro17]

I have a device that does not normally support random read operations. It has several reading modes. The fastest one is sequential read mode, which reads files in parts, one at a time, each part cannot be larger than 64k. In this mode I can't set the offset and length. I can only start reading the file from the beginning, and cancel it at any time if I need to.
As far as I understand, the system does not use this mode of operation, even when copying a file normally. The ReadFiles function is called continuously until the file is copied completely and the maximum buffer size is only 1048576 (I also wanted to know if it can be changed in any way).
There is also a second function that can read a file from any offset and any length, but it is much slower.
What is the most efficient way I can implement such a file system? I want the best sequential read performance. Maybe create a caching mechanism that would store the beginning of a few megabytes of each file, and give them to the system on demand? And when a sequential read is detected, can I use another mode of operation that will cache the file to memory and then copy it to the system buffers in parallel? Sounds complicated, maybe there is a better way?

[Liryna]

Hi @megapro17 ,
I was also going to suggest using a cache to match your requirements and the system behavior.
You could implement a read ahead logic that is triggered by the first read and continue to read sequentially while populating your cache for future possible reads.

[Rondom]

I think ideally, we should add an option to Dokany to use Windows caching. OSes have spent quite some time on optimising caching and readahead and writeback, so it would be good to be able to leverage built-in capabilities of the OS. I remember that it had been requested before, but I cannot find the issue for it, if it exists...

[LTRData]

There is really no generic "Windows caching" for file systems. Each file system implements its own cache logic. There are higher level cache systems like prefetch that work with many file systems though, but that is kind of a different thing and does not cache all kinds of files.

[lostmsu]

@LTRData I came back to #1214 and this comment. Is prefetch something that we need to implement, or does it already work with Dokany without a separate effort?

[LTRData]

@LTRData I came back to #1214 and this comment. Is prefetch something that we need to implement, or does it already work with Dokany without a separate effort?

I have not checked whether it currently works with Dokan because it is typically not relevant. It only caches executable files to increase performance when starting up applications etc.

The kind of cache discussed in this thread is different and it needs to temporarily store the beginning of each file to avoid unnecessary random location reads. That needs to be implemented by the file system.

[megapro17]

I have to implement cache logic manually? Is there any infomation or code that could help me?

[Liryna]

None of the samples have a cache logic unfortunately.
What kind of cache are you looking for ? Memfs is somewhat of a memory cache...