Release memory when space is freed from Ramdisk

[Lord-Malgus]

Thank you for this tool. It works great.

Is there anyway to free the memory allocated to the Ramdisk as data is deleted from it? The driver/system seems to hold on to the memory once allocated never releasing it.

[tmcdos]

This is handled by the driver while my tool is just a nice GUI to configure it. I don't think it ever releases the memory but I am not good enough with C++ and could be wrong. Here is the source code - if you can find how to ask the driver to release the memory, I can then find a way to add this feature to the GUI.
But I am not very optimistic. And in my case I am totally okay to have 8-10 GB of RAMdisk out of 32 GB total :)
I believe ImDisk was able to reserve memory on demand and then release it - but it has the issues which led to the birth of this GUI :))
Arsenal-Image-Mounter (src).zip

[Lord-Malgus]

Thank you for the quick response. ImDisk supported this feature with its 'dynamic allocation' option. Perhaps that is just no longer available. I will take a look at the arsenal source. Much obliged.

[alainbryden]

The current FAQ for Arsenal says that the "free mode" does have dynamic memory allocation.

For me personally, this is the only reason I haven't switched to your solution, despite having to work around the odd app falling apart when GetFinalPathNameByHandle isn't available.

If at any point you dig into this and find a way to surface it, I would be very interested!

[tmcdos]

I am using the GUI with driver v13.12.41.506 from May 20, 2019. The latest version probably has more features but I have not tried it. Since I have no issues with the old version of the driver - I have not invested time to update the GUI to use the new features of the latest driver.
If someone wants to contribute with a PR (and the PR still works with Delphi 7 and Win7) - they are welcome.
By the way, you can try the extended kernel for Win7 VxKex which might be able to solve the app falling caused by GetFinalPathNameByHandle. I am using Node.js v18 with VxKex on Win7 and it works - otherwise it chokes on missing GetHostnameW.

[igorbaryshev]

The Arsenal's aim_cli tool is very nice: I created a dynamic VHD image natively in Windows 11 with a drive letter, label and filesystem, attached right it right there as well, put default content, detached the drive, and what's left is to just mount it with aim_cli.exe --ramdisk --filename=RamDiskName.vhd!
Dynamically allocated RAM is definitely its greatest perk, I can set the image to a very large size without worrying that it will fill up the RAM and I'd have to restart every time I need more ram free.
Though, I imagine, you also want it to load the contents of a specific folder. But that probably can be changed as well, seeing how you already did the work to copy content over to the drive.

[tmcdos]

I guess I will have a deeper look at the newest aim_cli and try to discover how does it communicate with the driver in order to create a dynamic RAM-disk. It's probably something small.
You correctly noticed that I prefer persisting the RAM-disk inside a regular folder in the filesystem instead of VHD image because I want to be able to easily add or remove files/subfolders in/from that folder instead of mounting/unmounting the VHD image.

[tmcdos]

@igorbaryshev Can you confirm the following:

1. You mount a VHD image whose size is e.g. 100 Mb to a RAM-disk whose max size is e.g. 6 Gb
2. The RAM usage at this moment is low
3. You copy a large MP4 file to the RAM-disk, e.g. 4-5 Gb
4. The RAM usage increases - at least 4-5 Gb
5. You delete all files from the RAM-disk
6. The RAM usage instantly decreases to a very low amount

[igorbaryshev]

@tmcdos yes, pretty much this.
Maybe not instantly-instantly, but within a couple seconds it's freed up.

[tmcdos]

@igorbaryshev Does it also happen when you do not initialize the RAM-disk from a VHD but just create it empty, format it, copy large file and then delete it?

[igorbaryshev]

@igorbaryshev Does it also happen when you do not initialize the RAM-disk from a VHD but just create it empty, format it, copy large file and then delete it?

If you mean just running command to create ramdisk without any image, then no.
Ran .\aim_cli --ramdisk --disksize=2G and it immediately allocated all 2GiB

If you mean creating the VHD via aim_cli and mounting that, then it should be the same as with mounting any other image.
Just need to specify that it's dynamic upon creation.

Syntax to create a new disk image file:
aim_cli --create --filename=imagefilename --disksize=size [--variant=fixed|dynamic] [--mount]
Size in bytes, can be suffixed with for example M or G for MB or GB.

This one should do it, and you can continue formatting it however you need.

[tmcdos]

No, I do not use VHD (or any other) image(s) for my RAM-disk. I just wanted to make sure that you and/or other users can confirm that the Arsenal driver really releases the grabbed memory when it is not needed - or it does not (as I suspect).
As far as I can see in the source code of the driver, there is only a function for extending the LU (logical unit) but not for shrinking a LU. And TRIM commands are performed by the driver (actually passed-through to the filesystem) only in proxy mode and in image mode - but not otherwise.
What I observe on my own PC is that the RAM-disk begins by using just enough memory to hold the NTFS filesystem structures. Then, if I copy some files on the RAM-disk - memory usage increases because the driver allocates memory to hold the files. But when I delete all files from the RAM-disk - memory is not released and returned to the OS. And I believe that this is by design - at least in the open-source version of the driver.
I am not qualified enough to change the driver but even if I was - the driver must be digitally signed and I do not have such a certificate.
My conclusion is that this feature request should go to the author of the Arsenal driver.

[tmcdos]

Mark from Arsenal replied to me and forwarded a response from Olof.
Olof confirms my suspicion that the driver performs TRIM/unmap only for VHD/proxy:

Our basic RAM disk options implemented in the driver do not support unmap/trim. The entire memory size for the RAM disk is allocated as one single memory block when the RAM disk is created. Windows lazy-commits parts of virtual memory, so it might look like it starts with very little memory allocation, but it rather depends on what memory sizes you are looking at (committed, reserved etc).

We have dynamically allocated RAM disks as well in AIM, but they use several other components to work in the expected way. You need to start with a VHD image file as template and then create a RAM disk based on it, either advanced menu in the GUI, or something like:

aim_ll -a -f \\?\vhdaccess\??\awealloc\??\c:\image.vhd

This creates a dynamically allocated RAM disk based on contents of C:\image.vhd. Memory allocation increases as files are added and decreases as files are deleted. This is implemented by not allocating all memory as one single memory block, but instead each block of the VHD image is allocated separately. As VHD blocks are zeroed and moved to physical end of image in response to unmap/trim, the VHD image size shrinks and the last block at the end of underlying memory can be freed as well.

Since I am scratching my own itch with this GUI, I have particular goals which do not align well with VHD image:

1. I want to preload the RAM-disk with the contents of a given folder but once preloaded - I definitely do not want the folder (or a VHD image file in the root filesystem) to be constantly updated by the driver
2. At PC shutdown I want the RAM-disk to be persisted back to the same folder that was used for preloading - but with the ability to exclude some folders (specified in the GUI config) which are considered temporary work and can be safely thrown away
3. I prefer using a bare folder for preloading instead of VHD image not only because it does not require mounting/unmounting but also because I want the preloading/persisting folder to be modified only at shutdown and not at any other time to avoid SSD wearing out. I use this RAM-disk mainly to avoid the SSD wear-out (mostly by Chrome and Firefox but also by the TEMP files), speed is on 2nd place.

So apparently the only option for dynamic RAM allocation is

aim_ll -a -f \\?\vhdaccess\??\awealloc\??\c:\image.vhd

as suggested by @igorbaryshev

[alainbryden]

Thanks for chasing down that reply.

It seems an uncharacteristic regression since ImDisk always supported dynamic memory allocation, even without use of an image, and the same developer is still championing both the old and new project. Oh well 🤷‍♂️

(screenshots for posterity)

Seems like it's time for us to leave @tmcdos in peace and take up our campaign with @LTRData ;)

[LTRData]

Thanks for chasing down that reply.

It seems an uncharacteristic regression since ImDisk always supported dynamic memory allocation, even without use of an image, and the same developer is still championing both the old and new project. Oh well 🤷‍♂️

Seems like it's time for us to leave @tmcdos in peace and take up our campaign with @LTRData ;)

I am not the developer of ImDisk Toolkit (in your screenshots). It is an addition by v77 and supports for example dynamically allocated RAM disks.

[LTRData]

Mark from Arsenal replied to me and forwarded a response from Olof. Olof confirms my suspicion that the driver performs TRIM/unmap only for VHD/proxy:

Our basic RAM disk options implemented in the driver do not support unmap/trim. The entire memory size for the RAM disk is allocated as one single memory block when the RAM disk is created. Windows lazy-commits parts of virtual memory, so it might look like it starts with very little memory allocation, but it rather depends on what memory sizes you are looking at (committed, reserved etc).

We have dynamically allocated RAM disks as well in AIM, but they use several other components to work in the expected way. You need to start with a VHD image file as template and then create a RAM disk based on it, either advanced menu in the GUI, or something like:

aim_ll -a -f \\?\vhdaccess\??\awealloc\??\c:\image.vhd

This creates a dynamically allocated RAM disk based on contents of C:\image.vhd. Memory allocation increases as files are added and decreases as files are deleted. This is implemented by not allocating all memory as one single memory block, but instead each block of the VHD image is allocated separately. As VHD blocks are zeroed and moved to physical end of image in response to unmap/trim, the VHD image size shrinks and the last block at the end of underlying memory can be freed as well.

Since I am scratching my own itch with this GUI, I have particular goals which do not align well with VHD image:

1. I want to preload the RAM-disk with the contents of a given folder but once preloaded - I definitely do not want the folder (or a VHD image file in the root filesystem) to be constantly updated by the driver
2. At PC shutdown I want the RAM-disk to be persisted back to the same folder that was used for preloading - but with the ability to exclude some folders (specified in the GUI config) which are considered temporary work and can be safely thrown away
3. I prefer using a bare folder for preloading instead of VHD image not only because it does not require mounting/unmounting but also because I want the preloading/persisting folder to be modified only at shutdown and not at any other time to avoid SSD wearing out. I use this RAM-disk mainly to avoid the SSD wear-out (mostly by Chrome and Firefox but also by the TEMP files), speed is on 2nd place.

Maybe there was some kind of misunderstanding here. The VHD image is not modified by the driver when it is used for creating dynamically allocated RAM disks. It is only accessed when the RAM disk is created when it is copied into memory. After that, modifications happen only in memory and TRIM commands sent from AIM driver (in image mode in this case) causes the VHD driver to allocate and deallocate blocks dynamically. But since the VHD image is located in memory only, changes do not reach physical disk but instead just allocates and deallocates RAM.

I have used this myself for many years now and never had any issues with it. I personally prefer to not copy anything back from RAM disk to physical disk on shutdown, but I understand that some people want that. It could probably be possible by scheduling something that copies some directories right before the RAM disk is removed. Also, right after creating the RAM disk from a mostly empty VHD, you could copy your on-disk directory structure to the RAM disk. (This will not update the VHD.)

So apparently the only option for dynamic RAM allocation is

aim_ll -a -f \\?\vhdaccess\??\awealloc\??\c:\image.vhd

as suggested by @igorbaryshev

Yes, or aim_cli --ramdisk --filename=c:\image.vhd as suggested above.

[tmcdos]

Yes, it was my misunderstanding about the VHD updating - Mark clarified this in a second email to me.
My struggle with improving the GUI to offer VHD image support (e.g. like @LTRData suggests using an empty VHD image and then additionally preload another folder from the regular filesystem) is related to my suspicion that the VHD max size (which is probably set when creating the VHD image) probably determines the size of the RAM-disk which seems a bit inconvenient to me as a user since I would prefer to enter the RAM-disk size through the GUI (like it was in ImDisk Toolkit)

Of course once the user decides about the RAM-disk size - it is not changed so often but still having to resize the VHD image (or at least this is my suspicion - I could be again misunderstanding) feels uncomfortable to me. When I get more free time, I will add ability to choose a VHD image through the GUI for those who want to have dynamic RAM allocation.

The reason I've also implemented the ability to persist the RAM-disk contents on shutdown (which happens through the Windows service that is part of this project) is that sometimes I download some files from Internet on the RAM-disk and then forget to copy them on a non-volatile storage. So this is just a safety measure for my particular usecase.

[LTRData]

My struggle with improving the GUI to offer VHD image support (e.g. like @LTRData suggests using an empty VHD image and then additionally preload another folder from the regular filesystem) is related to my suspicion that the VHD max size (which is probably set when creating the VHD image) probably determines the size of the RAM-disk which seems a bit inconvenient to me as a user since I would prefer to enter the RAM-disk size through the GUI (like it was in ImDisk Toolkit)

I think that v77 is also working on migrating ImDisk Toolkit to use Arsenal Image Mounter with the goal of providing similar features. I think it could take some time to complete though because there is surely a lot of work needed.

[tmcdos]

If I knew C/C++ better, I could help. But I do not ... (I do not even like it, but that's another story)