Request: CHD format support for disc images

[ryz]

Hey again,

related to the 7z request, it'd be great if disc images for games that do have them the CHD format would be supported.

I'm sure CHD is well known as it's supported by many RetroArch cores already. Just in case here's some info:

CHD stands for "Compressed Hunk of Data" and is a format for storing disk images, developed by the MAME team.

libchdr from @rtissera provides a sample library:
https://github.com/rtissera/libchdr

Some of the benefits of the CHD format:

• Lossless compression
• Better compression than bin/cue/iso/etc.
• Streamable format, doesn't need to decompressed, just play
• Its very easy to convert entire folders of disc images with some batch file magic
• Single file for every disc
• Supports redbook audio in FLAC compression
• Non-destructive, images can be easily reverted to their original format
• It's becoming the defacto standard for libretro cores with support for MAME, Saturn, PCEngine/RurboGrafx, SegaCD/MegaCD, PS1 and Dreamcast cores

Thanks!

[schellingb]

It is certainly on my want list.

The need for adding 4 external library dependencies to what is currently 0 pains me a bit... So if I were to do I'd look into how to better integrate it into this project so it doesn't end up as a pain to both maintain and build for various platforms.

Probably once I'm done with fixing/improving immediate issues that are reported and improved MIDI features like adding support for external MIDI devices, this and 3dfx are the things I want to look at.

[kashkash18]

Out of curiosity, I converted 3 CDs from Command and Conquer
BIN CUE images are 1.69 GB and CHD images are 1.17 GB

It saves a lot of space. I hope you will be able to integrate it in the future.
A lot of space will be freed up with a large number of games.

[schellingb]

Keep in mind using the currently available ZIP compression (which can compress both the BIN/CUE/ISO files as well as the game installation) also results in a very similar 30% size reduction on C&C CDs.

I get that CHDs are probably faster than my ZIP streaming, but that can be improved I think. Also once we support CHDs, I'm sure some people will end up compressing the already compressed CHDs inside a ZIP file which would be a performance disaster with no size benefits. I guess the core could refuse to mount such files? Or show a warning?

Ideally an already compressed file is only stored in the ZIP (stored uncompressed) and not re-compressed, but that is a manual ZIP operation not many users even know about.

[TheDatadrainer]

Hi.
I disagree using CHD for CD-ROM compression with DOSBox Pure. As schellingb says, people will surely embed the CHD file into the ZIP file and it'll be a mess. And the purpose of this core is to have every static data in one archive file.
Maybe a way to get it right. I mean to get best file size and perfs would be to use a folder named DISCLABEL.ISO with ISO/BIN content extracted alongside the game folder and mounted as ISO-9660 CD. With optional only the content needed by the game ; eg. removing unnecessary data like trailers, pdf, etc.
That means having an automount feature for .ISO folder if detected. The same way, be able to mount such folders from menu too, as MOUNT allows it instead of IMGMOUNT in classical DOSBox use.
Additionally using the DISCLABEL part as the volume name. But the only two problems I see with that could be with games checking the disc label to launch. I don't know if games like that exists, but for a multi CD-ROM game with the same label for every disc or with a label containing DOS forbidden characters, that would be a problem. For such a game, a .conf file is required. The other problem is for audio tracks... Maybe with a cue sheet DISCLABEL.CUE file pointing to ripped audio files with the first track ignored if a .ISO folder is detected instead of a BIN file. That means DOSBox Pure to be able to read audio files. Some forks of DOSBox can, I don't know for this one.
Anyway. Only after doing that, the whole content can be compressed to a ZIP file or copied from a mounted empty HDD CHD (not CD). And only here CHD is really relevant, from my opinion.
I hope to have been clear.
Regards.

[Formedras]

Keep in mind using the currently available ZIP compression (which can compress both the BIN/CUE/ISO files as well as the game installation) also results in a very similar 30% size reduction on C&C CDs.

CHD may or may not provide much in the way of size reduction compared to using the ZIP format's DEFLATE on an ISO that's pure data, but there are other use cases where CHD can provide benefits. For example, if a game uses Red Book audio, such as with Quake, then the fact that CHD can compress those tracks with FLAC can provide a much smaller file size than trying to DEFLATE those tracks instead.

Ideally an already compressed file is only stored in the ZIP (stored uncompressed) and not re-compressed, but that is a manual ZIP operation not many users even know about.

Yes, it's true that most users wouldn't know about changing compression ratio or algorithm per-file... or at all, for that matter. But that's no reason to omit a useful feature that skilled users could work with. If it was, then the ZIP format wouldn't even have these features in the first place.

In addition, the supreme use case for DOSBox Pure honestly isn't for every user to create their own ZIP or DOSZ packages, although it's certainly necessary at this point. Where Pure will excel is users downloading DOSZ packages over the internet that have already been made by other users, or preferably by companies like CD Projekt (GOG) for standalone forks of DOSBox Pure. And in this use case, it can be expected that the preparer would be better versed in packing games. Therefore, regular users can still benefit from advanced features that they don't know how to use.

[casasfernando]

Bumping this one since it seems to be stalled for more than a year.
Having CHD support in Dosbox-Pure would be awesome.
Then I will have a viable alternative to Dosbox-X.

[Darknior]

I'm very interested buy this feature too, it help to save a lot of place.

[trotsky40]

Still hoping this is on the roadmap. CHD support would help make my file management easier as not all multidisk games seem to run well in ZIP files.

[schellingb]

@trotsky40 The main issue with this is that I can't come up with a way to link up a ZIP file and its CHD files. If it were just a single CHD file we could somewhat easily define the linking by filename:

• MyDosGame.zip
• MyDosGame.chd

This would mount the CHD on D: alongside the ZIP. But what if there are multiple CHD files for multi-cdrom games? How could that be intuitive? As stated before, letting users put CHD inside the ZIP is a performance nightmare due to people ending up adding a useless second compression of the CHD inside the ZIPs.

[trotsky40]

For my use case (may not be standard), that wouldn’t be an issue. My user provided files involving CDs are all installation/play discs, so there is no need to link a CHD to a zip. I install the game from the discs so the others files are stored in the “pure.zip” save files. I would use an M3U file to link the multi-disc CHD and the. “C:\” files are stored elsewhere.

[Formedras]

This would mount the CHD on D: alongside the ZIP. But what if there are multiple CHD files for multi-cdrom games? How could that be intuitive?

For multidisc linking, you could have a subextension, like what ZIP/RAR/7Z utilities do for multipart archives. Say it was Riven, despite its being a Win95 game:

• Riven.zip
• Riven.Disc1.chd
• Riven.Disc2.chd
• Riven.Disc3.chd
• Riven.Disc4.chd
• Riven.Disc5.chd

Or for a theoretical game that lets you use two CD-ROM drives, and you only have to switch one drive's discs (let's call it "Weddi"):

• Weddi.zip
• Weddi.D.chd
• Weddi.E.Disc1.chd
• Weddi.E.Disc2.chd

Heck, let's redefine the ZIP over to its own hard drive CHD (remember, CHD's original meaning was "Compressed Hard Drive"); let's use the Weddi example again:

• Weddi.C.chd
• Weddi.D.chd
• Weddi.E.Disc1.chd
• Weddi.E.Disc2.chd

Potentially, in that last example, you could put those into a ZIP (or TAR) file with no compression.
Yes, throw up a warning if there is compression, but ultimately you would have to just let the user screw themself over, performance-wise, if they insist on doing it anyway.

[schellingb]

@Formedras I'm not sure we can use CHD as the C: harddrive. As far as I know, CHD don't contain files, they contain file systems, right? Can CHD files be writable?

[Formedras]

@schellingb No, CHDs are not editable. But... do you really want to in the first place? Part of the point of game archives is to consistently have a pristine copy. One would then expect that filesystem changes from user gameplay would be a separate save file. (CHD does have "Parent CHDs", but I'm not sure that it would be suitable for this.)

But yeah, if you want your hard drive archive to be directly writable, CHD isn't the path for that drive. If keeping it an image, a sparse VHDX would be preferable, but that's probably more work than it's worth.

[schellingb]

Yes, to use the CHD as the C: drive we'd need some kind of saving of changes. We do something similar when using ZIP files as the D: drive in an installed operating system. But... I'm not really convinced this gives us many benefits over just ZIP files. It's not like a common thing for DOS emulators to use CHD files anyway so I don't really see a reason to push that.

If CHD support lands in DOSBox Pure, it'll probably be for CD ROM images initially. Maybe showing a warning about compressed CHD additionally compressed inside ZIP is enough. I still worry that not many users would know what to do if the error asked them to store the CHDs without compression in a ZIP. Many basic ZIP tools (those built into the OS for example) might not even be able to make such a ZIP file.

[Formedras]

I still worry that not many users would know what to do if the error asked them to store the CHDs without compression in a ZIP. Many basic ZIP tools (those built into the OS for example) might not even be able to make such a ZIP file.

Sometimes the only answer is to teach the user yourself. (And if they don't wanna learn, then let them suffer.) Along with the warning, provide a couple of links. One to instructions on the DOSBox Pure wiki on GitHub, one to a YouTube or DailyMotion video tutorial.

[TheDatadrainer]

Just trying to understand what's going on.
@Formedras, CHD are Compressed Hunk of Data, meaning they are essentially an assembly of small blocks of data compressed with different compression algorithm. They were created by the MAME team to address storage for HDDs and CDs alike to get very fast access to the data with a small memory footprint. Nothing compared to a big solid Deflate L9 file.
@everybody, Here is a concrete example. If you take a DOS game like Under a Killing Moon. The game is on 4 CDs. The installation program uses 10 MB on HDD and let you chose a different CD-ROM drive by disc. For DOSBox Pure it means managing 4 CHD and 1 ZIP (installation+save) files to the minimum if I'm correct. I think the installation can be put in another ZIP file or CHD file, adding one file to manage, with the advantage of having the saves and config file externally.
A way to do that would be to mount every file through the RetroArch core UI. Or as @Formedras proposed having a naming scheme allowing the core to do that automatically. But that is the same for CUE+BIN file.
As I said before, a solution with all files (CD contents + installation) in one ZIP file would be more practical. But that said having CHD would be more efficient. For two years now, CHD support have dramatically increase everywhere, almost every computers and CD consoles Libretro cores support it and even MiSTer supports it. So that is definitely something to consider today.
And to avoid having CHD compressed into ZIP it can be easily done by excluding CHD files from the listed content to core gets, just printing a message telling the case was detected and the file where excluded as they must be external to the archive.
So where is the problem here?

[schellingb]

Having the CD images in the ZIP file is the most convenient. It requires minimum to no explanation because it generally "just works". It requires no special file naming scheme and no M3U file, so zero friction to get started. There is no need to run a mount command in the command line or even select files in a file browser.

CHD is not simple (in terms of code complexity). If it gets added to this core, it needs to be widely accessible. I don't consider writing M3U files accessible. The normal user uses the core downloader in RetroArch to use this core. It doesn't come with a manual so it needs to be intuitive.

For example telling a user "CHD file in ZIP is deflate compressed please use store" is not acceptable. While technically correct, it is nonsense to most users. A libretro core can only show very short messages, it's not meant to teach a user complex file operations.

But.... most games have only 1 CD image. For those we can just load the .chd name with the same name. I think that is simple enough. For multiple cd images I still need to come up with a solution that works for me. Maybe .Disc2.chd .Disc3.chd? Maybe .chd2 .chd3? Maybe something else...

[schellingb]

If only multiple CD images could be stored in one CHD file...

[casasfernando]

I’m following this very interesting discussion but I feel I may be missing something after reading: https://docs.libretro.com/library/dosbox_pure/

Why handling CHD images would be so different from handling ISO files?
I mean, for multi-disc games m3u files are already supported by the core right? And perfectly documented in the core doc page, so why is inaccessible to use them?
RetroArch users are probably familiar with m3u files for multi-disc ganes since other popular cores (eg: beetle-psx) use them as well.

About the “CHD in ZIP file” concern, wouldn’t be enough to document this in the core doc page and simply ignore any CHD file found inside the ZIP maybe with a very short error message?

I’m currently using DOSBox-X mainly for it’s CHD support and didn’t have any issues at all with single or multi-disc games in CHD files but would love to try this core (and probably switch over) if it ever gets CHD support as well.

Just my two cents and thanks a lot for all the hard work!

[schellingb]

OK, I think we can try for the following:

• CHD stored in ZIP without compression, use like ISO
• CHD stored in ZIP with compression, show short error, user might have to look up manual or ask in a forum etc.
• CHD with same name as ZIP, load automatically
• Multi disc needs M3U (first line .ZIP file, .CHD on second, third, and so on)

Keep in mind this is just a plan, it still needs plenty of implementation work to get this done.

But I'm looking forward to @casasfernando contributing the update to that libretro manual page 😆

[casasfernando]

@schellingb you can count with my contribution to the manual page while I wish I would be skilled enough to contribute with something more meaningful, like actual code. 😉
Regarding the plan above, I would completely avoid allowing CHD in ZIP files. I think it can only cause problems and unnecessary confusion to users. Bear in mind that the default when creating a CHD is to use compression. So my suggestion would be:

• CHD stored in ZIP without compression, use like ISO
• CHD stored in ZIP with compression, show short error, user might have to look up manual or ask in a forum etc.
• CHD stored in ZIP, ignore/abort loading, show short error, user might have to look up the clearly documented by @casasfernando manual page or ask in a forum etc.
• CHD with same name as ZIP, load automatically
• Multi disc needs M3U (first line .ZIP file, .CHD on second, third, and so on)

Would love to read what other people participating in the discussion think about this approach.

[TheDatadrainer]

@casasfernando. I think @schellingb want to use CHD the same way as ISO are. Meaning embedded in a ZIP file. It seems an important feature for him, and I respect that even I don't really like it. But as the reading perf seems bad with high compression level. The only way is to use non-compressed ZIP. I agree it is confusing. I know Beetle PSX can show a message about the reading speed being to slow, maybe there should be a message like this and allowing CHD in a compressed archive to be loaded, but to write in the manual it is not the recommended way to proceed as there can be issues.
About using CHD reading implementation, come to my mind the project libchdr by Romain Tisserand if it is not possible to use the libretro API CHD implementation for some reason.
And indeed using M3U/M3U8 files is common for multiple CD games on Beetle PSX and Beetle Saturn for example. Nothing frightening for the end user I think. Again just need instructions in the manual page. The actual explanation being quite succinct.
@schellingb. For me a CHD file should always have the .chd extension as they can be used on multiple systems/emulators. The solution given by @Formedras is the good one I think.

Or for a theoretical game that lets you use two CD-ROM drives, and you only have to switch one drive's discs (let's call it "Weddi"):

* Weddi.zip

* Weddi.D.chd

* Weddi.E.Disc1.chd

* Weddi.E.Disc2.chd

With three attributes, preferably in the following order: the name of the software matching the ZIP filename, the disc number [optional] and the drive letter to mount the disc into [optional]. With the same logic, inside or outside the main ZIP file as said previously. About the use of a M3U file. I don't know... Should it be optional or mandatory when multiple media are needed? How it is managed when the CD or floppy images are embedded into the ZIP file? I never tried it so I don't know.
The advantage of putting the drive letter in the filename is to assign multiple drive or force a letter without the need of a M3U file. It can be automated by detecting the files and setting the following behavior: If no letter present, it should be possible to cycle CDs on one default drive set by DOSBox Pure (this is what currently exists). But if a letter is present the CD(s) should be assigned to the said drive. The only issue I see, is how to cycle between CDs on multiple drives if several are assigned because I don't know if the RetroArch API allow such a thing: dynamic menu item addition and deletion.
Just thinking here as that is another future implementation to make and it is not related exclusively to CHD.
But what do you think about all of that? Is this a good way to do it?

ps: Just for information about DVD image support for CHD. As far as I know creating CHD of DVDs is still not supported by chdman. But some games are using DVD on Windows 95. The only choice here is ISO or "fortunately" using CDs as all games I know existing in DVD also exists in CD version (an example is Blade Runner).

[casasfernando]

ps: Just for information about DVD image support for CHD. As far as I know creating CHD of DVDs is still not supported by chdman. But some games are using DVD on Windows 95. The only choice here is ISO or "fortunately" using CDs as all games I know existing in DVD also exists in CD version (an example is Blade Runner).

I'm a bit confused about this last comment and don't want to deviate the main subject, but can't help myself from asking what do you mean chdman doesn't support DVD? I believe that as long as you have an ISO or BIN/CUE image, chdman doesn't really care about the original media type, and should be able to convert it to CHD, but I may be wrong tough. I have several PS2 games (DVD) converted from ISO or BIN/CUE into CHD working without any issues.

[rtissera]

@casasfernando MAME 0.255 added explicit DVD media type in chdman. It's basically 2048 hunk size for MODE1/2048 which is the format on all DVD. It's not yet added in libchdr but it's a couple of lines patch.

[schellingb]

There's no downside to supporting uncompressed CHD in ZIPs though. It's technically not hard to do. Two good reason to do it. I don't think it can be confusing to users as there is not really a chance they ended up accidentally not compressing the CHDs. Such a ZIP needs to be deliberate created. Or maybe I'm wrong, maybe ZIP tools these days are smart and figure out that compressing the CHD file is a losing game and automatically store them uncompressed? That would be even better. I like having 1 full game in a single ZIP file. It's THE feature of DOSBox Pure. If we end up adding a huge complex library dependency like libchdr then I want it to work with the principles of this core.

Anyway, now people are talking about supporting DVDs and emulating multiple CDROMs. Please don't do this in this issue. These features both are very separate requests and both would require large engineering efforts. DOSBox Pure image mounting is built around the "Disc Control" feature of libretro. That feature sadly is built very much for a console and thus only ever thinks about there being one drive. Thus we ended up with only ever one CD-ROM being mounted on D: or one floppy disk on A:. But, I don't think that's a big issue. Most real computers didn't have multiple CD-ROMs. I don't think a real DOS game exists that requires multiple CD-ROMs. Switching the mounted image takes pressing 2 or 3 hotkeys in RetroArch and can be done in a second. Being bound to this simplicity might be limiting in some forms, but maybe overall the core is easier to use due to it.

Looking for complex file patterns is not something we can do. So we can't blindly scan for .CHD files with the same prefix followed by multiple arguments like disc numbers and drive letters. For example, this core runs on Xbox Series consoles via UWP which provides an application only VERY SLOW access to the file system. Because we don't know if CHD files exist, the scan for them would need to be done on every startup. Thus it's either a fixed series of names (.chd, .2.chd, .3.chd) or M3U files.

[schellingb]

* Weddi.E.Disc1.chd

Another problem with a naming like this is capitalization. We don't really want to end up in a place where on Windows "game.disc1.chd" works but on Linux it doesn't. Linux usually runs with case sensitive file systems so we need to be mindful of that. Yes, that applies to the file extension, too, and is quite annoying. I tend to match the capitalization on the main content file (be it ZIP, Zip or zip) and just hope for the best.

[TheDatadrainer]

@rtissera
@casasfernando MAME 0.255 added explicit DVD media type in chdman. It's basically 2048 hunk size for MODE1/2048 which is the format on all DVD. It's not yet added in libchdr but it's a couple of lines patch.

Ho Great. That's good news. Need to update my MAME as I have just the version below. Thanks for the information :)

[TheDatadrainer]

ps: Just for information about DVD image support for CHD. As far as I know creating CHD of DVDs is still not supported by chdman. But some games are using DVD on Windows 95. The only choice here is ISO or "fortunately" using CDs as all games I know existing in DVD also exists in CD version (an example is Blade Runner).

I'm a bit confused about this last comment and don't want to deviate the main subject, but can't help myself from asking what do you mean chdman doesn't support DVD? I believe that as long as you have an ISO or BIN/CUE image, chdman doesn't really care about the original media type, and should be able to convert it to CHD, but I may be wrong tough. I have several PS2 games (DVD) converted from ISO or BIN/CUE into CHD working without any issues.

CUE+BIN is normally limited to CD-ROM images. But ISO file can be anything except the hybrid ISO-9660+CD-DA.

[TheDatadrainer]

There's no downside to supporting uncompressed CHD in ZIPs though. It's technically not hard to do. Two good reason to do it. I don't think it can be confusing to users as there is not really a chance they ended up accidentally not compressing the CHDs. Such a ZIP needs to be deliberate created. Or maybe I'm wrong, maybe ZIP tools these days are smart and figure out that compressing the CHD file is a losing game and automatically store them uncompressed? That would be even better. I like having 1 full game in a single ZIP file. It's THE feature of DOSBox Pure. If we end up adding a huge complex library dependency like libchdr then I want it to work with the principles of this core.

Anyway, now people are talking about supporting DVDs and emulating multiple CDROMs. Please don't do this in this issue. These features both are very separate requests and both would require large engineering efforts. DOSBox Pure image mounting is built around the "Disc Control" feature of libretro. That feature sadly is built very much for a console and thus only ever thinks about there being one drive. Thus we ended up with only ever one CD-ROM being mounted on D: or one floppy disk on A:. But, I don't think that's a big issue. Most real computers didn't have multiple CD-ROMs. I don't think a real DOS game exists that requires multiple CD-ROMs. Switching the mounted image takes pressing 2 or 3 hotkeys in RetroArch and can be done in a second. Being bound to this simplicity might be limiting in some forms, but maybe overall the core is easier to use due to it.

Looking for complex file patterns is not something we can do. So we can't blindly scan for .CHD files with the same prefix followed by multiple arguments like disc numbers and drive letters. For example, this core runs on Xbox Series consoles via UWP which provides an application only VERY SLOW access to the file system. Because we don't know if CHD files exist, the scan for them would need to be done on every startup. Thus it's either a fixed series of names (.chd, .2.chd, .3.chd) or M3U files.

I was talking about DVD only for information that games exists on this support. I was not asking to implement it :p
But anyway thank you for all the useful information you have given. Especially about having multiple drive. Knowing it is not an option help to see what can be done, no need to open an issue for that then. I forgot RetroArch is available on very different platforms and the lowest common denominator is the target. So it will be CDs and floppies images embedded into the archive only, If the reading speed is not a problem with embedded CHD, so be it! That's totally okay for me.

About games on multiple CD-ROM, to my knowledge, none require to have multiple CD-ROM drives. A few can use up to four if configured to do so. I think using the cycling system of libretro is a safe assumption.

About deflate compressors, I don't know if modern implementation are smart enough to drop the compression on files with random content. It can be done with LZMA, LZMA2. It is by design with PA and other 'smart' compressors as they compress each file with the best algorithm after detecting the format but they are very very slow.

Finally, for information, many PCs during 97-2001 had two CD-ROM drives. Often it was a fast CD-ROM drive coupled with a burner.

[Torinde]

Related:

• supported in https://github.com/MartyShepard/DOSBox-Optionals
• supported in Support for CHD format joncampbell123/dosbox-x#1822
• Support MAME's Compressed Hunks of Data (CHD) filesystem format dosbox-staging/dosbox-staging#166
• [Feature Request] Add support for parent chd files libretro/beetle-psx-libretro#587
• Support chd parent/clone loading rtissera/libchdr#17

[alexb3d]

supporting DVDs and emulating multiple CDROMs

As I commented in this post, the support is random, DOSBox does not support DVDs, I guess this comes from RetroArch.

Emulating multiple CR-ROMs can be a big mistake. Windows 9x is very primitive with paths and many games ask for installation drive, if you change the letter, it doesn't work.
Personally it happened to me a lot. I didn't know anyone who had multiple CD-ROMs, but if you had a reader drive and a burner, if you installed on one and then loaded it on another, it didn't work.

I understand that it may be more comfortable, not changing discs like Steam, but emulation is about memories and this is also part of the retro experience, changing CDs, FDs, is like blowing the acetate disc before putting it in the disc player.