-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug fixes and improvements #54
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not entirely sure what exactly is mismatching, but with my existing test case XMLs produced from dumpsxiso without your PR, built images aren't matching Redump anymore.
It's probably a good idea to ensure compatibility with older XMLs. For reference, here's one from Advan Racing:
<iso_project image_name="Advan Racing (Japan).bin" cue_sheet="Advan Racing (Japan).cue">
<track type="data">
<identifiers system="PLAYSTATION" application="PLAYSTATION" volume="ADVANRACING" publisher="ATLUS" creation_date="1998101321000000+0" modification_date="1998101321000000+0"/>
<license file="test_advan/license_data.dat"/>
<default_attributes gmt_offs="36" xa_attrib="32" xa_perm="1365" xa_gid="0" xa_uid="0"/>
<directory_tree gmt_offs="0">
<file name="SYSTEM.CNF" source="test_advan/SYSTEM.CNF" type="data"/>
<dir name="TEX" source="test_advan/TEX" gmt_offs="0"/>
<dir name="ADVAN" source="test_advan/ADVAN" gmt_offs="0"/>
<file name="MOVIE.EXE" source="test_advan/MOVIE.EXE" type="data"/>
<file name="SELECT.EXE" source="test_advan/SELECT.EXE" type="data"/>
<dir name="TEX">
<file name="COCK2.TMD" source="test_advan/TEX/COCK2.TMD" type="data"/>
<file name="COCK1.TMD" source="test_advan/TEX/COCK1.TMD" type="data"/>
<file name="REFSEL.TIM" source="test_advan/TEX/REFSEL.TIM" type="data"/>
</dir>
<file name="ARMRG.MRG" source="test_advan/ARMRG.MRG" type="data"/>
<file name="ARMRG.HED" source="test_advan/ARMRG.HED" type="data"/>
<dir name="XA" source="test_advan/XA" gmt_offs="0"/>
<file name="QS.EXE" source="test_advan/QS.EXE" type="data"/>
<file name="OPT.EXE" source="test_advan/OPT.EXE" type="data"/>
<file name="SLPS_016.89" source="test_advan/SLPS_016.89" type="data"/>
<file name="QA.EXE" source="test_advan/QA.EXE" type="data"/>
<dir name="CDDA" source="test_advan/CDDA" gmt_offs="0">
<file name="TRACK_02.WAV" trackid="02" type="da"/>
</dir>
<dir name="TEX">
<file name="ADVANMK.TIM" source="test_advan/TEX/ADVANMK.TIM" type="data"/>
<file name="LOADING.TMD" source="test_advan/TEX/LOADING.TMD" type="data"/>
</dir>
<dir name="XA">
<file name="ADVANX2.XA" source="test_advan/XA/ADVANX2.XA" type="mixed"/>
<file name="ADVANX1.XA" source="test_advan/XA/ADVANX1.XA" type="mixed"/>
<file name="ADVANX0.XA" source="test_advan/XA/ADVANX0.XA" type="mixed"/>
<file name="ADVANX3.XA" source="test_advan/XA/ADVANX3.XA" type="mixed"/>
</dir>
<dir name="ADVAN">
<file name="ATLUS.STR" source="test_advan/ADVAN/ATLUS.STR" type="mixed"/>
<file name="OPEN.STR" source="test_advan/ADVAN/OPEN.STR" type="mixed"/>
<file name="END.STR" source="test_advan/ADVAN/END.STR" type="mixed"/>
<file name="CM2FIN.STR" source="test_advan/ADVAN/CM2FIN.STR" type="mixed"/>
<file name="CM1FIN.STR" source="test_advan/ADVAN/CM1FIN.STR" type="mixed"/>
<file name="CM0FIN.STR" source="test_advan/ADVAN/CM0FIN.STR" type="mixed"/>
<file name="CM3FIN.STR" source="test_advan/ADVAN/CM3FIN.STR" type="mixed"/>
<file name="CM4FIN.STR" source="test_advan/ADVAN/CM4FIN.STR" type="mixed"/>
<file name="CM5FIN.STR" source="test_advan/ADVAN/CM5FIN.STR" type="mixed"/>
<file name="1993.STR" source="test_advan/ADVAN/1993.STR" type="mixed"/>
<file name="1992.STR" source="test_advan/ADVAN/1992.STR" type="mixed"/>
<file name="1991.STR" source="test_advan/ADVAN/1991.STR" type="mixed"/>
<file name="1994.STR" source="test_advan/ADVAN/1994.STR" type="mixed"/>
<file name="1995.STR" source="test_advan/ADVAN/1995.STR" type="mixed"/>
<file name="1996.STR" source="test_advan/ADVAN/1996.STR" type="mixed"/>
<file name="1997.STR" source="test_advan/ADVAN/1997.STR" type="mixed"/>
</dir>
<dir name="QSEL_B" source="test_advan/QSEL_B">
<file name="2DN1_TIM.HED" source="test_advan/QSEL_B/2DN1_TIM.HED" type="data"/>
<file name="2DN1_TIM.MRG" source="test_advan/QSEL_B/2DN1_TIM.MRG" type="data"/>
<file name="2DN2_TIM.HED" source="test_advan/QSEL_B/2DN2_TIM.HED" type="data"/>
<file name="2DN2_TIM.MRG" source="test_advan/QSEL_B/2DN2_TIM.MRG" type="data"/>
<file name="2DN3_TIM.HED" source="test_advan/QSEL_B/2DN3_TIM.HED" type="data"/>
<file name="2DN3_TIM.MRG" source="test_advan/QSEL_B/2DN3_TIM.MRG" type="data"/>
<file name="2DSS_TIM.HED" source="test_advan/QSEL_B/2DSS_TIM.HED" type="data"/>
<file name="2DSS_TIM.MRG" source="test_advan/QSEL_B/2DSS_TIM.MRG" type="data"/>
<file name="2DST_TIM.HED" source="test_advan/QSEL_B/2DST_TIM.HED" type="data"/>
<file name="2DST_TIM.MRG" source="test_advan/QSEL_B/2DST_TIM.MRG" type="data"/>
<file name="2DT1_TIM.HED" source="test_advan/QSEL_B/2DT1_TIM.HED" type="data"/>
<file name="2DT1_TIM.MRG" source="test_advan/QSEL_B/2DT1_TIM.MRG" type="data"/>
<file name="2DT2_TIM.HED" source="test_advan/QSEL_B/2DT2_TIM.HED" type="data"/>
<file name="2DT2_TIM.MRG" source="test_advan/QSEL_B/2DT2_TIM.MRG" type="data"/>
<file name="2DT3_TIM.HED" source="test_advan/QSEL_B/2DT3_TIM.HED" type="data"/>
<file name="2DT3_TIM.MRG" source="test_advan/QSEL_B/2DT3_TIM.MRG" type="data"/>
<file name="DB_TIM.HED" source="test_advan/QSEL_B/DB_TIM.HED" type="data"/>
<file name="DB_TIM.MRG" source="test_advan/QSEL_B/DB_TIM.MRG" type="data"/>
<file name="GRID_TIM.HED" source="test_advan/QSEL_B/GRID_TIM.HED" type="data"/>
<file name="GRID_TIM.MRG" source="test_advan/QSEL_B/GRID_TIM.MRG" type="data"/>
<file name="OPT_TIM.HED" source="test_advan/QSEL_B/OPT_TIM.HED" type="data"/>
<file name="OPT_TIM.MRG" source="test_advan/QSEL_B/OPT_TIM.MRG" type="data"/>
<file name="SPC_TIM.HED" source="test_advan/QSEL_B/SPC_TIM.HED" type="data"/>
<file name="SPC_TIM.MRG" source="test_advan/QSEL_B/SPC_TIM.MRG" type="data"/>
<file name="SPP_TIM.HED" source="test_advan/QSEL_B/SPP_TIM.HED" type="data"/>
<file name="SPP_TIM.MRG" source="test_advan/QSEL_B/SPP_TIM.MRG" type="data"/>
<file name="SPS_TIM.HED" source="test_advan/QSEL_B/SPS_TIM.HED" type="data"/>
<file name="SPS_TIM.MRG" source="test_advan/QSEL_B/SPS_TIM.MRG" type="data"/>
</dir>
<dir name="QSEL_A" source="test_advan/QSEL_A">
<file name="2DC_TIM.HED" source="test_advan/QSEL_A/2DC_TIM.HED" type="data"/>
<file name="2DC_TIM.MRG" source="test_advan/QSEL_A/2DC_TIM.MRG" type="data"/>
<file name="ADG_TIM.HED" source="test_advan/QSEL_A/ADG_TIM.HED" type="data"/>
<file name="ADG_TIM.MRG" source="test_advan/QSEL_A/ADG_TIM.MRG" type="data"/>
<file name="AFT_TIM.HED" source="test_advan/QSEL_A/AFT_TIM.HED" type="data"/>
<file name="AFT_TIM.MRG" source="test_advan/QSEL_A/AFT_TIM.MRG" type="data"/>
<file name="AG_TIM.HED" source="test_advan/QSEL_A/AG_TIM.HED" type="data"/>
<file name="AG_TIM.MRG" source="test_advan/QSEL_A/AG_TIM.MRG" type="data"/>
<file name="CLS_TIM.HED" source="test_advan/QSEL_A/CLS_TIM.HED" type="data"/>
<file name="CLS_TIM.MRG" source="test_advan/QSEL_A/CLS_TIM.MRG" type="data"/>
<file name="COSC_TIM.HED" source="test_advan/QSEL_A/COSC_TIM.HED" type="data"/>
<file name="COSC_TIM.MRG" source="test_advan/QSEL_A/COSC_TIM.MRG" type="data"/>
<file name="COS_TIM.HED" source="test_advan/QSEL_A/COS_TIM.HED" type="data"/>
<file name="COS_TIM.MRG" source="test_advan/QSEL_A/COS_TIM.MRG" type="data"/>
<file name="PRE_TIM.HED" source="test_advan/QSEL_A/PRE_TIM.HED" type="data"/>
<file name="PRE_TIM.MRG" source="test_advan/QSEL_A/PRE_TIM.MRG" type="data"/>
<file name="SEL_TIM.HED" source="test_advan/QSEL_A/SEL_TIM.HED" type="data"/>
<file name="SEL_TIM.MRG" source="test_advan/QSEL_A/SEL_TIM.MRG" type="data"/>
<file name="SETT_TIM.HED" source="test_advan/QSEL_A/SETT_TIM.HED" type="data"/>
<file name="SETT_TIM.MRG" source="test_advan/QSEL_A/SETT_TIM.MRG" type="data"/>
<file name="SUBJ_TIM.HED" source="test_advan/QSEL_A/SUBJ_TIM.HED" type="data"/>
<file name="SUBJ_TIM.MRG" source="test_advan/QSEL_A/SUBJ_TIM.MRG" type="data"/>
<file name="S_L_TIM.HED" source="test_advan/QSEL_A/S_L_TIM.HED" type="data"/>
<file name="S_L_TIM.MRG" source="test_advan/QSEL_A/S_L_TIM.MRG" type="data"/>
<file name="T_CH_TIM.HED" source="test_advan/QSEL_A/T_CH_TIM.HED" type="data"/>
<file name="T_CH_TIM.MRG" source="test_advan/QSEL_A/T_CH_TIM.MRG" type="data"/>
<file name="VS_TIM.HED" source="test_advan/QSEL_A/VS_TIM.HED" type="data"/>
<file name="VS_TIM.MRG" source="test_advan/QSEL_A/VS_TIM.MRG" type="data"/>
</dir>
<dir name="QSEL_C" source="test_advan/QSEL_C">
<file name="RNK_TIM.HED" source="test_advan/QSEL_C/RNK_TIM.HED" type="data"/>
<file name="RNK_TIM.MRG" source="test_advan/QSEL_C/RNK_TIM.MRG" type="data"/>
</dir>
<dir name="ADVAN">
<file name="EXIV.STR" source="test_advan/ADVAN/EXIV.STR" type="mixed"/>
<file name="VIP.STR" source="test_advan/ADVAN/VIP.STR" type="mixed"/>
<file name="CHA.STR" source="test_advan/ADVAN/CHA.STR" type="mixed"/>
</dir>
<dir name="TEX">
<file name="BAD.TMD" source="test_advan/TEX/BAD.TMD" type="data"/>
<file name="COOL.TMD" source="test_advan/TEX/COOL.TMD" type="data"/>
<file name="GREAT.TMD" source="test_advan/TEX/GREAT.TMD" type="data"/>
<file name="AWFUL.TMD" source="test_advan/TEX/AWFUL.TMD" type="data"/>
<file name="GOOD.TMD" source="test_advan/TEX/GOOD.TMD" type="data"/>
<file name="HOLDOUT.TMD" source="test_advan/TEX/HOLDOUT.TMD" type="data"/>
<file name="NICE.TMD" source="test_advan/TEX/NICE.TMD" type="data"/>
<file name="POOR.TMD" source="test_advan/TEX/POOR.TMD" type="data"/>
</dir>
<dummy sectors="150"/>
</directory_tree>
</track>
<track type="audio" trackid="02" source="test_advan/CDDA/TRACK_02.WAV"/>
</iso_project>
Strange because I leaved default presets if the new xml strings are missing, but I will look at it. Also I'm planing to push another fixes and changes I have made after checking the issue with your old xml. |
I checked my Advan Racing checksum afterwards and it seems like, while matching my test case, it seems to be including the audio track in the .bin file and therefore it still doesn't match Redump. It should be easier for you to test against my GT1 XML instead, as that has no audio tracks and the rebuilt image matches Redump: <iso_project image_name="Gran Turismo (Europe) (En,Fr,De,Es,It) (EDC).bin" cue_sheet="Gran Turismo (Europe) (En,Fr,De,Es,It) (EDC).cue">
<track type="data">
<identifiers system="PLAYSTATION" application="PLAYSTATION" volume="GRAN_TURISMO" publisher="SONY COMPUTER ENTERTAINMENT INC." creation_date="1998041412583300+36"/>
<license file="test_gt1/license_data.dat"/>
<default_attributes gmt_offs="36" xa_attrib="56" xa_perm="17" xa_gid="4000" xa_uid="3025"/>
<directory_tree xa_perm="1365" xa_gid="0" xa_uid="0">
<file name="SYSTEM.CNF" source="test_gt1/SYSTEM.CNF" type="data"/>
<file name="SCES_009.84" source="test_gt1/SCES_009.84" type="data" xa_perm="273"/>
<file name="GTOS.EXE" source="test_gt1/GTOS.EXE" type="data" xa_perm="273"/>
<file name="GTMAIN.EXE" source="test_gt1/GTMAIN.EXE" type="data" xa_perm="273"/>
<file name="GTMENU.EXE" source="test_gt1/GTMENU.EXE" type="data" xa_perm="273"/>
<file name="GTEND.EXE" source="test_gt1/GTEND.EXE" type="data" xa_perm="273"/>
<file name="REPLAY.DAT" source="test_gt1/REPLAY.DAT" type="data"/>
<file name="SYSTEM.DAT" source="test_gt1/SYSTEM.DAT" type="data" xa_perm="273"/>
<file name="GAMEFONT.DAT" source="test_gt1/GAMEFONT.DAT" type="data"/>
<file name="MESSAGES.DAT" source="test_gt1/MESSAGES.DAT" type="data" xa_uid="1000"/>
<file name="TITLE.DAT" source="test_gt1/TITLE.DAT" type="data"/>
<file name="CAR.DAT" source="test_gt1/CAR.DAT" type="data" xa_perm="273" xa_uid="1000"/>
<file name="CARCADE.DAT" source="test_gt1/CARCADE.DAT" type="data" xa_perm="273" xa_uid="1000"/>
<file name="CARINF.DAT" source="test_gt1/CARINF.DAT" type="data"/>
<file name="COURSE.DAT" source="test_gt1/COURSE.DAT" type="data" xa_perm="273" xa_uid="1000"/>
<file name="BG.DAT" source="test_gt1/BG.DAT" type="data" xa_perm="273" xa_uid="1000"/>
<file name="GAMEMENU.DAT" source="test_gt1/GAMEMENU.DAT" type="data" xa_uid="1000"/>
<file name="PITMENU.DAT" source="test_gt1/PITMENU.DAT" type="data" xa_uid="1000"/>
<file name="ARCADE.DAT" source="test_gt1/ARCADE.DAT" type="data"/>
<file name="ARCADE2.DAT" source="test_gt1/ARCADE2.DAT" type="data"/>
<file name="MENU_IMG.ARC" source="test_gt1/MENU_IMG.ARC" type="data" xa_uid="1000"/>
<file name="MENU_RAW.ARC" source="test_gt1/MENU_RAW.ARC" type="data" xa_uid="1000"/>
<file name="SOUND.DAT" source="test_gt1/SOUND.DAT" type="data"/>
<file name="MUSIC.DAT" source="test_gt1/MUSIC.DAT" type="mixed" xa_perm="273"/>
<file name="STREAM.DAT" source="test_gt1/STREAM.DAT" type="mixed" xa_uid="1000"/>
<file name="MEASURE.QA" source="test_gt1/MEASURE.QA" type="data" xa_perm="273"/>
<dummy sectors="150" type="2336"/>
</directory_tree>
</track>
</iso_project> |
I just checked Advan Racing and I got a 1:1 copy, what I think is currently happening is releated to the date of files nor a XML thing, because I changed the behaviour of how they are extracted/stored. I think you alredy has the files previously with a date offsted to your local time and the changes I made does not modify the date at build time, instead modify it at extract time so it can be used with tools like CDmage preserving the same date. |
Support for games built with newer mastering tool
Now correctly generates the descriptor's subheader, year and file order for games built with newer Sony's mastering tool. Changed the way directory records are written to a linear write instead of a recursive write. Changed the way descriptor directories are written from Depth-First Search to Breadth-First Search. The -pt command now support dummies entries. Removed some unnecessary functions. Some minor fixes. |
Tested this and the difference is the dummy form2, I deprecated the |
I reverted back to use I think this can now be safely merged. Also added some QoL in these new commits And some minor fixes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good after the new changes, thank you for putting so much work into this!
However, two remarks that made me not approve it yet:
- I'm still a little concerned about the file timestamp changes. While your changes make perfect sense in terms of usability, I am worried that breaking "working copies" for the existing users may not be a good idea. Do you think this change can be somehow annotated in the XML file so only "new" unpacks with
dumpsxiso
get the time offset changes? - Do you think it makes sense to make CMake configs more consistent by putting Release build outputs in the Release subdirectory? Right now those land in
build/
while Debug lands inbuild/Debug
.
@spicyjpeg Do you have any additional comments?
As for the first point, yes I can incluide it in the xml, however it wouldn't be better to preserve the old timestamp format as an argument? Or maybe both (xml and argument)? Tell me what do you think. As for the second point, yes u are right, I noticed that behaviour recently when I switched to linux. When compiling with MSVS the binaries were output in a "Release" folder but no with GCC. |
I partly agree, but my point is that care should be taken to avoid cases where people have their existing XML files and working directories, then update mkpsxiso, and their built ISOs "break". There are people who rely on mkpsxiso providing them accurate rebuilds to reduce the delta patch sizes, so with an update like this their xdeltas will inflate in size only because they have not re-unpacked the original ISO, which IMO they should not have to do. |
Ok now the mkpsxiso timestamp generation should be retro-compatible. Instead of adding a new string to the xml, I just did a check on one of the new features, so since these string are new they shouldn't be present in older xml's. It would also be good to close all the issues mentioned in this PR if it is merged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok now the mkpsxiso timestamp generation should be retro-compatible. Instead of adding a new string to the xml, I just did a check on one of the new features, so since these string are new they shouldn't be present in older xml's.
Excellent idea!
I ran my tests again with your changes and now they work as expected - HS and CTR have some other differences I haven't yet investigated, but those are not related to your work and appear in the current version too: CTR likely needs unpacking again once your PR is in, as per your form2 changes:
Thank you for all the work! @spicyjpeg do you have any additional comments? If not I think we can merge it soon and make a new release.
@N4gtan I have one more favor to ask please. With how many iterations your PR had it's not trivial to infer what exactly is changed by only looking at the commit history - could you prepare an user-friendly changelog out of your change so we could include it in the release?
Is the following okay or you mean a more simplified log? Full changelog:
Changes only to dumpsxiso
Changes only to mkpsxiso
|
Looks good, I can work with that. |
v2.10 with your changes is now out: https://github.com/Lameguy64/mkpsxiso/releases/tag/v2.10 Thanks a lot once again! That was a lot of work. |
Updated Actions
Created a release and debug cmake profile
Updated readme compile instruccions
Update example.xml for the new format
Avoid read funcionts being call past EoF
Fixed some compiling warnings
Added ability to maintain EDC Form 2 integrity via xml, fixes this #46
Fix posible crash if end map iterator cannot be dereferenced
Dummy sectors now saves the subheader value to the xml, so it can regenerate the correct values for non-standart (0x00/0x20) sectors
Added a HACK to check if there is a post gap dummy and some warnings if the gap is a non-standart 150 one, fixes this #44
Now outputs by default an .xml file with the name of the input is no -s arg is given, this is a QOL for easy drag & drop
Fixed calculation length of dir/file entries, fixes this #52 and this #23
Removed some duplicated lines
Dirs entries are now sort before the calculation of LBA tree
Now the program supports read/update timestamps prior to 1970 in windows
Now files are extracted/built with the original timestamps instead of being offsetted previously and then re calculated at build time. This is for better compatibily with other CD authoring tools
Added ability to maintain File Flags for games with non-standart (0x00/0x02) flags, like this #53
Fixed Root date year. Root overflows dates past 1999, but there are games from 2003 that has correct (0x67) root year, maybe that got fixed in a newer version of sony's mastering tool? I need more evidence