Upload version 3.3

Changelog for v3.3 release:
- fixed regression regarding deduplication of consecutive data lines
  (added in v3.2) messing up disassembly split into separate files
  (i.e. reconstructed source files) (fixes issue #16)
- prevent very long lines when deduplicating consecutive data lines by
  truncating hex output/display + appending '..'
- added support for regions with multiple access sizes when generating/
  outputting possible hints for code objects
- extended pretty printer (modules/module_pretty_print.py) to produce
  hex dumps of bytes and other bytes-like objects
- extended file writer (module_miscellaneous.py) to create folders for
  destination path if missing
- fixed regex strings in 're.match' and 're.search' calls producing
  'SyntaxWarning's with Python 3.12+ due to invalid escape sequences
  (https://stackoverflow.com/a/52335971/1976617)
- applied various minor changes (console output, code formatting,
  comments, etc.)

Author: Fonic

CHANGELOG.md

@@ -1,3 +1,13 @@
+## Changelog for v3.3 release
+
+- fixed regression regarding *deduplication of consecutive data lines* (added in v3.2) messing up disassembly split into separate files (i.e. reconstructed source files) (fixes issue #16)
+- prevent very long lines when deduplicating consecutive data lines by truncating hex output/display + appending '..'
+- added support for regions with multiple access sizes when generating/outputting possible hints for code objects
+- extended pretty printer (`modules/module_pretty_print.py`) to produce hex dumps of bytes and other bytes-like objects
+- extended file writer (`module_miscellaneous.py`) to create folders for destination path if missing
+- fixed regex strings in `re.match` and `re.search` calls producing `SyntaxWarning`s with Python 3.12+ due to invalid escape sequences (https://stackoverflow.com/a/52335971/1976617)
+- applied various minor changes (console output, code formatting, comments, etc.)
+
 ## Changelog for v3.2 release
 
 - added algorithm to *deduplicate consecutive data lines* in formatted disassembly (*greatly* reduces disassembly size for data objects)
@@ -38,3 +48,7 @@
 - initial release
 - monolithic (everything in one single source file)
 - originally named 'wcdctool' (*Watcom Decompilation Tool*)
+
+##
+
+_Last updated: 08/31/24_

Executables/HARVEST.EXE

@@ -5,7 +5,7 @@
 # ----------------------------------------------------------------------------
 #                                                                            -
 #  Created by Fonic <https://github.com/fonic>                               -
-#  Date: 06/20/19 - 04/06/22                                                 -
+#  Date: 06/20/19 - 08/31/24                                                 -
 #                                                                            -
 # ----------------------------------------------------------------------------
 
@@ -361,4 +361,34 @@ Object 2:
 #3) offset = 0000D902H, length = 00000096H, type = data, mode = bytes, comment = Default highscore table; structure: 3 chars initials, 1 byte wins, 1 byte wins, 0, 4 bytes score (?)
 3) offset = 0000D902H, length = 00000096H, type = data, mode = struct:chars[3]:bytes[3]:dword, comment = Default highscore table (3 chars initials, 1 byte wins, 1 byte wins, 0, 4 bytes score) (?)
 4) offset = 0000E7A4H, length = 00000138H, type = data, mode = strings, comment = List of strings for keyboard keys
-5) start = 00024B40H, end = 00024BA0H, type = data, mode = comment, comment = Emulation of registers A0-A14 / B0-B14 of original arcade machine
+5) start = 00024B40H, end = 00024BA0H, type = data, mode = comment, comment = Emulation of registers A0-A14 / B0-B14 of original arcade machine's architecture (TMS34010)
+
+
+                           Global Info (section 1)
+==============================================================================
+
+#
+# MK1.EXE seems to contain significantly less named labels in its debug info
+# (likely limited to exported symbols only, i.e. code blocks that are called
+# from other modules and thus need to be exported) compared to MK2.EXE. As a
+# result, there are functions like 'WHO_IS_ALONE' which run a lot longer than
+# they should and actually span over other, unnamed functions (for which there
+# are no named symbols available in debug info).
+#
+# Note that this is just a stub for demonstration, there are likely many more
+# missing symbols + long functions that require splitting. Since MK1 and MK2
+# are quite similar (MK2 seems to have used and expanded on MK1's code base),
+# a thorough comparison might help reveal further missing symbols.
+#
+# For details, refer to:
+# https://github.com/fonic/wcdatool/issues/15#issuecomment-2308410432
+#
+
+Name:  GAME_FINISHED
+  address      = 0001:0000ca2e
+  module index = 12
+  kind:          (code)
+Name:  GAME_OVER
+  address      = 0001:0000cb9e
+  module index = 12
+  kind:          (code)

Hints/MK1.EXE.txt

@@ -5,7 +5,7 @@
 # ----------------------------------------------------------------------------
 #                                                                            -
 #  Created by Fonic <https://github.com/fonic>                               -
-#  Date: 06/20/19 - 07/25/23                                                 -
+#  Date: 06/20/19 - 08/31/24                                                 -
 #                                                                            -
 # ----------------------------------------------------------------------------
 
@@ -101,11 +101,12 @@ Object 2:
 208) start = 0003419EH, end = 000341B6H, type = data, mode = string, comment = String
 209) start = 000341B6H, end = 000341C6H, type = data, mode = string, comment = String
 210) start = 000341C6H, end = 000341D1H, type = data, mode = string, comment = String
+211) start = 000922ECH, end = 0009234CH, type = data, mode = comment, comment = Emulation of registers A0-A14 / B0-B14 of original arcade machine's architecture (TMS34010)
 # Code in data object
 # -> just a stub, there are many more
 # -> may be removed once this has been automated using tracing disassembler
-211) start = 0001D97DH, end = 0001D991H, type = code, mode = default, comment = Code (?)
-212) start = 0001D991H, end = 0001D9A5H, type = code, mode = default, comment = Code (?)
-213) start = 0001D9A5H, end = 0001D9DBH, type = code, mode = default, comment = Code (?)
-214) start = 0001D9DBH, end = 0001DA54H, type = code, mode = default, comment = Code (?)
+300) start = 0001D97DH, end = 0001D991H, type = code, mode = default, comment = Code (?)
+301) start = 0001D991H, end = 0001D9A5H, type = code, mode = default, comment = Code (?)
+302) start = 0001D9A5H, end = 0001D9DBH, type = code, mode = default, comment = Code (?)
+303) start = 0001D9DBH, end = 0001DA54H, type = code, mode = default, comment = Code (?)
 #...

Hints/MK2.EXE.txt

@@ -24,13 +24,13 @@ Thus, I began writing my own tool. What originally started out as *mkdecomptool*
 
 Note that while wcdatool performs the tasks it is designed for quite well, it is not intended to compete with or replace high-end tools like *IDA Pro* or *Ghidra*.
 
-## Current state and future development
+## Current state / future development
 
-Wcdatool is *work in progress*. You can tell from looking at the source code - there's tons of TODO, TESTING, FIXME, etc. flying around. Also, it is relatively slow as performance has not been the main focus ([Cython](https://cython.org/) might be utilized in the future to increase performance).
+Wcdatool works quite well in its current state - you'll get a well-readable, reasonably structured disassembly output (*objdump* format, *Intel* syntax). Check out issues [#9](https://github.com/fonic/wcdatool/issues/9) and [#11](https://github.com/fonic/wcdatool/issues/11) for games other than *Mortal Kombat* that wcdatool worked nicely for thus far. **Please note that wcdatool works best when used on executables that contain debug symbols.** If you come across other *unstripped* *Watcom*-based DOS applications that may be used for further testing and development, please let me know.
 
-Nevertheless, it works quite well in its current state - you'll get a well-readable, reasonably structured disassembly output (*objdump* format, *Intel* syntax). Check out issues [#9](https://github.com/fonic/wcdatool/issues/9) and [#11](https://github.com/fonic/wcdatool/issues/11) for games other than *Mortal Kombat* that wcdatool worked nicely for thus far. Please note that wcdatool works best when used on executables that contain debug symbols. If you come across other *unstripped* *Watcom*-based DOS applications that may be used for further testing and development, please let me know.
+**However, the current approach has reached its EOL.** There is no point in advancing it any further (aside from fixing bugs), as there are limits inherent to the fundamental design that cannot be overcome easily. Thus, the next major goal is to cleanly *rewrite the disassembler module* and transition from *static code disassembly* to *execution flow tracing* (e.g. *Mortal Kombat 2* executable contains code within its data object, which is neither discovered nor analyzed with the current approach). Also, instead of treating objects separately, a *linear unified address space* containing all object data shall be implemented. This will allow to *apply fixups on a binary level*, which should simplify dealing with references that cross object boundaries and with placeholders (stubs) that are replaced via fixups at run time.
 
-The *next major goal* is to cleanly rewrite the disassembler module and transition from *static code disassembly* to *execution flow tracing* (e.g. *Mortal Kombat 2* executable contains code within its data object, which is neither discovered nor processed with the current approach).
+Last but not least, wcdatool in its current state is relatively slow, as performance has not been the main focus during development. [Cython](https://cython.org/) might be utilized in the future to increase performance.
 
 ## Output sample
 
@@ -97,17 +97,18 @@ There are multiple ways to use *wcdatool*, but the following instructions should
 
 7. Have a look at the results in `wcdatool/Output`:
    - File `<name-of-executable>_zzz_log.txt` contains *log messages* (same as console output, but without coloring/formatting)
-   - Files `<name-of-executable>_disasm_object_x_disassembly_plain.asm` contain *plain disassembly*
-   - Files `<name-of-executable>_disasm_object_x_disassembly_formatted.asm` contain *formatted disassembly*
-   - Folder `<name-of-executable>_modules` contains *formatted disassembly split into separate files* (this attempts to reconstruct the application's original source files if corresponding debug information is available)
+   - Files `<name-of-executable>_disasm_object_x_disassembly_plain.asm` contain *plain disassembly* (unmodified *objdump* output, useful for reference)
+   - Files `<name-of-executable>_disasm_object_x_disassembly_formatted.asm` contain *formatted disassembly* (this is arguably the most interesting/useful output)
+   - Files `<name-of-executable>_disasm_object_x_disassembly_formatted_deduplicated.asm` contain *formatted deduplicated disassembly* (same as above, but with data portions being compressed for increased readability where applicable)
+   - Folder `<name-of-executable>_modules` contains *formatted disassembly split into separate files* (same as above, additionally attempts to reconstruct an application's original source files if corresponding debug information is available)
 
    **NOTE:** if you are new to assembler/assembly language, check out this [x86 Assembly Guide](https://www.cs.virginia.edu/~evans/cs216/guides/x86.html)
 
 8. Refine the output by analyzing the disassembly, updating the object hints and re-running *wcdatool* (i.e. loop steps 5-8):
-   - Identify and add hints for regions in code objects that are actually data (look for `; misplaced item` comments, `(bad)` assembly instructions and labels with `; access size` comments)
+   - Identify and add hints for regions in code objects that are actually data (look for `; misplaced item` comments, `(bad)` assembly instructions and labels with trailing `; access size` comments)
    - Identify and add hints for regions in data objects that are actually code (look for `call`/`jmp` instructions in code objects with fixup targets pointing to data objects)
    - Check section `Possible object hints` of *wcdatool*'s console output / log file for suggestions (not guaranteed to be correct, but likely a good starting point)
-   - *The ultimate goal here is to eliminate all (or at least most) warnings issued by wcdatool*. Each warning points out a region of the disassembly that does currently seem flawed and therefore requires further attention/investigation. Note that there is a *cascading effect* at work (e.g. a region of data that is falsely intepreted as code may produce bogus branches, leading to further warnings), thus warnings should be tackled one (or few) at a time from first to last with *wcdatool* re-runs in between
+   - *The ultimate goal is to eliminate all (or at least most) warnings issued by wcdatool*. Each warning points out a region of the disassembly that does currently seem flawed and therefore requires further attention/investigation. Note that there is a *cascading effect* at work (e.g. a region of data that is falsely intepreted as code may produce bogus branches, leading to further warnings), thus warnings should be tackled one (or few) at a time from first to last with *wcdatool* re-runs in between
 
    **NOTE:** this is by far the most time-consuming part, but *crucial* to achieve good and clean results (!)
 
@@ -153,4 +154,4 @@ If you want to get in touch with me, give feedback, ask questions or simply need
 
 ##
 
-_Last updated: 08/12/23_
+_Last updated: 08/31/24_