Use Kaitai Struct

[spotlightishere]

It's been discussed to use a kaitai struct for modifying, as this would presumably greatly assist with string manipulation and other design concerns going forward. However, building kaitai_struct_cpp_stl_runtime has proven to be a project of its own. It currently utilizes CMake which devkitPro chokes on, and other endianness + iconv concerns have arisen.

But it'd be nice. :)

Attached is a sample pd.dat ksy within a spoiler tag because GitHub apparently does not support the .ksy extension.

pd.dat

meta:
  id: pd
  file-extension: dat
  endian: be
seq:
  - id: file_header
    type: header
    size: 0x80
  - id: info
    type: info_block
    size: 0x256F
  - id: kana
    type: kana_block
    size: 0x040c
types:
  header:
    seq:
      - id: magic
        contents: 'PDFF'
      - id: version
        contents: [2]
        doc: <|
          This is checked by all known PD SDKs to be less than or equal to 2.
          It's assumed that 1 exists with lesser features - presumably without profiles.
  info_block:
    seq:
      - id: magic
        contents: 'INFO'
      - id: block_size
        type: u4
      - id: intentionally_null
        type: u4
      - id: preset_values
        contents: [0x01, 0x08, 0x00]
        doc: These do not appear to be read yet are set if uninitialized.
      - id: first_timestamp
        type: u8
      - id: profile_name
        type: str
        size: 57
        encoding: UTF-16BE
        doc: This is used within additional profiles. We do not support it.
      - id: first_name
        type: str
        size: 64
        encoding: UTF-16LE
        doc: Tyically written with Romaji.
      - id: surname
        type: str
        size: 64
        encoding: UTF-16LE
        doc: Tyically written with Romaji.
      - id: postal_code
        type: str
        size: 34
        encoding: UTF-16LE
      - id: unknown_one
        type: u2
        doc: Observed to only be 0x19e.
      - id: state_or_prefecture
        type: str
        size: 64
        encoding: UTF-16LE
      - id: city
        type: str
        size: 64
        encoding: UTF-16LE
      - id: address
        type: str
        size: 256
        encoding: UTF-16LE
      - id: apartment_number
        type: str
        size: 256
        encoding: UTF-16LE
      - id: phone_number
        type: str
        size: 64
        encoding: UTF-16LE
      - id: email_address
        type: str
        size: 252
        encoding: UTF-16LE
      - id: padding
        contents: [0, 0, 0]
      - id: second_timestamp
        type: u8

  kana_block:
    seq:
      - id: magic
        contents: 'KANA'
      - id: block_size
        type: u4
      - id: preset_value
        contents: [0x01, 0x00, 0x00, 0x00]
        doc: <|
          The literal 0x1 is additionally written over 1 byte via memcpy.
          As this area has been memset to 0 prior to run and never modified,
          we assume the next few bytes are null as well.
      - id: first_name
        type: str
        size: 64
        encoding: UTF-16LE
      - id: surname
        type: str
        size: 64
        encoding: UTF-16LE

[zurgeg]

Well then port CMake to Wii
In all seriousness though, can't CMake be customized to run certain commands? IIRC the DevKitPro makefiles basically call g++ for a different arch. Also CMake (IIRC) allows for custom command running, so we can use that o2dol thing or whatever(TM) (or create a library, idk)
EDIT: https://stackoverflow.com/questions/17275348/how-to-specify-new-gcc-path-for-cmake will allow us to use the DevkitPro version of G++/CC

[spotlightishere]

Thankfully devkitPro provides somewhat useful cmake definitions which we could use, but it often mixes the building system's definitions with those provided by devkitPPC. We'd have to take a look at where things are causing interfering and go from there, I'm not certain upstream definitions are complete enough to cross compile with :/

[spotlightishere]

I was able to fully integrate kaitai_struct_cpp_stl_runtime with little modification as of 40adb33. However, libiconv spews out quite a curious error later into building. This is reproducible with both 1.16 and 1.15.

powerpc-eabi-gcc -DHAVE_CONFIG_H -DEXEEXT=\"\" -I. -I.. -I../lib  -DDEPENDS_ON_LIBICONV=1 -DDEPENDS_ON_LIBINTL=1 -I/opt/devkitpro/portlibs/ppc/include  -O2 -mcpu=750 -meabi -mhard-float -ffunction-sections -fdata-sections -c -o areadlink.o areadlink.c
In file included from /opt/devkitpro/devkitPPC/powerpc-eabi/include/signal.h:6,
                 from ./signal.h:52,
                 from /opt/devkitpro/devkitPPC/powerpc-eabi/include/time.h:178,
                 from ./time.h:41,
                 from ./sys/stat.h:44,
                 from ./fcntl.h:58,
                 from careadlinkat.h:23,
                 from areadlink.c:27:
./signal.h:578:1: error: expected identifier or '(' before 'const'
  578 | _GL_FUNCDECL_SYS (sigismember, int, (const sigset_t *set, int sig)
      | ^~~~~~~~~~~~~~~~
./signal.h:578:1: error: expected ')' before '&' token
  578 | _GL_FUNCDECL_SYS (sigismember, int, (const sigset_t *set, int sig)
      | ^~~~~~~~~~~~~~~~
./signal.h:578:1: error: expected ')' before '!=' token
  578 | _GL_FUNCDECL_SYS (sigismember, int, (const sigset_t *set, int sig)
[...]

An offending line within our local srclib/signal.in.hmay look similar to the following:

/* Test whether a given signal is contained in a signal set.  */
# if @HAVE_POSIX_SIGNALBLOCKING@
/* This function is defined as a macro on Mac OS X.  */
#  if defined __cplusplus && defined GNULIB_NAMESPACE
#   undef sigismember
#  endif
# else
_GL_FUNCDECL_SYS (sigismember, int, (const sigset_t *set, int sig)
                                    _GL_ARG_NONNULL ((1)));
# endif
_GL_CXXALIAS_SYS (sigismember, int, (const sigset_t *set, int sig));
_GL_CXXALIASWARN (sigismember);

in which HAVE_POSIX_SIGNALBLOCKING is semi-rightfully set to 0.

The comment regarding macOS provides insight regarding what's going on here - the sig* family are macros. The normal definition is never used, as signal support does not exist under the Wii.

> grep -r "sigismember" /opt/devkitpro/devkitPPC/powerpc-eabi/include
/opt/devkitpro/devkitPPC/powerpc-eabi/include/sys/signal.h:int sigismember (const sigset_t *, int);
/opt/devkitpro/devkitPPC/powerpc-eabi/include/sys/signal.h:#define sigismember(what,sig) (((*(what)) & (1<<(sig))) != 0)

We can simply remedy this by

sed -i 's/HAVE_POSIX_SIGNALBLOCKING = 0/HAVE_POSIX_SIGNALBLOCKING = 1/' srclib/Makefile

which is a cheap hack at best, but a working hack nonetheless.

The majority of iconv-related issues are due to iconv(8), not the library itself. Who would've guessed?

[spotlightishere]

Shortly after resolving this, I ran into easily thousands of other utility related issues.
Due to such, I've gone ahead and copied the iconv implementation from musl.

[spotlightishere]

Integrated as of dc758b9.

[oscie57]

yay