Debug symbols for Godot builds

[bruno-garcia]

Sentry has custom debug file repositories. For example, for Android and Apple symbols, which are managed by Sentry itself.
Godot doesn't have a symbol server, but we could collect those symbols and provide them as part of a Godot custom symbol server to Sentry customers.

But for that, Godot builds would need to generate debug files. That's tracked here:

• Distribute official builds with full debugging symbols godotengine/godot-proposals#1342

[limbonaut]

I'll summarize my findings in this post.

I think proposal #1342 means to provide separate builds with debug symbols, which are not meant for game release distribution.

Currently, Godot for Windows is built with scons lto=full option for performance reasons, particularly it is claimed that it results in a better GDScript VM performance than MSVC builds. This option is used with MinGW/GCC toolchain to produce Windows release artifacts, in particular. It is mentioned in some places, that such builds using this option do not produce reliable debug symbols. In my testing, however, Godot builds produced using the following command scons target=template_release lto=full debug_symbols=yes result in usable symbols, at least on Linux platform with GCC 14.2.1. With a simple crash project using OS.crash() via GDScript, I was able to debug the program without much of an issue.

Verifying Linux build using GDB and addresses from a stack trace

Reading symbols from godot.linuxbsd.template_release.x86_64.ltofull...
(gdb) info symbol 0x056cc1a
GDScriptInstance::callp(StringName const&, Variant const**, int, Callable::CallError&) + 522 in section .text
(gdb) info symbol 0x38739c0
Variant::clear()::needs_deinit in section .rodata
(gdb) info symbol 0x2b05cae
core_bind::OS::crash(String const&) + 46 in section .text

It's a different story with MinGW/GCC version 13.1.0 on Arch Linux. Using the stable version of mingw-w64 from Arch Linux repos, the builds produced are considerably smaller and do not appear to contain usable debug symbols. A typical debug build of Godot is around 1Gib+ in size. Windows build I compiled that should contain debug symbols amounts to 100Mib, which is slightly larger than a typical release binary. The Windows build was produced using MinGW/GCC 13.1.0 on Arch Linux with the following options: scons target=template_release lto=full debug_symbols=yes use_mingw=yes.
Here is the output of objdump -tC:
godot.windows.template_release.symbols_dump_mingw_13_1.txt.zip

Using MinGW/GCC version 14.2.0 on Windows (installed via scoop), the binary produced is ~90MB in size, which again is too small for a debug build. Symbols also are not findable by GDB/addr2line using the addresses from a stack trace - it seems to be true for both lto=full and lto=none. Unsure why.

Verifying Windows build using GDB and addresses from a stack trace

Reading symbols from .\bin\godot.windows.template_release.x86_64.exe...
(gdb) info symbol 0x29f24b8
No symbol matches 0x29f24b8.
(gdb) info symbol 0x3417d34
No symbol matches 0x3417d34.
(gdb) info symbol 0x11373cc
No symbol matches 0x11373cc.
(gdb)

Related code:

• Source lines for scons lto option in Godot
• Source lines for scons production option

Typical file sizes for reference

• Editor binary ~125M
• Editor symbols ~1.5G
• Editor symbols (zipped) ~435M
• Template binary ~77M
• Template symbols ~1.1G
• Template symbols (zipped) ~294M

Related:

• Distribute official builds with full debugging symbols godotengine/godot-proposals#1342
• Debug symbols for Google Play godotengine/godot#63060
• Distribute official builds with full debugging symbols godotengine/godot-proposals#1342 (comment)
• Add support for Microsoft debug symbols server with official binaries godotengine/godot-proposals#8861
• Comments
  • Add global crash handler/logger godotengine/godot-proposals#1896 (comment)
  • Add global crash handler/logger godotengine/godot-proposals#1896 (comment)

[bruno-garcia]

the binary produced is ~90MB in size

Wasn't the debug information output written on a separate file?
I wouldn't expect the final executable size to increase much if at all. But I do expect additional debug files to be created.

If the debug files are not by default split, it's an additional step after compilation (except on Windows, where exe/pdbs are already created separately. Although that might not be the case with MinGW/GCC where it outputs debug info in DWARF format, and might require explicit parameter to split executable and debug info.

[limbonaut]

Wasn't the debug information output written on a separate file?
I wouldn't expect the final executable size to increase much if at all. But I do expect additional debug files to be created.

That is the size of the binary with the debug symbols section included, no separate file. Godot has an option for split separate_debug_symbols=yes|no which is off by default. lto=full seems to "eat" symbols away on MinGW/GCC, whichever version I tried.

[bruno-garcia]

I see. I wonder if @vaind has experience with MinGW/GCC and lto=full.
Without Godot building with debug info, our attempt to improve the debugability of Godot will be quite limited.

[vaind]

no experience with LTO on mingw I'm afraid

[bruno-garcia]

We got some instructions on the docs on how to build Godot with debug symbols. And how to upload those to Sentry to get proper/readable stack traces from crash dumps:

• https://docs.sentry.io/platforms/godot/configuration/stack-traces/

[bruno-garcia]

Since there's nothing we can do at this point other than document how to build Godot with debug files, which we did here:

https://docs.sentry.io/platforms/godot/configuration/stack-traces/

We'll close this as done.