Skip to content

Conversation

@smoogipoo
Copy link
Contributor

@smoogipoo smoogipoo commented Jun 9, 2025

I've gone super deep into this. Enough so that I can't write up everything I've done to try and resolve the issue, and at the end of the day I have no idea why things are as they are. I'll try to condense my findings.

Here's a test. Download the latest Tachyon release and perform the following using the entitlements on master:

> xattr -dr com.apple.quarantine osu\!b.app

> codesign -f -s - --entitlements ~/Repos/osu-deploy/osu.entitlements -o runtime --deep osu\!b.app
osu!b.app: replacing existing signature

> ./osu\!b.app/Contents/MacOS/osu\!
Failed to load /Users/smgi/Downloads/osu!b.app/Contents/MacOS/libhostfxr.dylib, error: dlopen(/Users/smgi/Downloads/osu!b.app/Contents/MacOS/libhostfxr.dylib, 0x0001): tried: '/Users/smgi/Downloads/osu!b.app/Contents/MacOS/libhostfxr.dylib' (code signature in <FDA96162-34BD-3443-9267-5E39A0CABDC2> '/Users/smgi/Downloads/osu!b.app/Contents/MacOS/libhostfxr.dylib' not valid for use in process: mapping process and mapped file (non-platform) have different Team IDs), '/System/Volumes/Preboot/Cryptexes/OS/Users/smgi/Downloads/osu!b.app/Contents/MacOS/libhostfxr.dylib' (no such file), '/Users/smgi/Downloads/osu!b.app/Contents/MacOS/libhostfxr.dylib' (code signature in <FDA96162-34BD-3443-9267-5E39A0CABDC2> '/Users/smgi/Downloads/osu!b.app/Contents/MacOS/libhostfxr.dylib' not valid for use in process: mapping process and mapped file (non-platform) have different Team IDs)
The library libhostfxr.dylib was found, but loading it from /Users/smgi/Downloads/osu!b.app/Contents/MacOS/libhostfxr.dylib failed
  - Installing .NET prerequisites might help resolve this problem.
     https://go.microsoft.com/fwlink/?linkid=2063366

Now, using the entitlements in this PR:

> $ codesign -f -s - --entitlements ~/Repos/osu-deploy/osu.entitlements -o runtime --deep osu\!b.app
osu!b.app: replacing existing signature

> $ ./osu\!b.app/Contents/MacOS/osu\!
# Runs!

Now, the error from the first attempt is still different from what happens if you try to run the app directly after downloading, which is:

Failed to create CoreCLR, HRESULT: 0x8007000C

I don't know exactly why this is but I expect it's due to the adhoc signature. I can reproduce this error using my developer certificate. Thus, I'm only using the above as a demonstration that simply changing the entitlements fixes the issue.

From what I can surmise:

  • The error only happens on osx-x64 or Rosetta, and to fix it we need to add this entitlement.
  • The entitlement is itself stated as a requirement for .NET: https://learn.microsoft.com/en-au/dotnet/core/install/macos-notarization-issues#default-entitlements At least, it's stated as a requirement for the "appHost" which would be the osu! executable itself, but it likely goes beyond that for the case where you replace more than just the host. .NET Entitlements.
  • Why does the app work when we don't provide the entitlement? I haven't figured this one out. The thing is, the output of codesign -d --entitlements - osu\!.app/Contents/MacOS/libcoreclr.dylib don't include that entitlement on either release. As far as every tool I've used can tell me, the two files look practically identical between the releases. I'd expect that re-signing would lock down the app further, yet...
  • I recognise that this entitlement seems sketchy to add, but there is precedent for this being done [1] [2] [3] [4] [5].

@balazserdeszlogmein
Copy link

You should never add this entitlement to any public facing application/executable. It's quite literally there what your problem is, your dylib is signed with a certificate from a different team than what the main executable is signed with.

Use library load constraints on the main executable to allow dylibs signed by the team of the dylib's signing certificate, or sign the dylib and the main executable with the same certificate.

@peppy
Copy link
Member

peppy commented Jun 10, 2025

If that statement holds, the simple fix is likely to drop the certificate on the satori lib files before running vpk (I guess it's skipping signing because another signature is present? Weird the same issue doesn't apply to dotnet library files as well?)

@smoogipoo
Copy link
Contributor Author

smoogipoo commented Jun 10, 2025

Here's the codesigning info for a few builds:

To be clear, we only replace libcoreclr.dylib, libclrjit.dylib, and System.Private.CoreLib.dll with our own builds, and only in 2025.607.1. We do not replace the main executable (osu!).

Unless I'm missing something, everything looks signed correctly to me. The only difference I can find is Runtime Version on the "bad" build on the replaced files is 14.5.0 versus 14.2.0 on the "good" build. But I've also tested signing with --runtime-version 14.2.0 to no effect.

your dylib is signed with a certificate from a different team

I can understand this in context of the ad-hoc certificate I used in the OP and I've tried to mention as much but I guess my wording was confusing. I had tested with a developer certificate which gets past that and to the actual error our users are seeing:

Failed to create CoreCLR, HRESULT: 0x8007000C

This is already happening from code executing inside libcoreclr.dylib so that lib is getting loaded and should prove that signing is working correctly, but it's trying to load some external library that fails validation. I'd wager that's libicu or libSystem.Native.dylib based on this code segment I found in the dotnet repo.
It's possible there's some different lib being loaded specifically through our builds that isn't being loaded by the production build, but I couldn't find anything. DYLD_* vars also don't seem to help.

or sign the dylib and the main executable with the same certificate

This is done automatically by the --deep argument. Both in what I've tested with locally, and with what the packer does for production builds: https://github.com/velopack/velopack/blob/10c80a835e8b8ceef42f546b3ed5506d35d7428d/src/vpk/Velopack.Packaging.Unix/OsxBuildTools.cs#L19-L53

If this weren't the case, then we'd expect the app to also fail on osx-arm64 or for previous builds to also fail on osx-x64, but that is not what is happening.

@smoogipoo
Copy link
Contributor Author

I'll look into it a little bit more and see if I can find something specific inside CoreCLR itself, or perhaps a build difference (could be an Xcode SDK difference)... As far as DYLD goes:

Good:

dyld[64949]: dlopen("/Users/smgi/Downloads/osu!a.app/Contents/MacOS/libcoreclr.dylib", 0x00000001)
dyld[64949]:       dlopen(libcoreclr.dylib) => 0x1dad40
dyld[64949]: dlsym(0x1dad40, "DllMain")
dyld[64949]:      dlsym("DllMain") => 0x10bb67b50
dyld[64949]: dlsym(0x1dad40, "PAL_RegisterModule")
dyld[64949]:      dlsym("PAL_RegisterModule") => NULL
dyld[64949]: dladdr(0x10be45170, 0x3099f9340)
dyld[64949]: _dyld_image_count() => 307
dyld[64949]: _dyld_get_image_header(0) => 0x102c47000
dyld[64949]: dladdr(0x102c47000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(1) => 0x7ff811e6c000
dyld[64949]: dladdr(0x7ff811e6c000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(2) => 0x7ff811eeb000
dyld[64949]: dladdr(0x7ff811eeb000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(3) => 0x7ff8209ac000
dyld[64949]: dladdr(0x7ff8209ac000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(4) => 0x7ff8209a6000
dyld[64949]: dladdr(0x7ff8209a6000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(5) => 0x7ff82095f000
dyld[64949]: dladdr(0x7ff82095f000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(6) => 0x7ff820989000
dyld[64949]: dladdr(0x7ff820989000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(7) => 0x7ff82097d000
dyld[64949]: dladdr(0x7ff82097d000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(8) => 0x7ff811c9b000
dyld[64949]: dladdr(0x7ff811c9b000, 0x3099f92c0)
dyld[64949]: _dyld_get_image_header(9) => 0x7ff811d98000
dyld[64949]: dladdr(0x7ff811d98000, 0x3099f92c0)
...

Bad:

dyld[64978]: dlopen("/Users/smgi/Downloads/osu!b.app/Contents/MacOS/libcoreclr.dylib", 0x00000001)
dyld[64978]:       dlopen(libcoreclr.dylib) => 0x1aed40
dyld[64978]: dlsym(0x1aed40, "DllMain")
dyld[64978]:      dlsym("DllMain") => 0x10dda8cb0
dyld[64978]: dlsym(0x1aed40, "PAL_RegisterModule")
dyld[64978]:      dlsym("PAL_RegisterModule") => NULL
dyld[64978]: _dyld_find_unwind_sections(0x7ff820972f24, 0x30e005530)
dyld[64978]: _dyld_find_unwind_sections(0x7ff811efd233, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x10de02136, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x10de0208b, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x10de8f025, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9539, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x7ff820972f24, 0x30e0054b0)
dyld[64978]: _dyld_find_unwind_sections(0x7ff811efd233, 0x30e0040b0)
dyld[64978]: _dyld_find_unwind_sections(0x10de02136, 0x30e0040b0)
dyld[64978]: _dyld_find_unwind_sections(0x10de0208b, 0x30e0040b0)
dyld[64978]: _dyld_find_unwind_sections(0x10de8f025, 0x30e0040b0)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9539, 0x30e0040b0)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9bc4, 0x30e005350)
dyld[64978]: _dyld_find_unwind_sections(0x7ff820972f24, 0x30e0055b0)
dyld[64978]: _dyld_find_unwind_sections(0x7ff811efd3cb, 0x30e0041b0)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9c03, 0x30e0041b0)
dyld[64978]: _dyld_find_unwind_sections(0x7ff820972f24, 0x30e005530)
dyld[64978]: _dyld_find_unwind_sections(0x7ff811efd3cb, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9c03, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9c09, 0x30e0053d0)
dyld[64978]: _dyld_find_unwind_sections(0x7ff820972f24, 0x30e0055b0)
dyld[64978]: _dyld_find_unwind_sections(0x7ff811efd3cb, 0x30e0041b0)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9c1d, 0x30e0041b0)
dyld[64978]: _dyld_find_unwind_sections(0x7ff820972f24, 0x30e005530)
dyld[64978]: _dyld_find_unwind_sections(0x7ff811efd3cb, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9c1d, 0x30e004130)
dyld[64978]: _dyld_find_unwind_sections(0x10e1a9c23, 0x30e0053d0)
Failed to create CoreCLR, HRESULT: 0x8007000C

@smoogipoo
Copy link
Contributor Author

smoogipoo commented Jun 10, 2025

CLR built in debug mode asserts:

Assert failure(PID 8876 [0x000022ac], Thread: 52773526 [0x3254296]): !"ClrVirtualProtect of code page failed"
    File: /Users/runner/work/Satori/Satori/src/coreclr/vm/threads.cpp Line: 1210
    Image: /Users/smgi/Downloads/osu!c.app/Contents/MacOS/osu!

Corresponds to this code: https://github.com/ppy/Satori/blob/cf8046d24861d63ce23fff3073f233c4e8baaa07/src/coreclr/vm/threads.cpp#L1200-L1212

This part wasn't directly touched by Satori, however that code segment is gone from .NET since dotnet/runtime#114982. Not sure how changing the entitlements resolves this...

But I'm not sure where to go from here. Don't use Satori on osx-x64? I'll find some time to report upstream in case it's an issue with Satori itself...

@smoogipoo
Copy link
Contributor Author

Closing this one - we won't deploy Satori to osx-x64 for now.

@smoogipoo smoogipoo closed this Jun 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants