Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Not for review] Thunks: Implement data layout analysis and struct repacking. Add 32-bit support to Wayland and Vulkan #3135

Closed
wants to merge 48 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
aecea05
TODODROP. DISABLE TESTS
neobrain Aug 22, 2023
34fce64
Thunks/gen: Track data types passed across architecture boundaries
neobrain Aug 24, 2023
89cf474
Thunks/gen: Add data layout analysis
neobrain Aug 24, 2023
c0ba993
Thunks/gen: Add detection logic for data layout differences
neobrain Aug 24, 2023
561fa3f
TODOFINISH. Add new annotations everywhere
neobrain Aug 3, 2023
dc3588d
unittests/ThunkLibs: Add data layout tests
neobrain Aug 10, 2023
fa53e2a
TODO APPLY TO OTHER TESTS TOO. TODOSQUASH: Also check host layout in …
neobrain Aug 22, 2023
1dcdc53
unittests/Thunks: Bump libclang requirement to C++20
neobrain Aug 22, 2023
cd798a5
TODOSPLIT test cases. unittests/ThunkLibs: Add struct repacking tests…
neobrain Sep 6, 2023
307deed
Thunks: Carry annotations in callback wrappers of host functions
neobrain Aug 8, 2023
0c5e4f7
Thunks: Introduce an intermediate guest_layout wrapper to unpack call…
neobrain Aug 8, 2023
1639db5
Thunks: Implement ptr_passthrough annotation
neobrain Sep 19, 2023
5b3c977
unittests/thunks: Add ptr_passthrough tests
neobrain Sep 6, 2023
5d7da30
TODOFINISH. Disable this test for now
neobrain Aug 22, 2023
bee6537
Thunks/gen: Add assume_compatible/is_opaque annotations
neobrain Aug 24, 2023
42cc086
unittests/thunks: Add tests for assume_compatible/is_opaque annotations
neobrain Sep 6, 2023
ba0aec0
TODO FIX BUILD. Thunks/gen: Enforce type compatibility for function p…
neobrain Aug 8, 2023
a80c60c
TODOMOVE. Various 32-bit fixes
neobrain Aug 24, 2023
7425f7e
TODOMOVE. Add support for asynchronous callbacks
neobrain Aug 7, 2023
cd70f04
TODODROP. Partially revert "Add support for asynchronous callbacks"
neobrain Aug 7, 2023
d5f5f43
Thunks/xcb: Drop unused and incomplete support for asynchous callbacks
neobrain Aug 15, 2023
a108aeb
Thunks/gen: Emit layout information for types passed across thunk bou…
neobrain Aug 14, 2023
cf41a02
TODO CONSIDER SQUASHING, or avoid introducing guest_layout placeholde…
neobrain Aug 15, 2023
b3695eb
TODOMOVE. Thunks: Avoid recompiling thunk interfaces on FEXLoader cha…
neobrain Sep 22, 2023
59f0de0
Thunks/gen: Also use guest_layout for return values
neobrain Aug 24, 2023
f204124
Thunks/gen: Implement automatic struct repacking
neobrain Aug 15, 2023
f6796a4
Thunks: Handle type differences for Guest->Host calls through functio…
neobrain Aug 15, 2023
af5d765
Thunks/gen: Implement assisted struct repacking
neobrain Aug 22, 2023
c4900fd
TODOSQUASH. Add unwrap_host helper
neobrain Aug 22, 2023
cea56eb
TODOSQUASH. Fix things that turned up during testing
neobrain Aug 22, 2023
1b3cea4
TODOSQUASH: unittests: Clean up automatic struct repacking tests
neobrain Aug 23, 2023
f8778df
unittests/Thunks: Add tests for assisted struct repacking
neobrain Aug 23, 2023
239c010
Add more assisted repacking tests
neobrain Sep 6, 2023
23cc9bb
unittests/thunks: Add tests for opaque types
neobrain Aug 23, 2023
fe624b8
TODOSQUASH. For struct repacking, const must be stripped off from poi…
neobrain Aug 23, 2023
6db1efd
unittests/Thunks: Add assume_compatible_data_layout test
neobrain Aug 23, 2023
a14b70b
unittests/thunks: Add assume_compatible_data_layout tests
neobrain Aug 23, 2023
657ca8f
unittests/Thunks: Add test for divergent function parameters
neobrain Aug 24, 2023
08c8b38
TODO ENSURE THERE'S A REPLACEMENT. Deprecate callback_guest annotation
neobrain Aug 31, 2023
715bd3a
TODOFINISH. Thunks/gen: Map guest integer types to fixed-size equival…
neobrain Aug 31, 2023
43fe4a1
TODOFINISH. Disable struct padding for packed arguments
neobrain Aug 31, 2023
44df42d
TODOSQUASH. Simplify this code. Not sure if correct
neobrain Aug 31, 2023
68266de
TODOFINISH. Fix 32-bit jemalloc symbols
neobrain Sep 4, 2023
77d84ac
TODO SPLIT host.h changes. Thunks/libwayland: Add 32-bit specific bits
neobrain Sep 4, 2023
dbdb8a6
TODOFINISH. Thunks/vulkan: Add 32-bit support
neobrain Sep 4, 2023
17426aa
Revert "TODOSPLIT test cases. unittests/ThunkLibs: Add struct repacki…
neobrain Sep 6, 2023
02518f2
TODOSQUASH. Thunks: Add more tests
neobrain Sep 6, 2023
ca023e7
TODOFINISH. Thunks/gen: Avoid generating layout conversion code for n…
neobrain Sep 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
TODOSPLIT test cases. unittests/ThunkLibs: Add struct repacking tests…
…; Add tests for void pointer parameters
neobrain committed Sep 21, 2023
commit cd798a50c78219c01b5557f0a1a0ff46d07be532
100 changes: 94 additions & 6 deletions unittests/ThunkLibs/generator.cpp
Original file line number Diff line number Diff line change
@@ -73,7 +73,7 @@ struct Fixture {
* It will be prepended to "code" before processing and also to the generator output.
*/
SourceWithAST run_thunkgen_guest(std::string_view prelude, std::string_view code, bool silent = false);
SourceWithAST run_thunkgen_host(std::string_view prelude, std::string_view code, bool silent = false);
SourceWithAST run_thunkgen_host(std::string_view prelude, std::string_view code, GuestABI = GuestABI::X86_64, bool silent = false);
GenOutput run_thunkgen(std::string_view prelude, std::string_view code, bool silent = false);

const std::string libname = "libtest";
@@ -236,13 +236,13 @@ SourceWithAST Fixture::run_thunkgen_guest(std::string_view prelude, std::string_
/**
* Generates host thunk library code from the given input
*/
SourceWithAST Fixture::run_thunkgen_host(std::string_view prelude, std::string_view code, bool silent) {
SourceWithAST Fixture::run_thunkgen_host(std::string_view prelude, std::string_view code, GuestABI guest_abi, bool silent) {
const std::string full_code = std::string { prelude } + std::string { code };

// These tests don't deal with data layout differences, so just run data
// layout analysis with host configuration
auto data_layout_analysis_factory = std::make_unique<AnalyzeDataLayoutActionFactory>();
run_tool(*data_layout_analysis_factory, full_code, silent);
run_tool(*data_layout_analysis_factory, full_code, silent, guest_abi);
auto& data_layout = data_layout_analysis_factory->GetDataLayout();

run_tool(std::make_unique<GenerateThunkLibsActionFactory>(libname, output_filenames, data_layout), full_code, silent);
@@ -282,17 +282,27 @@ SourceWithAST Fixture::run_thunkgen_host(std::string_view prelude, std::string_v
auto& filename = output_filenames.host;
{
std::ifstream file(filename);
const auto current_size = result.size();
const auto prelude_size = result.size();
const auto new_data_size = std::filesystem::file_size(filename);
result.resize(result.size() + new_data_size);
file.read(result.data() + current_size, result.size());
file.read(result.data() + prelude_size, result.size());

// Force all functions to be non-static, since having to define them
// would add a lot of noise to simple tests.
while (true) {
auto pos = result.find("static ", prelude_size);
if (pos == std::string::npos) {
break;
}
result.replace(pos, 6, " "); // Replace "static" with 6 spaces (avoiding reallocation)
}
}
return SourceWithAST { std::string { prelude } + result };
}

Fixture::GenOutput Fixture::run_thunkgen(std::string_view prelude, std::string_view code, bool silent) {
return { run_thunkgen_guest(prelude, code, silent),
run_thunkgen_host(prelude, code, silent) };
run_thunkgen_host(prelude, code, GuestABI::X86_64, silent) };
}

TEST_CASE_METHOD(Fixture, "Trivial") {
@@ -525,3 +535,81 @@ TEST_CASE_METHOD(Fixture, "VariadicFunctionsWithoutAnnotation") {
"template<auto> struct fex_gen_config {};\n"
"template<> struct fex_gen_config<func> {};\n", true));
}

TEST_CASE_METHOD(Fixture, "StructRepacking") {
auto guest_abi = GENERATE(GuestABI::X86_32, GuestABI::X86_64);
INFO(guest_abi);

// All tests use the same function, but the prelude defining its parameter type "A" varies
const std::string code =
"#include <thunks_common.h>\n"
"void func(A*);\n"
"template<auto> struct fex_gen_config {};\n"
"template<> struct fex_gen_config<func> : fexgen::custom_host_impl {};\n";

SECTION("Pointer to struct with consistent data layout") {
CHECK_NOTHROW(run_thunkgen_host("struct A { int a; };\n", code, guest_abi));
}

SECTION("Pointer to struct with unannotated pointer member with inconsistent data layout") {
const auto prelude =
"#ifdef HOST\n"
"struct B { int a; };\n"
"#else\n"
"struct B { int b; };\n"
"#endif\n"
"struct A { B* a; };\n";

SECTION("Parameter unannotated") {
// TODO
CHECK_THROWS(run_thunkgen_host(prelude, code, guest_abi, true));
}
}

SECTION("Pointer to struct with pointer member of consistent data layout") {
std::string type = GENERATE("char", "short", "int", "float");
REQUIRE_NOTHROW(run_thunkgen_host("struct A { " + type + "* a; };\n", code, guest_abi));
}

SECTION("Pointer to struct with pointer member of opaque type") {
const auto prelude =
"struct B;\n"
"struct A { B* a; };\n";

// Unannotated
REQUIRE_THROWS_WITH(run_thunkgen_host(prelude, code, guest_abi), Catch::Contains("incomplete type"));
}
}

TEST_CASE_METHOD(Fixture, "VoidPointerParameter") {
auto guest_abi = GENERATE(GuestABI::X86_32, GuestABI::X86_64);
INFO(guest_abi);

SECTION("Unannotated") {
const char* code =
"#include <thunks_common.h>\n"
"void func(void*);\n"
"template<> struct fex_gen_config<func> {};\n";
if (guest_abi == GuestABI::X86_32) {
// CHECK_THROWS_WITH(run_thunkgen_host("", code, guest_abi, true), Catch::Contains("unsupported parameter type", Catch::CaseSensitive::No));
} else {
// Pointee data is assumed to be compatible on 64-bit
CHECK_NOTHROW(run_thunkgen_host("", code, guest_abi));
}
}

SECTION("Unannotated in struct") {
const char* prelude =
"struct A { void* a; };\n";
const char* code =
"#include <thunks_common.h>\n"
"void func(A*);\n"
"template<> struct fex_gen_config<func> {};\n";
if (guest_abi == GuestABI::X86_32) {
// TODO
CHECK_THROWS_WITH(run_thunkgen_host(prelude, code, guest_abi, true), Catch::Contains("unsupported parameter type", Catch::CaseSensitive::No));
} else {
CHECK_NOTHROW(run_thunkgen_host(prelude, code, guest_abi));
}
}
}