staticdata: a pre-relocated system image

[levy]

Stacks on #63068; its two commits are under this one.

Applying the relocations of a system image writes almost every page of it: 47,566 page faults and 88 ms of a 105 ms jl_init on a 368 MB application image (#63065). When the image is linked into a non-PIE program it is at the same address at every start, so the relocation can be done once, at build time, and written into the program file.

• --sysimage-prelink=yes, when writing an image, reserves room for the residual list: the pointers a file cannot hold. Without it the image is unchanged, byte for byte.
• --output-prelinked <file>, on a program that holds such an image, restores it, writes the program with the restored image to <file>, and exits. Linux only, through /proc/self/exe; the fast path itself is portable.

A start of the written file applies the residual list, fills the gvar slots, registers the native code and runs the fixup list; it reads neither relocation list. The runtime refuses, with a message and exit code 1: an image pre-relocated for another address, an image without native code, an image with several code variants, and a second pre-relocation.

Measured on master, page cache evicted, minimum of 15 pinned runs, with #63070:

program	start	minor faults	resident
hello world	74.9 → 7.1 ms	30,582 → 2,487	163 → 66 MB
a network simulator	101.8 → 7.9 ms	48,209 → 2,824	245 → 106 MB

The remaining faults are the dynamic loader (about 1,150) and jl_init (about 1,200). The image pages stay clean file pages, so processes of the same program share them.

Tests: test/prelink (make -C test prelink, after test/embedding) builds a program with its own image, pre-relocates it, runs both, checks that at most 64 of 12 million pointers stay on the residual list, and checks the two refusals it can build; 1 min 36 s from clean. test/cmdlineargs.jl covers the options, the no-room refusal and the record. A devdoc section is added to sysimg.md. The full suite passes; cmdlineargs was run separately because rr does not support this CPU.

Reproduction: an 8-line image script and a 40-line C launcher; julia --sysimage-prelink=yes --output-o app.o.a image.jl, cc -fPIC -no-pie ..., ./app --julia-args --output-prelinked=app2.

Disclosure: developed with Claude Code (Opus 5) under my direction. It wrote the code and this text; the measurements were run on my machine. I reviewed the changes. The commits carry an Assisted-by trailer.

[xal-0]

I've added myself as a reviewer on the related PRs here. For transparency: I've been working on an improved system image format that will let us do this (among other things, there's a slop branch from the JuliaCon hackathon at ss/fixed-addr-sysimage-slop) but I would like to cannibalize any good ideas that may be here. Do you have any human-authored documentation or prompts?

[levy]

You can cannibalize these however you like. I'm happy that I could beat the C++ version of the simulator in startup time from a way larger binary... :) BTW, I also happen to have an incremental sysimage compiler which compiles changes into binary form in a few seconds instead of minutes...

Here comes the AI agent collected response:

No human-authored design document, and no prompt library: the constraints and the design decisions were mine, made in conversation while measuring, and the tool wrote the code and the prose. The devdoc section in this PR is the closest thing to written documentation. Here is the short form of what the work found, in case any of it is useful for the new format. Take whatever you like.

The cost is per page written, not per pointer. The type-tag pass writes one word per object but touches 41,796 of 43,140 data pages, and every touch is a copy-on-write fault. That is why a smaller image scales the cost down but does not remove it, and why partial fixes gain nothing.

Four classes of pointer have to become final together. Their pages overlap almost completely, so leaving any one class behind keeps the image dirty:

class	pointers	pages	pages no other class writes
nothing	681,878	37,656	9,424
symbols	412,766	24,460	2,977
small boxed integers	43,090	12,274	45
runtime entry points	29,167	8,531	0
union	1,166,903	40,963 of ~43,000	—

Each class needs a different answer. nothing and the boxed integers are serialized as objects; julia_init allocates a nothing before the root task exists, so the root task's fields are re-pointed at the image's one after the relocations, by a walk over the task layout rather than a field list. Symbols become objects too, and the image's tree becomes the process symbol table, which forces jl_init_common_symbols to run after the restore. The runtime entry points cannot be in a file at all, so the image carries a thunk per entry point and a slot table: a start fills 72 slots in one page instead of writing 25,831 pointers over 8,500 pages.

State the contract, or the fast path rots. A pre-relocated start applies the residual list, fills the global variable slots, registers the native code, and runs the fixup list; it reads neither relocation list and neither memory reference list. Anything later that adds work to those passes has to add it to the residual list too. The devdoc says this so a future change has something to violate.

Two ordering traps. The function pointers must be resolved before the fixup list runs, because the fixup list writes the first pointers into the image that a file cannot hold. And the global variable slots live in .bss, which no file holds, so they are filled at every start either way.

Trimming pulls the other way. A trimmed binary is under 2 MB and the symbol table of Base is 2.6 MB, so a trimmed image keeps the tag encoding here. Pruning the table to the symbols an image actually references would give both, and would need the search tree rebuilt in serialization order rather than copied. That is the piece I would do next.

What is left after all of it. A pre-relocated hello world takes 2,487 minor faults: about 1,150 before main runs, where the loader maps libjulia-internal, libjulia-codegen and libLLVM and reads their symbol tables, and about 1,200 in jl_init, mostly anonymous pages that the collector heap, the root task stack and the module initializers write. The image is no longer the expensive part, which is worth knowing before optimizing it further.

Happy to rebase onto whatever shape you land, or to close these if the new format subsumes them.

[levy]

I've added myself as a reviewer on the related PRs here. For transparency: I've been working on an improved system image format that will let us do this (among other things, there's a slop branch from the JuliaCon hackathon at ss/fixed-addr-sysimage-slop) but I would like to cannibalize any good ideas that may be here. Do you have any human-authored documentation or prompts?

I have just checked your branch, I'm really happy you are working on it. I do think that Julia binaries can start in milliseconds, an SDL window comes up for me in 100ms. This was a big minus when deciding if a Julia port of a C++ communication network simulator is feasible or not. I do understand that Julia is used from the REPL and we can do that, but not all customers in all cases use the system like that. But this risk has been mitigated, great!

[xal-0]

BTW, I also happen to have an incremental sysimage compiler which compiles changes into binary form in a few seconds instead of minutes...

Very cool! I've also got some WIP stuff for improving the situation here. The plan with the objcache has always been to use it for AOT compilation as well, but there's some legwork that needs to be done in order to do it properly. What approach worked for you?