[Documentation] static core linking

[penzn]

@pchickey @lukewagner, we had a discussion of static linking in context of WebAssembly/WASI#549, and I just wanted to make sure I understood it correctly:

Currently static core linking works by shipping static libs or wasm 'object files' in the component binary and then running wasm-ld in the runtime during component instantiation.

Is this an accurate description? Can you please point me to where this is currently documented?

[pchickey]

wasm-ld is able to statically link wasm object files and archive files (static libs) into a core wasm module.

You can then transform a core wasm module into a component using a tool such as wit-component.

All of these things happen ahead of time, before the component is loaded into a runtime for instantiation.

[penzn]

So what is the relationship between wasm-ld and wit-component? I thought you mentioned that the latter uses the former or something like that. Let me know if there is already a place to read about it, so I don't bug you with questions already answered elsewhere :)

[alexcrichton]

I'm not aware of any specific documentation myself, but these tools are quite low-level so it's sort of similar to asking what the relationship is between clang and ld in a sense. It's probably best to understand the tools from an inputs/outputs perspective where:

• wasm-ld takes in compiler object files, those generated by clang or rustc, and generates a core wasm module
• wit-component takes a core wasm module and some type information, previously embedded in the core wasm module, and generates a component

The wit-component support is expoed through the wasm-tools component new subcomand (in this repo) and you can explore that with wasm-tools component new -h. For wasm-ld you can also explore with wasm-ld -h and binaries I know are available from the wasi-sdk repository but if you already have LLD installed you may already have wasm-ld installed as well.

[penzn]

@pchickey made a reference to wasm-ld being used by either a wit tool or runtime, and I am trying to follow up on the mechanics of that.

I know what wasm-ld does and what wasm 'object files' really are.

[pchickey]

wasm-ld is not used by any of the wit tools or runtimes. On our call last night I described the process by which which you can statically link programs prior to getting involved with any component runtime.

If you desire a component which contains a single statically linked core module, you should create that core module using whatever wasm-object file (meaning, in almost all cases, llvm) based toolchain, and use wasm-ld to link all of the objects in that program together (statically). None of what I described is specific to the component model at all - this is the mechanism through which the overwhelming majority of wasm modules are linked today. The llvm team documented the object file format here: https://github.com/WebAssembly/tool-conventions/blob/main/Linking.md

You can then take that core module and transform it into a component by using wit-component to add the component binary format type information.

At this time there is no mechanism to take already-existing components (meaning, any set of components containing more than one core module) and change their representation into a single core module using a single linear memory. If you wanted to create such a thing, merging the core modules into a single linear memory would have to implement its own page table / virtual addressing scheme internal to wasm, which would be slower than letting a wasm runtime do that translation on your behalf to implement multiple linear memories, since the runtime can leverage actual virtual memory and an internal-to-wasm implementation cannot.

[penzn]

OK, I guess I completely misunderstood that. So if the linking is going to be done the same way it is done today, does it interface with component model, i.e. does component model encode any of the core module composition? There is this example in mvp/Explainer.md:

(component
  (core module $A
    (func (export "one") (result i32) (i32.const 1))
  )
  (core module $B
    (func (import "a" "one") (result i32))
  )
  (core instance $a (instantiate $A))
  (core instance $b (instantiate $B (with "a" (instance $a))))
)

This would not mean that $a is linked into $b, right? Also, when we have a library that has to be statically linked for performance reasons, would that be done entirely outside of component model?

[lukewagner]

@penzn That example is admittedly a bit synthetic; if you were able to generate that component, you might as well have statically linked $A and $B together into a single core module (e.g., using wasm-ld), producing just:

(component
  (core module $AB
    ... $A and $B linked together via wasm-ld
  )
  (core instance $ab (instantiate $AB))
)

This simple example doesn't have any imports/exports, but in a more realistic example, the core module produced by wasm-ld would contain imports for all symbols annotated with import_name or export_name, and then these imports/exports would be wired up via lower/lift into component-level imports/exports.

That being said, if you did leave $A and $B as separate core modules in the original example, that would be like load-time dynamic linking in a native setting: separate compiled objects produced by wasm-ld, linked together at runtime. As another variation, if you imported $A (instead of nesting it), then that starts to look like the shared-everything dynamic linking example, for sharing $A code with multiple components.

[penzn]

I see, how this integrates with components is similar to how we currently use 'external' functions, with the linking staying pretty much the same (difference is in how the interfaces are really produced). Can we add this to the docs, maybe changing this particular example? I can try to help.

[lukewagner]

Yes, that's a good idea; it seems like how static linking fits into the component model is a generally pretty hot topic these days :) Maybe the best way would be a new StaticLinking.md example (linked to from within the main explainer in the relevant points like the example you're looking at) so that this StaticLinking.md can better introduce the problem from scratch and explain how it all works. That'd be awesome to have your help writing it up, otherwise I'm happy to take a crack at it some time after WasmCon.

[lukewagner]

I started this diagram to try to capture the different phases of linking that we've been talking about and how they fit together in the overall developer workflow. I'm imagining a new examples/StagesOfLinking.md that embeds this diagram and then talks through how it works (analogous to the existing examples/SharedEverythingDynamicLinking.md). @penzn does that look/sound right to you?

[penzn]

@lukewagner that looks right, thank you for putting it together! I think this diagram is great, though I think there is another angle that can be useful (this might need a diagram, maybe just a note in the text), specifically how would this work from 'using a component' point of view. Say I have a WIT file, implementation of that component, and a consumer of that component - I can compile both the implementation and consumer into a bunch of .o/.a files, but would I be able to link them together, or would that be tricky due to the need to maintain memory boundaries?

[lukewagner]

@penzn Using core wasm multi-memory (which it looks like is in both Firefox and Chrome now), it's always possible to take an arbitrary component and "fuse" it into a single core wasm module that imports/exports the Canonical ABI of the original WIT. Is that what you're asking about, or something else?