Skip to content

Commit 5a9685b

Browse files
moosingin3spacemoosingin3space
committed
Add terminology and boundaries RFC
1 parent 40b7b31 commit 5a9685b

File tree

1 file changed

+224
-0
lines changed

1 file changed

+224
-0
lines changed

‎text/0000-terminology-and-boundaries.md‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
- Feature Name: Terminology and Boundary Definition
2+
- Start Date: 2018-10-11
3+
- RFC PR: (leave this empty)
4+
- Nebulet Issue: (leave this empty)
5+
6+
# Summary
7+
[summary]: #summary
8+
9+
The Nebulet operating system, being so different from popular operating systems
10+
of today, will require new terminology to refer to abstraction boundaries.
11+
Terms such as "userspace" and "kernelspace" will carry too much baggage from
12+
existing OSes, which is likely to confuse potential contributors. This RFC
13+
proposes we define the following terms to refer to parts of our OS, adapted
14+
from [EROS](https://web.archive.org/web/20160412201504/http://www.coyotos.org/docs/misc/eros-structure.pdf):
15+
the _kernel_, which implements threading and inter-proces communication (IPC)
16+
as a wasm module, the _nucleus_, which implements a small wasm JIT natively,
17+
and the _firmament_, which specifies IPC interfaces for drivers to implement.
18+
Applications will be implemented atop these abstractions.
19+
20+
# Motivation
21+
[motivation]: #motivation
22+
23+
Today's common operating systems generally use hardware-based isolation
24+
approaches, such as protection rings on x86 or processor modes on ARM, to
25+
protect the kernel's memory from tampering. In colloquial operating systems
26+
discussion, the phrases "kernelspace" and "ring 0" have become synonymous,
27+
as have the phrases "userspace" and "ring 3". As Nebulet uses software-based
28+
isolation and not protection rings, these terms will be unfortunately
29+
overloaded, confusing potential contributors. The world of microkernel-based
30+
research OSes has provided some alternative terminology, some of which is
31+
potentially useful here.
32+
33+
# Guide-level explanation
34+
[guide-level-explanation]: #guide-level-explanation
35+
36+
The Nebulet operating system architecture is defined in terms of four
37+
components:
38+
39+
- The _kernel_, which implements the core functionality of a microkernel OS:
40+
thread/process scheduling and isolation, and inter-process communication
41+
(IPC). These two pieces of functionality will be further specified in future
42+
RFCs.
43+
- The _nucleus_, which implements a small WebAssembly runtime in native code. A
44+
nucleus will implement the minimal set of hardware interfaces required by the
45+
kernel and will likely resemble an exokernel on its surface. A nucleus need not
46+
be implemented in "ring 0", it could be implemented as a Linux userspace
47+
program.
48+
- The _firmament_, which specifies interfaces for hardware drivers. Drivers in
49+
Nebulet, like in any microkernel OS, are just processes, which means that a
50+
sufficiently flexible operating system will require standardized driver
51+
interfaces so as to make applications hardware-independent.
52+
- _Applications_ are software compiled to WebAssembly that the user runs.
53+
54+
**Example 1:** a [UTM](https://en.wikipedia.org/wiki/Unified_threat_management)
55+
appliance running on an Atheros SoC based on Nebulet might use:
56+
57+
- A nucleus compiled for ARM, possibly making use of the SoC's secure boot chip
58+
to implement a tamper-resistant boot.
59+
- The official Nebulet kernel.
60+
- Ethernet drivers that implement the Data Plane firmament specification.
61+
- A cryptographic accelerator driver that uses a custom cryptography chip on
62+
the SoC, that implements the Cryptography firmament specification.
63+
- A firewall implemented as a Nebulet application that uses the Data Plane
64+
firmament IPC interfaces.
65+
- A VPN implemented as a Nebulet application that uses both the Cryptography and
66+
Data Plane firmament IPC interfaces.
67+
68+
**Example 2:** an x86-64 laptop based on Nebulet might use:
69+
70+
- A nucleus compiled for x86-64, with some integration with UEFI Secure Boot.
71+
- The official Nebulet kernel.
72+
- Graphics drivers that implement the GPU firmament specification.
73+
- Wi-Fi drivers that implement the Data Plane firmament specification.
74+
- An implementation of Layer 3+ networking, standardized as the Sockets
75+
firmament specification.
76+
- A Vulkan library that uses the GPU firmament specification.
77+
- A desktop environment implemented as a Nebulet application that statically
78+
links the Vulkan library into its wasm module. The desktop environment
79+
provides an IPC protocol for opening windows and drawing widgets. This
80+
particular protocol is not part of the firmament, as it is not relevant to a
81+
hardware driver.
82+
- Firefox, which uses the desktop environment's IPC interface to create a window
83+
onscreen, statically links to the Vulkan library for rendering, and uses the
84+
Sockets firmament specification.
85+
86+
**Example 3:** a cluster of [TALOS II 2U servers](https://www.raptorcs.com/content/TL2SV1/intro.html)
87+
being used as a machine learning laboratory might use:
88+
89+
- A nucleus compiled for POWER9, that integrates with OpenBMC.
90+
- The official Nebulet kernel.
91+
- NVIDIA graphics drivers that implement the GPGPU firmament specification.
92+
- Ethernet drivers that implement the Data Plane firmament specification.
93+
- An implementation of the Sockets firmament.
94+
- TensorFlow for clusters ported to a Nebulet application that uses the GPGPU
95+
and Socket firmaments.
96+
- Jupyter and the entire SciPy stack (lol) ported to Nebulet, using the Sockets
97+
firmament and many ad-hoc application-layer IPC interfaces.
98+
99+
Note that in all examples, we can re-use the Nebulet kernel and firmament
100+
interfaces.
101+
102+
# Reference-level explanation
103+
[reference-level-explanation]: #reference-level-explanation
104+
105+
The nucleus must provide a thin set of abstractions over the hardware, in
106+
particular the CPU. These abstractions must include:
107+
- Access and control over CPU power states, if applicable.
108+
- Access and control over CPU interrupts, if applicable.
109+
- Access and control of the MMU, if applicable.
110+
- Access and control over CPU registers if necessary for common drivers on
111+
the target hardware.
112+
113+
The nucleus can also implement part of a verified boot chain if such a feature
114+
is desirable. In this case, the nucleus will be verified by the bootloader and
115+
it will verify the Nebulet kernel and other startup data, such as initial
116+
ramdisk or commandline.
117+
118+
The Nebulet kernel will use these functions to implement the core functionality
119+
of threading, scheduling, process isolation, and inter-process communication.
120+
121+
A firmament specification will describe the IPC interfaces and datatypes that
122+
an application using that piece of hardware will interact with. The Data Plane
123+
firmament specification used in the above examples might be defined as follows
124+
(using Rust-like syntax):
125+
126+
```rust
127+
// The supporting structs and enums are assumed to have
128+
// functions that can be called easily from the receiver side.
129+
// We'll need to flesh out the IPC system for this to be in any
130+
// way realistic.
131+
struct MacAddr([u8; 6]);
132+
enum Ethertype {
133+
Arp,
134+
Ipv4,
135+
Ipv6,
136+
}
137+
138+
struct EthernetPacket {
139+
data: Bytes,
140+
}
141+
142+
impl EthernetPacket {
143+
pub fn dst(&self) -> &MacAddr;
144+
pub fn src(&self) -> &MacAddr;
145+
pub fn ethertype(&self) -> &Ethertype;
146+
pub fn next_layer(&self) -> &L3Packet;
147+
}
148+
149+
interface EthernetNic {
150+
fn send(&mut self, pkt: EthernetPacket) -> Result<()>;
151+
fn send_inplace(&mut self, func: impl Fn(&mut EthernetPacket) -> Result<()>) -> Result<()>;
152+
fn recv(&mut self) -> Future<Result<EthernetPacket>>;
153+
}
154+
```
155+
156+
An interface is implemented by the driver side and used by the application.
157+
Firmament interfaces could be provided to the application on initialization,
158+
like in [CloudABI](https://cloudabi.org), or they could be dynamically queried,
159+
depending on the security model, which is yet to be specified.
160+
161+
# Drawbacks
162+
[drawbacks]: #drawbacks
163+
164+
Implementing all drivers as WebAssembly applications may not be able to provide
165+
maximum performance along the critical path of an application, in particular,
166+
networking- and GPU-intensive applications may find such performance limits to
167+
be problematic.
168+
169+
Specifying all classes of hardware as firmament drivers will be a tall order,
170+
depending on how wide interest in Nebulet is. The RFC process should allow for
171+
upstreaming the most useful ones in the community, though, as Nebulet becomes
172+
used in domains the original designers did not anticipate.
173+
174+
The terminology chosen is rather alien, and not typically used in the context of
175+
operating systems research and development. This may confuse seasoned operating
176+
systems developers who could be valuable assets to this project.
177+
178+
# Rationale and alternatives
179+
[rationale-and-alternatives]: #rationale-and-alternatives
180+
181+
- **Why is this design the best in the space of possible designs?** This design
182+
introduces architectural boundaries that are sensible for a unique operating
183+
system such as Nebulet.
184+
- **What other designs have been considered and what is the rationale for not
185+
choosing them?**
186+
- A _monolithic kernel_ design was not chosen because this niche is already
187+
filled extremely well by existing OSes such as Linux. Running WebAssembly
188+
in kernel space on Linux is actively being pursued through the
189+
[WasmJIT](https://github.com/rianhunter/wasmjit) project.
190+
- **What is the impact of not doing this?** Architectural boundaries would be
191+
allowed to emerge organically. This will likely result in an overspecialized
192+
operating system based on its initial applications, or a poorly-documented
193+
set of semi-specified, semi-organic interfaces, both limiting code re-use and
194+
diluting the value proposition of Nebulet.
195+
196+
# Prior art
197+
[prior-art]: #prior-art
198+
199+
This terminology and boundary specification was mostly inspired by
200+
[EROS](https://web.archive.org/web/20160412201504/http://www.coyotos.org/docs/misc/eros-structure.pdf).
201+
Some other microkernel operating systems, such as [Robigalia](https://robigalia.org),
202+
have chosen to adopt similar terminology.
203+
204+
The concept of an operating system kernel making use of a JIT-compiled environment
205+
dates back to the mainframe days, with IBM mainframes including
206+
hardware-assisted JIT compilation, which enabled fast, architecture-independent
207+
applications. More recently, Microsoft Research has experimented with
208+
Singularity, an operating system using the .NET virtual machine. Likewise, Sun
209+
Microsystems developed a Java-based operating system in the late 90s.
210+
211+
# Unresolved questions
212+
[unresolved-questions]: #unresolved-questions
213+
214+
- How do we handle cases where pure WebAssembly drivers aren't fast enough?
215+
- Should we allow native drivers to be loaded into the nucleus?
216+
- Should we allow native drivers to be statically compiled into the nucleus?
217+
If so, does the nucleus also implement the firmament interfaces?
218+
- Can we offer (possibly privileged) nucleus interfaces that mitigate this in
219+
common cases?
220+
- What should the process of standardizing firmament interfaces look like? Can
221+
we re-use the RFC process? Can we make templates for firmament RFCs?
222+
- How does this interact with our IPC system and security model? Both must be
223+
specified independently of this RFC.
224+
- How do applications acquire handles to firmament functions?

0 commit comments

Comments
 (0)