Clean up cargo features: feature vs option vs const boolean

[qinsoon]

I am talking about cargo features, our runtime options, and const booleans in the code. They all change MMTk's behaviours. But they have obvious differences, and I think we should be careful about this.

The following is what I propose:

1. Adding const booleans is a trivial change, as they are not public to the users. We should feel free to add as long as they are useful to developers in the future.
2. We have to use cargo features if
   1. we need conditional compilation (e.g. optional base spaces, alloc bit).
   2. our users (bindings) want something out of box for every build (e.g. non-moving immix).
3. We have to use runtime options if we want to configure something at runtime.
4. If we can freely choose between cargo features and runtime options, we should prefer using runtime options as they do not require a rebuild.

With this proposal, we may consider making the following changes:

• sanity: feature -> runtime option. It just involves scheduling sanity work. There is a check in every ProcessEdgesWork, we need to make sure this does not introduce overhead.
• perf_counter: feature -> (removed). perf_counter only works on linux, we can use cfg(target_os = "linux") instead. We also have runtime options to set perf events, which should be sufficient.
• object_pining: feature -> (removed). We can just include all the methods about object pinning.
• nogc_no_zeroing: feature -> const boolean.
• single_worker: feature -> (removed). We can just use the options threads to set worker threads.
• immix_zero_on_release: feature -> const boolean. This is just an option for developing and debugging.

[wks]

object_pinning requires an additional pinning bit metadata for some spaces. For Immix, it also requires a metadata load during trace_object_with_opportunistic_copy. So there is a run-time cost.

We may add a const USE_OBJECT_PINNING: bool in ObjectModel so that ImmixSpace::is_pinned can elide the load from VM::VMObjectModel::LOCAL_PINNING_BIT_SPEC when USE_OBJECT_PINNING is false.

The VM::VMObjectModel::LOCAL_PINNING_BIT_SPEC metadata can be set as an "in-header" metadata so that it doens't take extra side metadata space if the VM binding doesn't intend to use object pinning.

[wks]

We may also introduce a NotUsed case to MetadataSpec in addition to InHeader and OnSide.

 pub enum MetadataSpec {
+    NotUsed,
     InHeader(HeaderMetadataSpec),
     OnSide(SideMetadataSpec),
 }

It will be a more explicit way to express that "this metadata is never used in this VM binding". Any attempt to access NotUsed metadata will result in a panic.

• VMs that do not need object pinning can set LOCAL_PINNING_BIT_SPEC as NotUsed.

The following practices are possible. They may be usseful for debugging or during development, but should not be encouraged. I don't think it is right to confine a VM to non-copying or non-generational GC algorithms.

• VMs that do not support copying GC can set LOCAL_FORWARDING_POINTER_SPEC and LOCAL_FORWARDING_BITS_SPEC to NotUsed;
• VMs that do not support generational GC can set GLOBAL_LOG_BIT_SPEC and LOCAL_LOS_MARK_NURSERY_SPEC as "NotUsed";

[qinsoon]

object_pinning requires an additional pinning bit metadata for some spaces. For Immix, it also requires a metadata load during trace_object_with_opportunistic_copy. So there is a run-time cost.

We may add a const USE_OBJECT_PINNING: bool in ObjectModel so that ImmixSpace::is_pinned can elide the load from VM::VMObjectModel::LOCAL_PINNING_BIT_SPEC when USE_OBJECT_PINNING is false.

The VM::VMObjectModel::LOCAL_PINNING_BIT_SPEC metadata can be set as an "in-header" metadata so that it doens't take extra side metadata space if the VM binding doesn't intend to use object pinning.

Yeah. This is a valid point. We probably should keep the object_pinning feature then.