Make nodes boot independently

[martijnbastiaan]

After merging #1100, we'll have all our boot/reset logic in software which will make it much easier to work with. That PR doesn't change the reset logic in any way, it just moves it around. It currently works as follows:

1. "Boot" CPU
   1. Gets programmed by the host
   2. Programs the clock boards
   3. Gets transceiver block out of reset (enabling the bittide domain / other CPUs)
   4. Activates each transceiver channel and waits until they've negotiated a link with their neighbors.
   5. Prints "all done" message to UART.
2. Clock control CPU
   1. Gets programmed by the host
   2. Calibrates clocks
   3. Prints "all done" message to UART.
   4. (Keeps calibrating clocks.)
3. Management unit CPU
   1. Gets programmed by the host
   2. Centers elastic buffers
   3. Sets the channels to "user" mode. This makes the channels accept data from the outside world instead of their negotiation state machinery.
   4. Prints UGNs captured by hardware component to UART.

The host would typically move on to program a demo/test, but that's out of scope for this issue. In any case, all these steps are sequential:

I.e., there is a hard cut-off point at waiting for all transceiver channels to come up and another one waiting for all clocks to become stable. Ideally, the diagram would look like this:

To get there, we have to do two things:

• Remove the "channels ok" cut-off point. Clock control needs to only operate on active channels. The current software already accounts for this, we just need to make sure that that link mask reaches the software correctly.
• Remove the "everything calibrated" cut-off point. This one is more tricky as we're balancing two requirements: all our components are placed after elastic buffers and capturing the UGN relies on capturing the very first "real" word out of the EB. To keep the UGNs valid buffers can't overflow, but if we remove the cut-off point clock frequencies won't be stable yet so they naturally will. The MUs will have to continuously move the elastic buffers to their midpoints and keep track of how many elements they remove/add and adjust the UGNs accordingly.

[martijnbastiaan]

For the second TODO, I think we need to do the following:

Hardware changes
We need to give the MU access to CC. This is a bit confusing, but what needs to happen is that the LHS of the Circuit of callistoSwClockControlC will need to accept a:

( ToConstBwd Mm
, Wishbone dom 'Standard (SwcccRemBusWidth otherWb) (Bytes 4)
)

This says that callistoSwClockControlC is a subordinate, i.e., a manager can talk to it -- in this case the management unit. This might look a bit weird as there are now two ToConstBwd Mm on the LHS. One will represent the whole device (the one that's already there), while the other one will just carry the memory map information of clockControlWb. This makes sense, because the latter will need to be inserted somewhere in the memory region of the management unit, hence the need to carry it around separately. Anyway, after adding that to the type signature, we should of course accept it as an argument here:

bittide-hardware/bittide/src/Bittide/ClockControl/CallistoSw.hs

Line 88 in 20af7f1

circuit $ \(mm, (jtag, Fwd linkMask, Fwd linksOk)) -> do

Now we have a problem, because we have a manager interface coming in as an argument, but we also have one coming from the CPU:

bittide-hardware/bittide/src/Bittide/ClockControl/CallistoSw.hs

Line 90 in 20af7f1

( [ clockControlBus

We'll need to arbitrate that away, similar to what we do for the transceivers:

bittide-hardware/bittide-instances/src/Bittide/Instances/Hitl/SwitchDemo/BringUp.hs

Lines 108 to 110 in 20af7f1

	transceiverWb <-
	withRefClockResetEnable arbiterMm
	-< [muTransceiverWb, bootTransceiverWb]

It's probably a good idea to list CC's bus first, as the arbiter is left-biased.

Software changes
I think they should be reasonably straight-forward, but one thing to keep in mind is that clock control does need access to its clock control component (naturally), so the management unit should be careful not to give CC enough space on the bus.