Skip to content

Conversation

@resistor
Copy link
Contributor

@resistor resistor commented Jul 7, 2025

These are already correctly handled by the AC ROM as reading/writing the cap metadata
bits for GPCRs. This diff also extends that handling to access the same bits on CSRs as well.

These are already correctly handled by the AC ROM as reading/writing the cap metadata
bits for GPCRs. This diff also extends that handling to access the same bits on CSRs as well.
@elliotb-lowrisc
Copy link
Contributor

elliotb-lowrisc commented Jul 9, 2025

Thanks for this fix. As this is a "vendored" file it will need some additional work to avoid your changes being lost next time someone runs the vendoring script to update the vendored files. You've got two options:

  1. Create a patch file in "vendor/patches/cheriot_debug_module". There's already one in there for reference. The format is simply the output of git diff with paths made relative to the source repo.

Or

  1. Update the upstream repo and then re-run the vendoring script using util/vendor.py --update --verbose hw/vendor/cheriot_debug_module.vendor.hjson to pull in the changes.

@resistor
Copy link
Contributor Author

resistor commented Jul 9, 2025

Opened a PR on the upstream repository: CHERIoT-Platform/cheriot-dbg-module#3

@elliotb-lowrisc
Copy link
Contributor

Thanks for getting the upstream changes made @resistor. Would you like to update this PR to vendor in the changes or would you prefer I created a PR to do that?

@resistor
Copy link
Contributor Author

resistor commented Aug 4, 2025

Thanks for getting the upstream changes made @resistor. Would you like to update this PR to vendor in the changes or would you prefer I created a PR to do that?

If you could do it, it’s much more likely to be done correctly.

@elliotb-lowrisc
Copy link
Contributor

Hi @resistor, I'm having some trouble getting the modified debug module to work, and am wondering if the MaxAar change has had an unintended consequence.

I've vendored in the updated debug module into sonata-system locally and have been trying to connect to it using OpenOCD and gdb, but keep getting an illegal instruction exception. Specifically, the message reported by Verilator is Illegal instruction (hart 0) at PC 0xb0000348: 0x39053023. The PC indicates it is executing an Abstract Command (addresses 0x338 to 0x35f) from position abstract_cmd[2][31:0]. Decoding the instruction value reveals a CSC instruction with a cs2 value of 16, outside of the RV32E register range of 0 - 15. I think the RTL line for this instruction is dm_mem.sv:435, where a store operation occurs. However, I think this is just a symptom, not the cause.

I believe the underlying issue is that the increase in MaxAar has caused OpenOCD to think it is talking to an RV64I core rather than an RV32E. The OpenOCD output used to say XLEN=32, misa=0x40901016, but now says XLEN=64, misa=0x0. A comment in dm_mem.sv seems to indicate that OpenOCD attempts a 64-bit (aarsize=3) access to determine whether the core is 64-bit or 32-bit. I suspect that increasing MaxAar to 4 causes this access attempt to not trigger the 'command error' signal that OpenOCD checks for. An unfortunate side-effect.

I've not yet thought up a way around this issue (assuming my diagnosis is correct), but perhaps you can as you know more about what this change is ultimately trying to achieve.

@resistor
Copy link
Contributor Author

resistor commented Aug 5, 2025

Hi @resistor, I'm having some trouble getting the modified debug module to work, and am wondering if the MaxAar change has had an unintended consequence.

I've vendored in the updated debug module into sonata-system locally and have been trying to connect to it using OpenOCD and gdb, but keep getting an illegal instruction exception. Specifically, the message reported by Verilator is Illegal instruction (hart 0) at PC 0xb0000348: 0x39053023. The PC indicates it is executing an Abstract Command (addresses 0x338 to 0x35f) from position abstract_cmd[2][31:0]. Decoding the instruction value reveals a CSC instruction with a cs2 value of 16, outside of the RV32E register range of 0 - 15. I think the RTL line for this instruction is dm_mem.sv:435, where a store operation occurs. However, I think this is just a symptom, not the cause.

I believe the underlying issue is that the increase in MaxAar has caused OpenOCD to think it is talking to an RV64I core rather than an RV32E. The OpenOCD output used to say XLEN=32, misa=0x40901016, but now says XLEN=64, misa=0x0. A comment in dm_mem.sv seems to indicate that OpenOCD attempts a 64-bit (aarsize=3) access to determine whether the core is 64-bit or 32-bit. I suspect that increasing MaxAar to 4 causes this access attempt to not trigger the 'command error' signal that OpenOCD checks for. An unfortunate side-effect.

I've not yet thought up a way around this issue (assuming my diagnosis is correct), but perhaps you can as you know more about what this change is ultimately trying to achieve.

Could you try using this fork of OpenOCD, which we've been customizing for talking to CHERIoT? It should handle the issue you're seeing correctly. https://github.com/cherioT-Platform/openocd

@elliotb-lowrisc
Copy link
Contributor

Thanks, I managed to build the OpenOCD fork and it seemed to detect the XLEN and misa, but GDB wouldn't connect. I assume some incompatibility has cropped-up due to the CHERIoT changes. The only message I could see was the cryptic (to me) Truncated register 32 in remote 'g' packet. I'll try to build LLDB instead for CHERIoT-ness.

Do you recommend any particular commit of CHERIoT-Platform/llvm-project for me to build? One with any required LLDB changes that is also fairly stable perhaps. I'm currently building the latest (3fcf44fb), but it would be good to have a known-good version to fall-back to if I encounter problems.

@elliotb-lowrisc
Copy link
Contributor

My first issue seems to be that I cannot seem to read general-purpose registers using the modified debug module and OpenOCD. A loop counter in my test program (reg a4) is reading as zero using the new DM & CHERIoT-OpenOCD, whereas it read correctly using the previous DM & normal-OpenOCD (in both LLDB and GDB). Some other registers are also reading different values. Perhaps the values will make more sense to you than to me:

Previous DM + OpenOCD + GDB:

(gdb) info reg
ra             0x1000c6	0x1000c6
sp             0x120000	0x120000
gp             0x1001f0	0x1001f0
tp             0x0	0x0
t0             0x1e8480	2000000
t1             0x0	0
t2             0x0	0
fp             0x0	0x0
s1             0x100088	1048712
a0             0x80000000	-2147483648
a1             0xff	255
a2             0x80000004	-2147483644
a3             0xff	255
a4             0x1e2a57	1976919
...
t6             0x0	0
pc             0x100142	0x100142

Modified DM + CHERIoT-OpenOCD + CHERIoT-LLDB:

(lldb) reg r
general:
      zero = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        ra = 0x5f3e0000 (v:? 0x5f3e0000-0x5f3e0000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        sp = 0x7e3e0000 (v:? 0x7e3e0000-0x7e3e0000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        gp = 0x7e3e0000 (v:? 0x7e3e0000-0x7e3e0000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        tp = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        t0 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        t1 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        t2 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        fp = 0x7e3e0000 (v:? 0x7e3e0000-0x7e3e0000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        s1 = 0x5e3e0000 (v:? 0x5e3e0000-0x5e3e0000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        a0 = 0x7e000800 (v:? 0x7e000800-0x7e000800 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        a1 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        a2 = 0x7e001004 (v:? 0x7e001000-0x7e001000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        a3 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        a4 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
...
        t6 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        pc = 0x00100156  cheri_sanity`entry_point + 60
      priv = 0x03

(Note that processor state is probably not identical between these two register snapshots as they were two different Verilator runs and halted at different places)

Currently I'm using $CHERIOT_LLVM_BIN/lldb --one-line 'gdb-remote 3333' sw/cheri/build/checks/cheri_sanity for LLDB and ~/openocd/src/openocd -f util/verilator-openocd-cfg.tcl (see: https://github.com/lowRISC/sonata-system/blob/main/util/verilator-openocd-cfg.tcl) for OpenOCD.

I'm not familiar with LLDB, so perhaps I've made a mistake somewhere.

Do you have any ideas why register reading is not working?

@resistor
Copy link
Contributor Author

resistor commented Aug 6, 2025

The formatting you're seeing from lldb is in Cheriot capability decoding format, showing the cap metadata. If you'd prefer to view it in a more GDB-like format, just pass --format hex to the register read command.

@resistor
Copy link
Contributor Author

resistor commented Aug 6, 2025

I don't know offhand why you would be getting different results. What path is it taking through the AC command ROM?

@elliotb-lowrisc
Copy link
Contributor

This seems to be the section where it is reading a4 ("c14"). Looks like the register does have an expected value (0x001e3d01) at the time the abstract command stores it (time 24957650000):

    24956300000	    998248	b0000824	f1402473	csrrs	x8,mhartid,x0	  x0:0x00000000  x8=0x00000000
    24956400000	    998252	b0000828	10852023	csw	x8,256(c10)	 x10:0xb0000000+0x000000000  x8:0x00000000 PA:0xb0000100 store:0x000000000
    24956425000	    998253	b000082c	00a40433	add	x8,x8,x10	  x8:0x00000000 x10:0xb0000000  x8=0xb0000000
    24956550000	    998258	b0000830	40044403	clbu	x8,1024(c8)	  x8:0xb0000000+0x000000000  x8=0x00000001 PA:0xb0000400 load:0x00000001
    24956600000	    998260	b0000834	00147413	andi	x8,x8,1	  x8:0x00000001  x8=0x00000001
    24956650000	    998262	b0000838	02041c63	bne	x8,x0,b0000870	  x8:0x00000001  x0:0x00000000
    24956850000	    998270	b0000870	10052223	csw	x0,260(c10)	 x10:0xb0000000+0x000000000  x0:0x00000000 PA:0xb0000104 store:0x000000000
    24956875000	    998271	b0000874	03a0055b	CH.cspecialr	c10,scr26	  x0:0x00000000+0x000000000 x10=0x80000000+0x17e000800
    24956950000	    998274	b0000878	0390045b	CH.cspecialr	c8,scr25	  x0:0x00000000+0x000000000  x8=0x00000000+0x17e3e0000
    24957000000	    998276	b000087c	a85ff06f	CH.cjal	c0,b0000300	  x0=0x00000000+0x000000000
    24957150000	    998282	b0000300	0380006f	CH.cjal	c0,b0000338	  x0=0x00000000+0x000000000
    24957300000	    998288	b0000338	03a5005b	CH.cspecialw	scr26,c10	 x10:0x80000000+0x17e000800  x0=0x00000000+0x000000000
    24957350000	    998290	b000033c	00000517	CH.auipcc	c10,0x0	 x10=0xb000033c+0x15e3e0000
    24957400000	    998292	b0000340	00c55513	srli	x10,x10,0xc	 x10:0xb000033c x10=0x000b0000
    24957450000	    998294	b0000344	00c51513	slli	x10,x10,0xc	 x10:0x000b0000 x10=0xb0000000
    24957650000	    998302	b0000348	38e53023	CH.csc	c14,896(c10)	 x10:0xb0000000+0x000000000 x14:0x001e3d01+0x000000000 PA:0xb0000380 store:0x0001e3d01+0x000000000
    24957675000	    998303	b000034c	00000013	addi	x0,x0,0	  x0:0x00000000  x0=0x00000000
    24957700000	    998304	b0000350	00000013	addi	x0,x0,0	  x0:0x00000000  x0=0x00000000
    24957750000	    998306	b0000354	00000013	addi	x0,x0,0	  x0:0x00000000  x0=0x00000000
    24957800000	    998308	b0000358	03a0055b	CH.cspecialr	c10,scr26	  x0:0x00000000+0x000000000 x10=0x80000000+0x17e000800
    24957850000	    998310	b000035c	00100073	-->ebreak	
    24958050000	    998318	b0000800	00c0006f	CH.cjal	c0,b000080c	  x0=0x00000000+0x000000000
    24958200000	    998324	b000080c	0ff0000f	fence	iorw,iorw	
    24958250000	    998326	b0000810	0394005b	CH.cspecialw	scr25,c8	  x8:0x00000000+0x17e3e0000  x0=0x00000000+0x000000000
    24958300000	    998328	b0000814	03a5005b	CH.cspecialw	scr26,c10	 x10:0x80000000+0x17e000800  x0=0x00000000+0x000000000
    24958350000	    998330	b0000818	00000517	CH.auipcc	c10,0x0	 x10=0xb0000818+0x15e3e0000
    24958400000	    998332	b000081c	00c55513	srli	x10,x10,0xc	 x10:0xb0000818 x10=0x000b0000
    24958450000	    998334	b0000820	00c51513	slli	x10,x10,0xc	 x10:0x000b0000 x10=0xb0000000
    24958500000	    998336	b0000824	f1402473	csrrs	x8,mhartid,x0	  x0:0x00000000  x8=0x00000000
    24958600000	    998340	b0000828	10852023	csw	x8,256(c10)	 x10:0xb0000000+0x000000000  x8:0x00000000 PA:0xb0000100 store:0x000000000
    24958625000	    998341	b000082c	00a40433	add	x8,x8,x10	  x8:0x00000000 x10:0xb0000000  x8=0xb0000000
    24958750000	    998346	b0000830	40044403	clbu	x8,1024(c8)	  x8:0xb0000000+0x000000000  x8=0x00000000 PA:0xb0000400 load:0x00000000
    24958800000	    998348	b0000834	00147413	andi	x8,x8,1	  x8:0x00000000  x8=0x00000000
    24958850000	    998350	b0000838	02041c63	bne	x8,x0,b0000870	  x8:0x00000000  x0:0x00000000
    24958900000	    998352	b000083c	f1402473	csrrs	x8,mhartid,x0	  x0:0x00000000  x8=0x00000000
    24958950000	    998354	b0000840	00a40433	add	x8,x8,x10	  x8:0x00000000 x10:0xb0000000  x8=0xb0000000
    24959050000	    998358	b0000844	40044403	clbu	x8,1024(c8)	  x8:0xb0000000+0x000000000  x8=0x00000000 PA:0xb0000400 load:0x00000000
    24959100000	    998360	b0000848	00247413	andi	x8,x8,2	  x8:0x00000000  x8=0x00000000
    24959150000	    998362	b000084c	fa041ce3	bne	x8,x0,b0000804	  x8:0x00000000  x0:0x00000000
    24959200000	    998364	b0000850	fd5ff06f	CH.cjal	c0,b0000824	  x0=0x00000000+0x000000000
    24959350000	    998370	b0000824	f1402473	csrrs	x8,mhartid,x0	  x0:0x00000000  x8=0x00000000

Let me know if another instruction trace section would be of use.

@resistor
Copy link
Contributor Author

resistor commented Aug 6, 2025

I don't see anything wrong with the instruction trace. I assume the offset is correct for the store?

I might not be able to get to it for a few days, but is it possible for to reproduce this? If so I can try debugging cheriot openocd/lldb and see what's going on.

@elliotb-lowrisc
Copy link
Contributor

Could it be these shift instructions (time 24957400000 and 24957450000 in the trace above)?

    // clr lowest 12b -> DM base offset
    abstract_cmd[1][31:0]  = HasSndScratch ? dm::srli(5'd10, 5'd10, 6'd12) : dm::nop();
    abstract_cmd[1][63:32] = HasSndScratch ? dm::slli(5'd10, 5'd10, 6'd12) : dm::nop();

Looks like it is loading a capability into c10, using non-capability operations on it, then trying to use it to store data (which presumably fails):

    24957350000	    998290	b000033c	00000517	CH.auipcc	c10,0x0	 x10=0xb000033c+0x15e3e0000
    24957400000	    998292	b0000340	00c55513	srli	x10,x10,0xc	 x10:0xb000033c x10=0x000b0000
    24957450000	    998294	b0000344	00c51513	slli	x10,x10,0xc	 x10:0x000b0000 x10=0xb0000000
    24957650000	    998302	b0000348	38e53023	CH.csc	c14,896(c10)	 x10:0xb0000000+0x000000000 x14:0x001e3d01+0x000000000 PA:0xb0000380 store:0x0001e3d01+0x000000000

@elliotb-lowrisc
Copy link
Contributor

I might not be able to get to it for a few days, but is it possible for to reproduce this? If so I can try debugging cheriot openocd/lldb and see what's going on.

Sure, in what environment would you like to reproduce this? If you are familiar with sonata-system, I could put together some instructions.

@resistor
Copy link
Contributor Author

resistor commented Aug 6, 2025

Could it be these shift instructions (time 24957400000 and 24957450000 in the trace above)?

    // clr lowest 12b -> DM base offset
    abstract_cmd[1][31:0]  = HasSndScratch ? dm::srli(5'd10, 5'd10, 6'd12) : dm::nop();
    abstract_cmd[1][63:32] = HasSndScratch ? dm::slli(5'd10, 5'd10, 6'd12) : dm::nop();

Looks like it is loading a capability into c10, using non-capability operations on it, then trying to use it to store data (which presumably fails):

    24957350000	    998290	b000033c	00000517	CH.auipcc	c10,0x0	 x10=0xb000033c+0x15e3e0000
    24957400000	    998292	b0000340	00c55513	srli	x10,x10,0xc	 x10:0xb000033c x10=0x000b0000
    24957450000	    998294	b0000344	00c51513	slli	x10,x10,0xc	 x10:0x000b0000 x10=0xb0000000
    24957650000	    998302	b0000348	38e53023	CH.csc	c14,896(c10)	 x10:0xb0000000+0x000000000 x14:0x001e3d01+0x000000000 PA:0xb0000380 store:0x0001e3d01+0x000000000

I had the same question the first time I looked at the AC ROM. It turns out that cap exceptions are suppressed when we are in debug mode, so that code works. It's also unchanged by this patch.

@resistor
Copy link
Contributor Author

resistor commented Aug 6, 2025

I might not be able to get to it for a few days, but is it possible for to reproduce this? If so I can try debugging cheriot openocd/lldb and see what's going on.

Sure, in what environment would you like to reproduce this? If you are familiar with sonata-system, I could put together some instructions.

I have a Sonata board. I could also try to run a simulation with Verilator if you can give me instructions. I'm normally a software type working on the toolchain, not a Verilog person.

@resistor
Copy link
Contributor Author

resistor commented Aug 6, 2025

One other thing that might be worth trying is connection to OpenOCD over telnet on port 4444 and controlling it directly via this command set: https://openocd.org/doc/html/General-Commands.html

That should make it possible to determine if the register values are wrong in OpenOCD even without LLDB involved.

@elliotb-lowrisc
Copy link
Contributor

Thanks for the tip about the telnet interface, it helped me find the root issue and get register reads working.

Unfortunately, this seems to have been yet another bug that has already been fixed in the upstream PULP debug module. In this old version of the code, the location in the debug module where the data from a register read is stored doesn't support storing a 64-bit value using two 32-bit stores. The second 32-bit store simply overwrites the first. This appears to have been fixed by pulp-platform/riscv-dbg#94.

Another update to the CHERIoT-Platform debug module to either cherry-pick this fix or rebase onto a more recent version would be much appreciated.

Here's the working LLDB register read output, if that's of interest:

(lldb) reg r
general:
      zero = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        ra = 0x001000c6 (v:? 0x00000000-0x100000000 l:0x100000000 o:0x4[IRQ disable return sentry] p: G R-cgm- Xa ---)
        sp = 0x00120000 (v:? 0x00000000-0x100000000 l:0x100000000 o:0x0[unsealed] p: G RWc-ml -- ---)
        gp = 0x001001f0 (v:? 0x00000000-0x100000000 l:0x100000000 o:0x0[unsealed] p: G RWc-ml -- ---)
        tp = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        t0 = 0x001e8480 (v:? 0x001e8400-0x001e8400 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        t1 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        t2 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        fp = 0x00000000 (v:? 0x00000000-0x100000000 l:0x100000000 o:0x0[unsealed] p: G RWc-ml -- ---)
        s1 = 0x00100088 (v:? 0x00000000-0x100000000 l:0x100000000 o:0x0[unsealed] p: G R-cgm- Xa ---)
        a0 = 0x80000000 (v:? 0x80000000-0x80000004 l:0x4 o:0x0[unsealed] p: G RWc-ml -- ---)
        a1 = 0x000000ff (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        a2 = 0x80000004 (v:? 0x80000004-0x80000008 l:0x4 o:0x0[unsealed] p: G RWc-ml -- ---)
        a3 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        a4 = 0x001e4fb3 (v:? 0x001e4e00-0x001e4e00 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        a5 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
...
        t6 = 0x00000000 (v:? 0x00000000-0x00000000 l:0x0 o:0x0[unsealed] p: - ------ -- ---)
        pc = 0x00100144  cheri_sanity`entry_point + 42
      priv = 0x03

@elliotb-lowrisc
Copy link
Contributor

I think these debug module changes will be left out of the v1.3 release at this point. We're keen to get the recent performance improvements released. We can do a v1.4 release once the debug module is working nicely with Sonata, hopefully not long after.

@elliotb-lowrisc
Copy link
Contributor

@resistor would it help if I opened the fix cherry-pick PR on the CHERIoT-Platform debug module?

@resistor
Copy link
Contributor Author

@elliotb-lowrisc Unfortunately Colin is on holiday at the moment. I'm hoping to get him to weigh in here once he's back.

@elliotb-lowrisc
Copy link
Contributor

Ah, fair enough. I hope he's having a good break

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants