minstret reads wrong value after a pipeline flush with WritebackStage=1

[georgerennie]

When using the opentitan config which has WritebackStage=1, reading the value of minstret (number of executed instructions) directly after the pipeline is flushed reads a value that is one too high.

Steps to reproduce the issue

I have created a branch with a reproducer program in examples/sw/simple_system/minstret_bug. This program (in asm.S) demonstrates the bug by flushing the pipeline with a csr write and with a jump in between minstret reads. It can be built with the standard Verilator simple system build steps, using the opentitan config to demonstrate the bug.

Reproducer bash commands

fusesoc --cores-root=. run --target=sim --setup --build lowrisc:ibex:ibex_simple_system $(util/ibex_config.py opentitan fusesoc_opts)

make -C examples/sw/simple_system/minstret_bug

./build/lowrisc_ibex_ibex_simple_system_0/sim-verilator/Vibex_simple_system --meminit=ram,examples/sw/simple_system/minstret_bug/minstret_bug.elf

cat ibex_simple_system.log

The core are these sequences which read minstret with a forced flush inbetween. These values are subsequently printed to ibex_simple_system.log. For each case we expect to see n, n+2 and n+3 printed as there is one instruction between the first and second minstret reads.

csrw_flush:
        nop
	csrr s0, minstret
        # write mstatus csr WPRI bit, flushing pipeline
	csrsi mstatus, 1
	csrr s1, minstret
	csrr s2, minstret
	ret

jump_flush:
        nop
	csrr s0, minstret
        # jump, flushing pipeline
	j jump_flush_continue
jump_flush_continue:
	csrr s1, minstret
	csrr s2, minstret
	ret

Observed Behavior

When using the opentitan config, this results in the following log. Even though an extra instruction has occured between the second and third values, they incorrectly read the same value.

0000003D
00000040
00000040

0000011D
00000120
00000120

Expected Behavior

minstret should return the number of instructions which have been executed, which should be monotonically increasing. Using the small config, the log meets these expectations. it is not currently clear to me why the second set of values start at 0x11E instead of 0x11D in this config or whether this is related to the bug.

0000003D
0000003F
00000040

0000011E
00000120
00000121

My Environment

EDA tool and version:

Verilator 5.032 2025-01-01
Bug originally found using a version of riscv-formal.

Operating system:

Manjaro Linux with 6.6.65-1-MANJARO kernel.

Version of the Ibex source code:

bbda8c8

[marnovandermaas]

Thanks for making this issue from a quick look through the RTL I see that there is a speculative and non-speculative version of the instructions retired:
https://github.com/lowRISC/ibex/blob/master/rtl/ibex_wb_stage.sv#L154-L162
Which are used to update minstret as follows:
https://github.com/lowRISC/ibex/blob/master/rtl/ibex_cs_registers.sv#L1355-L1362

I'd need to look at the waves on how this all feeds into the ID stage.