How to use RISCV RVV in Gem5?

[love-lilly]

Currently I have been working with Gem5 for months. And now I want to use the support for RVV in RISCV isa after finding it in branch . I'm confused if I want to run some instructions like 512b SIMD to support 8 64b compute, what should I do with Gem5, and how to construct the binary which will be compiled into RVV instructions. Are there any test tutorials？

[hnpl]

Hi there, OS support and compiler support for RVV are rather new so unfortunately, there are not many tutorials that I can refer to. Though, here are roughly the steps needed to be done to use RVV with gem5.

Here are a few pointers on how to compiling RISC-V binaries with RVV instructions,

• I saw some document in gcc 13+ saying that it supports code generation with RVV instructions using RISC-V vector intrinsics. I assume that if you're using RISC-V vector intrinsics, you can use gcc-13 cross compiler to compile the binary. However, I've never been able to use gcc-13 for auto-vectorization, so I use llvm + gcc instead, which is similar to this. So, if you'd like to use llvm+gcc setup, you can follow the following steps,
• Downloading this dockerfile: https://github.com/hnpl/containers/blob/main/llvm-rv64gcv.dockerfile.
• Building the docker container, docker build -f llvm-rv64gcv.dockerfile -t gem5/llvm-rv64gcv .
• Running the docker container and building the RISC-V binary with vector instructions. Suppose that you're compiling rvv_test.cpp,

  docker run -u $UID:$GID --volume $PWD:/workdir -w /workdir --rm -it gem5/llvm-rv64gcv
  (docker) /riscv/_install/bin/clang++ rvv_test.cpp -o rvv_test.bin -O2 -march=rv64gcv -menable-experimental-extensions -mllvm --riscv-v-vector-bits-min=128 -mno-relax -static

You can add more flags to the compilation command. It is important to use the -static flag so that all necessary code are compile to 1 executable file.

• Testing if the compiled file has vector instructions,

  (docker) /riscv/_install/bin/riscv64-unknown-linux-gnu-objdump -D -j.text rvv_test.bin | grep -i "vset"

If you binary has vector instructions, it should show some vsetvli instructions like this,

2a:   0d9076d7                vsetvli a3,zero,e64,m2,ta,ma
86:   0d907757                vsetvli a4,zero,e64,m2,ta,ma
f0:   0d907757                vsetvli a4,zero,e64,m2,ta,ma

Here are a few pointers on how to use the binary with gem5 SE mode,

• Compiling gem5: Make sure that you're using the current develop branch. The current stable branch does not have RVV support. Other than that, you can compile gem5 as usual,

   scons build/RISCV/gem5.opt -j`nproc`

• System configuration: you don't need to do anything special as RVV support is enable by default (exception: you should not use the RISC-V Matched board from the standard library as the U74 core model does not support RVV). Here is an example configuration,

# Copyright (c) 2023 The Regents of the University of California
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met: redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer;
# redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution;
# neither the name of the copyright holders nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

"""
Usage: ./build/RISCV/gem5.opt rvv_test.py

"""

import m5
from m5.objects import Root

from gem5.utils.requires import requires
from gem5.components.boards.simple_board import SimpleBoard
from gem5.components.memory import DualChannelDDR4_2400
from gem5.components.processors.simple_processor import SimpleProcessor
from gem5.components.processors.cpu_types import CPUTypes
from gem5.isas import ISA
from gem5.simulate.simulator import Simulator
from gem5.resources.resource import obtain_resource, FileResource
from gem5.components.cachehierarchies.classic.private_l1_private_l2_cache_hierarchy import (
  PrivateL1PrivateL2CacheHierarchy,
)

requires(isa_required=ISA.RISCV)

cache_hierarchy = PrivateL1PrivateL2CacheHierarchy(
  l1d_size="16kB", l1i_size="16kB", l2_size="256kB"
)

memory = DualChannelDDR4_2400(size="3GB")

processor = SimpleProcessor(
  cpu_type=CPUTypes.TIMING, isa=ISA.RISCV, num_cores=2
)

board = SimpleBoard(
  clk_freq="3GHz",
  processor=processor,
  memory=memory,
  cache_hierarchy=cache_hierarchy,
)

board.set_se_binary_workload(
  FileResource("rvv_test.bin"),
  arguments = [] # add arguments here if the binary requires arguments
)

simulator = Simulator(board=board)
simulator.run()

• Verifying the configuration: note that RVV is vector-length agnostic vector architecture, so the vector length (VLEN) is dependent on your architecture, and in this case, gem5's setting. By default, the vector length is 256 bits. You can add the following lines to the above configuration (above the board=SimpleBoard line) to use a different vector length, e.g.,

for core in processor.cores:
    core.core.isa[0].vlen = 512

You should also check the m5out/config.ini to see whether enable_rvv is set and the vlen value is as desired.

• Also, note that due to way vector memory instruction is implemented, currently gem5 does not support VLEN > 512 bits.

RVV in FS mode.

gem5 does support RVV in FS mode, though it requires using more recent versions of Linux kernel and OpenSBI as well as some modification to the gem5 code base. I'll fill out this section if anyone'd like to use RVV in FS mode.

gem5 should be able to support RVV instructions in FS mode from the v23.1.0.0 release. You can start from this configuration to run an example of running OpenSBI+Linux+Ubuntu20.04 in gem5. The riscv-ubuntu-20.04-boot workload is a predefined workload as detailed here, https://resources.gem5.org/resources/riscv-ubuntu-20.04-boot. To run the simulation with a customized disk image, or a modified Linux/OpenSBI version, you can change the input to the board.set_workload() function in the example configuration.

[AlutionXian]

Hello there, I hope it's not too much of an intrusion, but I'm quite intrigued by your comment regarding the implementation of RVV in FS mode. As I'm currently involved in a project that necessitates RVV simulations under FS mode, any insights or experiences you could share would be immensely valuable to me. Thank you for considering my request.

[hnpl]

Hi there, please see the answer here https://github.com/orgs/gem5/discussions/1159#discussioncomment-9759519

[grigohas]

Hello, i did what you said and i succesfully changed the vlen to 512 but when i am trying to change the elen to 128 i get an error that has to be <64. Is that because of gem5 ?

[grigohas]

yeah but i changed the max of elen on riscvisa.py and i did the scon again and i dont get an error. is it correct?

[hnpl]

Even though ELEN can be changed to > 64, I suspect the underlying arithmetic of the gem5's RVV implementation is 64-bit arithmetic, so there won't be support for 128-bit arithmetic.

Having said that, I haven't been up-to-date with RVV implementation. Please try and let me know if it works correctly.

[grigohas]

Okay thank you. How can i know that it is working correctly? I dont get an error and also elen changed in config file at gem_out folder.

[hnpl]

I guess you can do some tests with a small binary with 128-bit arithmetic and verify the results.

[grigohas]

do you have any binary in that form ?