ruby garnet The injection rate and the number of flits in the results do not correspond.

[Zch106]

Hi all, I'm new to gem5 and have run into some problems while using Garnet. I hope to get some responses from you.

I conducted experiments based on the examples from this website: https://www.gem5.org/documentation/general_docs/ruby/garnet_synthetic_traffic/

I run the command below：

../../build/NULL/gem5.debug ../../configs/example/garnet_synth_traffic.py \
    --num-cpus=64 \
    --num-dirs=64 \
    --network=garnet \
    --topology=Mesh_XY_Chiplet\
    --mesh-rows=8 \
    --sim-cycles=200000000 \
    --synthetic=uniform_random \
    --sys-clock=1GHz \
    --ruby-clock=1GHz \
    --router-latency=2 \
    --link-latency=2 \
    --vcs-per-vnet=4 \
    --routing-algorithm=1 \
    --injectionrate=0.01

I simulated a total of 20,000,000 cycles. According to expectations, this should generate approximately 20,000 * 0.01 * 64 = 12.8K packets. However, in the stats.txt file under the m5out folder:
system.ruby.network.packets_injected::total 1966

No matter how I adjust the injectionrate, there is a significant discrepancy between the final number of packets_injected and what was expected. Additionally, the average_packet_latency values are barely affected by changes in the injection rate, which is quite different from my expectations.

I attempted to resolve this issue by modifying the sendPkt function in the GarnetSyntheticTraffic.cc file.

void
GarnetSyntheticTraffic::sendPkt(PacketPtr pkt)
{
    if (!cachePort.sendTimingReq(pkt)) {
        retryPkt = pkt; // RubyPort will retry sending
    }
    numPacketsSent++;
    DPRINTF(GarnetSyntheticTraffic,"numPacketsSent = %d , and id = %d\n",
            numPacketsSent, id);
}

I used the --debug-flag=GarnetSyntheticTraffic option and found in the debug.log that each CPU sent approximately 200 packets, which aligns with the expected number of 20000 * 0.01 * 64 = 12.8K packets. However, for some reason, these packets seem not to have reached the Garnet network.

In the section 'Implementation of Garnet synthetic traffic' on this post [https://www.gem5.org/documentation/general_docs/ruby/garnet_synthetic_traffic/], it describes the process of CPUs sending packets as follows: 'Every cycle, each CPU performs a Bernoulli trial with a probability equal to –injectionrate to determine whether to generate a packet or not. If –num-packets-max is non-negative, each CPU stops generating new packets after generating –num-packets-max number of packets. The injector terminates after –sim-cycles.'

However, in the debug.log, I observed that it seems all 64 CPUs send packets at the same time, as shown in the following results:

19548000: system.cpu12: Generated packet with destination 63, embedded in address fc0
19548000: system.cpu12: numPacketsSent = 176 , and id = 12
19548000: system.cpu11: Generated packet with destination 63, embedded in address fc0
19548000: system.cpu11: numPacketsSent = 176 , and id = 11
19548000: system.cpu10: Generated packet with destination 63, embedded in address fc0
19548000: system.cpu10: numPacketsSent = 176 , and id = 10
19548000: system.cpu09: Generated packet with destination 63, embedded in address fc0
19548000: system.cpu09: numPacketsSent = 176 , and id = 9
19548000: system.cpu08: Generated packet with destination 63, embedded in address fc0
19548000: system.cpu08: numPacketsSent = 176 , and id = 8
19548000: system.cpu07: Generated packet with destination 63, embedded in address fc0
19548000: system.cpu07: numPacketsSent = 176 , and id = 7
19548000: system.cpu06: Generated packet with destination 63, embedded in address fc0

This has left me very confused, as different CPUs generate packets with the same destination at the same time, which seems to indicate that all CPUs are sharing the sendAllowedThisCycle parameter within the GarnetSyntheticTraffic::tick() function. I believe the correct behavior should be that different CPUs send packets at different ticks, independently calculating the probability of sending a packet.

Any commend would be quite helpful. I hope someone can answer my questions!! Thank you very much in advance!!!!!! I am also willing to share my experimental data.