Benchmark Details

[agnesnatasya]

Hi,

I am interested in replicating the benchmark setup as detailed in the Assise paper, and I would like to ask some details about the NFS and CephFS configuration.

In the experimental configuration part , it is stated that

• Ceph
  • "machines are used as OSD and MDS replicas in Ceph"
  • "Ceph's cluster managers run on 2 additional testbed machines"
• NFS
  • "NFS uses only one machine as server"

For Ceph,

1. If the machine number is 2, as used in the write latency microbenchmark
   a. What is the number of data pool, metadata pool, and MDS replicas respectively?
2. What does the 'Ceph cluster manager' mentioned in the paper refer to, is it the MDS replicas?
   a. If yes, does it mean that the MDS replica number will be 2, because it is run on 2 additional testbed machines?
3. As the kernel buffer cache is limited to 3GB for all FS, is this only for the kernel page cache size? Are there any specifications on the MDS cache?

For NFS,

1. If the machine number is 2, does it mean that there are 1 client and 1 server, or 2 clients and 1 server in the cluster?

For both,

1. I am interested to know how do you set this linux page cache size to 3GB and make sure that the other clients' kernel buffer cache is totally empty before reading? I tried some options when mounting NFS, but it still looks like that the other client does some pre-reading on other writes that happen on other clients, as the benchmark value that I get for a Read-HIT and Read-MISS is the same.

Thank you very much for the kind help!

[wreda]

Adding Jongyul (@yulistic), who should be able to tell you about our Ceph/NFS configuration.

[yulistic]

For Ceph,

1. In the write latency microbenchmark, the number of data pools was 2. We deployed 1 MDS for latency microbenchmark because there are few metadata operations in the benchmark. For the scalability microbenchmark and mail server experiment, we used the same number of MDS replicas with the number of server machines.
2. It corresponds to Ceph's MON and MGR daemons. Conceptually, "cluster manager" doesn't handle data and metadata. It only deploys, controls, and monitors distributed file system processes (daemons).
3. Only kernel page cache size was limited. We used the default configuration for MDS cache limit.

For NFS,

1. One client and one server.

For both,

1. I wasn't able to find a way to directly restrict page cache size. Instead, we restrict the total memory usage of microbenchmarks. The size of the page cache used by microbenchmark can be obtained after its execution. With this observation, we set the limit of total memory usage of microbenchmark (with cgroup) so that the microbenchmark uses roughly 3 GB of page cache consequently.
   You can clear the buffer cache with echo 3 > /proc/sys/vm/drop_caches.

[agnesnatasya]

Hi Jongyul,

Thank you very much for the details of the benchmark. Do you mind if I clarify about the experimental setup for measuring read latency and LevelDb performance specifically?

LevelDB Application Benchmark

• Assise
  I am running Assise using 3 different hot replicas. When I set up using 3 replicas, Assise seems to stuck at some point of the replication. I tested using the normal iotest and also LevelDb. I run the LevelDb benchmark under bench/leveldb/mlfs directory using the command ./run_bench.sh fillseq, and the program seems to hang.
  For information, I am able to run and complete if Assise uses 2 different hot replicas.

Read Microbenchmark

• NFS and Ceph. NFS: 1 client and 1 server. Ceph: 2 OSD replicas. Using fio library
  • HIT
    • Client in NFS, Node 1 in Ceph: Write then immediately read

       fio --name=test --bs=<bs> --readwrite=write --size=1G --filename=/mnt/local_dir/bench_1.txt # write
       fio --name=test --bs=<bs> --readwrite=read --size=1G --filename=/mnt/local_dir/bench_1.txt #immediately read
      

  • MISS
    • Server in NFS, Node 1 in Ceph: Write

       fio --name=test --bs=<bs> --readwrite=write --size=1G --filename=/mnt/local_dir/bench_1.txt
      

    • Client in NFS, Node 2 in Ceph: Clean cache and read

       echo 3 > /proc/sys/vm/drop_caches
       fio --name=test --bs=<bs> --readwrite=read --size=1G --filename=/mnt/local_dir/bench_1.txt
      

• Assise: Run on 2 hot replicas
  • HIT
    • Node 1

       MLFS_DIGEST_TH=100 ./run.sh iobench_lat wr 1000M <BS> 1 
      

  • MISS
    • Node 1

      ./run.sh iobench_lat sw 1000M <BS> 1  # writes so that the file exist
      

    • Node 2

      ./run.sh iobench_lat sr 1000M <BS> 1 # read from another node.
      

Is the above configuration correct as the result that I got from the above tests are different from the one presented in the paper, and it does not reflect hit and miss in most cases. The results that I have are as follows

• NFS: The result for both Hit and MIss is closer to the Hit result presented in the paper for most block sizes
• Ceph: The result for both Hit and Miss is closer to the Miss result presented in the paper for most block sizes
• Assise: The result does not seem correct as the Hit method might have a higher latency than the Miss method.

May I ask for help regarding the discrepancies on benchmark results as stated above? Thank you!

[yulistic]

Please compare your results with the raw latency of DRAM and your IB or RoCE network. The miss latency should include the network crossing overhead which is much higher than local DRAM access (hit latency). Then, you will be able to figure out which configuration (miss or hit) is incorrect.

The numbers were measured with microbenchmarks: bench/micro. It seems better to use the same benchmark to reproduce the numbers.

Mentioning @wreda for Assise results.

[wreda]

I am running Assise using 3 different hot replicas. When I set up using 3 replicas, Assise seems to stuck at some point of the replication. I tested using the normal iotest and also LevelDb. I run the LevelDb benchmark under bench/leveldb/mlfs directory using the command ./run_bench.sh fillseq, and the program seems to hang.
For information, I am able to run and complete if Assise uses 2 different hot replicas.

This could be a bug in the 3-replica configuration. Feel free to open another issue for this with error logs/stack trace, and I'll be happy to take a look.

Read Microbenchmark

• Assise: Run on 2 hot replicas

  • HIT

    • Node 1

       MLFS_DIGEST_TH=100 ./run.sh iobench_lat wr 1000M <BS> 1 
      

  • MISS

    • Node 1

      ./run.sh iobench_lat sw 1000M <BS> 1  # writes so that the file exist
      

    • Node 2

      ./run.sh iobench_lat sr 1000M <BS> 1 # read from another node.
      

Is the above configuration correct as the result that I got from the above tests are different from the one presented in the paper, and it does not reflect hit and miss in most cases. The results that I have are as follows

• Assise: The result does not seem correct as the Hit method might have a higher latency than the Miss method.

May I ask for help regarding the discrepancies on benchmark results as stated above? Thank you!

The configuration for Assise looks fine. Note, however, that for higher IO sizes (> 4KB) you might experience worse performance in the HIT case since it needs to do multiple hash table lookups. In any case, you can try running LibFS with profiling enabled for both HIT and MISS: MLFS_PROFILE=1 ./run.sh iobench_lat. This will provide a more fine-grained performance breakdown, which might help us pinpoint the cause of the discrepancy.

EDIT: Upon further thought, the HIT performance is likely worse here since your file size is not small enough to fit inside the log (assuming you're using the default log size of 1 GB). This causes the file to spillover to the other caches. To avoid this, either reduce your file size or increase the log size. For example, at 4 KB IO and a log size of 1 GB, your file should be ≤ 256 MB (to account for any metadata overheads).