Setting up a cluster with 8 nodes

[rohankadekodi]

Hello,

I wish to create an Assise cluster with 8 nodes, with 3-way replication of data. How do I set up the nodes with replicas manually? I am guessing that we need to make certain modifications in libfs/src/distributed/rpc_interface.h.

For example, should I set g_n_hot_reps = 8 (the total number of nodes in the cluster)?

How do I configure which SharedFS replicates which parts of the cached file system namespace?

Thanks for your help.

[wreda]

Our artifact currently doesn't support multiple namespaces, but you can likely still get this working with a few tweaks.

The quickest way is to create separate KernFS configurations for each replica group and assign them to distinct NVM namespaces. You should set g_n_hot_reps = 3 since you're doing 3-way replication. You might need to assign different port numbers to your KernFS instances if they're sharing the same machines. You can set this using the environment variable PORTNO for both LibFS and KernFS.

I'd be interested in adding proper support for namespaces, so any PR requests are also welcome here.

[rohankadekodi]

For a single namespace, is there a way to run with 4 or more machines, with 3-way replication of data in the current prototype? For example, say I want to run RocksDB or Filebench with 4 nodes, how can I configure the cluster?

I can give a shot at namespace support soon.

[wreda]

You can try setting g_n_hot_reps = 4 and then manually override the replication factor in LibFS by setting the environment variable MLFS_RF to 3. This will skip the last replica in the chain.

This is a bit hackish though, so no guarantee it'll work out-of-the-box. You can write back here if you run into issues, and I'll help debug.

[rohankadekodi]

I tried this, and all the kernfs instances segfaulted at:
digest_logs()->digest_inode()->mlfs_mark_inode_dirty()->rb_insert()->inode_cmp()

I can also provide access to this cluster if that helps.

[wreda]

Thanks for the update. I'll check on my end first and get back to you.

[wreda]

I wasn't able to get this working properly yet, and it will likely require some non-trivial changes. I'd recommend for now that you limit your setup to the number of replicas.

I'll keep this issue open and will take another stab once I have free cycles.