Why dont separate replication work from noobaa core

[lzw06061139]

I deploy noobaa on k8s with noobaa operator, and there are one noobaa-core and one noobaa-endpoint pod running on the k8s on default. There are running webserver, bg_workers and hosted_agents on the noobaa-core pod.

I want to copy data from one bucket to another, the source bucket(A) contains billions of objects, the target bucket is empy(B). So I create two namespacestores/bucketclasses/obcs, one of them for the source bucket(A), the other for the target bucket(B), and set replication-policy(the destination_bucket is target bucket B) option when create obc for the source bucket(A) . All the above done, and check the copy progress every now and then, objects in source bucket(A) are copied to target bucket(B), but it‘s seems too slow。

I browser the noobaa core code， get the replication process work like that: The noobaa core starts a background scheduled task that periodically reads replication rules from the replication store. For each rule, an asynchronous function is started, which reads 1000 objects from the source bucket and 1000 objects from the destination bucket each time, and then copies the difference data to the destination bucket. After waiting for a period of time (default 10 seconds), the next cycle begins.

if you have lots of objects(eg, billions) to copy ， copying them using the above way is undoubtedly inefficient and time-consuming. So why don't separate the replication worker from noobaa core, and run it on a different pod, scale it independently, and optimaize the replication logic so there replication pods can deal with the rules concurrenty(or even deal with one rule concurrently) .

[nimrod-becker]

There is no reason why not, just priorities for the project. We'd be happy to get such a contribution if this is interesting for you.