Open
Description
We currently ignore the costs of transferring non-Chunk
task inputs, but that's silly; they have a real transfer cost, but it's incurred immediately at task transfer to the worker. We should calculate the sizes of these inputs (which we can also cache locally) and factor them into the costs in estimate_task_costs
. This should cause the scheduler to bias tasks with heavy non-Chunk
inputs to the scheduler's worker (but on a different thread).