Skip to content

Task cost estimation: Treat non-Chunk inputs as residing on scheduler #330

Open
@jpsamaroo

Description

@jpsamaroo

We currently ignore the costs of transferring non-Chunk task inputs, but that's silly; they have a real transfer cost, but it's incurred immediately at task transfer to the worker. We should calculate the sizes of these inputs (which we can also cache locally) and factor them into the costs in estimate_task_costs. This should cause the scheduler to bias tasks with heavy non-Chunk inputs to the scheduler's worker (but on a different thread).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions