Skip to content

feat: Keep progress markers separate from underlying task data #464

@starpit

Description

@starpit

Currently, we cannot stream the input stream directly from s3 into the stdin of the worker. This is because currently the inbox->processing->outbox flow (needed by the workstealer) is not separate from the data to be processed. We will need to keep them separate, somehow. So that the inbox->processing->outbox flow is done with markers, and the underlying data is placed in a separate folder. This will also avoid potentially expensive copying of large data blobs through those stages.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions