Is there a way to create some kind of sub-tasks to track an operation duration without forking it into a separate @task?

[mgcrea]

I'm running ML training jobs and I would ideally like to track each phase of the training in the run UI like if they were separate sub-tasks without having to use actual @task since their results would not be easily serialisable.

question: should I not be worried about the non-serialisable results of my tasks? And use @task anyway? Is there another way to do that?

Currently my UI lacks inner insight of the train process:

So in that case I would need something like:

@task: load-dataset
@subtask: prepare-dataset (returns a dataset)
@subtask: load-model (returns a model instance)
@subtask: run-train (returns a save-able model instance)
@task: upload-results

An API like console.group / groupEnd could be nice:

start_task_group("load-model")
log.info("loading model ...")
end_task_group()
start_task_group("train-model")
log.info("training ...")
end_task_group()

which could allow nested grouping.

[desertaxle]

Hey @mgcrea! If you turn off result persistence for your child tasks then you should be able to achieve what you’re looking for. If result persistence is disabled, then Prefect won’t try to serialize you results.