How to re-enqueue a minion task?

[fluca1978]

I'd like to enqueu a task that will perform a "batch" of operations, and then exit. But then I need to execute another batch unless the all work is done (this is data retrieval task).
So my idea would be to enqueue again the same task (or reschedule for execution) until all it's done.
But what is the best way to do this?
I was wondering to do something like:

my $result = $minion->job( $tasks{ $name } )->info->{ result };
if ( $result != 'COMPLETE' ) {
   $minion->enqueue( $tasks{ $name } );
}

but I don't think this will trigger a new execution. Otherwise, is there a way to re-perform (or re-run) an already compelted job as a new one?

[kraih]

Use dependencies. Make one job that creates a batch of jobs and one followup job that depends on all the batch jobs. The followup job can then check if more batches are needed.

[fluca1978]

I'm trying with parents and children, but I need a short example of documentation with this kind of approach because I'm not able to figure it out.
I've done this:

• scraper is the task supposed to do '$batch` amount of work
• master is the task responsible to create all the scrapers depending on the total amount of work
• observer is in charge to see if the scraper tasks have completed enough work, otherwise schedule again a master.

my $amount = 2000;
my $batch = 300;

my $scraper_task = sub {
    my ( $job, $batch) = @_;
    sleep( int( rand( 2 ) ) );
    say "Job " . $job->id . " completed $batch";
    $job->finish( $batch  );
};


my $batch_creator_task = sub {
    my ( $job, $batch ) = @_;
    my $tasks_to_generate = int( $amount / $batch ) + 1;
    for ( 1 .. $tasks_to_generate ) {
	my $id = $job->minion->enqueue( 'scraper', [ $batch ]  );
	say "Master enqueued job $id";
    }

    $job->finish( "Master completed" );
};


my $observer_task = sub {
    my ( $job ) = @_;
    my $done = 0;
    my $jobs = $job->minion->jobs( { tasks => [ 'scraper' ] } );
    while ( my $info = $jobs->next) {
	say "$info->{id}: $info->{state} $info->{ result }";
	$done += $info->{ result };
    }


    if ( $done <= $amount ) {
	my $master = $job->minion->enqueue( 'creator', [ $batch ] ); # start enqueueing other jobs
    }
    else {
	say "ALL DONE";
    }

};

my $minion = Minion->new ( Pg => 'postgresql://luca:luca@venkman/testdb' );
$minion->add_task( creator => $batch_creator_task );
$minion->add_task( scraper => $scraper_task );
$minion->add_task( observer => $observer_task );

my $master = $minion->enqueue( 'creator', [ $batch ] ); # start enqueueing other jobs
my $observer = $minion->enqueue( 'observer' );

$minion->perform_jobs;

however, even with this simple example I hit a problem: how to limit the observer to get information from the last scheduled tasks and not all the previous tasks? This is what you were referring to as dependencies, but I don't know how to implement it.

[kraih]

I'm using that approach in Cavil: https://github.com/openSUSE/cavil/blob/a527755a49ccd3feaa0cdca14227bf3901c99f8e/lib/Cavil/Task/Index.pm#L55