Adding the ability to trigger task run manually.

[FreCap]

Motivation: in processing pipelines where tasks can be time-intensive, and multiple queues/tasks are fighting for resources, the lib users can benefit from implementing a more complex pulling strategy and trigger executions manually. Hence we start exposing a public executeDueTasks() method.

Changes:

• Exposing executeDueTasks as public
• renaming Scheduler.start() to Scheduler.startConsumer(), keeping backward compatibility, to reflect that only the "consumer-machines" will have to start this process
• add the ability to disable consuming with SchedulerBuilder.disableStartConsumingTasksOnStart(), in such a way that lifecycle threads can handle heartbeat and dead executions, but not computation heavy threads (of the executions), which can be triggered manually

Additional:

• feature tested
• all methods changes are backward compatible.

[kagkarlsson]

Thank you for contributing!

I think I need to understand this use-case better in order to evaluate this feature/pull-request. Could you describe some scenario where this would be useful (in more detail than above)? I assume you can't just increase the thread-count?

Also, a small heads-up here, this will likely have some conflicts with another PR I am looking to merge soon: #175

[FreCap]

@kagkarlsson thanks for the comment!

One of the use cases, is when you have a set of tasks that always have to be run before the others.
Let's say you have 3 different tasks:

1. "put off fire"
2. "send some texts"
3. "get other"

You want to run all of the "put off fire" fire before going to any other task.
All of those tasks have different dataclasses.

Having executeDueTasks can facilitate greatly situations where we have to handle this case by building a coordinator

[kagkarlsson]

You want to run all of the "put off fire" fire before going to any other task.
All of those tasks have different dataclasses.

Wouldn't this be solved by adding the priority-feature?

Also, do you mean that 1) must run completely before 2) should start running?

[FreCap]

In the other post you put it in better words than mine:
Do you require to be able to set priority on instance-level, or would task-level work? I.e., for a given task, all instances have the same priority, both are needed

The difference is that at instance-level once the priority is given, that is the priority. At task level instead it can be flexible and some task can gain precedence over other with the change of time. For this reason it can be worth it just exposing the API and the user can run complex scheduling on his own instead of limiting the possible usages.

An example of tasks can change priority over time is: the task "compute-news-midnight" with all of its instances has more priority than "compute-news-midday", and but in another point in time the other is more important to complete

[maratik123]

It seems that Scheduler.executeDue() should be also public. It is required for tests to fire scheduled tasks (without jumping around with semaphores (or something like that) to wait execution)

[kagkarlsson]

Check out ManualScheduler fre. 15. okt. 2021 kl. 21:39 skrev maratik123 ***@***.***>:

…

It seems that Scheduler.executeDue() should be also public. It is required for tests to fire scheduled tasks (without jumping around with semaphores (or something like that) to wait execution) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#184 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABKEHAWHSMOUDLQ6Z5VVM3UHB7PJANCNFSM4X6O2GXQ> .