Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a Persistent Job Scheduler #1486

Closed
alexanderkiel opened this issue Feb 9, 2024 · 0 comments
Closed

Implement a Persistent Job Scheduler #1486

alexanderkiel opened this issue Feb 9, 2024 · 0 comments

Comments

@alexanderkiel
Copy link
Member

alexanderkiel commented Feb 9, 2024

For issues like #1442 we need a persistent job scheduler. We already have a scheduler module.

The jobs are used for the following scenarios:

  • (re)building indices
  • index cleanup
  • calculating statistics
  • calculating bloom filters

All jobs have the following properties:

  • there execution is idempotent so that they can be rerun partially

What the jobs can't do:

  • non-idempotent tasks like sending messages only once

The new functionality should provide:

  • persistent job states
    • job's that run longer should survive restarts
    • a job should be able to persist it's state regularly to be able to continue after a restart
    • jobs states should be persisted in reasonably long intervals like minutes not milliseconds to not to overwhelm storage
    • the idea is to have one column family were the key is the job id and the value is the job state
    • a type to be able to execute the right code
    • an status of ready, cancelled, in-progress, failed and completed
  • a multi-method that provides the implementation of each job type
@alexanderkiel alexanderkiel self-assigned this Feb 9, 2024
alexanderkiel added a commit that referenced this issue Mar 25, 2024
alexanderkiel added a commit that referenced this issue Mar 26, 2024
alexanderkiel added a commit that referenced this issue Mar 26, 2024
alexanderkiel added a commit that referenced this issue Mar 26, 2024
alexanderkiel added a commit that referenced this issue Mar 26, 2024
alexanderkiel added a commit that referenced this issue Mar 29, 2024
alexanderkiel added a commit that referenced this issue Mar 29, 2024
alexanderkiel added a commit that referenced this issue Mar 29, 2024
alexanderkiel added a commit that referenced this issue Mar 29, 2024
alexanderkiel added a commit that referenced this issue Mar 29, 2024
alexanderkiel added a commit that referenced this issue Mar 30, 2024
alexanderkiel added a commit that referenced this issue Mar 30, 2024
alexanderkiel added a commit that referenced this issue Mar 30, 2024
alexanderkiel added a commit that referenced this issue Mar 30, 2024
alexanderkiel added a commit that referenced this issue Apr 7, 2024
alexanderkiel added a commit that referenced this issue Apr 7, 2024
alexanderkiel added a commit that referenced this issue Apr 7, 2024
alexanderkiel added a commit that referenced this issue Apr 7, 2024
alexanderkiel added a commit that referenced this issue Apr 7, 2024
alexanderkiel added a commit that referenced this issue Apr 7, 2024
alexanderkiel added a commit that referenced this issue Apr 8, 2024
alexanderkiel added a commit that referenced this issue Apr 9, 2024
alexanderkiel added a commit that referenced this issue Apr 11, 2024
alexanderkiel added a commit that referenced this issue Apr 12, 2024
alexanderkiel added a commit that referenced this issue Apr 13, 2024
alexanderkiel added a commit that referenced this issue Apr 14, 2024
alexanderkiel added a commit that referenced this issue Apr 14, 2024
alexanderkiel added a commit that referenced this issue Apr 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

1 participant