List view
**Goals** - Simplify integration into applications and improve developer ergonomy - Reduce external dependencies (ideally to defaults in rails) **Rationale for "next"** The current Aeternitas gem has not been actively maintained and presents a number of limitations: - It is tightly coupled to Sidekiq and Redis, complicating upgrades and alternative job processing backends. - Developers have found the API difficult to understand or extend. - Many of its advanced features are underutilized or overly complex for typical use cases. This milestone aims to explore and define a path forward—whether by significantly refactoring Aeternitas or replacing it with a leaner alternative that aligns with the current ecosystem. **Main functionality provided by existing Aeternitas** 1. Automated scheduling of jobs to crawl (single) webpages (`Aeternitas::Pollable`) - defines commen methods (`poll`) which are then picked up by jobs - stores statistics and calculates next scheduled time - next schedule can adopt dynamically to number of results retrieved etc. 2. Storing raw data of crawl results for further processing (`Aeternitas::Source`) - default store are file compressed on disk - storage adapter may be changed (implemented) 3. Limiting the requests per timeframe(ratelimit via `Aeternitas::Guard`) - to avoid accidential DDoS against a Publisher (and avoid complaints) - Can be configured per pollable and static or dynamic (proc) - can also be used to distribute crawling load over larger time period (limit concurrency + extra cooldown time) - if configured via Pollable a raised lock error automatically reenqueues the polling job (by default the worker may continue processing other jobs) - if needed a lock can be configured to cause worker to sleep to avoid thrashing (always reenqueing the same job) 4. Web UI statistics via [aeternitas_web_ui](https://github.com/FHG-IMW/aeternitas_web_ui)
No due date•7/7 issues closed