It would be nice to have an epi archive backed by disk / many files + we talked about the current exploration framework based on the targets package + annoyances going through epix_slide when also looping across things other than forecast_date (+ annoyances with epix_slide when the slide function outputs errors/etc.) I think this points to:
- epi_archive should be something more general that can have various backends. I sketched some backends we might imagine here.
- Maybe we want just a smarter/faster
epix_as_of rather than epix_slide. (epix_slide isn't smart at all yet, but it'd be easier than making epix_as_of smarter. Making epix_as_of smarter probably involves caching the last query & result or some sort of partition of the data, while epix_slide would get to order things as it likes, partition once, etc.)
- We often want a way to let slide computations output errors/warnings instead/alongside "real" results, rather than have
epix_slide completely give up or spit out all the warnings without associated forecast_as_ofs at the end. I think purrr might have some useful helper functions here (purrr::safely?).
- Would we want a
derived_epi_archive that does some of this targets/targets-like stuff, or based on some (hypothetical?) library that does parallelism with only one pool of workers & adjusts for BLAS/gurobi/etc. using parallelism to prevent CPU/swap thrashing / OOM killing (investigate coro?)? It might still have some of the awkwardness of epix_slide, but might make some things look nicer; e.g., metaforecaster's forecasts would be a derived_epi_archive maybe atop a merge_epi_archive atop (a) the derived_epi_archive holding the base forecaster's forecasts and (b) the response data's ram_epi_archive.
Alternatively, if we don't have derived or cached archived concepts, can we integrate our slide computations with a framework that does? E.g., targets.
It would be nice to have an epi archive backed by disk / many files + we talked about the current exploration framework based on the targets package + annoyances going through
epix_slidewhen also looping across things other than forecast_date (+ annoyances with epix_slide when the slide function outputs errors/etc.) I think this points to:epix_as_ofrather thanepix_slide. (epix_slideisn't smart at all yet, but it'd be easier than making epix_as_of smarter. Makingepix_as_ofsmarter probably involves caching the last query & result or some sort of partition of the data, whileepix_slidewould get to order things as it likes, partition once, etc.)epix_slidecompletely give up or spit out all the warnings without associatedforecast_as_ofs at the end. I thinkpurrrmight have some useful helper functions here (purrr::safely?).derived_epi_archivethat does some of this targets/targets-like stuff, or based on some (hypothetical?) library that does parallelism with only one pool of workers & adjusts for BLAS/gurobi/etc. using parallelism to prevent CPU/swap thrashing / OOM killing (investigatecoro?)? It might still have some of the awkwardness ofepix_slide, but might make some things look nicer; e.g., metaforecaster's forecasts would be aderived_epi_archivemaybe atop amerge_epi_archiveatop (a) thederived_epi_archiveholding the base forecaster's forecasts and (b) the response data'sram_epi_archive.Alternatively, if we don't have derived or cached archived concepts, can we integrate our slide computations with a framework that does? E.g.,
targets.