Speed up queries via materialized view

[Muhammad-Moiz626]

Description

Our data model with multiple beliefs per event time (multiple horizons as well as sources) makes queries computationally expensive. Most-used are queries where we look only for the most recent beliefs about each event, when given any moment in time as max belief time.

Pre-computation with Postgres' materialized views can make queries much faster. This PR will enable hosts to use this performance enhancement.

• Created migrations file to create materialized view.
• Use the MV in queries.
• Use SQL Alchemy for MV.
• Automate the refreshing of the MV.
• Make MV optional / configurable.
• Document the process

...

Further Improvements

• the full queries (used in playbacks, getting fulls set of beliefs for each event) might also enjoy a materialized view, but maybe it is just too much data, then we should just limit playback to shorter time ranges
• give hosts a range of options (e.g. refresh time)
• actually move to TimeScaleDB, which auto-refreshes their materialized views on inserts

Related Items

Closes 108

...

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[Flix6x]

Some early review comments. Nice to see the refresh logic taking shape.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #29463391 | 📁 Comparing e7b4599 against latest (2bbb109)

🔍 Preview build

Show files changed (4 files in total): 📝 4 modified | ➕ 0 added | ➖ 0 deleted

File	Status
changelog.html	📝 modified
configuration.html	📝 modified
_autosummary/flexmeasures.data.models.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.models.time_series.html	📝 modified

[Flix6x]

Exciting work! Thanks for addressing my earlier comments. Just a short code review for now, so you don't have to wait for me testing. Looking forward to testing tomorrow.

[nhoening]

Also a quick review from me, before having tested myself, mostly about concepts.

I also suggest moving get_timed_belief_min_v() to timely-beliefs and add basic documentation there (not much more than the SQL command so anybody reading only that could in principle use it)

[Flix6x]

This FLEXMEASURES_MVIEW_REFRESH_INTERVAL setting requires a short discussion, @Muhammad-Moiz626 @nhoening. I already scoped some options with @Ahmad-Wahid. I suggest to spin out that discussion on this comment thread.

There is currently no automation that runs the CLI command to refresh the mviews. Our go-to approach is to do this via a cronjob, for which we'd essentially have to move this setting in the cron job configuration. It doesn't then make sense to have that setting live in FlexMeasures, as FlexMeasures is not setting up cronjobs itself. I believe the config setting is, in this PR, only used to inform the user on the refresh rate in the UI. I think that is nice. If we want to keep that, then an alternative to this config setting may be to use the tooling we have that saves (in the db) the last time a specific CLI command was run successfully, and report that time in the UI instead.