Resample beliefs about instantaneous sensors #118

Flix6x · 2022-10-31T12:49:36Z

This PR implements resampling for a BeliefDataFrame describing an instantaneous sensor. It does so by updating the data frequency without changing the event resolution. At some point, I'd like to start introducing terminology to distinguish between the resolution of events and the frequency of the data, but not in this PR.

…aking into account DST transitions

nhoening

This is probably super useful, but can be much clearer.

timely_beliefs/tests/test_belief_utils.py

timely_beliefs/beliefs/classes.py

nhoening · 2022-11-06T09:23:07Z

timely_beliefs/beliefs/utils.py

@@ -749,3 +751,21 @@ def extreme_timedeltas_not_equal(
    if isinstance(td_a, pd.Timedelta):
        td_a = td_a.to_pytimedelta()
    return td_a != td_b
+
+
+def downsample_first(df: pd.DataFrame, resolution: timedelta) -> pd.DataFrame:


This function name was zero helpful to me. Comment also didn't help a lot.

Is this downsampling, and in case of non-obvious cases opting for the first event?

Also, handling DST correctly is so crucial, we could even consider it in to he name.

Thanks for raising this issue. For me it boils down to a more fundamental question: does resampling affect the data frequency or the event resolution? Hopefully these examples illustrate the question:

Let's say there is a sensor recording the average wind speed over a period of 3 seconds. That is, its event resolution is 3 seconds. And let's say a record is made every second, so we get a sliding window of measurements. That is, its data frequency is 1 second. (To be explicit, I'm following pandas terminology here, which measures frequency in units of time, say, s rather than 1/s.)

Now we want to resample and save the results to a new sensor that records the maximum 3-second wind gust in a 10 minute period, for every sequential period of 10 minutes (this is actually a useful measure, e.g. reported by KNMI). That is, the new sensor has an event resolution of 10 minutes, and also a data frequency of 10 minutes.

This resampling operation is downsampling, with both the event resolution and the data frequency being downsampled to 10 minutes. Something like .resample(pd.Timedelta("PT10M")).max() should do the trick to transform the data, and the event resolution would be updated to 10 minutes.

Now consider a sensor recording instantaneous temperature readings, once per hour. That is, its event resolution is 0 hours and its data frequency is 1 hour. What would it mean to resample to 30 minutes? I see two possibilities:

We measure instantaneous temperature every 30 minutes. The event resolution stays 0 hours and the data frequency becomes 30 minutes. Something like .resample(pd.Timedelta("PT30M")).ffill() (or .interpolate()) would do the trick to transform the data. The event resolution would not change. We are upsampling (with regards to the data frequency).

We measure average temperatures over a period of 30 minutes. Some combination of .resample().interpolate().rolling().mean() should do the trick to transform the data. The event resolution would be updated to 30 minutes. We are upsampling (with regards to the data frequency) and downsampling (with regards to the event resolution).

For example, given temperature readings:

10 °C at 1 PM 12 °C at 2 PM

With linear interpolation, option 1 yields:

10 °C at 1 PM 11 °C at 1.30 PM 12 °C at 2 PM

With linear interpolation, option 2 yields:

10.5 °C average between 1 PM and 1.30 PM 11.5 °C average between 1.30 PM and 2 PM

For now, I'm gravitating towards a policy like "in timely-beliefs, event resampling, by default, updates both the data frequency and the event resolution for non-instantaneous sensors, and only the data frequency for instantaneous sensors." But I'd like to support updating the event resolution for instantaneous sensors, too, maybe only when explicitly requested. For example, consider instantaneous battery SoC measurements every 5 minutes. When resampled to a day, one might not really be interested to know the SoC at midnight every day (.first(), and keep the original zero event resolution), but rather something like the daily min or max SoC (.min()/.max(), and update the event resolution to 1 day).

I understand the distinction now (between treating data frequency and event resolution)

W.r.t. your last paragraph,

I can imagine resampling policies, which can be chosen by way of general configuration setting or even per sensor attribute. You named two policies here, hopefully we could keep that simple for now. And you know which default you'd prefer.

The text you've written can probably enter the timely-beliefs documentation, so everybody can learn about our thoughts here.

Finally, you made no suggestion for a better function name yet.

resample_to_frequency or resample_frequency?

I renamed the function.

I can imagine resampling policies

I now made the util function accept different resampling methods, and restricted its use in resample_events to BeliefsDataFrames containing as many events as data points. Solving the general case is a lot harder, but this should already support most use cases.

The text you've written can probably enter the timely-beliefs documentation, so everybody can learn about our thoughts here.

I opened #123 for this.

nhoening · 2022-11-06T09:24:30Z

timely_beliefs/beliefs/utils.py

+
+
+def downsample_first(df: pd.DataFrame, resolution: timedelta) -> pd.DataFrame:
+    """Resample data representing instantaneous events.


This assumption (instantaneous events) is never checked. Or that the index is even a DateTimeIndex

Can you reply on this?

I'm more leaning towards making this a private function. Would that take away your concern?

Not quite. If it matters so much, why not add an assert statement? I'm guessing you wouldn't be sure what the outcomes would be if df wasn't instantaneous?

I opened #124 as a follow-up.

timely_beliefs/beliefs/utils.py

…ld use a different method, such as pad or interpolate)

…t-instantaneous-sensors # Conflicts: # timely_beliefs/beliefs/utils.py

Flix6x · 2022-11-15T18:49:29Z

FYI After you get a chance to reply to our open comments, I plan to clear up the terminology in the docstrings with regards to the sensor resolution vs data frequency.

…ing should use a different method, such as pad or interpolate)" This reverts commit 707ea63.

…y resampling per unique offset, and support multiple resampling methods, with some of them updating the event resolution of the resulting BeliefsDataFrame

…rame

…' method.

nhoening

I approve with the logic in place here, but I still don't grok something about the instantaneous case.

timely_beliefs/tests/test_belief_utils.py

nhoening · 2022-11-21T21:51:14Z

timely_beliefs/beliefs/classes.py

@@ -1416,6 +1434,13 @@ def resample_events(
            return self
        df = self

+        # Resample instantaneous sensors
+        # The event resolution stays zero, but the event frequency is updated


Actually, the event resolution doesn't stay zero in all cases, now, right?

It does here, because the resampling method defaults to asfreq.

nhoening · 2022-11-21T22:29:18Z

timely_beliefs/beliefs/classes.py

@@ -1416,6 +1434,13 @@ def resample_events(
            return self
        df = self

+        # Resample instantaneous sensors


I can't explain why your new function only applies to instantaneous sensors. Can you add that?

And why is that the one case in which we take so much care of DST transitions? (this resample_events function here has two other cases)

Note: The NB about keep_only_most_recent_belief=True applies to only one case, the reader might assume it's about all of them.

nhoening · 2022-11-21T22:30:42Z

timely_beliefs/beliefs/utils.py

+
+
+def downsample_first(df: pd.DataFrame, resolution: timedelta) -> pd.DataFrame:
+    """Resample data representing instantaneous events.


Not quite. If it matters so much, why not add an assert statement? I'm guessing you wouldn't be sure what the outcomes would be if df wasn't instantaneous?

Flix6x · 2022-11-21T23:06:41Z

Not quite. If it matters so much, why not add an assert statement? I'm guessing you wouldn't be sure what the outcomes would be if df wasn't instantaneous?

You guessed correctly. I really intended this PR to be only about resampling instantaneous sensor data. That means I just haven't investigated yet whether all of this DST handling logic would be useful for resampling non-instantaneous sensor data, too. I'll gladly make that a follow-up ticket.

One extra complexity is that the function currently works on regular DataFrames, too. They don't have a sensor as metadata.

nhoening

Thanks for the clarifications!!

Flix6x added 6 commits October 29, 2022 11:34

Support downsampling instantaneous sensors

55341e8

More robust implementation of resampling instantaneous sensor data, t…

7f13512

…aking into account DST transitions

Add tests

53a78dc

Add NotImplementedError case

4f5de41

Simplify util function

13f1fbc

Expand docstring

66ee209

Flix6x added the Resampling Dealing with downsampling and/or upsampling label Oct 31, 2022

Flix6x self-assigned this Oct 31, 2022

Flix6x added 2 commits October 31, 2022 13:50

isort

a904765

Fix test

f3feabc

Flix6x marked this pull request as ready for review October 31, 2022 13:01

Flix6x requested a review from nhoening October 31, 2022 13:02

nhoening requested changes Nov 6, 2022

View reviewed changes

Flix6x added 7 commits November 9, 2022 13:37

Add two regular (non-DST-transition) resampling cases

167f007

Move index reset dance to within util function

3537784

Remove upsampling example in a test for downsampling (upsampling shou…

707ea63

…ld use a different method, such as pad or interpolate)

Remove redundante call to drop_duplicates

e262dd8

Merge remote-tracking branch 'origin/main' into resample-beliefs-abou…

15f6429

…t-instantaneous-sensors # Conflicts: # timely_beliefs/beliefs/utils.py

typo

d1bbb5b

Clarify inline comment

cc00933

Flix6x mentioned this pull request Nov 15, 2022

Allow for upsampling of instantaneous measurements FlexMeasures/flexmeasures#136

Closed

Flix6x added 8 commits November 17, 2022 12:05

Add inline explanation about handling DST transitions

9330d16

Revert "Remove upsampling example in a test for downsampling (upsampl…

b951c78

…ing should use a different method, such as pad or interpolate)" This reverts commit 707ea63.

Add new property: event_frequency

7bf2217

Rename function to resample instantaneous events, rewrite its logic b…

5395a6d

…y resampling per unique offset, and support multiple resampling methods, with some of them updating the event resolution of the resulting BeliefsDataFrame

Add a lot more test cases for resampling instantaneous sensor data

3bd3d35

Test resolution and frequency for resample instantaneous BeliefsDataF…

1d98721

…rame

Expand test for resampling instantaneous BeliefsDataFrame with 'first…

b3a70b9

…' method.

clarifications

bb703f9

Flix6x added 3 commits November 21, 2022 16:10

Do not cast floats to ints (loss of information)

5621506

Restrict resampling to BeliefsDataFrames with 1 row per event

9dec4da

isort and flake8

ee00e40

Flix6x mentioned this pull request Nov 21, 2022

Expand documentation regarding our distinction between event resolution and event frequency #123

Open

Flix6x requested a review from nhoening November 21, 2022 16:22

Merge branch 'main' into resample-beliefs-about-instantaneous-sensors

23353ea

Flix6x added this to the 1.15.0 milestone Nov 21, 2022

nhoening requested changes Nov 21, 2022

View reviewed changes

Flix6x added 3 commits November 22, 2022 09:11

Rename test

ed18518

Clarify docstring of function parameters

f943d06

Expand resample_events docstring

a4fca7e

Flix6x mentioned this pull request Nov 22, 2022

Streamline how resampling over DST transitions is handled for non-instantaneous sensor data #124

Open

nhoening approved these changes Nov 22, 2022

View reviewed changes

Flix6x merged commit 47a28a0 into main Nov 22, 2022

Flix6x deleted the resample-beliefs-about-instantaneous-sensors branch November 22, 2022 20:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resample beliefs about instantaneous sensors #118

Resample beliefs about instantaneous sensors #118

Flix6x commented Oct 31, 2022

nhoening left a comment •

edited by Flix6x

Loading

nhoening Nov 6, 2022

Flix6x Nov 9, 2022

nhoening Nov 15, 2022 •

edited by Flix6x

Loading

nhoening Nov 15, 2022

Flix6x Nov 21, 2022

nhoening Nov 6, 2022

nhoening Nov 21, 2022

Flix6x Nov 21, 2022

nhoening Nov 21, 2022

Flix6x Nov 22, 2022

Flix6x commented Nov 15, 2022

nhoening left a comment

nhoening Nov 21, 2022

Flix6x Nov 21, 2022

nhoening Nov 21, 2022

nhoening Nov 21, 2022

Flix6x commented Nov 21, 2022

nhoening left a comment



		def downsample_first(df: pd.DataFrame, resolution: timedelta) -> pd.DataFrame:
		"""Resample data representing instantaneous events.

Resample beliefs about instantaneous sensors #118

Resample beliefs about instantaneous sensors #118

Conversation

Flix6x commented Oct 31, 2022

nhoening left a comment • edited by Flix6x Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nhoening Nov 15, 2022 • edited by Flix6x Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Flix6x commented Nov 15, 2022

nhoening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Flix6x commented Nov 21, 2022

nhoening left a comment

Choose a reason for hiding this comment

nhoening left a comment •

edited by Flix6x

Loading

nhoening Nov 15, 2022 •

edited by Flix6x

Loading