-
Couldn't load subscription status.
- Fork 2
75 hybrid time series buffer #103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Conversation
|
Note to self:
|
…. In HybridAxisArrayBuffer, additionally add `searchsorted` which operates on the concatenated axis data.
… write times -- resample can do that on its own.
…p doing this on its own.
|
I was hoping to defer adding different overflow strategies until later, but then I realized that one of the primary purposes of the ResampleProcessor is to return even-interval data for smoother processing downstream, so allowing the buffer to drop samples was antithetical to that. So I implemented The recent smattering of commits improves upon the buffer classes then refactors resample.py to use them. This did not impact performance when using While I didn't get the performance boost I was hoping for, I did achieve the other primary objective of this effort. By centralizing the buffer-handling code, I was able to simplify resample.py (less ~40 lines of code, though I added that much back in docstrings) and I found and fixed a bug that I might have missed otherwise. Oh and another result is that the resample module is now Array API compatible (on the data... still expects np array for the CoordinateAxis timestamps but only in a couple lines). I'll continue this effort on sampler.py after I step away for a bit. |
…xis now behaves like np.searchsorted
…rther deferring flushes until absolutely needed. This merited exposes the update_strategy option to the resample settings.
|
Sampler now refactored to use the new buffers. This resulted in about 30 fewer lines of code, Array API support, and support for messages with their primary axis being a CoordinateAxis (i.e., irregularly sampled data). I'm not going to include the Window node in this PR. That one is a little too complex and will delay this further. Ready for review! |
…data; copy directly from deque to buffer.
|
Sorry I've slept on this for so long, but it looks like this PR has stabilized and is likely good to go. I haven't reviewed the code in detail, but I'm alright to merge this as a new feature -- my (minor) comments below can be addressed in future PRs if at all.
I'll mark this as ok-to-go and we can fix issues as they pop up in |
This adds
HybridBuffer-- A stateful, FIFO buffer that combines a deque for fast appends with a contiguous circular buffer for efficient, advancing reads. It also addsHybridAxisArrayBufferwhich does the same for AxisArrays.I will leave this in draft for now because I'm going to further develop
HybridAxisArrayBufferso it is suitable in: