Question on extracting data for a specific time range

[Merlencer]

Affiliation
ENN

Question
I have a set of raw data with a sampling rate of 500 kHz, starting at -0.25 seconds and ending at 6.999998 seconds, which is a segment of data inserted using makesegment. Now, I want to select only the data after 0 seconds in another node while keeping the sampling rate unchanged. How can I achieve this using a formula in the node? After using [0.::], it becomes [0.:*], and after reading, the frequency is defaulted to 1 Hz.

Raw Data

another node

[mwinkel-dev]

Hi @Merlencer -- Segmented signals do support a "resampling" feature. I will investigate to find out why your 500 kHz signal is being resampled at 1 Hz.

[mwinkel-dev]

Hi @Merlencer -- I have reproduced the issue you encountered. I have 7 segments, each spanning one second of fake 10 Hz data. Using an expression similar to what you described, it is indeed doing 1 Hz sampling of the data.

I am now working on the solution.

[mwinkel-dev]

Hi @Merlencer -- In this example, "SEG" is a node that contains segmented data (the 7 segments mentioned above, each is one second long). And "SUBSET" is the derived portion based on "SEG".

In the "SUBSET" node, I entered the following TDI code to obtain just the data from 2.5 to 4.5 seconds.

_rate = 40;  /* Samples per second */
_period = 1.0 / _rate;
SEG[2.5 : 4.5 : _period];

That works OK, however because of round-off error it doesn't always sample at exactly the same points as the raw data in "SEG". Give this a try and see if it works OK for your use case.

If it doesn't, then we'll have to use a more elaborate solution for extracting a subset that doesn't do any resampling.

[mwinkel-dev]

Hi @Merlencer,

Using the "Nyquist rate" (or higher) yields a better result for the "SUBSET" node. In other words, _rate = 40 * 2;, works well. (This example uses 4 samples per cycle for a 10 Hz signal, thus a sample rate of 40 samples/sec and a Nyquist rate of 80 samples/sec.) Of course, using the Nyquist rate also means that the derived "SUBSET" has more data points than the corresponding portion of the original "SEG" signal.

One can also apply the above approach to everything following a specific point in time, such as SEG[2.5 : * : _period].

If you do not want the segmented signal to be resampled, then it appears that the only option is to do some post processing that reads the segmented signal, extracts the desired subset, and writes it as unsegmented data to a new node. However, I'm still investigating to see if there are other workarounds.

[mwinkel-dev]

Hi @Merlencer,

You might also want to experiment with the various resampling methods. To do so, set the MDSPLUS_DEFAULT_RESAMPLE_MODE environment variable to one of the following:

• Average (this is the default)
• MinMax
• Interpolation
• Closest
• Previous

For example, export MDSPLUS_DEFAULT_RESAMPLE_MODE=Closest.

[Merlencer]

Hi @Merlencer -- 在这个例子中，“SEG”是一个包含分段数据的节点（上述7个段，每个段长一秒）。而“SUBSET”是基于“SEG”的派生部分。

在“SUBSET”节点中，我输入了以下TDI代码，只获取2.5秒到4.5秒的数据。

_rate = 40;  /* Samples per second */
_period = 1.0 / _rate;
SEG[2.5 : 4.5 : _period];

这个方法还行，但由于舍入误差，它并不总是在“SEG”原始数据的完全相同点采样。试试看，看看是否适合你的使用场景。

如果不行，那我们就得用更复杂的方法来提取一个不做重采样的子集。

I understand this method, but my sampling rate may not be fixed. It seems that there is currently no way to solve this problem, right? Is there any chance of considering optimizing this issue in the future?

[mwinkel-dev]

Hi @Merlencer -- Yes, segmented records have some limitations. We have been discussing ways to improve the recording of such data, but it will likely take considerable time to implement those ideas. Thus, this issue will likely be upgraded to a "feature request" so that we consider your use case when designing those improvements.

If the sampling rate only varies from one shot to another shot, then the approach described above can be made to work. All that needs to be done is to have _rate = <node_in_tree_that_has_the_digitizer_sample_rate>.

However, I gather that your requirements are more complex, namely that within a single shot, the segmented signal can have different sample rates over the duration of the shot. Perhaps it starts at 300 kHZ, then jumps to 500 kHz for many seconds, and then reverts to 300 kHz.

It is my understanding that these are your requirements:

1. Ability to record 500 kHz signals for long periods of time (how long? how much data per signal?).
2. The recording digitizer does not provide a timestamp for each sample (i.e., the time for each sample is instead inferred from the start time and the sampling rate).
3. Similar to 1 above, however the signal might use varying sample rates (e.g., a different rate during the middle of a shot than at the beginning).
4. Regarding 3, a signal that consists of N segments, might use sample rate X for one segment, a different sample rate Y for the next segment, and so forth. However, within a single segment the sample rate will not vary.
5. Another node in the tree should be able to have an expression that specifies a subset of the segmented signal, consisting of a start time and optional end time, but with no resampling of the data (i.e., it will return the original data for the time frame of interest).

If I have misunderstood any of the above requirements, please post a correction to this issue. We want to make sure we fully understand your requirements.

In the meantime, the guaranteed workaround is to use a program to do the following:

• read the data (segment by segment),
• do the subsetting, and
• write the subset to a new node (but not as a segmented signal).

MDSplus presently does have the ability to use custom "resampling" functions. Thus it might be possible to achieve the same result by extending MDSplus with a custom "resampling" function that does not actually resample the segmented signal, but instead returns the original data for the subset of interest. Note that the custom function would only be attached to the appropriate signal nodes. Meaning that your 500 kHz segmented signals would use the custom function, but all other segmented signals would use the default resampling function. We will have to study the source code to see if this is a viable solution or not.

[Merlencer]

Hi @Merlencer -- Yes, segmented records have some limitations. We have been discussing ways to improve the recording of such data, but it will likely take considerable time to implement those ideas. Thus, this issue will likely be upgraded to a "feature request" so that we consider your use case when designing those improvements.

If the sampling rate only varies from one shot to another shot, then the approach described above can be made to work. All that needs to be done is to have _rate = <node_in_tree_that_has_the_digitizer_sample_rate>.

However, I gather that your requirements are more complex, namely that within a single shot, the segmented signal can have different sample rates over the duration of the shot. Perhaps it starts at 300 kHZ, then jumps to 500 kHz for many seconds, and then reverts to 300 kHz.

It is my understanding that these are your requirements:

1. Ability to record 500 kHz signals for long periods of time (how long? how much data per signal?).
2. The recording digitizer does not provide a timestamp for each sample (i.e., the time for each sample is instead inferred from the start time and the sampling rate).
3. Similar to 1 above, however the signal might use varying sample rates (e.g., a different rate during the middle of a shot than at the beginning).
4. Regarding 3, a signal that consists of N segments, might use sample rate X for one segment, a different sample rate Y for the next segment, and so forth. However, within a single segment the sample rate will not vary.
5. Another node in the tree should be able to have an expression that specifies a subset of the segmented signal, consisting of a start time and optional end time, but with no resampling of the data (i.e., it will return the original data for the time frame of interest).

If I have misunderstood any of the above requirements, please post a correction to this issue. We want to make sure we fully understand your requirements.

In the meantime, the guaranteed workaround is to use a program to do the following:

• read the data (segment by segment),
• do the subsetting, and
• write the subset to a new node (but not as a segmented signal).

MDSplus presently does have the ability to use custom "resampling" functions. Thus it might be possible to achieve the same result by extending MDSplus with a custom "resampling" function that does not actually resample the segmented signal, but instead returns the original data for the subset of interest. Note that the custom function would only be attached to the appropriate signal nodes. Meaning that your 500 kHz segmented signals would use the custom function, but all other segmented signals would use the default resampling function. We will have to study the source code to see if this is a viable solution or not.

Yes, your understanding is correct, and we hope this feature can be implemented. Additionally, is [starttime::] considered a bug that needs to be fixed? Because the sampling rate of the data actually obtained through this formula is 1 Hz, instead of the actual sampling rate.

[mwinkel-dev]

Hi @Merlencer -- Thanks for confirming the requirements.

As for whether [starttime::] is a bug or not, all I can say at the moment is that it definitely doesn't work as you expected nor as I expected. (The 1 Hz sample rate was quite a surprise to me.). I will be discussing this with the rest of the software team. When a decision is reached, I will post it here.

[mwinkel-dev]

Hi @Merlencer -- The source code has special logic to handle the condition when the delta is zero (i.e., when _period is omitted in the above example). My conjecture is that the special logic is supposed to return the subset of data as per your requirements. Instead, it is definitely sampling at 1 Hz. In summary, this issue likely is a bug (and not a new feature request).

The next step is to use the debugger to determine why the special logic is sampling at 1 Hz.

[Merlencer]

Hi @Merlencer -- The source code has special logic to handle the condition when the delta is zero (i.e., when _period is omitted in the above example). My conjecture is that the special logic is supposed to return the subset of data as per your requirements. Instead, it is definitely sampling at 1 Hz. In summary, this issue likely is a bug (and not a new feature request).

The next step is to use the debugger to determine why the special logic is sampling at 1 Hz.

Additionally, I would like to consult about a question: does the tdi function currently support interpolation or fitting methods?

[mwinkel-dev]

Hi @Merlencer -- I will also investigate your new question when conducting experiments with the debugger.

There is a XTreeDefaultResample() function in one of the core libraries; the function is used with segmented signals. It has various modes, including linear interpolation. Presumably, all APIs provided by MDSplus (TDI *.fun, TDI *.py, Python, Java, C, etcetera) provide access to that function.

[mwinkel-dev]

Hi @Merlencer -- The debugger experiments have identified the general area of the code where the 1 Hz sampling occurs. More investigation will be done to locate the exact cause.