Counting with len is slow when slicing.

[itrobinson]

I am counting the number of values in a channel in a large file.

This statement is very fast

len( tdms['group_name']['channel_name'] )

but these equivalent statements are very slow

len( tdms['group_name']['channel_name'][:] )
len( tdms['group_name']['channel_name'][0:-1] )
len( tdms['group_name']['channel_name'][0:-1:1] )
len( tdms['group_name']['channel_name'][slice(None)] )

Similarly, these statements will all be very slow

len( tdms['group_name']['channel_name'][0:-100] )
len( tdms['group_name']['channel_name'][0:-1:2] )

(In the above tdms is a nptdms.TdmsFile object.)

Is there any feature built into npTDMS that can quickly count slices on a channel?

Why does this matter? I often test a script by running it on a small regularly-spaced subset of a large data file. Slicing is the easiest way to do this. Then I remove the slice and leave the script running, sometimes for hours, on the full data file.

[adamreeve]

Hi @itrobinson

This gets a TdmsChannel object and doesn't require reading any data, the required metadata was already loaded in TdmsFile.load:

tdms['group_name']['channel_name']

Getting the length is fast because it's stored in the metadata:

npTDMS/nptdms/tdms.py

Lines 521 to 524 in 6cb5a80

	def __len__(self):
	""" Returns the number of values in this channel
	"""
	return self._length

Any time you index into the channel with a slice, however, data is read from the channel and returned as a numpy array. This should be faster for smaller slices that read less data, but will always be a lot slower than getting the length of the channel.

You should be able to easily compute the length of a slice if you know the channel length. You could leverage Python's built-in range type which can quickly compute its own length and only stores the start, stop and step.

Eg:

channel_length = len(tdms['group_name']['channel_name'])
slice_length = len(range(channel_length)[0:-1:2])