[Bug] The number of declared samples in the recording diverged from the one obtained when loading audio (offset is 0.0)

[johnchienbronci]

@pzelasko

sample rate: 16k
The total length of the sound file: 48.92 s

Read a segment of the audio file:
offset: 0.0
duration: 4.948
and the following error will occur:

[Suppressed AudioLoadingError] Error message: The number of declared samples in the recording diverged from the one obtained when loading audio (offset=0.0, duration=4.948). This could be internal Lhotse's error or a faulty transform implementation. Please report this issue in Lhotse and show the following: diff=-703509, audio.shape=(1, 782677), recording=Recording(id='/shorts-20230702.wav_ln25.0', sources=[AudioSource(type='file', channels=[0], source=shorts-20230702.wav')], sampling_rate=16000, num_samples=79168, duration=4.948, channel_ids=[0], transforms=[LoudnessNormalization(target=-25.0)])

[extra info] When calling: Recording.load_audio(args=(Recording(id='shorts-20230702.wav_ln-25.0', so
urces=[AudioSource(type='file', channels=[0], source=shorts-20230702.wav')], sampling_rate=16000, n
um_samples=79168, duration=4.948, channel_ids=[0], transforms=[LoudnessNormalization(target=-25.0)]),) kwargs={'channels': 0, 'offset': 0.0, 'duration': 4.948})        
                                                                                                                                
[extra info] When calling: MonoCut.load_audio(args=(MonoCut(id='shorts-20230702.wav-000000-000495_ln-25.0', start=0.0, duration=4.948, channel=0, supervisions=[SupervisionSegment(id='shorts-20230702.wav-000000-000495_ln-25.0', recording_id='shorts-20230702.wav_ln-25.0', start=0, duration=4.948, channel=0, text='xxxxx', language=None, speaker=None, gender=None, custom={'tokens': array([1272, 1032, 1920, 1032,  198,  202,  996, 2002, 2011, 2009, 1060,                                                                                         
       1941, 1908, 1160, 1945, 1675, 1602, 1137,  996,   12,  844, 1003,                                                                            
         18, 1123, 1955, 1545, 1434, 1064,  996, 2003, 2001, 2003, 2004,                                                                            
       1173, 1521, 1412])}, alignment=None)], features=None, recording=Recording(id='shorts-20230702
.wav_ln-25.0', sources=[AudioSource(type='file', channels=[0], source='shorts-20230702.wav')], sampling_rate=16000, num_samples=79168, duration=4.948, channel_ids=[0], transforms=[LoudnessNormalization(target=-25.0)]), custom={'text': 'xxxxx', 'dataloading_info': {'rank': 0, 'world_size': 1, 'worker_id': None}}),) kwargs={}) 

If the offset is set to 0.1, there will be no error messages.

export LHOTSE_AUDIO_DURATION_MISMATCH_TOLERANCE=0.2, can't not resolve it.

How can I solve this problem?

[pzelasko]

Your manifest seems to be wrong, you say:

The total length of the sound file: 48.92 s

but the Recording manifest inside MonoCut says:

recording=Recording(id='shorts-20230702.wav_ln-25.0', ..., sampling_rate=16000, num_samples=79168, duration=4.948, ...)

The Recording duration should be 48.92 s

[johnchienbronci]

I think MonoCut is correct because I have specified the offset and duration.
kwargs={'channels': 0, 'offset': 0.0, 'duration': 4.948}

audio.shape=(1, 782677) should be changed to (1, 79168)

[pzelasko]

MonoCut seems OK but the Recording inside has incorrect metadata. You can check with lhotse.qa.validate(cut.recording, load_data=True)

[johnchienbronci]

check resut:

I don't understand why the audio offset is always 0.0, and duration is always None, regardless of the specified offset value.

[rank0]: lhotse.audio.utils.AudioLoadingError: The number of declared samples in the recording diverged from the one obtained when loading audio (offset=0.0, duration=None). This could be internal Lhotse's error or a faulty transform implementation. Please report this issue in Lhotse and show the following: diff=-703509, audio.shape=(1, 782677), recording=Recording(id='shorts-20230702.wav', sources=[AudioSource(type='file', channels=[0], source='shorts-20230702.wav')], sampling_rate=16000, num_samples=79168, duration=4.948, channel_ids=[0], transforms=None)

[rank0]: [extra info] When calling: Recording.load_audio(args=(Recording(id='shorts-20230702.wav', sources=[AudioSource(type='file', channels=[0], source='shorts-20230702.wav')], sampling_rate=16000, num_samples=79168, duration=4.948, channel_ids=[0], transforms=None),) kwargs={})

[pzelasko]

Set the actual duration and num samples of the audio file in the Recording object. Cut is a slice, so the recording must always have full audio metadata.

[johnchienbronci]

Ok.
but why should full samples be compared with cut samples, cause the error?

diff=-703509, audio.shape=(1, 782677), ...

This problem only occurs when the offset is < 0.00097 and the length of the full audio file is different from the length of the clipped audio file.

it's normal when offset >= 0.00097

[pzelasko]

There’s quick-path code optimization when Cut’s start is 0 and duration == Recording.duration that loads the full audio without slicing. Since your metadata is wrong it errors out. When start is nonzero it applies Cut start and duration and manages to load the slice as expected.

[johnchienbronci]

Ok,
Is this a bug in Lhotse or is it an error due to a problem with the function used?
if function used issue, how can i fix it?

[pzelasko]

It’s not a bug in Lhotse. I don’t know how you created the Recording object but the issue is likely there somewhere.

[johnchienbronci]

I use nemo project to load lhotse dataset (set use_lhotse=True)
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/data/audio_to_text_lhotse.py

[pzelasko]

Can you show the Nemo manifest line for that recording?

[johnchienbronci]

{"audio_filepath": "/a/b/c.wav", "text": "efg", "offset": 0, "duration": 7.755, "sampling_rate": 16000}
{"audio_filepath": "/a/b/c.wav", "text": "abcd", "offset": 10, "duration": 5.0, "sampling_rate": 16000}
{"audio_filepath": "/a/b/d.wav", "text": "abcd", "offset": 0, "duration": 2.0, "sampling_rate": 16000}
....