[Q&A] Handling long-form audio and speaker labels in Sortformer training manifest

[LilDevsy0117]

I am preparing training data for the Sortformer diarization model and have a few questions regarding the handling of long-form audio recordings:

Virtual Slicing via Manifest: If I have a long audio file (e.g., 5 minutes), is it sufficient to define segments in the manifest using only the offset and duration fields (e.g., 0-90s, 90-180s, etc.) without physically cutting the audio into separate files? Will the model train correctly using these offset-based entries?

Partial Speaker Presence in Segments: Suppose a 5-minute recording has 4 unique speakers in total. However, in a specific 90-second segment (e.g., the first 90 seconds), only 2 of those speakers are actively talking.

Is it okay to include segments where only a subset of the session's total speakers is present?

In the manifest entry for that 90s segment, should the num_speakers field reflect the number of speakers present in that specific segment (2), or the total speakers in the original long recording (4)?

[pzelasko]

CC @tango4j

[tango4j]

1. Virtual Slicing via Manifest: Correct, you do NOT need to physically cut the audio. The offset and duration fields in the manifest handle this — the NeMo dataloader will read only the specified segment from the full audio file. The RTTM file is also read with the same offset/duration window. NO NEED to edit RTTM files. The dataloader will capture the offset + duration from RTTM

2. Partial Speaker Presence: Yes, it is perfectly fine for a segment to contain only a subset of the session's speakers. The num_speakers field should reflect the number of speakers in that specific segment, not the total in the full recording. The model needs to know how many speakers to expect in the segment it is actually processing.

You don't even have to prepare the "num_speakers" in the json because diarization data loader automatically figures out.

Check out this function in audio_to_diar_label.py [Link]

To make a better understanding, let me give you a real-life example

Here is one example manifest line (JSON-line format):
total in the full recording is 4 but for this segment, it is , "num_speakers": 3
EN2001a.Mix-Headset.wav is 45 min long audio file.
EN2001a.Mix-Headset.rttm is also annotating 45 min long audio file.

{"uniq_id": "EN2001a.Mix-Headset#116#932.92#90.0", "offset": 932.92, "duration": 90, "num_speakers": 3, "audio_filepath": "/path/to/wav/EN2001a.Mix-Headset.wav", "rttm_filepath": "/path/to/rttm/EN2001a.Mix-Headset.rttm"}

What you need to prepare: You need to prepare these json lines with "speech existing" segment in your training dataset with RTTMs. It could be 30min, 90min, 12.5min audio with RTTM - all you need to do is (1) get the segmentation (2) make uniq_ids for the segment so that dataloader does not get confused. (something like "EN2001a.Mix-Headset#116#932.92#90.0")

Hope this helps.
Let me know this clears up your inquiry.

[LilDevsy0117]

Hi @tango4j, thank you so much for the detailed explanation! It really helped me clear up my doubts about the manifest structure.

I have one follow-up question regarding the speaker density in training segments. If I am training a Sortformer model from scratch for 4 speakers, but all of my 90-second training segments contain only 3 or fewer active speakers (even though the original long recordings have 4), will the model still be able to learn to diarize 4 speakers effectively?

I am concerned that if the model never sees a single 90-second window where all 4 speakers are present, it might not learn to utilize the 4th output slot or handle 4-way speaker transitions properly. Do I need to ensure that a certain portion of my training segments actually contains the maximum number of speakers (4)?