bug in bsrnn_feats.py

[haha010508]

in bsrnn_feats.py at tfmap_emb mode, the embedding is n x192, so the att_weights is 192 x192, but the enroll_mag is n x T x 257, so, i think this can not matmul? am i right?
tf_map = torch.matmul(att_weights, enroll_mag)

[KyleZhang1118]

In tfmap_emb, the frame-level speaker embeddings (generated by the speaker encoder without the pooling layer) are used, not the speaker embedding. The dimension of the former is n x 512 x T_e, then the att_weights is n x T_m x T_e, the enroll_mag is n x T_e x 257.

[haha010508]

thanks for your reply, so theECAPA_TDNN output before pooling is 1536, you change here to change the dim to 512? am i right? if so can you use pretrain model?
`ECAPA_TDNN
class ECAPA_TDNN(nn.Module):

def __init__(self,
             channels=512,
             feat_dim=80,
             embed_dim=192,
             pooling_func='ASTP',
             global_context_att=False,
             emb_bn=False):
    super().__init__()

    self.layer1 = Conv1dReluBn(feat_dim,
                               channels,
                               kernel_size=5,
                               padding=2)
    self.layer2 = SE_Res2Block(channels,
                               kernel_size=3,
                               stride=1,
                               padding=2,
                               dilation=2,
                               scale=8)
    self.layer3 = SE_Res2Block(channels,
                               kernel_size=3,
                               stride=1,
                               padding=3,
                               dilation=3,
                               scale=8)
    self.layer4 = SE_Res2Block(channels,
                               kernel_size=3,
                               stride=1,
                               padding=4,
                               dilation=4,
                               scale=8)

    cat_channels = channels * 3
    **out_channels = 512**
    self.conv = nn.Conv1d(cat_channels, out_channels, kernel_size=1)
    self.pool = getattr(pooling_layers, pooling_func)(
        in_dim=out_channels, global_context_att=global_context_att)
    self.pool_out_dim = self.pool.get_out_dim()
    self.bn = nn.BatchNorm1d(self.pool_out_dim)
    self.linear = nn.Linear(self.pool_out_dim, embed_dim)
    self.emb_bn = emb_bn
    if emb_bn:  # better in SSL for SV
        self.bn2 = nn.BatchNorm1d(embed_dim)
    else:
        self.bn2 = nn.Identity()`

[KyleZhang1118]

In Ecapa-TDNN, three outputs (each with 512 dimensions) from the SE_Res2Block are combined to form the final representation with 1536. In my TSE test, the output from the last SE_Res2Block performs best rather than the concatenation of the representation with 1536, so the former is used here. And the pre-trained model can still be used.

[haha010508]

I got it, thanks. And another qustion: in your test, do you found the speaker embedding (ECAPA-TDNN=192 dim) will help improve the TSE performance, if so, i do not found you use the feature in your code bsrnn_feats.py, why? And how to use it to fuse to mixed signal spectrogram. And if i use all feature, stft spectrogram, ECAPA-TDNN embedding(last SE_Res2Block output 512 dim), and 192 dim speaker embedding, can i got the best performance? thanks! looking forward to your reply.

[KyleZhang1118]

In my test, the speaker embedding (192) doesn't further improve the performance with the lower features. More details can be found in the paper "Multi-Level Speaker Representation for Target Speaker Extraction".

[haha010508]

OK, thanks for your reply, so you retain the ECAPA-TDNN use win=512 ponits/(16kHz) stride=128ponits FBank for keep same with BSRNN STFT? right? Could you share the pretrain model? thanks a lot!

[KyleZhang1118]

The pre-trained Ecapa-TDNN model provided by Wespeaker can be directly applied here. Though the FFT coefficient is not consistent with the BSRNN, we don't observe the difference in terms of the result.

[haha010508]

OK， thanks for your reply!!