Roberta hidden_states[-1] == Bert pooler_output?

[Yuji-github]

I want to feed the last layer hidden state which is generated by RoberTa.
However, when I try this code:

out = pretrained_roberta(dummy_input["input_ids"], dummy_input["attention_mask"], output_hidden_states=True)
out = out.hidden_states[-1]
out = nn.Dense(features=3)(out)

Is that equivalent of pooler_output in Bert?

pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task.

[marcvanzee]

Hi @Yuji-github, are you using HuggingFace? In that case it might be easier to ask it in their repo, since we do not own these pretrained models.

Also you seem to mention some torch in your question, is that intended?

[Yuji-github]

Thank you @marcvanzee, I will ask it on Hugging Face =)