[BMG][OOB] hf_Reformer performance drop

[jianyizh]

🐛 Describe the bug

hf_Reformer inference with bs=128 performance regression from 505ms to 623ms due to pytorch/pytorch@e2917a3

Versions

b580