disable transformers pooler (#481)

# Description

The pooler is always run in in the transformers code even when the
outputs aren't used. And in our case, we instantiate the pooler outside
of the transformers code to use the vLLM code. In a small test that I'm
using the total time goes from 1.8s to 1.2s,.

Signed-off-by: Max de Bayser <[email protected]>

Author: maxdebayser

vllm_spyre/v1/worker/spyre_model_runner.py

@@ -1384,11 +1384,9 @@ def load_model(self, prompt_lens: Iterable[int],
             if hasattr(class_model, "bert"):
                 self.model = class_model.bert
                 self._pooler = PoolerAdapter(self.model.pooler)
-                self.model.pooler = None
             elif hasattr(class_model, "roberta"):
                 self.model = class_model.roberta
                 self._pooler = PoolerAdapter(_cls)
-                self.model.pooler = None
             else:
                 raise ValueError(
                     f"Unsupported model {self.model_config.model}: Expected "
@@ -1397,6 +1395,11 @@ def load_model(self, prompt_lens: Iterable[int],
         else:
             raise ValueError(f"Unsupported task {task}")
 
+        # Disable pooler because in transformers it's
+        # always run even tough we don't use the outputs
+        # directly.
+        self.model.pooler = None
+
         model_class_name = type(self.model).__name__
         self.is_roberta = "roberta" in model_class_name.lower()