Playing with Llama2

[mikix]

I wanted to talk more about what we're doing with LLMs, and how that would interact with Cumulus.

How it would work

Much like the current Covid ETL task sends a note to cTAKES and gets a response, we'd likely make a new task that sends notes plus a study-specific prompt to an LLM. And maybe does some further post-processing on it (like a classifier to get the verbose LLM response down to a yes/no answer - again, very similar to how our Covid task post-process cTAKES responses with a negation transformer).

You can play with Llama2 today!

We have a toy/demo task in the ETL today, which just summarizes clinical notes. Here's how to run it:

1. Accept the llama2 license
2. Make a Hugging Face account and create an API read token
3. Run the following like you would the Covid task (pointing the ETL at your DocRefs):

export HUGGING_FACE_HUB_TOKEN=xxx
docker compose up llama2 --wait  # this will take several minutes
docker compose run cumulus-etl --task hftest__summary ...

You'll want a machine/GPU at least as powerful as a g5.xlarge AWS EC2 instance. This is currently only using the 13b llama2 model to make it easier to test. But if we used this in any study, we'd probably need to use the beefier 70b model version, which requires much more GPU power.

What's next

We're continuing to play with this - we may introduce a new task for a study that uses a specific prompt. We're also exploring using more cNLP transformer models. Comparing the two approaches will be interesting. At some point, also comparing with chat gpt would be interesting.

Let us know if you played with this and have thoughts!