Fix: Discard KV cache for last token before reusing prompt cache for prompt + suffix#79

Open

sannat17 wants to merge 2 commits intohuggingface:mainfrom sannat17:fix/prompt_reuse_recipe

+31-22

Commits on Oct 21, 2024

Fix: Discard KV cache for last token before reusing prompt cache with suffix added to prompt
sannat17
committed
Show the difference between naively reusing prompt cache and using the fixed version with the kv cache for last token being discarded
sannat17
committed