How do you deal with the 'too many tokens' problem? Like, For 13b, max tokens I'... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		MPSimmons on May 15, 2023 \| parent \| context \| favorite \| on: Run Llama 13B with a 6GB graphics card How do you deal with the 'too many tokens' problem? Like, For 13b, max tokens I've gotten it to is 508 - is there a way to increase the number it will accept?

MPSimmons on May 15, 2023 [–]

Oh, I figured this out. It runs with 512 as the default context, but you can increase it with -c

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact