PrincessCory@lemmy.wtf to Ollama - Local LLMs for everyone!@lemmy.worldEnglish · 4 days agoAnyone alive here?message-squaremessage-square8linkfedilinkarrow-up113arrow-down15file-text
arrow-up18arrow-down1message-squareAnyone alive here?PrincessCory@lemmy.wtf to Ollama - Local LLMs for everyone!@lemmy.worldEnglish · 4 days agomessage-square8linkfedilinkfile-text
minus-squarefloquant@lemmy.dbzer0.comlinkfedilinkEnglisharrow-up2·edit-22 days agoOllama uses a context length of 4096 by default, if the conversation goes on for longer the older parts start getting removed. You can change that via environment variable, command line option, or even at request time (num_ctx iirc) On 16GB you should be able to set 64k with no issue, I run either 32k or 64k depending on the model and usecase on 12GB
Ollama uses a context length of 4096 by default, if the conversation goes on for longer the older parts start getting removed.
You can change that via environment variable, command line option, or even at request time (
num_ctxiirc)On 16GB you should be able to set 64k with no issue, I run either 32k or 64k depending on the model and usecase on 12GB