Wanted to ask a question about ollama, but i think i hear only echo here 😅

  • floquant@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    2 days ago

    Ollama uses a context length of 4096 by default, if the conversation goes on for longer the older parts start getting removed.

    You can change that via environment variable, command line option, or even at request time (num_ctx iirc)

    On 16GB you should be able to set 64k with no issue, I run either 32k or 64k depending on the model and usecase on 12GB