Not even an engineer and the thought of rounding pi to 4 triggered me so hard, good job

Ollama uses a context length of 4096 by default, if the conversation goes on for longer the older parts start getting removed.

You can change that via environment variable, command line option, or even at request time (num_ctx iirc)

On 16GB you should be able to set 64k with no issue, I run either 32k or 64k depending on the model and usecase on 12GB

I noticed that whatever llm i use, they all have prity limited “memory” of how long they remember our conversacion. I mean if chat is wery short, they tend to remember what we were talking wery well. But if chat gets a bit longer they quickly start removing upper parts of the chat and remember only some part abowe our curent stage.

Thats why they easly get lost, and dont know anymore what we were talking about and what they need to answer now…

Yup. The context window only goes so far.

If you have the memory to spend, you can get it pretty far out, depending on the model (which is also a factor). I’m currently mostly using a Llama 3 derived-model (AnubisLemonade) with a 128k context window on a 128GB Framework Desktop. IIRC, there have been some subsequent models that can go substantially further, though I understand that the response quality degrades on them. The problem is that systems with that much memory available have gotten really expensive over the past nine months or so and are probably going to stay expensive at least through the end of 2027.

I use llama.cpp these days, not ollama, but I’d guess that it’s probably possible to configure ollama for a larger context window as well, up to the maximum that your model can handle, though I can’t give specifics as to configuration.

searches

https://docs.ollama.com/faq

How can I specify the context window size?

By default, Ollama uses a context window size of 4096 tokens. This can be overridden with the OLLAMA_CONTEXT_LENGTH environment variable. For example, to set the default context window to 8K, use:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

If you don’t have the VRAM for a larger context window with whatever model you’re using:

• I don’t do it, but it’s possible to, with some engines, place some model layers in main memory and have the CPU do them, and place some model layers in VRAM.

checks

It looks like ollama can do this; apparently it will use as much video memory as it can, and fall back to system memory.

https://docs.ollama.com/faq

The Processor column will show which memory the model was loaded into:

• 100% GPU means the model was loaded entirely into the GPU
• 100% CPU means the model was loaded entirely in system memory
• 48%/52% CPU/GPU means the model was loaded partially onto both the GPU and into system memory

If you’re using something like SillyTavern, from my reading, users on /r/SillyTavernAI have generally tried to occasionally summarize the past conversation and insert that into the context. There are some plugins that try to do this automatically; my impression from past reading is that they weren’t very good.