No problem. Note that there is one other notable technique to expand the amount of relevant context used specifically in SillyTavern, but it comes comes with its own drawbacks: RAG (Retrieval Augmented Generation).
The idea here is that the system uses some kind of heuristic for what information must be relevant, and then allocates part of the context window to including that information; that portion now cannot be used to send history. SillyTavern uses what it calls “lorebooks” for RAG. This means that a given “world” has a set of keywords and then associated strings. If it sees a keyword in your prompt, then it inserts the string into the context window.
So…okay. Say you want it to know that Smaug is a black dragon. You’d have something like:
keyword: “Smaug” Text: “Smaug is a fire-breathing black dragon, with a vast wingspan.”
If SillyTavern sees “Smaug”, it’ll stick that into your context window, up near the top, before submitting a prompt to ollama. So for any prompt from you that mentions “smaug”, it’ll never “forget” that Smaug is a black dragon, because that reminder will always be inserted into the context window.
Now, there are drawbacks. One of these is that SillyTavern also uses a K-V cache; this avoids needing to recompute most of a prompt during prompt processing.
Typically, SillyTavern prefixes the prompt you type with as much conversation history as it can as part of your prompt submitted to the LLM engine. Other than the last bit, the bit you just typed, which is now at the end, nothing changes from prompt to prompt. But…this only works if all of the prompt (up to the text you just added at the end) is unchanged from something that you’ve already submitted. If RAG is in use and a keyword is matched and an insertion is triggered, that won’t be the case, because it’ll be modifying the context from where it was last prompt.
On my system, using RAG at all, and thus making the K-V cache potentially useless, is a huge loss. I work with large context windows. Invalidating an entire 128k context window means that SillyTavern needs to go back and process the entire prompt over again, instead of just the new text that I just added. It’d take ages to respond to each prompt, once you get enough history.
But…if you’re working with a small context window (which reduces prompt processing time) and have high memory bandwidth, you might be willing to live with that recomputation.
That won’t give you more history per se, unless you’re willing to summarize (or use some sort of auto-summarization system). But it will let the LLM have more information about whatever you’re currently prompting about.
And RAG isn’t really subject to any particular size limitations. As long as you’re only inserting so much into a given context window, you could theoretically have gigabytes of data, with the LLM being provided with only a little relevant data on each prompt. I mean, you could have a database that has information on every bit of Tolkien’s world or something like that.
I can’t offer much guidance on effective use of RAG. For my situation, it was preferable to just blow a bunch of memory, use a large context window and the K-V cache and avoid RAG. That way, the LLM “just remembers lots of history”; less work on my end. But…then you can only have it remember history (or something that you’ve inserted into history). You can’t have some system inserting data into the context from a potentially huge database. And you need a bunch of memory to make a large context window doable.
No problem. Note that there is one other notable technique to expand the amount of relevant context used specifically in SillyTavern, but it comes comes with its own drawbacks: RAG (Retrieval Augmented Generation).
The idea here is that the system uses some kind of heuristic for what information must be relevant, and then allocates part of the context window to including that information; that portion now cannot be used to send history. SillyTavern uses what it calls “lorebooks” for RAG. This means that a given “world” has a set of keywords and then associated strings. If it sees a keyword in your prompt, then it inserts the string into the context window.
So…okay. Say you want it to know that Smaug is a black dragon. You’d have something like:
keyword: “Smaug” Text: “Smaug is a fire-breathing black dragon, with a vast wingspan.”
If SillyTavern sees “Smaug”, it’ll stick that into your context window, up near the top, before submitting a prompt to ollama. So for any prompt from you that mentions “smaug”, it’ll never “forget” that Smaug is a black dragon, because that reminder will always be inserted into the context window.
Now, there are drawbacks. One of these is that SillyTavern also uses a K-V cache; this avoids needing to recompute most of a prompt during prompt processing.
Typically, SillyTavern prefixes the prompt you type with as much conversation history as it can as part of your prompt submitted to the LLM engine. Other than the last bit, the bit you just typed, which is now at the end, nothing changes from prompt to prompt. But…this only works if all of the prompt (up to the text you just added at the end) is unchanged from something that you’ve already submitted. If RAG is in use and a keyword is matched and an insertion is triggered, that won’t be the case, because it’ll be modifying the context from where it was last prompt.
On my system, using RAG at all, and thus making the K-V cache potentially useless, is a huge loss. I work with large context windows. Invalidating an entire 128k context window means that SillyTavern needs to go back and process the entire prompt over again, instead of just the new text that I just added. It’d take ages to respond to each prompt, once you get enough history.
But…if you’re working with a small context window (which reduces prompt processing time) and have high memory bandwidth, you might be willing to live with that recomputation.
That won’t give you more history per se, unless you’re willing to summarize (or use some sort of auto-summarization system). But it will let the LLM have more information about whatever you’re currently prompting about.
And RAG isn’t really subject to any particular size limitations. As long as you’re only inserting so much into a given context window, you could theoretically have gigabytes of data, with the LLM being provided with only a little relevant data on each prompt. I mean, you could have a database that has information on every bit of Tolkien’s world or something like that.
I can’t offer much guidance on effective use of RAG. For my situation, it was preferable to just blow a bunch of memory, use a large context window and the K-V cache and avoid RAG. That way, the LLM “just remembers lots of history”; less work on my end. But…then you can only have it remember history (or something that you’ve inserted into history). You can’t have some system inserting data into the context from a potentially huge database. And you need a bunch of memory to make a large context window doable.