Caching

With a model on, core keeps each of the model's answers in a cache. The next page view that needs the same answer gets it from there and doesn't wait for the model. Cohorts decide which page views share an answer. With no provider, there's nothing to cache.

What the cache holds

An entry holds the model's answer and the time it was written. It holds no page view and no prompt. SpecCache has the exact shape.

Entries can still hold shopper data. Deleting a shopper's data says when, and what to do about it.

Choosing where answers are kept

• Leave cache out, and each server process keeps answers in its own memory. Call createMemorySpecCache to change how long they last and how many it holds.
• To share answers between servers, pass a store of your own, such as Redis, through SpecCache.
• To keep nothing, pass createNullSpecCache().

Your own store keeps entries for as long as you tell it to. Core sets no expiry on it. What it holds, and for how long, is yours to tell your shoppers about.

When the model wrote something wrong

Core serves a cached answer until it expires. When one is wrong, you can:

• drop that one entry, if your store has delete
• shorten the expiry, so a bad entry ends sooner
• set provider: null, so nothing new is generated and every page view gets the block core builds without a model

To drop one entry, you need its key. onEvent reports the key on every event, so log it when the model writes a new answer.

import { createAnthropicProvider } from '@rudra-js/anthropic';
import { createComponentGenerator, createMemorySpecCache } from '@rudra-js/core';

const cache = createMemorySpecCache();

export const generator = createComponentGenerator({
  provider: createAnthropicProvider({ apiKey: process.env.ANTHROPIC_API_KEY! }),
  cache,
  onEvent(event) {
    if (event.source !== 'llm' || !event.calledModel) return;
    console.log('new answer', event.key, event.violations);
  },
});

export async function dropAnswer(key: string) {
  await cache.delete?.(key);
}

• The cache sits in its own variable, so you can call delete on it later.
• The log line holds the key and what core's checks took out of the answer.
• Call dropAnswer with the key, and the next page view that needs it asks the model again.
• With several servers, keep the cache in a shared store, so one delete clears the entry for all of them.

In per-shopper mode, shorten the expiry instead. Each shopper has entries of their own, and nothing ties a shopper to their keys.