SpecCache

import { SpecCache } from '@rudra-js/core';

SpecCache lets you keep the model's answers in a store of your own, such as Redis, so all your servers share them. Write an object with these methods and pass it to createComponentGenerator as cache.

export interface SpecCache { get(key: string): Promise<CachedSpec | undefined>; set(key: string, cached: CachedSpec): Promise<void>; delete?(key: string): Promise<void>; }

Reference

SpecCache

Properties

• get: Called with the request's key before core asks the model. Resolve with the entry set stored under it, or undefined when there's none.
• set: Called after the model answers, with a key and the entry to store under it. The entry holds:
  • spec: The model's answer, before core's checks. Core checks it each time it's served from the cache.
  • generatedAt: When the model answered, as a Date.now() timestamp.
• deleteoptional: Removes the entry under key. Core never calls it. Call it yourself to drop one bad answer, by the key that onEvent reported for it. The next request asks the model again.

Returns

Each method returns a promise. get resolves to the entry or undefined. set and delete resolve once the store is done.

Caveats

• Entries can hold shopper data, so give them an expiry. In 'per-shopper' mode the model writes each answer from one shopper's signals, and its text can repeat their searches. Nothing maps a shopper to their keys, so you can't delete one shopper's entries on request.
• Core checks what get returns. An entry without the shape above counts as a miss.
• A get that throws, or takes longer than cacheTimeoutMs, counts as a miss too. The page still renders.
• Core doesn't wait for set, and it ignores an error from it. Log failed writes inside set if you want to see them.

Usage

import { createAnthropicProvider } from '@rudra-js/anthropic';
import { createComponentGenerator, type CachedSpec, type SpecCache } from '@rudra-js/core';

interface RedisClient {
  get(key: string): Promise<string | null>;
  set(key: string, value: string, options: { EX: number }): Promise<string | null>;
  del(key: string): Promise<number>;
}

const entries = new Map<string, string>();

const redis: RedisClient = {
  async get(key) {
    return entries.get(key) ?? null;
  },
  async set(key, value) {
    entries.set(key, value);
    return 'OK';
  },
  async del(key) {
    return entries.delete(key) ? 1 : 0;
  },
};

export const redisSpecCache: SpecCache = {
  async get(key) {
    const value = await redis.get(`rudra:${key}`);
    return value === null ? undefined : (JSON.parse(value) as CachedSpec);
  },
  async set(key, cached) {
    await redis.set(`rudra:${key}`, JSON.stringify(cached), { EX: 60 });
  },
  async delete(key) {
    await redis.del(`rudra:${key}`);
  },
};

export const generator = createComponentGenerator({
  provider: createAnthropicProvider({ apiKey: process.env.ANTHROPIC_API_KEY! }),
  cache: redisSpecCache,
});

Troubleshooting

Source on GitHub@rudra-js/core on npm