OpenRouter support added as well. To clarify, the 2 in the code is supposed to be from the cache immediately before the previous message (if available) regardless of what's going on with injections at depth (which should all presumably have user role for Claude?), hence why depth is strict role switches (unless I'm doing something absurdly stupid here, always possible).

(which should all presumably have user role for Claude?)

I doubt that's a requirement. It's on user messages in the docs because naturally the last message in a request would just be user (solo chat being the typical use case). User sends on turn 1. Then turn 2. The current and previous breakpoints happen to be on the user; you get the idea.

hence why depth is strict role switches (unless I'm doing something absurdly stupid here, always possible)

https://docs.anthropic.com/en/release-notes/api#october-8th-2024
Anthropic recently loosened restrictions to allow for consecutive same-role messages, but they say those messages will be combined into single message, which makes me worry whether consecutive assistant messages will break the cache. If it does, then we'll have to default to caching only the user messages to be safe. If not, well, caching assistant messages will help group chats where most messages are from various characters.

Also, we are allowed up to 4 breakpoints, so we should use 4 (edit: if not enableSystemPromptCache else 3 uh, 2). This will allow users to edit 7 messages back (after 4th last user turn) and keep the cache.

#2693

My original idea was system prompt + last 3 user messages, which would allow you to restart the chat assuming the sys prompt is at least 1024 tokens

I very intentionally only used 2 breakpoints to avoid breaking the system prompt caching option (which uses up to 2 breakpoints).

EDIT: Anyway the current setup is entirely rational. It's customizable, it hits good caches between swipes and between messages, assuming people aren't going back and editing the chat history. Extremely optimized for 90% of usecases other than swiping 50 times (but still optimized for that).

EDIT 2: Exclusively caching on user messages might be a good heuristic, I'll sleep on it. Group chats were always a mess and will remain a mess and I refuse to waste more than 5 neurons on them.

System prompt caching uses 2? My bad. But where and why?

One breakpoint on the sysprompt and another for the tools. It doesn't NEED both, but if I were using a third breakpoint I'd just place it on the prefill to optimize swipes all the way (with a third configuration option).

EDIT: To clarify the prefill breakpoint would be more of a "might as well" thing because your prefill shouldn't be big enough for the caching to make a difference, heuristically, and not worth breaking a feature that already exists.

Funny drawing to explain the intended cache hits.

Thx. Sorry about the eslint thing.