kubus/hatchat
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Track prompt cache hits and stabilize request prefix #5

Merged
kubus merged 1 commit from t3code/prompt-cache-tracking into main 2026-09-29 16:02:26 +00:00
Conversation 0 Commits 1 Files changed 12 +291 -25
kubus commented 2026-09-29 16:02:21 +00:00
Owner
Copy link

Summary

  • Surface cached input tokens in Usage (cachedTokens), aggregating them in addUsage while keeping the field absent when no provider reports cache stats.
  • Add cacheHitRate in core, clamped to 0-1, derived from cachedTokens / inputTokens and undefined when input is unknown.
  • Parse cache usage in the OpenAI-compatible provider via extractCachedTokens, preferring prompt_tokens_details.cached_tokens and falling back to DeepSeek's prompt_cache_hit_tokens.
  • Emit cacheHitRate and cachedTokens in the web UI: compact N% cached session label plus a cached count/rate in the usage tooltip.
  • Stabilize the prompt-cache prefix by sorting tool specs by name in both ToolRegistry.toToolSpecs() and the provider request body.
  • Send a stable per-session prompt_cache_key (cacheKey) from the agent so providers route same-prefix requests to the same cache shard.

Testing

  • Added unit tests for cacheHitRate, addUsage cached handling, and cache detail formatting in packages/core/src/messages.test.ts and apps/web/src/tokens.test.ts.
  • Added extractCachedTokens tests covering nested OpenAI, DeepSeek fallback, and missing stats, plus streamed usage assertions in packages/provider-openai/src/provider.test.ts.
  • Added a stable tool-order test in packages/kernel/src/registries.test.ts and verified sorted wire tools plus prompt_cache_key in the provider request body.
  • Not run: full monorepo test suite.
## Summary - Surface cached input tokens in `Usage` (`cachedTokens`), aggregating them in `addUsage` while keeping the field absent when no provider reports cache stats. - Add `cacheHitRate` in core, clamped to 0-1, derived from `cachedTokens / inputTokens` and `undefined` when input is unknown. - Parse cache usage in the OpenAI-compatible provider via `extractCachedTokens`, preferring `prompt_tokens_details.cached_tokens` and falling back to DeepSeek's `prompt_cache_hit_tokens`. - Emit `cacheHitRate` and `cachedTokens` in the web UI: compact `N% cached` session label plus a cached count/rate in the usage tooltip. - Stabilize the prompt-cache prefix by sorting tool specs by name in both `ToolRegistry.toToolSpecs()` and the provider request body. - Send a stable per-session `prompt_cache_key` (`cacheKey`) from the agent so providers route same-prefix requests to the same cache shard. ## Testing - Added unit tests for `cacheHitRate`, `addUsage` cached handling, and cache detail formatting in `packages/core/src/messages.test.ts` and `apps/web/src/tokens.test.ts`. - Added `extractCachedTokens` tests covering nested OpenAI, DeepSeek fallback, and missing stats, plus streamed usage assertions in `packages/provider-openai/src/provider.test.ts`. - Added a stable tool-order test in `packages/kernel/src/registries.test.ts` and verified sorted wire tools plus `prompt_cache_key` in the provider request body. - Not run: full monorepo test suite.
- Parse cached input tokens from OpenAI/OpenRouter and DeepSeek usage shapes
- Accumulate cachedTokens and derive a clamped cacheHitRate in core
- Show cache hit rate in session totals and usage tooltips
- Send a stable prompt_cache_key per session and sort tools by name
kubus merged commit 3a80308707 into main 2026-09-29 16:02:26 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kubus/hatchat!5
No description provided.