Enabling Cache Control for Anthropic API Requests in Portkey
Enabling Cache Control for Anthropic API Requests in Portkey
At a glance
The community member is asking if Portkey has a way to enable Cache Control for all requests sent to the Anthropic API, similar to the automatic caching feature in OpenAI. They mention that they are using Portkey with HARPA AI, and they don't have a way to set cache control otherwise, so they can't use caching.
In the comments, another community member responds that currently, the UI only handles this per message object, not for the entire request. They were considering implementing this feature, but since OpenAI doesn't have anything comparable, they kept it on hold. Another community member expresses that this would be a great feature and they will communicate it to the product team.
The final comment suggests that OpenAI's caching works well for the community member's team, and they would find a global "ON" or similar option for prompt caching in Portkey to be very helpful, as they use large-prompt automations that could benefit from this feature.
Does Portkey have any way to enable Cache Control for all requests sent for Anthropic API? Like there's now a simple button in the playground, so maybe some config option that will add all input to the cache control. Basically to try and make it work like OpenAI's caching; automatic and always on.
The way we're using Portkey with HARPA AI, we don't really have a way to set cache control otherwise, so we can't use caching.
Hey @Anshul , currently no, the UI also handles this per message object only, not for the entire request, we were thinking if we should implement this, but since openai doesn't have anything comparable we kept it on hold
OpenAI's caching works really well for our team because it's automatic. I checked with Portkey logs and it does work most of the time for us.
And thank you for considering this feature. Would be a lifesaver really. We use large-prompt automations that use 100k+ static tokens, so prompt caching would be a big benefit. It's just the app we use, HARPA AI, does not support prompt caching either, so a global ON or something would be very nice.
