Caching issues with multiple tool calls

Hey Hans! Welcome 👋

Is it possible to share some more info - example of requests where you got a miss v/s where you got a hit?

Portkey hashes your whole request body and stores it in our cache - whenever the same hash is seen again, we deliver the response from cache.

So, I wanted to understand if there are any minute differences coming in between the requests where you got a miss v/s hits.

cc @visarg who can explain more on Portkey cache behaviour with tools

Parallelly, if you can also share your request trace id in DM - I'll look into it specifically

Hi! The problem is not a miss, the problem is a hit and then tool calling is only called once when in the original the function was called many times.

Hmm need to dig deeper. Sharing more in DM

Hey @Hans Magnus - We have identified the issue. We are storing all the function calls in cache. But while retrieving it for stream=True calls, we are only sending the first one response. I have finalized the fix for this. Should be live today in sometime. Thanks for reporting the issue.

We have pushed a fix for this. Parallel function calling cache HITs should return all the tool calls in the response now. Please check and let us know.