Hi everyone! 👋
I’m working on a use case where I’m using the portkey.prompts.completions.create method to integrate AI model calls. Here’s a snippet of my code:
portkey = Portkey(
api_key= get_portkey_apikey(),
virtual_key="openai-key-...",
config='pc-cache-...'
)
ai_balancer_v2_response = portkey.prompts.completions.create(
prompt_id="pp-balancer-...",
variables={
"user_query": f"{msg_input}",
"system_prompt": {
"type": "text",
"text": balancer_prompt,
"cache_control": {"type": "cacheable"}
}
},
tools=tool_definitions,
tool_choice="required",
parallel_tool_calls=False,
max_tokens=1024
)The issue I’m encountering is related to the caching mechanism. It seems that the cache is invalidated whenever I change the user_query (dynamic input). However, I need the cache to remain valid for the system prompt (balancer_prompt), even when the user input changes.
In my use case, the system prompt is consistent across requests, and I want to benefit from caching for it, but the user_query will always be different and should not affect the caching.
