Welcome to Portkey Forum

Updated 2 months ago

Retry in Config

At a glance
Plain Text
class PortkeyClient(BasePortkeyClient):
    def __init__(self, config: PortkeyConfig = PortkeyConfig(), provider: str = "deepinfra"):
        super().__init__(config, provider)
        self._client = Portkey(api_key=self.config.api_key, virtual_key=self.config.virtual_key)

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_fixed(2),
        retry=retry_if_result(should_retry_content)
    )
    def invoke(self, messages: List[Union[SystemMessage, AIMessage, HumanMessage]],
               model: str = "llama_70b",
               **kwargs):
        self._client.config = PortkeyConfigObjects().invoke # Contains the ID of the config obj on portkey
        model_id = self.models.get_model_id(model)
        converted_messages = self._convert_langchain_messages(messages)

        try:
            response = self._client.chat.completions.create(
                messages=converted_messages,
                model=model_id,
                **kwargs
            )
            return response.choices[0].message
        except Exception as e:
            logger.error(f"Error in chat completion: {str(e)}")
            return f"An error occurred: {str(e)}"
V
M
14 comments
@MolBioBoy you could use retry in the Configs directly instead of using Tenacity right?
You're right - whatever is in the Config overrides your virtual key, model etc. settings
Doesnt seem to be working for me. When I misspell the model name in my pydantic class locally it doesnt seem to override this and use the config provided to it in the invoke method
I need tenacity here since I need to retry given some conditions
not just if it fails
Unless its possible to maybe create guardrails and retry if guardrail is triggered?
Yep of course you could do that
within the config itself i mean
I am also checking the fireworks -> deepinfra fallback myself
Got config working when i ran invoke with_options() rather than during class construction
can give you the traces if needed
Yeah traces will be useful. I'm unable to replicate the fallback failure. Even when I override the model slug at the time of making a request, if I have a faulty model slug in my override_params in Config, my fallback gets triggered
Add a reply
Sign up and join the conversation on Discord