Hey @susa . Have you tried adding the

vvisarg

Hey @susa . Have you tried adding the messages array in override_params in the config object? So your fallback config should look something like this:

Plain Text

{
  "strategy": {
    "mode": "fallback"
  },
  "targets": [
    {
      "virtual_key": "oai-vk",
      "override_params": {
        "model": "gpt-4"
      }
    },
    {
      "virtual_key": "oai-vk",
      "override_params": {
        "model": "gpt-3.5-turbo",
        "messages": [...override_prompt_for_gpt-3.5]
      }
    }
  ]
}

When you do this, Portkey will first try the gpt-4 model with the messages sent in the request body. And if that fails, then it will try gpt-3.5-turbo but with the new messages array that you add in the override_params object for that target. Please let me know if this works for you.

3 comments

VVrushank | Portkey

To take further what @visarg said, here's the Config that sets up fallback between two prompt templates, with Model A having 5 retry attempts:

Plain Text

{
  "strategy": { "mode": "loadbalance" },
  "targets": [
    {
      "prompt_id": "PROMPT_ID_1",
      "retry": { "attempts": 5 }
    },
    { "prompt_id": "PROMPT_ID_2" }
  ]
}

ssusa

Ah this is really cool, just what I needed. Will try it out

VVrushank | Portkey

Let us know how it goes!

Add a reply

Welcome to Portkey Forum

Hey @susa . Have you tried adding the