Portkey
Log in

Welcome to Portkey Forum

Updated 6 days ago

Understanding Fallback Configuration and Request Timeout Handling

AArya
·
Want to understand the fallback config, I simulated request timeout, where I get:

Plain Text
{
  "status": 408,
  "headers": {
    "content-type": "application/json",
    "x-portkey-cache-status": "MISS",
    "x-portkey-last-used-option-index": "config.targets[0]",
    "x-portkey-provider": "openai",
    "x-portkey-retry-attempt-count": "0",
    "x-portkey-trace-id": "9ac0fc87-562c-4b42-92e6-ad3cdb100880"
  },
  "body": {
    "error": {
      "message": "Request exceeded the timeout sent in the request: 12ms",
      "type": "timeout_error",
      "param": null,
      "code": null
    }
  },
  "responseTime": 1851,
  "lastUsedOptionJsonPath": "config.targets[0]"
}

My config does not include 408 in it on_status_codes list , yet the gateway fallbacks and uses the second model. What am I missing ?

Also, if I have on_status code under strategy and then nested within the target what gets preference ?
v
A
4 comments
vvisarg
Hey! Are you not passing on_status_codes in the config at all? Or the codes does not include 408 in it?

If on_status_codes is not passed, the fallback gets triggered for all non 2xx responses.
vvisarg
Regarding the preference,

  • If you pass retry on root level and again pass it on nested target, the nested ones take the preference. Else the retry settings are inherited from the parent target.
Just to avoid confusion, on_status_codes of fallback strategy is not applied/considered for the retry setting. They both operate separately with their own settings.
AArya
this was the config:
{
"strategy": {
"mode": "fallback",
"on_status_codes": [
401,
500,
503,
520,
524
]
},
"request_timeout": 360000,
"targets": [
{
"virtual_key": "open-ai-virtual-xxxx",
"override_params": {
"model": "gpt-4o-2024-08-06"
},
"request_timeout": 12,
"retry": {
"attempts": 1,
"on_status_codes": [
429,
408
]
}
},
{
"virtual_key": "anthropic-api-k-xxxx",
"override_params": {
"model": "claude-3-7-sonnet-20250219"
},
"request_timeout": 120000
},
{
"virtual_key": "anthropic-api-k-xxxx",
"override_params": {
"model": "claude-3-5-sonnet-20241022"
},
"request_timeout": 120000
}
],
"cache": {
"mode": "simple",
"max_age": 6
}
}
....

so 408 is not present at the root strategy level, however the behavior I observed was that gateway would fallback from open ai to anthropic.
vvisarg
Thats strange, it should not have done a fallback. Let me verify this on our end once. Do you have any trace_id where this happened?
Add a reply
Sign up and join the conversation on Discord