Portkey
Log in

Welcome to Portkey Forum

Updated yesterday

Extremely Odd Issue with OpenAI Gateway Timeout and Portkey Logs Gap

AArya
·
Extremely odd issue, My service got hung up for 10 minutes calling OpenAI via the gateway where the model specific timeout is set to 2 min:
Portkey Logs have a gap of 10 mins:
trace_id: c1d2e8a6-8e0c-4962-b9c2-322bcb6c1c38 (original)
trace_id: 5c3d907b-17d0-4631-bdfd-720684f4f61c (cached)

the second request that portkey logs show after 10 mins hits the cache, as it exactly the same, I would assume this is result of a retry.

the metadata on logs show the first request 5130 ms though.

However when I look into my tracing application, it confirmed what I encountered, and the gap is exactly when the second request shows up in the portkey logs and hits the cache.

I would assume the call should have timed out.
Attachment
image.png
s
A
2 comments
ssega
how are you configuring the timeout?
AArya
Plain Text
{
  "strategy": {
    "mode": "fallback"
  },
  "request_timeout": 360000,
  "targets": [
    {
      "virtual_key": "open-ai-virtual-xxxx",
      "override_params": {
        "model": "gpt-4o-2024-08-06"
      },
      "request_timeout": 120000,
      "retry": {
        "attempts": 1,
        "on_status_codes": [
          429
        ]
      }
    },
    {
      "virtual_key": "aws-bedrock-xxxx",
      "override_params": {
        "model": "us.anthropic.claude-3-5-sonnet-20241022-v2:0"
      },
      "request_timeout": 120000,
      "retry": {
        "attempts": 1,
        "on_status_codes": [
          429
        ]
      }
    },
    {
      "virtual_key": "aws-bedrock-xxxx",
      "override_params": {
        "model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0"
      },
      "request_timeout": 120000,
      "retry": {
        "attempts": 1,
        "on_status_codes": [
          429
        ]
      }
    }
  ],
  "cache": {
    "mode": "simple",
    "max_age": 259200
  }
}


here is the config its using
Add a reply
Sign up and join the conversation on Discord