Unexpected Error While Calling Portkey Gateway

Question

Here is the full traceback:2025-02-27 08:09:14.751
stream=stdout
2025-02-27 13:09:14.737 | ERROR | ai_core.llms.open_ai_wrapper:generate_text_response_async:436 | [Portkey Gateway] Unexpected error while calling Portkey Gateway with config pc-opeai-5393d5: 'ConnectTimeout' object has no attribute 'response'
2025-02-27 08:09:14.730
stream=stdout
AttributeError: 'ConnectTimeout' object has no attribute 'response'
2025-02-27 08:09:14.730
stream=stdout
└ 1
2025-02-27 08:09:14.730
stream=stdout
│ └
2025-02-27 08:09:14.730
stream=stdout
│ │ └
2025-02-27 08:09:14.730
stream=stdout
if remaining_retries > 0 and self._should_retry(err.response):
2025-02-27 08:09:14.730
stream=stdout
File "/app/.venv/lib/python3.12/site-packages/portkey_ai/_vendor/openai/_base_client.py", line 1601, in _request
2025-02-27 08:09:14.730
stream=stdout
└
2025-02-27 08:09:14.730
stream=stdout
│ └
2025-02-27 08:09:14.730
stream=stdout
return await self._request(
2025-02-27 08:09:14.730
stream=stdout
File "/app/.venv/lib/python3.12/site-packages/portkey_ai/_vendor/openai/_base_client.py", line 1554, in request
2025-02-27 08:09:14.730
stream=stdout
└
2025-02-27 08:09:14.730
stream=stdout
│ └
2025-02-27 08:09:14.730
stream=stdout
│ │ └
2025-02-27 08:09:14.730
stream=stdout
│ │ │ └ FinalRequestOptions(method='post', url='/chat/completions', params={}, headers={'X-Stainless-Raw-Response': 'true'}, max_retr...
2025-02-27 08:09:14.730
stream=stdout
│ │ │ │ └ False
2025-02-27 08:09:14.730
stream=stdout
│ │ │ │ │ └ openai.AsyncStream[portkey_ai._vendor.openai.types.chat.chat_completion_chunk.ChatCompletionChunk]
2025-02-27 08:09:14.730
stream=stdout
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
2025-02-27 08:09:14.730
stream=stdout
File "/app/.venv/lib/python3.12/site-packages/portkey_ai/_vendor/openai/_base_client.py", line 1860, in post

sega · Answer

Hey @Arya can jump on a quick 5 min call if required

Arya · Answer

sure

Arya · Answer

Dm'ed you

sega · Answer

Hey @Arya @Chandeep | Portkey is making a change so that this error is not thrown after timeout

sega · Answer

that said, I'm still surprised your code reaches this block because the default timeout in the SDK is 10 minutes

sega · Answer

how are you doing concurrent requests?

Arya · Answer

that is super odd, the whole execution of the process is about 2 min. It has a bunch of concurrent request though

Arya · Answer

.. the client is being created per request

sega · Answer

got it

sega · Answer

I'm still not sure why the SDK is throwing a httpx.TimeoutException, checking that

Arya · Answer

the rationale is that a client can use different config based on the workflow, so for the sake of simplicity I am taking the overhead of creating a new client. (Unless there is an option to pass config at request level)
PS: this was not an issue with using openai sdk directly before integrating portkey.

sega · Answer

It's weird that you're getting a connectionTimeout, but you're getting a log, that should simply not be possible

sega · Answer

I think the logs you're seeing and the requests that are erroring out are non overlapping

Arya · Answer

huh, may be its not the same request, just one of those other concurrent request within the same time window. let me get the exact timestamps, on when i noticed this first

Arya · Answer

I think you are right the logs are for other requests in the same timewindow, so basically error happens in sdk and possibly never propogates to the gateway. So in gateway logs I saw no errors.

Welcome to Portkey Forum

Unexpected Error While Calling Portkey Gateway