Troubleshooting Retries and Rate Limiting

On top of this @jbianchini1 - 429 errors are typically for rate limiting errors. But you said that you are grouping Portkey requests - how are you doing that?

Along with retries, are you using any other Configs option?

@rohit I don't have retries on, it only attempts the first time. Within the code, I send a request through to GPT-4o and a separate request through to Dall-E both handled through Portkey. The requests to GPT returned successfully, but received the 429 errors for the Dall-E requests. When I switched the code over to calling Dall-E directly, outside of Portkey, it returned without any issues or rate limit errors.

@Vrushank | Portkey - I'm not grouping them myself, I meant that when the requests are sent from Portkey to DALL-E, they're likely being grouped together and bucketed from the same sending source, so if there was a high volume of requests being sent at that time from other users within Portkey, that might be hitting the rate limits.

In terms of configs, I'm not using any, just setting tracking information in the headers:

headers: {
'x-portkey-api-key': portkeyApiKey,
'Content-Type': 'application/json',
'x-portkey-trace-id': currentSimulationId,
'x-portkey-metadata': {"_user":"${userEmail}"}
}

Got it @jbianchini1 thanks! Requests from Portkey would ideally not be grouped together because there is no fingerprint of Portkey on the final request, and the requests are sent anywhere from our 300 global servers - so all of them are not coming from a single source as well.

As @rohit shared, I'd suggest enabling the retry option once and see if that makes a difference - you can do something like

x-portkey-config:{ "retry": { "attempts": 3 } }

If the 429 errors still persist only when you're using Portkey, just let me know your request trace ID for the failed request. This behaviour is indeed strange, so I am thinking that a retry (or a fallback setup) should typically fix it.. but if it doesn't, we'll take a deeper look and let you know!