Reducing Latency in Portkey Route Measurements

Question

Ayo legends, happy new year almost. I'm trying to reduce latency because I think it's quite high.

I put some measurements in my portkey route, and comes down to this:

Final timing: {
auth: 455.3472920060158,
balanceCheck: 143.57929101586342,
messageProcessing: 0.00041601061820983887,
portkeyInit: 1.2397089898586273,
messagePrep: 0.03700000047683716,
toolsSetup: 0.0010839998722076416,
apiCallSetup: 4235.661958009005,
portkeyCall: 4235.673999994993,
timeToFirstToken: 4904.3970829844475,
totalTime: 14430.662541985512
}

It looks like main bottleneck is happening after calling portkey actually. Any tips? Am I messing something up in my config or something?

Vrushank | Portkey · Answer

Almost happy new year @Gijs! Unlikely that there's something explicitly wrong with the Config. Although, what are your numbers if you remove Portkey and directly call the provider?

Gijs · Answer

Ill try it

Gijs · Answer

Just to rule out some causes: Does streaming generally increase request latency? How about tool calls?

Vrushank | Portkey · Answer

Actually, I'm curious about streaming too, Last I had run some tests, I was almost close to writing a blogpost titled "Streaming Is All You Need" because I saw that streaming calls for the same requests typically had somewhat lower latencies.. But should rerun the tests and confirm! @Siddharth | Portkey we can do this as part of the LLMs in Prod series too!

Vrushank | Portkey · Answer

Tool calls should def increase latencies though

Welcome to Portkey Forum

Reducing Latency in Portkey Route Measurements