Welcome to Portkey Forum

Updated 2 weeks ago

Portkey: Integrating Llm Gateway And Observability

Hey friends, love Portkey so far especially since it integrates an LLM gateway and an LLM observability solution into a single product! A few issues I've discovered so far:
  • Integrating Sonnet 3.5, I've noticed that even though Portkey says it conforms to the OpenAI spec, for streaming responses I'm receiving an end_turn as finish reason, instead of stop like OpenAI would do
  • While the docs say I can attach the user property to the /v1/chat/completions request, the user property was only populated in the Portkey UI when I was using the _user metadata KV
  • Not a bug per-se, but more of a usability issue, but it took me forever to figure out why the "Create" button for my new Guardrail is disabled -> I didn't put a name for the Guardrail, but there was no indication in the UI that I was missing the name
V
f
2 comments
Thanks @flagbug - love that you love Portkey!

  1. This is expected, because in this case we decided not to unify any of the json values. Larger discussion on this is here
Please let me know what you think here!

  1. Ah this is a miss from us! It is indeed true that to render the user value in the UI, you need to pass it with _user key. We mention this here
  1. I've faced this myself! Thank you for sharing this. We'll figure out a fix for this
This is expected, because in this case we decided not to unify any of the json values. Larger discussion on this is here

Ah, I understand the engineering decision behind it, but it would be nice if this was mentioned somewhere in the docs 😄

Ah this is a miss from us! It is indeed true that to render the user value in the UI, you need to pass it with _user key. We mention this here

"If you pass the user key in the OpenAI request body, we will automatically also store it in Portkey’s _user key." -> that's what the docs say, so I was assuming I wouldn't have to set the _user metadata value at all and could rely on the OpenAI body value 🤔
Add a reply
Sign up and join the conversation on Discord