Hi, I had few suggestions for Portkey product which will make my team really productive
P0
Folders for Prompts (Currently, it become very difficult to organize the prompts)
Support for Hindi Unicode in displaying Prompt (Have already reported to Rohit)
P1
Personal Prompts and Team Prompts (So that navigating prompts does not become cluttered)
Keeping Production & Sandbox separate. (I am always worried if some changes in dev will break production. We use same prompt Id for Sandbox & Production. I would want to keep "text" of prompt same, but variables like "Model provider" or "Model Virtual" Key different.
Same for keeping Sandbox Keys and Production keys separate
Testing same prompt on multiple models and comparing the results.
P2
Evaluation/Test Framework. Maybe ability to rate prompt responses through another LLM Prompt.
I really like this small note which gets generated on every update, but the message is too generic whenever I update parameters ("Prompt Parameters updates"). Could we make it specific like what parameter was updated?
Thanks for the detailed suggestions @Siddharth Bulia! Really appreciate this. Sharing with the team, and will let you know here if/when we pick them up!
Folder for prompts is a hotly requested feature - we'll pick it up soon 😄
In the Model filter, you can find Saved Prompts at the bottom - all of your prompt templates that have requests withint the chosen timeframe will show up there. You can select one or more and see their analytics
But the feature I am actually looking is more like "Group By" Function. I want to compare which prompts are costing most money. Not needed right away, but I could suggest for the roadmap.
@Siddharth Bulia in the meanwhile, you can check out the Metadata tab on the Analytics page, where we show "Group By" - like summaries for any key:value pairs you send in the metadata. So you could achieve this for your prompt templates right now by passing a different custom metadata KV and then seeing the cost trends there
I will mostly just report every bug/feature which pop-up in my mind. 😅 You can always then prioritize in the order you find most suitable considering need of all other clients 🙂
