Restricting models

But I'm assuming you folks would like something more flexible, can you tell me what your use case for this is? I'll look at if this is possible with some changes

documentation about configs:
https://portkey.ai/docs/api-reference/inference-api/config-object

Right, we would need to whitelist the possible models as @bgeneto mentioned, not override to a specific one

It would be to avoid having controls on this in the cloud provider platform

That's right. More flexibility would be nice. The use case can be ample here.... Allowing only newest models, avoiding embedding models etc...

also this might be of interest to you guys, tracking cost/user on a centralized instance of librechat with portkey https://github.com/timmanik/librechat-for-portkey

Cool... I prefer the much more feature rich Open Webui (it has a whitelist by the way, but it is global, not per provider or key). I can write a custom function for open-webui in order to add metadata to the request so we can track costs in portkey.ai. I could also write a tutorial/article... Problem is, I've just checked my trial account in portkey.ai does not have spent enabled... So I can't test.

oh yes, I use openwebui myself xd

in Librechat you can restrict models per provider in the librechat.yml

you might want to try tweaking the openwebui code for this

link to docs for librechat with portkey: https://portkey.ai/docs/integrations/libraries/librechat

would you like us to enable a month long upgraded trial for you? @elentaure. has been a super active user in the forum, so we'd be happy to do this!
cc: @morsczx

Sure! I'm just arriving at portkey.ai (leaving litellm). It would be nice if you could activate spend/cost tracking for my account (me@bgeneto.com), even if it is only trial. This account is only for testing purposes. I need to be sure that I can use portkey.ai in production, I mean, if it has all the features we need for our Gen AI projects. @morsczx

Thanks @sega no need in my case, we have already enterprise agreement in our company as far as I know. We have a line of communication about this open as well with Ayush

Hi @sega I was thinking if I get some time to try creating a PR with a plugin to have a list of allowed models as a guardrail. Do you think that would be a good place for it? Also the other question, how can I debug plugins with a local instance of the gateway and without access to the UI to configure the guardrail? can the before_request_hooks be passed in the config object in the headers for example with a full content somehow instead of pointing to an id?

Yes, that'd be super, the pre request hooks have access to the Request body, so it can be parsed and used!
Yes correct, here's a guide on using writing custom guardrails https://portkey.ai/docs/product/guardrails/list-of-guardrail-checks/bring-your-own-guardrails, lemme know if you have any doubts on it

Thanks @sega but this means having the logic somewhere else and exposed with a webhook. What I meant was writing a plugin in the gateway codebase

similar to the regex one or the schema validator and so on. But with a logic to setup a list of allowed models and check it agains the model selected in the body

should be quite straight forward I guess, but Im not sure how to setup the dev environment for this case to be able to debug

Ok, I see that you can pass the object with the check and so on in the config for the hook

@sega why are hooks skipped for embeddings?

We implemented it for request and response body, but we’re working on making it generic now

ok. when thats available will the plugings work directly with embed function or will it require code modification? Also any ETA on that? 🙂

haven't planned on a timeline, but yes, they should, the idea is to have generic pre request and post request methods that can be used as hooks

@bgeneto dming you for some details

Ok, In the meantime Ive created a PR for a simple plugin to define a list of allowed models as guardrail: https://github.com/Portkey-AI/gateway/pull/695

sweet! I'll check it out!