Customizing Models with Portkey

Hey! Does this do the job? https://portkey.ai/docs/integrations/llms/byollm

Hmm. So I need to host it, and protect it w/ an API key, then route it all through you. Is that right?

Correct!

Portkey doesn't do any inferencing itself

We work as a middleware gateway + observability layer

What if I don't want to make the model inference available through a public gateway?

You can run Portkey Gateway locally with npx @portkey-ai/gateway and route to your inference API without exposing it publicly

The caveat is - this is the open source version, it doesn't come with observability or dashboard yet.

Okay, so self-host?

Yes!

But not the full Portkey experience. Just the Gateway

Hmm, okay

We are working on a way to let you run Portkey locally and still be able to plug to our hosted dashboard + observability suite

This is coming fairly soon!