Portkey Community

Hosted production plan server locations and GDPR compliance

I have a question. If I choose the hosted production plan, where are the servers located? I need to comply with GDPR, so it's required that Portkey has servers in the EU.

1 comment

LLuke Pighetti

Add Cost Per Thousand Requests to Analytics Charts

add cost per thousand requests to analytics charts

1 comment

LLuke Pighetti

Playground Comparison Saved for Later Use

allow us to save a playground comparison for later use, right now i have to recreate it every time i close my browser tab

2 comments

LLuke Pighetti

Comparing Prices and Running Quality Checks in Prompt Playground

hey all, how do I

compare price between two prompts in prompt playground?
run quality checks on prompts with criteria and multiple runs?

3 comments

HHarold Senzei

Analytics APIs Access

are the analytics apis open for all orgs? If not can you please enable it for our org

4 comments

SSreeram Sridhar

Evaluating Team Member Capabilities in Portkey Free Version

Can we do evals and create virtual keys for our team members in portkey free version?

1 comment

HHarold Senzei

Enabling JSON Mode on TogetherAI's Models

How do I enable json mode on togetherai's models? They have some models that support it but portkey's prompt ui does not allow adding json mode.

3 comments

kkoishore

OpenAI content filter

"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"refusal": "I'm sorry, I can't assist with that request."
},
"logprobs": null,
"finish_reason": "stop"
}
],

possible reasons for getting this repeatedly on the same request?

3 comments

ttractorjuice

Portkey's Google API Caching Support Expectations

When do we expect Portkey to support caching for the Google APIs?

3 comments

AAli Ranalvi

Mapping a Subdomain to Portkey's API Endpoint

Portkey Support: is there a way I can setup my own subdomain like api.mydomain.com to map to api.portkey.com? I tried doing this using a CNAME record and x-portkey-custom-host but running into this issue.

REASON: we're building AI functionality and on client machines, we need to whitelist outgoing endpoints. it's easier to get clearance for our domain vs portkey.

4 comments

aalgedonode

Pressing This But Option Doesn't Change for O3

pressing this but option doesn't change for o3

2 comments

aalgedonode

Improve json handling with these 3 requests

3 asks:
1) Stop sorting the keys in my JSON schema.
2) Add a copy button
3) Add a nice JSON viewer like in here

2 comments

ddeepanshu_11

Llm model router implementation for query routing

Is there any LLM model router implementation that folks use for determining which LLM to route based on user's query? Prefer open-source

1 comment

kkam

Troubleshooting Format Errors When Using OpenAI and Google APIs

I am using the OpenAI sdk to enable usage of new models/providers but using existing code. Here is a sample JSON object submitted to openai.chat.completions.create(obj) . The code works against OpenAI but I am getting format errors from Google. First I had to remove the $schema key from each tool function.parameters.$schema

I was under the impression that Portkey was suppose to translate the payload formats? Am I doing something wrong?

Current error: BadRequestError: 400 google error: * GenerateContentRequest.contents: contents is not specified

Plain Text

obj = {
  "messages": [
    {
      "role": "developer",
      "content": "I want to buy a hoodie with a fur lined hood. It needs a full zipper. Near Times Square in NYC. Where can I buy one today at lunch time?"
    }
  ],
  "model": "gemini-2.0-flash-001",
  "tool_choice": "auto",
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "searchGoogle",
        "parameters": {
          "type": "object",
          "properties": {
            "query": {
              "type": "string",
              "description": "The search query to send to google."
            }
          },
          "required": [
            "query"
          ],
          "additionalProperties": false
        },
        "strict": true,
        "description": "Run a search query against google for information from the web."
      }
    }
  ],
  "store": false
}

6 comments

kkam

Using Portkey with Google API Requires Actual API Key

I am experimenting with using the OpenAI sdk (since we have other tooling that relies on it) but using portkey to access other models. Specifically for Google I would have expected the following code to work, but instead of working with the virtual key, I need to specify the actual API key for Google. Am I doing something wrong? Do virtual keys only work with the Portkey sdk?

This fails with BadRequestError: 400 google error: API key not valid. Please pass a valid API key.

Plain Text

const openai = new OpenAI({
  apiKey: process.env["GOOGLE_VIRTUAL_API_KEY"],
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: "google",
    apiKey: process.env["PORTKEY_API_KEY"],
  })
});

If I switch out the GOOGLE_VIRTUAL_API_KEY env var instead for GOOGLE_API_KEY it works.

4 comments

HHet007

Potential Pricing Issue with Claude API

This 0.81 cents for 2k token is not right. claude api doens't cost this much. i think it's a bug

3 comments

GGlen_Gold_Cloud_Specialist

Portkey's Retrieval-Augmented Generation Capabilities and Implementation

Hello all,
I wanted to check if Portkey supports Retrieval-Augmented Generation (RAG) and whether it provides options to add documents or links as source material for retrieval. If these capabilities are available, could you share details on how to implement them and any relevant documentation or best practices?

2 comments

kkoishore

Production issue following latest update

@Vrushank | Portkey @Siddharth | Portkey

There is a production issue following the latest update. The prompt functionality has started breaking, and it seems that URLs are no longer being supported and that image inputs only expect base64 encoded images. Additionally, the LLM being used (GPT-4o) is encountering errors, with the dashboard displaying generation errors after the UI refresh for the same example that was working before the changes. Can you get this checked ASAP?

cc: @Sabbyasachi @visarg

4 comments

mmattbert

Guardrails Not Triggered in After Request Hooks

Hi everyone! I'm testing out guardrails, and am running into a problem where my guardrails are not triggered when added to the "after_request_hooks" in a config. The same guardrails are triggered when using a config that adds them to "before_request_hooks". For example, a simple regex guardrail that should fail validation if "wolf" is in the content:

This config triggers the guardrail with the user question: "Is the wolf the ancestor of the dog":
{
"retry": {
"attempts": 3
},
"before_request_hooks": [
{
"id": "pg-no-wolf-guardrail"
}
]
}

but this config does not trigger the guardrail if the LLM responds to a question with an answer that includes the word "wolf":
{
"retry": {
"attempts": 3
},
"after_request_hooks": [
{
"id": "pg-no-wolf-guardrail"
}
]
}

Are after_request guardrails not available in the free plan, or am I doing something wrong?

Thanks.

13 comments

fflagbug

Strange issue with empty responses from completions endpoint

I have a really strange issue where sometimes, with a large enough system message (or at least that's how I was able to reproduce it), I'm receiving an completely empty response from the completions endpoint (streaming).

The even weirder thing is that there's also no log showing up in the portkey UI, but I do receive a trace ID in the response headers. 46698b1e-479c-4287-a612-c249e20dee22 is such an example.

45 comments

aandrw.

Prompts render

So is https://portkey.ai/docs/product/prompt-library/retrieve-prompts not supported on the free plan? Getting a You do not have enough permissions to execute this request error when trying to fetch a prompt.

6 comments

aartyabra

Vertex structured outputs

Hi everyone!
I'm working with the openai client + portkey and I can't send structured_output type queries to vertex ai gemini models.
However sending the same queries to Azure Openai hosted model o1 works fine.
I'm attaching the request I send and the error I receive.
I'm using the openai==1.61.1 package version & 1.9.5 gateway proxy.
Will be glad for your assistance 🙂

25 comments

mmrKa

Setting Usage Limits and Fallback Options for Models within a Virtual Key

My user authenticates exclusively with an API key, and this key has an associated config. My user makes a request using the name of a model that has a virtual key associated with the provider that serves that model.

How can I set a limit on a specific model within a virtual key, so that if a rate or budget limit is exceeded, it fallsback to another model within the same virtual key or another one?

Something similar to what ChatGPT does when a user exceeds the usage limit for O1—deactivating that model while still allowing access to other models. In my case, I would like to have both possibilities: fallback to another model, whether within the same provider or another, and also something like what ChatGPT does—disabling a specific model after a certain usage threshold within a week, month, or day.

3 comments

KKrisD

Specifying Location for Gemini Models in Portkey

Hello all, just discovered Portkey. Looks exciting. I am trying to use Gemini models but I only have quota in us-central1. How do I specify the location in the Portkey calls for Gemini Models?

If I use OpenAI library directly, Google says I have to do this:

OpenAI Client

client = openai.OpenAI(
base_url=f"https://{location}-aiplatform.googleapis.com/v1/projects/{project_id}/locations/{location}/endpoints/openapi",
api_key=credentials.token,
)

response = client.chat.completions.create(
model="google/gemini-1.5-flash-002",
messages=[{"role": "user", "content": "Why is the sky blue?"}],
)

Same works with Portkey as well? I mean I specify location and project_id in base_url and everything will be taken care of?

2 comments

HHet007

the two bugs in portkey

Hello, i found this two bugs. correct me if i am wrong

when i select gemini flash 2 model in portkey i am not able to attach images with it but flash model supports image funcnility. I am migrating from sonnet to gemini so i need this feature.

Portkey is showing two flash model and i am not sure what is the difference, i think there is only one model available right now.

Thanks

11 comments

Welcome to Portkey Forum

Hosted production plan server locations and GDPR compliance

Add Cost Per Thousand Requests to Analytics Charts

Playground Comparison Saved for Later Use

Comparing Prices and Running Quality Checks in Prompt Playground

Analytics APIs Access

Evaluating Team Member Capabilities in Portkey Free Version

Enabling JSON Mode on TogetherAI's Models

OpenAI content filter

Portkey's Google API Caching Support Expectations

Mapping a Subdomain to Portkey's API Endpoint

Pressing This But Option Doesn't Change for O3

Improve json handling with these 3 requests

Llm model router implementation for query routing

Troubleshooting Format Errors When Using OpenAI and Google APIs

Using Portkey with Google API Requires Actual API Key

Potential Pricing Issue with Claude API

Portkey's Retrieval-Augmented Generation Capabilities and Implementation

Production issue following latest update

Guardrails Not Triggered in After Request Hooks

Strange issue with empty responses from completions endpoint

Prompts render

Vertex structured outputs

Setting Usage Limits and Fallback Options for Models within a Virtual Key

Specifying Location for Gemini Models in Portkey

OpenAI Client

the two bugs in portkey