Welcome to Portkey Forum

Updated 3 months ago

Providing image and tools to gemini using portkey

Hey, I was trying out to Gemini models to work for our agentic run. The run requires vision capabilities and tool calling. I'm getting error from the google, saying that the there is a format issue. So how can I provide image and tools to gemini using portkey.

I'm using portkey SDK for this and the format is based on OpenAI.
S
d
14 comments
See if this fixes your issue
https://discord.com/channels/1143393887742861333/1275330446296875068/1275791798127820903
or you can paste a code snippet if it is still not working
@sega I copied this JSON from Portkey log
Using the same format that works with OpenAI.

It is working with OpenAI and Anthropic. But not working with gemini-1.5-pro
This is the error google throwing in the response.
@darkprince is it only with this one image or with all the images you've tried
The same format and image works with OpenAI, Anthropic
@darkprince I think it's a problem with how your base64 is formatted, you might want to check by directly calling gemini if the formatting is correct.

I've used the image you gave after removing a trailing ""

Here's the working snippet with your image:
response:
Plain Text
{
    "id": "portkey-ebeda651-fbf2-49dd-ae17-33ec2a54ef3b",
    "object": "chat_completion",
    "created": 1724941823,
    "model": "Unknown",
    "provider": "vertex-ai",
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "The image depicts a screenshot of the Amazon India website. The homepage showcases a variety of product categories organized in a grid layout. \n\nHere's a breakdown of the visible sections:\n\n**Top Banner:** Features the Amazon logo, search bar, account and order details, and a prominent \"Prime Now\" button.  A subtle background suggests a focus on mobile phones.\n\n**\"Shop latest smartphones\" Section:** Displays three smartphones, highlighting the \"Never Settle\" tagline, likely promoting OnePlus devices.\n\n**Product Category Grid:** The majority of the screenshot showcases various product categories with representative images:\n\n* **Makeup products:** Features makeup brushes, eyeshadow palette, lipstick, and foundation.\n* **New looks for the new season:** Showcases two models (male and female) dressed in casual wear, hinting at new fashion arrivals.\n* **Do up your home:** Depicts a tastefully decorated living room with a sofa, plants, and side table.\n* **Smart gadgets by"
            },
            "index": 0,
            "finish_reason": "MAX_TOKENS"
        }
    ],
    "usage": {
        "prompt_tokens": 263,
        "completion_tokens": 200,
        "total_tokens": 463
    }
}
Okay. I used a google virtual key for the call. Seems like with your code it is specifically providing vertex ai, i thought google virtual key will be calling vertex, mayb that could be the issue. I will try to update the code to vertext provider and see what happens.
It worked. Thanks.
Umm this shouldn't matter, Virtual keys would work the same as providing auth
Add a reply
Sign up and join the conversation on Discord