Welcome to Portkey Forum

Home
Members
MolBioBoy
M
MolBioBoy
Offline, last seen last week
Joined November 4, 2024
heyo 🙂 Wondering if theres any way to use Portkey with smolagents?
12 comments
V
M
Also another bug from a while ago is that prompt templates on the UI dont work with deepinfra models. when you make a prompt and try to generate it in the playfround it errors out
2 comments
V
fireworks works but deepinfra shows as $0 even though theres been 10 ish USD spent
18 comments
s
M
V
Hi guys, just wanted to ask about whether or not the deepinfra cost calculation is being worked out? Still not showing up on my dashboard and im using llama 3.1 70b
3 comments
S
M
bonjour and happy new years! got a quick question. on the user param of the chat body schema, I can define the user etc and its fine. however let's say I have 3 platforms and the same user can access my app from any of these 3 platforms. currently im setting user=userid-platformid but for analytics theres no way for me to make more granular rules for example all users vs platform vs users on platform 1 etc. is there a way to do this on portkey or do I have to fetch data from portkey and run my own script?
1 comment
W
Hey guys 🙂 Is there any imports to get a message object from portkey? Langchain has AIMessage, HumanMessage etc and id prefer not to download the langchain package so I was wondering if you guys use some standardized message format without resorting to json
7 comments
V
s
M
Hey, im using langgraph and langsmith, any way to get the correct traces to portkey? not as individual logs but as a trace per graph?
3 comments
V
M
Plain Text
class PortkeyClient(BasePortkeyClient):
    def __init__(self, config: PortkeyConfig = PortkeyConfig(), provider: str = "deepinfra"):
        super().__init__(config, provider)
        self._client = Portkey(api_key=self.config.api_key, virtual_key=self.config.virtual_key)

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_fixed(2),
        retry=retry_if_result(should_retry_content)
    )
    def invoke(self, messages: List[Union[SystemMessage, AIMessage, HumanMessage]],
               model: str = "llama_70b",
               **kwargs):
        self._client.config = PortkeyConfigObjects().invoke # Contains the ID of the config obj on portkey
        model_id = self.models.get_model_id(model)
        converted_messages = self._convert_langchain_messages(messages)

        try:
            response = self._client.chat.completions.create(
                messages=converted_messages,
                model=model_id,
                **kwargs
            )
            return response.choices[0].message
        except Exception as e:
            logger.error(f"Error in chat completion: {str(e)}")
            return f"An error occurred: {str(e)}"
14 comments
M
V
Could just be me but API calls to portkey are taking 30-60 seconds. I use different providers so im unsure if my 3 different providers are having a problem or if its portkey?
4 comments
V
M
Any news on when the UI will support deepinfra on prompt templates? Also the addition of Llama3.3 on fireworks? I like to use the prompt template playground as a way to validate performance at a glance when changing models so would be great if this would be possible with a change on the UI rather than having to change model at runtime in my app
3 comments
M
s
Getting an error i havent been getting before with portkey
3 comments
C
M
ERROR:app.services.ai.portkey.client:Error in embedding batch 192 to 288: Error code: 500 - {'error': {'message': 'cohere error: internal server error, this has been reported to our developers. id 2f67ae23bc584dda09f0ade7ffe165d2', 'type': None, 'param': None, 'code': None}, 'provider': 'cohere'}
6 comments
s
M
Forgot to add. The only way the embedding works is if I use default settings by portkey, but I have no idea which model this uses and need to change model depending on input in some scenarios
13 comments
M
s
🥹 any input on this guys
8 comments
V
M
Could be an edge case, but the ability to choose models for prompt templates based on which API key is used. ie. I have 3 keys (dev, staging and prod). In staging I want to use llama 8b to save money while in prod I want 70b. Right now I need to implement this logic in my backend, but I could see value for something like dynamic model choice based on X (not necessarily API key but could be anything like user ID or something like that?)
4 comments
s
M
ability to filter the analytics for which prompt is used, for now i have to go through each log in a time range to figure out what i need to
4 comments
M
V
Different environments for prompts (ie. dev, staging, production)
5 comments
V
M
Also, side question. When sending a list of strings to be embedded, does the return maintain the order? ie. if I send "hello", "world", will the embeddings for hello be index 0 and world be 1?
2 comments
M
s
Unsure if its just me but when i click on logs pertaining to embeddings, the whole platform crashes.
9 comments
s
M
V
R
Hi guys, im struggling a little to use cohere on your platform. Fireworks works fine, but when I made a cohere virtual key it stops working.

For example, to create a prompt when I use fireworks it shows me all the models, but when I use cohere it throws an error saying "no foundation models available"

Similarly, when I try to create embeddings using cohere + explicitly naming the model I want to use, it gives me an error as well:

Plain Text
openai.BadRequestError: Error code: 400 - {'error': {'message': 'cohere error: invalid request: valid input_type must be provided with the provided model', 'type': None, 'param': None, 'code': None}, 'provider': 'cohere'}


This is how im using the library to generate embeddings:

Plain Text
    ...
    async def create_embedding(self, input: Union[str, List[str]], model: str):
        return await self._client.embeddings.create(input=input, model=model)
    ...
    
    #USAGE
    client = create_portkey_fireworks_client(client_type="sync", provider="cohere")
    input = ["I am happy", "I am sad"]
    x = client.create_embedding(input, model="embed-multilingual-v3.0")
    print(x)


Please let me know what im doing wrong :/
2 comments
V