Portkey's Integration with Llamaindex Query Engine

How does Portkey integrate with Llamaindex query engine? CC: @Vrushank | Portkey

10 comments

Since Portkey is integrated as a custom LLM within Llamaindex, you should be able to create a custom query engine with the LLM defined through Portkey client right?

Attachment

ddeepanshu_11

@Vrushank | Portkey can you give an example of the code with portkey and query engine

VVrushank | Portkey

Yes! Let me share today

ddeepanshu_11

sure perfect

ddeepanshu_11

@Vrushank | Portkey possible to share this asap, need it bit urgently?

VVrushank | Portkey

@deepanshu_11 tried out this with Portkey.

Here's the notebook: https://colab.research.google.com/drive/1BkbQh5RPQC7jNbM44xuH0D4QPG--U0yv?usp=sharing

Let me know what you think!

VVrushank | Portkey

Embedding calls are not currently logged.. but seems to work well except for that

ddeepanshu_11

@Vrushank | Portkey currently my llm = OpenAI(model="gpt-4", temperature=0.1, system_prompt=system_prompt, embed_model=OpenAIEmbedding(model="text-embedding-ada-002"))

Where can I add system_prompt and embed_model?

Also, thanks for the prompt response, you are fantastic 🙂

VVrushank | Portkey

Plain Text

import portkey
from llama_index.llms import Portkey

chat_client = Portkey(mode="single", api_key="PORTKEY_API_KEY")

system_prompt = [{"role": "system", "content": "..."}]

chat_client.add_llms(portkey.LLMOptions(provider="openai", model="gpt-4", temperature=0.1, virtual_key="OPENAI_VIRTUAL_KEY", messages=system_prompt
))

chat_llm = chat_client

embedding_client = Portkey(mode="single", api_key="PORTKEY_API_KEY")

embedding_client.add_llms(portkey.LLMOptions(provider="openai", model="text-embedding-ada-002", virtual_key="OPENAI_VIRTUAL_KEY"))

embedding_llm = embedding_client

I had no idea you could add system prompt + embedding model all directly to a single LLM instance of Llamaindex. How does that work? Can you point me to the Llamaindex doc on this? 😮

ddeepanshu_11

Lemme try this, not remember where I saw it, probably in one of their example in github

Add a reply

Welcome to Portkey Forum

Portkey's Integration with Llamaindex Query Engine