How do I create a config to associate a group of models with a specific user, considering that I need to create a hierarchy of models from more advanced to less advanced? The idea is that, after reaching a certain token usage limit for the first model, the system switches to the second model, then to the third, and so on. For example, if I’m using Groq, I would like the user to be able to use up to 200,000 tokens with DeepSeek R1. If they exceed that limit, they would be switched to the Llama 3.2 model. And if they exceed 500,000 tokens, they would then switch to Gemma 2. How do I implement this?
first you'd have to count your tokens with a tokenizer (honestly, you can do words/4 which is a decent measure instead of using different tokenizers like tiktoken for different models)
Thanks. I was looking for some ready solution. Then in your link I read just about it: Soon, Portkey will also support routing based on other critical parameters like input character count, input token count, prompt type, tool support, and more.
What about the issue of associating a group of models from a provider with a virtual key? I know that in LiteLLM, you can associate more than one provider, each with a specific group of models, within the limitation of a specific virtual key.
