Implementing a Stop Button for Streaming Chat Responses

Obviously I could just break out of the streaming loop on my side but then I suppose the full response would still keep streaming and I'd be charged for all tokens?

@Gijs do you mean the stop param is not working as intended with Portkey SDK?

Can you link me to the doc? I couldn't find it in the docs?

Just this - https://portkey.ai/docs/api-reference/inference-api/chat#body-stop

Lol ok. Don't know how I missed it

😄

This is actually a brilliant feature and it works perfectly!

Thanks 😄

Do you know how it's supposed to behave if you stop before the first chunks come in?

That's the only part that isn't working. If I stop before the stream starts it doesn't actually stop.

But might be an issue in my own code

afaik, that doesn't seem to be possible at the moment. Pinging @Griphook | Portkey if he knows!

Ok, yeah I DM'ed you as well.

I actually don't know how to use the stop param. What I did now is just close the SSE reader on the front-end.

The response still completes in the portkey log.

I passed STOP to portkey as the stop param, but how do I use this? When my user presses the stop button what should I do?

I think the stop param is listening for the model to say stop, not for my user to trigger it?

Actually:

1. Never mind about aborting before the stream starts. I'll just not make the button available to press until the stream starts

1. But how to actually stop the stream, I need some help on this. Can't figure it out so far

Yep correct. This is because it's not 2-way event stream right. So the stop param is useful only to stop the stream when a a keyword you define comes in the stream.

I wonder if @Griphook | Portkey knows some tricks to achieve this. @Vedant Agarwala curious if you've tackled this? (Context: How to arbitrarily stop an OpenAI chat completion stream, so that it doesn't send any more tokens?)

Alright. Yup awaiting input.

Here's some intel on langchain: https://github.com/langchain-ai/langchainjs/discussions/5258

What I'm currently going to ship out at least for now: Just break the loop in the front-end. My UX will be perfect, but:

1. Server will make some noise about unhandled abort:

⨯ unhandledRejection: TypeError: Invalid state: WritableStream is closed
at close (webpack:///src/app/api/chat/route.ts?46b7:219:21)
217 | await writer.write(encoder.encode('\n\nAn error occurred while generating the response.'));
218 | } finally {

219 | await writer.close();

| ^
220 | }
221 | })();
222 | {
code: 'ERR_INVALID_STATE'

1. In my portkey logs I can see the full reply continues to come in and so the API will charge me for all response tokens. Could be an issue with o1 model

I haven’t seen it to be possible. Basically, once you make the request, there is no way to send additional information like cancellation parameter.
In our case we just terminate the http request. So we get charged for the tokens, but never see the result.

Sad

Yeah, what you're expecting is not possible with a http stream,
@Gijs however you can do this with the realtime api which is a websocket connection:
https://platform.openai.com/docs/api-reference/realtime-client-events/response/cancel
portkey realtime api support: https://portkey.ai/docs/product/ai-gateway/realtime-api

Doesn't suit me, I need it to work across all models