I wrote a prompt today, which gives me occassionally 3k tokens full of line breaks as output (the prompt is supposed to create this much as text). Is it possible to understand if this was the original OpenAI output? I used the Portkey SDK in python, but saw the same results in portkey evals. It only happens sometimes.
Possible to share the trace id over DM? Can investigate if it's something to do with Portkey. In general, Portkey doesn't even touch the actual output - so whatever you see either on the playground or elsewhere is the actual LLM output.
I did, it is the same - 4k of empty lines and line breaks. Retrying helps, but I would still pay for the tokens. I am not really sure how that happens.
Well, it is difficult to pinpoint because it happens only sometimes and retrying the exact same prompt gives me normal results next time. Today I had it only twice out of about 20 runs. But it does happen relatively regularly.
I’d suggest taking the full input text and trying it out on OpenAI directly, removing Portkey. That way we can pinpoint and see if it’s a prompt engineering problem - which I have a hunch it is!
Well, my theory is that I am creating something like a memory overload on OpenAI's side because I am requiring large input and large output as well. There is no indication in the prompt that would yield line breaks as the output. But I also understand that it is not Portkey doing anything here.
The issue seemed to have been temporary. I had now about 70 runs in a row where it didn't occur anymore for the last 1,5 days without changing anything.
