Hi everyone!
A few days ago we moved from Gateway 1.5.3 to 1.9.8 and having issues with Bedrock Sonnet model.
With Gateway 1.5.3 we were setting
max_tokens=200_000, since
max_tokens parameter was required so we decided to set it to a big number.
Now with Gateway 1.9.8 we're getting the following error:
Error code: 400 - {\'error\': {\'message\': \'bedrock error: The maximum tokens you requested exceeds the model limit of 4096. Try again with a maximum tokens value that is lower than 4096.\', \'type\': None, \'param\': None, \'code\': None}, \'provider\': \'bedrock\'}If we remove the
max_tokens param or set it to 4096 or lower, the request is successful.
If we downgrade to Gateway 1.5.3 then even if we set
max_tokens=200_000 the request passes successfully.
So we wonder why is the request failing in Gateway 1.9.8? It seems that there is a logic somewhere that checks if
max_tokens_val > max_tokens_max_allowed_val.
Also we deduced that the problem starts happening in Gateway 1.8.0 where changes involving
max_tokens took place.
