Guardrails Not Triggered in After Request Hooks

Question

Hi everyone! I'm testing out guardrails, and am running into a problem where my guardrails are not triggered when added to the "after_request_hooks" in a config. The same guardrails are triggered when using a config that adds them to "before_request_hooks". For example, a simple regex guardrail that should fail validation if "wolf" is in the content:

This config triggers the guardrail with the user question: "Is the wolf the ancestor of the dog":
{
"retry": {
"attempts": 3
},
"before_request_hooks": [
{
"id": "pg-no-wolf-guardrail"
}
]
}

but this config does not trigger the guardrail if the LLM responds to a question with an answer that includes the word "wolf":
{
"retry": {
"attempts": 3
},
"after_request_hooks": [
{
"id": "pg-no-wolf-guardrail"
}
]
}

Are after_request guardrails not available in the free plan, or am I doing something wrong?

Thanks.

sega · Answer

hey Matt, basic guardrails like regex check are available in all plans

sega · Answer

can you check the log in your UI once, did you configure it as an async guardrail? then it would fail validation but would return response

sega · Answer

you have to select this option to deny the response

mattbert · Answer

Hi sega, I had tried switching back and forth between "deny" and "async", the after_request guardrail didn't trigger either way. When using the "before" config, which did trigger the guardrail, there is a "hook_results" section in the response JSON with a populated "before_request_hooks" list and an empty "after_request_hooks" list.

When using the "after" config, there is no "hook_results" section at all in the response JSON.

sega · Answer

@mattbert , after updating the guardrail, give it a minute for it to invalidate cache and try, it should work. Try with deny request as that is your usecase

mattbert · Answer

@sega I deleted all prompts, guardrails, and configs, and recreated new ones using "lion" as the forbidden word instead of "wolf". Also generated a separate min/max word count guardrail with associated configs. Still experiencing the same issue for all: the guardrail is only triggered in a before hook, not after hook. Could my after_hook configs be incorrect? I am testing these in each config's playground.

{
"retry": {
"attempts": 3
},
"cache": {
"mode": "simple"
},
"after_request_hooks": [
{
"id": "pg-keep-i-guardid"
}
]
}

sega · Answer

looks correct to me, I'll verify and get back

sega · Answer

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'X-portkey-provider: anthropic' \
--header 'x-portkey-virtual-key: ${VIRTUAL_KEY}' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: ${PORTKEY_API_KEY}' \
--header 'x-portkey-config: {"retry":{"attempts":3},"cache":{"mode":"simple"},"after_request_hooks":[{"id":"pg-wolf-c-ccee4e"}]}' \
--data '{ "model": "claude-3-5-haiku-latest", "max_tokens": 100, "stream": false, "messages": [ { "role": "system", "content": [ { "type": "text", "text": "you are helpful" } ] }, { "role": "user", "content": [ { "type": "text", "text": "say wolf" } ] } ]
}'

sega · Answer

response

sega · Answer

let me know if this doesn't help @mattbert

mattbert · Answer

Hi @sega , thanks, it is working via curl! The problem seems to be just with the config playground UI, so I'll stick with curl from now on.

sega · Answer

yo, really? let us take a look, that shouldn't be the case

sega · Answer

thanks for reporting this! :yodasip:

Welcome to Portkey Forum

Guardrails Not Triggered in After Request Hooks