[TAN-7583] Check content against tenant-defined moderation guidelines#14288
Open
kogre wants to merge 2 commits into
Open
[TAN-7583] Check content against tenant-defined moderation guidelines#14288kogre wants to merge 2 commits into
kogre wants to merge 2 commits into
Conversation
Adds an optional custom_guidelines plain-text setting to the flag_inappropriate_content feature. When present, the guidelines are injected into the toxicity detection prompt as an extra category, and violations are flagged with the new guideline_violation toxicity label. The detection model is now selected through LLMSelector's toxicity_detection use case (Claude Haiku 4.5 by default) instead of a hardcoded Claude 3 Haiku client, and the response is parsed from structured output instead of assistant-prefill prefix matching, since the RubyLLM-based model classes don't support prefills. The prompt template loses its claude_ prefix as it is no longer model-specific. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… switch Toxicity detection was the only consumer of the Bedrock converse client and its Claude 3 Haiku subclass. The aws-sdk-bedrockruntime gem stays, as CohereMultilingualEmbeddings still uses it directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allows a tenant's own moderation policy to be enforced by the AI inappropriate-content detection.
The guidelines are plain text stored in a new
custom_guidelinessetting on theflag_inappropriate_contentfeature and injected into a delimited section of the detection prompt, so the prompt scaffolding and output contract stay under our control.Changelog
Added
custom_guidelinessetting of the "Flag Inappropriate Content" feature (via Admin HQ; there is no in-product UI yet). Content that violates them shows up in the moderation queue as flagged with the newguideline_violationlabel, alongside the AI's reason.Changed
toxicity_detectionuse case), defaulting to Claude Haiku 4.5, instead of a hardcoded Claude 3 Haiku.