MAINT: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback by rlundeen2 · Pull Request #2053 · microsoft/PyRIT

rlundeen2 · 2026-06-19T02:41:44Z

AdversarialConversationManager

AdversarialConversationManager centralizes how multi-turn attacks talk to their adversarial chat. Today Red Teaming, Crescendo, TAP, and PAIR each hand-roll the same mechanics: holding the adversarial system prompt, building the per-turn message, sending it on a stable conversation id, and parsing the reply. The manager owns one adversarial conversation and centralizes all of that in one place.

Because it's one place, it's also where we decide what context the adversarial chat gets each turn — the objective, the latest score, and the objective target's last response — all via a single adversarial_prompt_template. The manager computes the per-turn feedback text in Python (handling blocked / errored / empty responses and optional score feedback) and exposes it to the template as feedback_text, so adding a new signal becomes a template change rather than per-attack code.

It also unifies the shared adversarial_chat JSON schema and is the single home for JSON retry, so schema-aware targets natively constrain the reply and every attack gets consistent parsing/retry behavior for free.

Multimodal feedback

The manager folds in the modality router, so adversarial context is no longer text-only:

Seed media (e.g. images) provided on turn 1 is forwarded to the adversarial chat and preserved into the message sent to the objective target.
The objective target's latest response is bucketed by data type when composing feedback (e.g. {{ message.text.converted_value }}).
Image-generation adversarial system prompts (Crescendo / Red Teaming / TAP) now refer to provided seed images as editable inputs instead of describing them, so image targets edit/compose the seeds rather than regenerate them from scratch.

Config surface

AttackAdversarialConfig now carries system_prompt (inline string or SeedPrompt), first_message, and adversarial_prompt_template; the legacy system_prompt_path is deprecated (removed in 0.17.0). None of this is visible to callers of the executors — the public attack APIs are unchanged.

Migrated executors

Every consuming executor now routes its adversarial conversation through the manager: Crescendo, Red Teaming, Tree of Attacks (and PAIR), Simulated Conversation, Context Compliance, and Role Play. The scenario attack_technique_factory is updated to match.

OLD PR description by @rlundeen2

AdversarialConversationManager is a way to simplify managing adversarial conversations, letting us centrally give various kinds of context to the adversarial chat.

Today each multi-turn attack (Red Teaming, Crescendo, TAP, PAIR) hand-rolls the same mechanics: holding the adversarial system prompt, building the per-turn message, sending it on a stable conversation id, and parsing the reply. This manager owns one adversarial conversation and centralizes all of that in one place.

Because it's one place, it's also where we decide what context the adversarial chat gets each turn — the objective, the latest score, and the objective target's last response (bucketed by data type, e.g. {{ message.text.converted_value }}) — all via a single adversarial_prompt_template. Adding a new signal (a new score field, multimodal pieces, etc.) becomes a template change rather than per-attack code.

It also unifies the shared adversarial_chat JSON schema and is the single home for JSON retry, so schema-aware targets natively constrain the reply and every attack gets consistent parsing/retry behavior for free.

Add a shared `adversarial_chat` JSON schema (next_message, rationale, last_response_summary) and wire the Crescendo, TAP, and PAIR adversarial-chat prompts onto it so their prompts are interchangeable. Parsers now read/return next_message and forward the schema to schema-aware targets via prompt metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The message normalizer already appends the response JSON schema when it is forwarded via prompt metadata, so the hand-written schema block in the Crescendo system prompts duplicated that instruction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Routes Crescendo and TAP adversarial sends through a real PromptNormalizer and a MockPromptTarget (which lacks native JSON_SCHEMA support) to verify the shared adversarial_chat schema is forwarded via prompt metadata and rendered into the prompt the adversarial chat receives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

into rlundeen2/unify-red-team-attack-schema

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…eam-attack-schema # Conflicts: # pyrit/datasets/executors/crescendo/crescendo_variant_1.yaml # pyrit/datasets/executors/crescendo/escalation_crisis.yaml # pyrit/datasets/executors/crescendo/therapist.yaml # pyrit/datasets/executors/pair/attacker_system_prompt.yaml # pyrit/datasets/executors/tree_of_attacks/adversarial_system_prompt.yaml # pyrit/datasets/executors/tree_of_attacks/image_generation.yaml

romanlutz

I love this! It solves lots of problems I've noticed but not really had a good handle on (yet!). That said, there are lots of conflicts with #1377. It's compatible if we adjust this or #1377. Some notes below.

API migration: #2053 renames seed_prompt → first_message and adds adversarial_prompt_template. This would probably need deprecation etc.
Responsibility boundary: #1377’s ModalityFeedbackRouter handles true multimodal forwarding (message pieces). #2053’s manager centralizes adversarial turn/schema logic but is text-message-centric in RedTeamingAttack. We'd need to generalize it to accept/build full Message objects (not just rendered text) so #1377 media forwarding remains intact.
First-turn placeholder flow: #1377 supports adversarial placeholders + seed media for edit-only starts; #2053’s red-teaming path bypasses that behavior unless explicitly reintroduced.
Rollout scope: apply manager only to red teaming first, or also refactor Crescendo/TAP to share the same adversarial-turn primitive.

Another thought that came to mind: Should there be an objective conversation manager then? It's worth considering, but as a follow-up if at all. There’s already ConversationManager; a new objective manager should only centralize objective-target send mechanics (rotation for single-turn targets, converter plumbing, execution context), not absorb attack-specific control flow (backtracking, tree branching, pruning).

So all in all: This PR is compatible with #1377 if integrated deliberately; a mechanical merge would likely regress #1377 multimodal behavior in red teaming. @rlundeen2

rlundeen2 · 2026-06-28T19:19:12Z

Would you want to finish #1377 first? This PR is one I'd probably abandon and start over but I like the strategy. I'd be happy to pick up, but be even happier to offload 😁

Another strategy would be to tackle the goals of #1377 but using this class, which I am equally happy with

Resolve the collision between PR microsoft#2053 (AdversarialConversationManager) and main microsoft#1377 (_ModalityFeedbackRouter) using Option A: the manager owns the adversarial send/parse and, when a modality router is injected, builds its outgoing message via router.build_adversarial_input_message so the adversarial model still sees generated media (preserves microsoft#1377's multimodal feedback loop). red_teaming keeps self._modality_router for the objective side (placeholder fill / build_objective_input_message). Preserve main's blocked/error/use_score_as_feedback semantics by baking them into the default adversarial prompt template (DEFAULT_ADVERSARIAL_ PROMPT_TEMPLATE) rendered through _MessageView, rather than the removed imperative _build_adversarial_prompt_async / _handle_adversarial_* helpers. Note: the default red_teaming adversarial system prompt (text_generation.yaml) declares no JSON schema, so the manager uses the raw-text path (adversarial reply is the next message verbatim); the schema/JSON-parse path only engages when a system prompt or first message declares a schema. Test reconciliation: - red_teaming: drop obsolete handler tests, rewire modality-router integration tests onto the manager/raw-text path, rename seed_prompt -> first_message. - attack_config: add coverage for all six default-template branches (blocked / error / text / text+feedback / no-text+feedback / empty). - simulated_conversation: cover schema vs raw-text next-message handling. - shared attack tests updated for the first_message rename. All 873 attack unit tests pass; ruff + ty clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move the blocked/error/empty/score-feedback control flow out of the per-turn Jinja template into a testable Python helper and render the template strictly. Detect blocked/errored pieces across all message pieces instead of only the first, fixing a bug where a blocked/errored later piece was masked by an earlier clean one. - adversarial_conversation_manager.py: replace the _MessageView / _MessageBucket template shim with _build_adversarial_feedback_text (+ _joined_text_value / _first_response_error); render via render_template_value (strict) exposing feedback_text + objective. - attack_config.py: simplify DEFAULT_ADVERSARIAL_PROMPT_TEMPLATE to "{{ feedback_text }}". - tests: relocate the template-branch coverage to TestBuildAdversarialFeedbackText (adding any-piece blocked/error cases) and update prompt assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… lifecycle Address review findings microsoft#4-microsoft#6 on the adversarial conversation manager: - microsoft#6 Schema-driven parsing (single source of truth): `_parse_adversarial_reply` derives its required/permitted keys from the resolved response schema (`required`, `properties`, `additionalProperties`) -- the same schema the manager forwards to constrain the target -- instead of a hard-coded copy of the adversarial_chat keys, so the parser cannot silently drift from adversarial_chat.yaml. Behavior is identical for the shared adversarial_chat schema; `next_message` is always required since the attack loop consumes it (enforced at parse time and, for the property, at construction). - microsoft#5 Naming: rename the manager's `adversarial_first_prompt_template` param/attr/ property to `first_message`, matching `AttackAdversarialConfig.first_message`. - microsoft#4 Lifecycle: build the AdversarialConversationManager once per execution in RedTeamingAttack (`_build_adversarial_manager`, constructed before the turn loop and threaded into `_generate_next_prompt_async`) instead of rebuilding each turn. - Drop the never-read `_adversarial_response_json_schema` field (keep the construction-time "declared on both" validation) and the dead `_DEFAULT_SEED_PROMPT` constant; refresh the stale `adversarial_prompt_template` comment. - Thread the resolved schema through the simulated_conversation parse call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Crescendo and TAP/PAIR each hand-rolled their own parser for the same adversarial_chat JSON schema. They now delegate to the shared _parse_adversarial_reply used by Red Teaming and Simulated Conversation, so validation and normalization stay consistent across every adversarial-chat executor. This fixes TAP/PAIR's missing camelCase normalization (the exact gap that once broke a Crescendo CI run) and makes them enforce the schema's required keys and additionalProperties instead of only checking next_message. The schema argument is optional, so schemaless custom adversarial prompts keep the permissive next_message-only contract. Also expands coverage: direct TAP node parser tests (camelCase accepted, strict-schema rejection of missing/extra keys, markdown stripping, schemaless-stays-lax), a manager next_message coercion test, a Red Teaming build-once test, and a Simulated Conversation invalid-JSON test. Removes the now-obsolete Crescendo _camel_to_snake test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rlundeen2's WIP renamed AttackAdversarialConfig.seed_prompt to first_message (field + factory internals) but left four tests in test_attack_technique_factory.py asserting the old field, causing AttributeError/TypeError. Update those tests to the new field name; the factory source was already migrated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…describe them When seed/reference images are provided, the crescendo, red_teaming, and tree_of_attacks image_generation adversarial system prompts now instruct the adversarial model to refer to those images by position and issue an edit/compose instruction, rather than describing their contents. Describing the contents made image-editing targets regenerate from scratch and drop the seeded subject's identity. The seed guidance is conditional, so the no-seed generation path is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Brings in 7 upstream commits (scenario DEFAULT strategy curation, dataset parameters on dataset configuration, initializer fixes, Azure SQL memory test fix, schema migration guard). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rlundeen2 · 2026-07-08T17:11:24Z

Right now this only does RedTeamAttack. Which is good! But are there followups for the other attacks? Or we could do it here.

RN Crescendo, TAP, simulated_conversation all use some of it, but should (eventually) use the AdversarialConversationManager to take care of the next turn generally.

rlundeen2 · 2026-07-08T17:19:43Z

                    "'system_prompt' takes precedence and 'system_prompt_path' is ignored."
                )

+    def get_json_schema(self) -> JsonSchemaDefinition | None:


There are several places we

resolve_adversarial_json_schema(system_prompt=..., first_message=...)

But this method should be used instead if we can.

rlundeen2 · 2026-07-08T17:33:14Z

    AttackAdversarialConfig,
    AttackConverterConfig,
    AttackScoringConfig,
+    resolve_adversarial_json_schema,


IMO we would simplify things if we eventually used JSON schemas for red_teaming attack also, and just used the same schema everywhere by default.

rlundeen2 · 2026-07-08T18:23:00Z

            objective=context.objective,
            max_turns=self._max_turns,
        )
        if not adversarial_system_prompt:


This should be removed and managed by AdversarialConversationManager; it should be responsible for all setup

rlundeen2 and others added 6 commits June 17, 2026 17:04

Merge commit 'refs/pull/2039/head' of https://github.com/microsoft/PyRIT

37963ed

into rlundeen2/unify-red-team-attack-schema

WIP: AdversarialConversationManager redesign + config

caef853

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

romanlutz reviewed Jun 27, 2026

View reviewed changes

rlundeen2 mentioned this pull request Jun 30, 2026

FEAT capability-aware multimodal feedback loop for Crescendo/RedTeaming/TAP #1377

Merged

Copilot AI added 7 commits July 4, 2026 07:47

romanlutz changed the title ~~Draft: AdversarialConversationManager~~ Draft: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback Jul 8, 2026

romanlutz changed the title ~~Draft: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback~~ MAINT: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback Jul 8, 2026

romanlutz marked this pull request as ready for review July 8, 2026 16:59

Merge branch 'main' into rlundeen2/unify-red-team-attack-schema

451ddaa

rlundeen2 commented Jul 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MAINT: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback#2053

MAINT: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback#2053
rlundeen2 wants to merge 14 commits into
microsoft:mainfrom
rlundeen2:rlundeen2/unify-red-team-attack-schema

rlundeen2 commented Jun 19, 2026 •

edited by romanlutz

Loading

Uh oh!

romanlutz left a comment

Uh oh!

rlundeen2 commented Jun 28, 2026 •

edited

Loading

Uh oh!

rlundeen2 commented Jul 8, 2026 •

edited

Loading

Uh oh!

rlundeen2 Jul 8, 2026

Uh oh!

rlundeen2 Jul 8, 2026

Uh oh!

rlundeen2 Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rlundeen2 commented Jun 19, 2026 • edited by romanlutz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AdversarialConversationManager

Multimodal feedback

Config surface

Migrated executors

Uh oh!

romanlutz left a comment

Choose a reason for hiding this comment

Uh oh!

rlundeen2 commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rlundeen2 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rlundeen2 Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

rlundeen2 Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

rlundeen2 Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rlundeen2 commented Jun 19, 2026 •

edited by romanlutz

Loading

rlundeen2 commented Jun 28, 2026 •

edited

Loading

rlundeen2 commented Jul 8, 2026 •

edited

Loading