Skip to content

xgrep-rule-creator: the cost model for taint rules - #13

Merged
chris-rock merged 2 commits into
mainfrom
docs/taint-rule-cost-model
Aug 23, 2026
Merged

xgrep-rule-creator: the cost model for taint rules#13
chris-rock merged 2 commits into
mainfrom
docs/taint-rule-cost-model

Conversation

@chris-rock

Copy link
Copy Markdown
Member

The skill covers correctness thoroughly and says nothing about what a rule costs to run — "Step 5: Optimize" is about simplifying patterns, not runtime. So a rule can pass every test in the workflow and still be a defect.

What's added

The one thing an author can act on at authoring time: engines pre-filter candidate files using the literal tokens in your sink pattern, and that gate dies silently when the sink is punctuation or a short type keyword. When it can't engage, cost is driven entirely by how often the source matches.

So the pathological shape needs both halves:

a source that matches everywhere + a sink with no distinctive literal

Either alone is fine — a broad source is cheap when the sink gates, and a syntax-only sink is cheap when the source is rare. That conjunction is the part that isn't obvious.

SKILL.md

  • new anti-pattern: "A taint rule whose sink has no distinctive literal", showing & $V flowing into *((int*)$P) — a sink whose only token is int, too short and common to filter on
  • new rationalization to reject: "It passes the tests, so it's done"

references/workflow.md (Step 5)

  • how to measure, with --time and a snippet that ranks rules by seconds-per-finding
  • why seconds-per-finding is the sort key: two rules with the same total cost aren't comparable if one returns thousands of findings and the other zero, yet total time ranks them adjacent
  • three questions to ask before shipping a taint rule, the first being "what literal token would an engine pre-filter my sink on?" — if the answer is "none", expect it to visit every file

The guardrail

Recorded explicitly, because acting on this data invites the wrong fix:

A rule that finds nothing may still be correct. The flaw may genuinely be absent from that codebase. What's wrong is the cost of reaching that answer — so make the negative answer cheap, never delete or weaken the rule.

Notes

🤖 Generated with Claude Code

@mondoo-code-review mondoo-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds documentation on taint rule cost modeling with clear guidance on performance profiling.

Comment thread skills/xgrep-rule-creator/references/workflow.md Outdated
chris-rock and others added 2 commits August 23, 2026 12:49
A rule can pass every test and still be a defect. The skill covered correctness
thoroughly and said nothing about what a rule costs to run, so "Step 5: Optimize"
was about simplifying patterns rather than about runtime.

Adds the one thing a rule author can act on at authoring time: engines pre-filter
candidate files using the literal tokens in the SINK pattern, and that gate dies
silently when the sink is punctuation or a short type keyword. When it cannot
engage, cost is driven entirely by how often the SOURCE matches — so the
pathological shape needs both halves:

  a source that matches everywhere + a sink with no distinctive literal

Either alone is fine. A broad source is cheap when the sink gates; a syntax-only
sink is cheap when the source is rare.

  - SKILL.md: a new anti-pattern showing the shape (`& $V` into `*((int*)$P)`,
    whose sink yields no usable token), and a new rationalization to reject —
    "it passes the tests, so it's done".
  - workflow.md Step 5: how to measure with `--time`, why seconds-per-finding is
    the sort key rather than total time, and three questions to ask before
    shipping a taint rule.

Also records the guardrail that matters when acting on this: a rule that finds
nothing may be CORRECT — the flaw may simply be absent. What is wrong is the cost
of reaching that answer, so the fix is to make the negative answer cheap, never
to delete or weaken the rule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review feedback on #13. The snippet used a chained conditional whose None branch
worked only by accident of evaluation order:

  per = 'INF' if n == 0 else ('%.2fs' % (...) if n else '-')

For n=None the first test is False and the second is falsy, so it printed '-' —
the right answer, reached by luck rather than intent, and it breaks the moment
someone reorders the branches.

Made explicit, and the distinction named in a comment: a missing findings count
means the binary did not report one; zero means the rule ran and found nothing.
Those are different answers and the whole point of the ranking is to notice the
second.

Verified for all three inputs (5 / 0 / absent).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@chris-rock
chris-rock force-pushed the docs/taint-rule-cost-model branch from 107b1d5 to bd67c07 Compare August 23, 2026 10:51
@chris-rock
chris-rock merged commit 1394435 into main Aug 23, 2026
3 checks passed
@chris-rock
chris-rock deleted the docs/taint-rule-cost-model branch August 23, 2026 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant