Hardening: fix inf-to-int and INT_MIN negation UB in identify.c

[TheNewJavaman]

Hardening: guard degenerate perspective transforms to avoid UB in identify.c

Found by Coqui GPU fuzzer.
AI usage: Claude Opus 4.6 assisted with analysis and write-up.

Summary

This is a non-security hardening issue.

A crafted image can drive quirc's perspective math into a degenerate state (den == 0), which produces ±inf in perspective_map(). Converting that to int is undefined behavior, and the resulting INT_MIN later triggers a second UB (-INT_MIN) in find_alignment_pattern().

Observed with UBSan (reproduced against v1.2):

• identify.c:602: negation overflow (-INT_MIN)

Classification

• Type: Undefined behavior / robustness bug
  • UB Doesn't recognize "custom" QR codes #1: conversion of non-finite floating-point (±inf) to int in perspective_map()
  • UB Decoding problems with large images #2: signed integer overflow on unary negation of INT_MIN (-INT_MIN) in find_alignment_pattern()
• Security classification: Non-security hardening
• Reason: no demonstrated OOB read/write, control-flow hijack, or memory corruption; primary effect is sanitizer abort / incorrect internal math state

Minimal trigger context

• Target: quirc v1.2 (542848dd6b9b0eaa9587bbf25b9bc67bd8a71fca) checked out from local bare repo/worktree
• Input format used by reproducer:
  • bytes 0-1: width (LE u16)
  • bytes 2-3: height (LE u16)
  • remaining bytes: grayscale pixels
• Crash input uses a 21x21 image payload that causes a singular perspective transform during grid/alignment estimation.

Root cause (UB chain)

1) perspective_map() can divide by zero

When den = c[6]*u + c[7]*v + 1.0 is zero (or near zero), x/y become ±inf.

Then:

ret->x = (int) rint(x);
ret->y = (int) rint(y);

Casting non-finite values to int is UB.

2) downstream arithmetic overflows on INT_MIN

Later in find_alignment_pattern(), an expression of the form:

...(a.x - b.x) * -(c.y - b.y)...

can evaluate -INT_MIN, which is signed overflow UB.

Impact

• With UBSan (non-recovering): process aborts (DoS in sanitizer-enabled deployments).
• Without sanitizers: behavior is compiler/arch dependent due to UB; commonly manifests as bad geometry estimate and decode failure.
• No memory safety primitive demonstrated.

Recommended hardening changes

A) Validate denominator and finiteness in perspective_map()

Before converting to integer coordinates:

• reject near-zero denominator (fabs(den) < eps)
• reject non-finite x/y (!isfinite(x) || !isfinite(y))

Then return a failure indicator (preferred), or write safe sentinel coordinates.

B) Use saturating conversion from floating-point to int

Convert via checked clamp (INT_MIN..INT_MAX) instead of direct cast from rint(...).

C) Avoid UB in size-estimate arithmetic

In find_alignment_pattern(), compute with wider signed type (int64_t) or guarded operations to avoid -INT_MIN overflow.

Reproduction

clang -O2 -g \
  -fsanitize=address,signed-integer-overflow \
  -fno-sanitize-recover=signed-integer-overflow \
  -I quirc/lib \
  quirc/lib/quirc.c quirc/lib/decode.c quirc/lib/identify.c \
  quirc/lib/version_db.c reproducer.c -o reproducer -lm

./reproducer crash_input.bin

Observed output (ubsan_output.txt):

quirc/lib/identify.c:602:36: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior quirc/lib/identify.c:602:36
Exit code: 1

Notes for maintainers

This report is intended as a robustness cleanup request, not a vulnerability disclosure.