Description:
Currently, the DUnitX Console Runner aggregates all negative outcomes into a single generic exit code (usually 1), or 100 for command-line errors.
While the console output provides a summary (Ignored/Passed/Leaked/Failed/Errored), the process exit code is binary (Success/Fail). This forces CI/CD pipelines to parse the text output or the XML log to distinguish between critical failures (e.g., Exceptions/Crashes) and manageable issues (e.g., Memory Leaks or Failed Assertions).
Proposal:
Introduce a mechanism (potentially opt-in via a switch like --exitmode:Bitmask) to return the exit code as a bitmask. This allows build scripts to make intelligent decisions (e.g., "Fail build on Errors, but only Warn on Leaks").
Proposed Bit Mapping:
The exit code would be calculated by OR-ing the following flags:
| Bit |
Hex |
Decimal |
Condition |
| 0 |
0x01 |
1 |
Internal/CMD Error (Bad arguments, runner crashed) |
| 1 |
0x02 |
2 |
Tests Errored (Unhandled Exceptions) |
| 2 |
0x04 |
4 |
Tests Failed (Assertion failures) |
| 3 |
0x08 |
8 |
Tests Leaked (Memory leaks detected) |
| 4 |
0x10 |
16 |
Tests Ignored (See note below) |
Logic:
ExitCode := 0;
if HasCmdLineError then ExitCode := ExitCode or $01;
if ErrorCount > 0 then ExitCode := ExitCode or $02;
if FailCount > 0 then ExitCode := ExitCode or $04;
if LeakCount > 0 then ExitCode := ExitCode or $08;
// Optional: Only if --fail-on-ignored is set
if IgnoreCount > 0 then ExitCode := ExitCode or $10;
Use Case Example:
A CI pipeline could execute the tests and handle the result like this (pseudocode):
run_tests.exe
CODE=$?
# If Bit 1 (Error) or Bit 2 (Fail) is set -> Critical Failure
if (( CODE & 6 )); then
echo "Tests Failed or Errored!"
exit 1
fi
# If Bit 3 (Leak) is set -> Warning only
if (( CODE & 8 )); then
echo "WARNING: Memory Leaks detected"
# Do not exit 1, allow build to proceed
fi
Benefits:
- Eliminates the need for fragile Log/XML parsing in build scripts.
- Allows distinct handling of Memory Leaks vs. Logic Failures.
- Allows differentiating between "Tests Failed" and "Runner Failed" (Bad Params).
Description:
Currently, the DUnitX Console Runner aggregates all negative outcomes into a single generic exit code (usually
1), or100for command-line errors.While the console output provides a summary (Ignored/Passed/Leaked/Failed/Errored), the process exit code is binary (Success/Fail). This forces CI/CD pipelines to parse the text output or the XML log to distinguish between critical failures (e.g., Exceptions/Crashes) and manageable issues (e.g., Memory Leaks or Failed Assertions).
Proposal:
Introduce a mechanism (potentially opt-in via a switch like
--exitmode:Bitmask) to return the exit code as a bitmask. This allows build scripts to make intelligent decisions (e.g., "Fail build on Errors, but only Warn on Leaks").Proposed Bit Mapping:
The exit code would be calculated by OR-ing the following flags:
0x010x020x040x080x10Logic:
Use Case Example:
A CI pipeline could execute the tests and handle the result like this (pseudocode):
Benefits: