Unit tests for command/option parsing correctness in CLI app?

[ptr727]

Looking for advice and example code on how to add unit tests for CLI parsing correctness?

Some way to provide a string, run parse on it, then test for success/failure and value presence, seems not too difficult, but is there a way to express the results in e.g. xUnit [Theory] section?

[zivkan]

What I do is have a method that creates the parser data structures and takes in Func<????, Task<int>> delegates that each command sets as its action. My Program.Main passes in the product code that executes the commands, while test code passes in a delegate that validates that all the function arguments have the expected values.

For example, something like:

[Theory]
[InlineData("quiet", LogLevel.Quiet)]
[InlineData("detailed", LogLevel.Detailed)]
public async Task ValidateFooBarVerbosity(string input, LogLevel expected)
{
    CliCommand parser = FooBarCommand.Create(ValidateArguments);
    _ = await parser.Parse($"foo -v {input}").InvokeAsync();
   
    async Task ValidateArguments(FooBarArguments args)
    {
        args.Verbosity.Should().Be(expected);
    }
}

Writing tests for error conditions is a bit different, instead of parser.Parse(...).InvokeAsync(), it's just var result = parser.Parser(...);, and there's some properties on the parse results data structure that you can validate that it caught an error.

This way I can write many exhaustive tests for the CLI parsing, without it actually executing the product code, so it's nice and fast to execute and doesn't have side effects.

[ptr727]

I modified my cli class to use a static delegate func pointing to the default implementation, in my test code I modify that to point to a local scoped function inside my test function, similar to your code, and that works.

It would still be easier to get access to the parsed parameters after calling Parse(), vs. having to call Invoke().

[jonsequitur]

The approach that I've generally recommended is to test against the ParseResult directly. In beta5, there's no bult-in binding support, so this will become the only way to test this.

[ptr727]

Can you share examples on how to parse the values in ParseResult, I've just a bit of time navigating the structure in the debugger, I can see the command, but I can't figure out how to to get access to global and command specific parameters?

Re no built-in binding support in beta 5, what does that mean?

[jonsequitur]

By calling ParseResult.GetValue (renamed in beta5 from GetValueForArgument / GetValueForOption), you can check the parsed value of any option or argument.

You can see lots of usage examples in these tests:

https://github.com/dotnet/command-line-api/blob/060374e56c1b2e741b6525ca8417006efb54fbd7/src/System.CommandLine.Tests/GetValueByNameParserTests.cs

https://github.com/dotnet/command-line-api/blob/e8282e765515d32c96ea6adb2ec83653ed1379a5/src/System.CommandLine.Tests/Binding/TypeConversionTests.cs

Re no built-in binding support in beta 5, what does that mean?

This refers to the functionality in earlier versions that allowed System.CommandLine to create an instance of a class with properties like Abc and Xyz bound by convention from options like --abc and --xyz.

[ptr727]

This refers to the functionality in earlier versions that allowed System.CommandLine to create an instance of a class with properties like Abc and Xyz bound by convention from options like --abc and --xyz.

Hmm, are you talking about System.CommandLine.NamingConventionBinder I use this, it is an easy way to have a single class with properties representative of all possible options, then I just pass the class around. To my knowledge this was the only way to do it for multiple options, at least when the I thing 8 limit was created. What will the alternative / migration path be?

For parsing, looks like Beta4 does not have a GetValueForOption variant that takes just the option string, and I need to recreate the Option object for every parameter, too much work to either recreate the option or to expose all options are reusable functions. Using a local function and Invoke is easier. I'll re-evaluate when Beta5 makes it to nuget.

[jonsequitur]

Hmm, are you talking about System.CommandLine.NamingConventionBinder

Yes.

I'll re-evaluate when Beta5 makes it to nuget.

It's available on the build feed here if you'd like the evaluate sooner:

https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet-libraries/NuGet/System.CommandLine/versions

[ptr727]

Yes.

What are my options for migration away from System.CommandLine.NamingConventionBinder?

Any docs for b4 to b5 migration available, I see lots of errors that need correcting in my code.