Add expect.JSON comparator

This changeset adds expect.JSON[T any], which will allow easier testing of json output and hopefully
remove a lot of boilerplate unmarshalling / assert in tests.

It also adds an extensive doc.md document about comparators (plan is trim down the main documentation
to a small set of simple examples, then link to these "advanced" docs for further reading), which
will allow for easier documentation maintenance and more approachable reading.

Signed-off-by: apostasie <[email protected]>

Author: apostasie

mod/tigron/expect/comparators.go

@@ -19,11 +19,13 @@
 package expect
 
 import (
+	"encoding/json"
 	"regexp"
 	"testing"
 
 	"github.com/containerd/nerdctl/mod/tigron/internal/assertive"
 	"github.com/containerd/nerdctl/mod/tigron/test"
+	"github.com/containerd/nerdctl/mod/tigron/tig"
 )
 
 // All can be used as a parameter for expected.Output to group a set of comparators.
@@ -69,3 +71,18 @@ func Match(reg *regexp.Regexp) test.Comparator {
 		assertive.Match(assertive.WithFailLater(t), stdout, reg, info)
 	}
 }
+
+// JSON allows to verify that the output can be marshalled into T, and optionally can be further verified by a provided
+// method.
+func JSON[T any](obj T, verifier func(T, string, tig.T)) test.Comparator {
+	return func(stdout, info string, t *testing.T) {
+		t.Helper()
+
+		err := json.Unmarshal([]byte(stdout), &obj)
+		assertive.ErrorIsNil(assertive.WithFailLater(t), err, "failed to unmarshal JSON from stdout")
+
+		if verifier != nil && err == nil {
+			verifier(obj, info, t)
+		}
+	}
+}

mod/tigron/expect/comparators_test.go

@@ -18,24 +18,45 @@
 package expect_test
 
 import (
+	"encoding/json"
 	"regexp"
 	"testing"
 
 	"github.com/containerd/nerdctl/mod/tigron/expect"
+	"github.com/containerd/nerdctl/mod/tigron/internal/assertive"
+	"github.com/containerd/nerdctl/mod/tigron/tig"
 )
 
 func TestExpect(t *testing.T) {
 	t.Parallel()
 
-	expect.Contains("b")("a b c", "info", t)
-	expect.DoesNotContain("d")("a b c", "info", t)
-	expect.Equals("a b c")("a b c", "info", t)
-	expect.Match(regexp.MustCompile("[a-z ]+"))("a b c", "info", t)
+	expect.Contains("b")("a b c", "contains works", t)
+	expect.DoesNotContain("d")("a b c", "does not contain works", t)
+	expect.Equals("a b c")("a b c", "equals work", t)
+	expect.Match(regexp.MustCompile("[a-z ]+"))("a b c", "match works", t)
 
 	expect.All(
 		expect.Contains("b"),
 		expect.DoesNotContain("d"),
 		expect.Equals("a b c"),
 		expect.Match(regexp.MustCompile("[a-z ]+")),
-	)("a b c", "info", t)
+	)("a b c", "all", t)
+
+	type foo struct {
+		Foo map[string]string `json:"foo"`
+	}
+
+	data, err := json.Marshal(&foo{
+		Foo: map[string]string{
+			"foo": "bar",
+		},
+	})
+
+	assertive.ErrorIsNil(t, err)
+
+	expect.JSON(&foo{}, nil)(string(data), "json, no verifier", t)
+
+	expect.JSON(&foo{}, func(obj *foo, info string, t tig.T) {
+		assertive.IsEqual(t, obj.Foo["foo"], "bar", info)
+	})(string(data), "json, with verifier", t)
 }

mod/tigron/expect/doc.md

@@ -0,0 +1,221 @@
+# Expectations
+
+Attaching expectations to a test case is how the developer can express conditions on exit code, stdout, or stderr,
+to be verified for the test to pass.
+
+The simplest way to do that is to use the helper `test.Expects(exitCode int, errors []error, outputCompare test.Comparator)`.
+
+```go
+package main
+
+import (
+    "testing"
+
+    "go.farcloser.world/tigron/test"
+)
+
+func TestMyThing(t *testing.T) {
+    // Declare your test
+    myTest := &test.Case{}
+
+    // Attach a command to run
+    myTest.Command = test.Custom("ls")
+
+    // Set your expectations
+    myTest.Expected = test.Expects(expect.ExitCodeSuccess, nil, nil)
+
+    // Run it
+    myTest.Run(t)
+}
+```
+
+### Exit status expectations
+
+The first parameter, `exitCode` should be set to one of the provided `expect.ExitCodeXXX` constants:
+- `expect.ExitCodeSuccess`: validates that the command ran and exited successfully
+- `expect.ExitCodeTimeout`: validates that the command did time out
+- `expect.ExitCodeSignaled`: validates that the command received a signal
+- `expect.ExitCodeGenericFail`: validates that the command failed (failed to start, or returned a non-zero exit code)
+- `expect.ExitCodeNoCheck`: does not enforce any verification at all on the command
+
+... you may also pass explicit exit codes directly (> 0) if you want to precisely match them.
+
+### Stderr expectations with []error
+
+To validate that stderr contain specific information, you can pass a slice of `error` as `test.Expects`
+second parameter.
+
+The command output on stderr is then verified to contain all stringified errors.
+
+### Stdout expectations with Comparators
+
+The last parameter of `test.Expects` accepts a `test.Comparator`, which allows testing the content of the command
+output on `stdout`.
+
+The following ready-made `test.Comparator` generators are provided:
+- `expect.Contains(string)`: verifies that stdout contains the string parameter
+- `expect.DoesNotContain(string)`: negation of above
+- `expect.Equals(string)`: strict equality
+- `expect.Match(*regexp.Regexp)`: regexp matching
+- `expect.All(comparators ...Comparator)`: allows to bundle together a bunch of other comparators
+- `expect.JSON[T any](obj T, verifier func(T, string, tig.T))`: allows to verify the output is valid JSON and optionally
+pass `verifier(T, string, tig.T)` extra validation
+
+### A complete example
+
+```go
+package main
+
+import (
+    "testing"
+    "errors"
+
+	"go.farcloser.world/tigron/tig"
+	"go.farcloser.world/tigron/test"
+    "go.farcloser.world/tigron/expect"
+)
+
+type Thing struct {
+	Name string
+}
+
+func TestMyThing(t *testing.T) {
+    // Declare your test
+    myTest := &test.Case{}
+
+    // Attach a command to run
+    myTest.Command = test.Custom("bash", "-c", "--", ">&2 echo thing; echo '{\"Name\": \"out\"}'; exit 42;")
+
+    // Set your expectations
+    myTest.Expected = test.Expects(
+        expect.ExitCodeGenericFail,
+        []error{errors.New("thing")},
+        expect.All(
+            expect.Contains("out"),
+            expect.DoesNotContain("something"),
+			expect.JSON(&Thing{}, func(obj *Thing, info string, t tig.T) {
+				assert.Equal(t, obj.Name, "something", info)
+			}),
+        ),
+    )
+
+    // Run it
+    myTest.Run(t)
+}
+```
+
+### Custom stdout comparators
+
+If you need to implement more advanced verifications on stdout that the ready-made comparators can't do,
+you can implement your own custom `test.Comparator`.
+
+For example:
+
+```go
+package whatever
+
+import (
+    "testing"
+
+    "gotest.tools/v3/assert"
+
+    "go.farcloser.world/tigron/tig"
+    "go.farcloser.world/tigron/test"
+)
+
+func TestMyThing(t *testing.T) {
+    // Declare your test
+    myTest := &test.Case{}
+
+    // Attach a command to run
+    myTest.Command = test.Custom("ls")
+
+    // Set your expectations
+    myTest.Expected = test.Expects(0, nil, func(stdout, info string, t tig.T){
+        t.Helper()
+        // Bla bla, do whatever advanced stuff and some asserts
+    })
+
+    // Run it
+    myTest.Run(t)
+}
+
+// You can of course generalize your comparator into a generator if it is going to be useful repeatedly
+
+func MyComparatorGenerator(param1, param2 any) test.Comparator {
+    return func(stdout, info string, t tig.T) {
+        t.Helper()
+        // Do your thing...
+        // ...
+    }
+}
+
+```
+
+You can now pass along `MyComparator(comparisonString)` as the third parameter of `test.Expects`, or compose it with
+other comparators using `expect.All(MyComparator(comparisonString), OtherComparator(somethingElse))`
+
+Note that you have access to an opaque `info` string, that provides a brief formatted header message that assert
+will use in case of failure to provide context on the error.
+You may of course ignore it and write your own message.
+
+### Advanced expectations
+
+You may want to have expectations that contain a certain piece of data that is being used in the command or at
+other stages of your test (like `Setup` for example).
+
+To achieve that, you should write your own `test.Manager` instead of using the helper `test.Expects`.
+
+A manager is a simple function which only role is to return a `test.Expected` struct.
+The `test.Manager` signature makes available `test.Data` and `test.Helpers` to you.
+
+Here is an example, where we are using `data.Get("sometestdata")`.
+
+```go
+package main
+
+import (
+    "errors"
+    "testing"
+
+    "gotest.tools/v3/assert"
+
+    "go.farcloser.world/tigron/test"
+)
+
+func TestMyThing(t *testing.T) {
+    // Declare your test
+    myTest := &test.Case{}
+
+    myTest.Setup = func(data test.Data, helpers test.Helpers){
+        // Do things...
+        // ...
+        // Save this for later
+        data.Set("something", "lalala")
+    }
+
+    // Attach a command to run
+    myTest.Command = test.Custom("somecommand")
+
+    // Set your fully custom expectations
+    myTest.Expected = func(data test.Data, helpers test.Helpers) *test.Expected {
+        // With a custom Manager you have access to both the test.Data and test.Helpers to perform more
+        // refined verifications.
+        return &test.Expected{
+            ExitCode: 1,
+            Errors: []error{
+                errors.New("foobla"),
+            },
+            Output: func(stdout, info string, t tig.T) {
+                t.Helper()
+
+                // Retrieve the data that was set during the Setup phase.
+                assert.Assert(t, stdout == data.Get("sometestdata"), info)
+            },
+        }
+    }
+
+    // Run it
+    myTest.Run(t)
+}
+```