aigw: better envoy launch error handling

[codefromthecrypt]

Right now, if you have an invalid config envoy will crash and there will be no output, and the process will hang. If you supply —debug, you can see why it crashed but it still hangs.

2025-10-08T07:57:23.453-0400	ERROR	provider	file/file.go:100	failed to reload resources initially	{"runner": "provider", "error": "failed to load resources from file /var/folders/qy/4hghk2xs5wd4jhjtg5ffxdwr0000gn/T/aigw-run/envoy-ai-gateway-resources/config.yaml: local validation error: spec.telemetry.accessLog.settings[0].format.omit_empty_values: Invalid value: value provided for unknown field"}
time=2025-10-08T07:57:26.318-04:00 level=WARN msg="Falling back to Envoy listener adminPort check" err="timeout waiting for child processes: no child process found"
time=2025-10-08T07:57:28.319-04:00 level=INFO msg="Waiting for Envoy to be ready..." err="dial tcp 127.0.0.1:1975: connect: connection refused"

A couple improvements we should make:

1. Ensure we always see ERROR from envoy in aigw output, as this explains failures. So, reconfigure so we never have zero output on ERROR.
2. Ensure when envoy crashes, we crash

• If possible, with the way cobra is working, have its failures cancel our cancelable context to end the goroutines
• If not, refactor out the subprocess detection code in the admin-lookup helper and crash unconditionally of admin server if the envoy subprocess died.

[nacx]

Point 2 requires changes in EG. I'll take a look at this.

[codefromthecrypt]

#1314 improves this

[nacx]

I've opened envoyproxy/gateway#7189 to provide a way to gracefully terminate when there are async runner errors. Once the PR is merged, we can apply the following patch here to terminate the process when certain runners fail:

diff --git a/cmd/aigw/run.go b/cmd/aigw/run.go
index 4c239887..a494a882 100644
--- a/cmd/aigw/run.go
+++ b/cmd/aigw/run.go
@@ -107,7 +107,7 @@ func run(ctx context.Context, c cmdRun, o runOpts, stdout, stderr io.Writer) err
        //
        // TODO: maybe make it skip if the certs are already there, but not sure if it's worth the complexity.
        certGenOut := &bytes.Buffer{}
-       certGen := root.GetRootCommand()
+       certGen := root.GetRootCommand(nil)
        certGen.SetOut(certGenOut)
        certGen.SetErr(certGenOut)
        certGen.SetArgs([]string{"certgen", "--local"})
@@ -198,7 +198,25 @@ func run(ctx context.Context, c cmdRun, o runOpts, stdout, stderr io.Writer) err
        // Now running the `envoy-gateway` CLI alternative below by passing `--config-path` to `egConfigPath`.
        // Then the agent will read the resources from the file pointed inside the config and start the Envoy process.

-       server := root.GetRootCommand()
+       runnerErrorHandler := func(err error) {
+               terminateOnErrorFromRunners := []string{
+                       string(egv1a1.LogComponentInfrastructureRunner), // host infra runner that starts the Envoy process.
+                       string(egv1a1.LogComponentProviderRunner),       // file provider that watches and reads config files.
+                       string(egv1a1.LogComponentGatewayAPIRunner),     // reads and translates configuration.
+               }
+
+               if re, ok := err.(interface{ Runner() string }); ok {
+                       for _, runner := range terminateOnErrorFromRunners {
+                               if re.Runner() == runner {
+                                       _, _ = fmt.Fprintf(stderr, "Shutting down on runner error: %v\n", err)
+                                       serverCancel()
+                                       return
+                               }
+                       }
+               }
+       }
+
+       server := root.GetRootCommand(runnerErrorHandler)
+
        // TODO: enable the log by default after the issue is resolved: https://github.com/envoyproxy/gateway/issues/6596
        if c.Debug {
                server.SetOut(stdout)

Testing this with an invalid config gracefully terminates the process:

$ ./out/aigw-darwin-arm64 run invalid-config.yaml
Shutting down on runner error: provider runner error: failed to load resources from file /var/folders/2p/k34gqngx4vn0_7x8xfygr8bm0000gn/T/aigw-run/envoy-ai-gateway-resources/config.yaml: local validation error: spec.telemetry.accessLog.settings[0].format.omit_empty_values: Invalid value: value provided for unknown field