Error Code 2: Internal Error (Assertion c != kNoColor failed. Topological sort failed, which means there's a circle in the graph.)

[zhluo94]

Description

I export a variant of the YOLO model that contains a loop for post-processing abd ran into this error for topological sort. However, the model passes onnx_graphsurgeon's toposort(), nor could I spot any loop when visualizing it. A link to the ONNX file is attached, as well as a minimal python script to reproduce the error:

graphsurgeon toposort passed
[03/04/2025-06:21:50] [TRT] [I] [MemUsageChange] Init CUDA: CPU +19, GPU +0, now: CPU 42, GPU 102 (MiB)
[03/04/2025-06:21:52] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +963, GPU +194, now: CPU 1160, GPU 296 (MiB)
[03/04/2025-06:21:52] [TRT] [I] ----------------------------------------------------------------
[03/04/2025-06:21:52] [TRT] [I] Input filename: ./model.onnx
[03/04/2025-06:21:52] [TRT] [I] ONNX IR version: 0.0.9
[03/04/2025-06:21:52] [TRT] [I] Opset version: 19
[03/04/2025-06:21:52] [TRT] [I] Producer name: pytorch
[03/04/2025-06:21:52] [TRT] [I] Producer version: 2.4.0
[03/04/2025-06:21:52] [TRT] [I] Domain:
[03/04/2025-06:21:52] [TRT] [I] Model version: 0
[03/04/2025-06:21:52] [TRT] [I] Doc string:
[03/04/2025-06:21:52] [TRT] [I] ----------------------------------------------------------------
[03/04/2025-06:21:52] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[03/04/2025-06:21:52] [TRT] [E] [topSort.cpp::trivialChoice::325] Error Code 2: Internal Error (Assertion c != kNoColor failed. Topological sort failed, which means there's a circle in the graph.)

Environment

TensorRT Version: 10.7.0

NVIDIA GPU: T4

NVIDIA Driver Version: 550.90.07

CUDA Version: 12.4

CUDNN Version: N/A

Operating System: Debian

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:
https://drive.google.com/file/d/1rll4p_ejFeZWbhbKaO607awIIvQjleJC/view?usp=sharing

Python script:

import tensorrt as trt
import onnx_graphsurgeon as gs
import onnx

onnx_filename = "./model.onnx"
graph = gs.import_onnx(onnx.load(onnx_filename))
graph.toposort()
print("graphsurgeon toposort passed")

logger = trt.Logger(trt.Logger.INFO)
builder = trt.Builder(logger)
config = builder.create_builder_config()
network = builder.create_network()

# Read ONNX file
parser = trt.OnnxParser(network, logger)
parser.parse_from_file(onnx_filename)

# Add optimization profile
inputs = [network.get_input(i) for i in range(network.num_inputs)]
profile = builder.create_optimization_profile()
min_shape = (1, 3, 32, 32)
opt_shape = (8, 3, 384, 640)
max_shape = (8, 3, 640, 640)
for inp in inputs:
    profile.set_shape(inp.name, min=min_shape, opt=opt_shape, max=max_shape)
config.add_optimization_profile(profile)

# Fail here "Topological sort failed, which means there's a circle in the graph"
engine = builder.build_serialized_network(network, config)

Steps To Reproduce

Download the onnx file and run the python script above

Commands or scripts:
Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

[kevinch-nv]

Thanks for the report. I was able to repro as well, filing an internal bug to track and fix this.