Hello,
I am developing a driver that uses the Lattice Sensor Bridge, and I am encountering issues during the driver and camera register debugging phase.
I lose the connection to the HSB when I debug and stop the capture program with the debugger, or when the program crashes due to a failed capture caused by incorrect camera settings. The problem is that a reboot is required to restore full connection with the HSB.
I can reproduce this issue with your hardware. I have set up the following configuration:
Hardware
- Host: AGX Thor DevKit
- Sensor Bridge: Lattice CPNX100-ETH-SENSOR-BRIDGE updated "sb_ip_version=0x2510"
- Cameras: Leopard LI-IMX274-MIPI-D-HOLOSCAN-090H dual camera module
- Camera 0: IP 192.168.0.2, interface mgbe0_0
- Camera 1: Not used
Versions
- Host OS: Jetson Linux R38.4
- Holoscan Sensor Bridge: v2.5
To reproduce the issue, I kill the capture program via the command line once the capture has started:
- In one terminal, I start the capture and wait for the visualization window to appear:
./build/GCC/examples/fusa_coe_imx274_player --sensor=0 --camera-mode=0
- In another terminal, I kill the program:
sudo killall -9 fusa_coe_imx274_player
I repeat steps 1) and 2) until step 1) no longer works (usually need between 0 and 10 tries).
I can see that I am no longer able to ping the HSB.
If I try to reboot the HSB (by unplugging the power supply), it does not help, I still cannot ping the HSB.
I can ping the HSB again if I unload and reload the tegra_capture_coe module:
sudo modprobe -r tegra_capture_coe
sudo modprobe tegra_capture_coe
However, even though I can ping the HSB again, the capture still does not work. I get the following error:
INFO 0.0000 hololink.cpp:1776 configure_hsb tid=0x17da -- HSB IP version=0x2510 datecode=0xecfb0000
ERROR 1.2078 arp_wrapper.cpp:62 arp_set tid=0x17da -- SIOCSARP operation failed: r=-1, e=1
INFO 1.2081 hololink.cpp:1776 configure_hsb tid=0x17da -- HSB IP version=0x2510 datecode=0xecfb0000
INFO 1.6433 native_imx274_sensor.cpp:60 configure tid=0x17da -- version=1
[warning] [application.cpp:289] Current stack size limit (8388608 bytes / 8192 KB) is below the recommended minimum (33554432 bytes / 32768 KB). Consider increasing it with 'ulimit -s 32768'. For Docker, use '--ulimit stack=33554432'
[info] [fragment.cpp:1170] Loading extensions from configs...
[info] [fusa_coe_imx274_player.cpp:93] Using interface=mgbe0_0, mac=00:50:C2:58:C4:2C
INFO 1.7239 fusa_coe_capture.cpp:302 configure tid=0x17da -- start=41152, bytes_per_line=5120, width=3840, height=2160, format=1, trailing_bytes=0
INFO 1.7240 packed_format_converter.cpp:248 configure tid=0x17da -- start_byte=0, bytes_per_line=5120, width=3840, height=2160, trailing_bytes=0
[info] [gxf_executor.cpp:397] Creating context
[info] [gxf_executor.cpp:2413] Activating Graph...
[info] [gxf_executor.cpp:2545] Running Graph...
[info] [gxf_executor.cpp:2547] Waiting for completion...
[info] [greedy_scheduler.cpp:191] Scheduling 5 entities
initialize: if_name=mgbe0_0
attrib.sensorMacAddr MAC address: 00:50:c2:58:c4:2c
FuSa: CoE: open channel if=mgbe0_0 mac=00:50:c2:58:c4:2c
openChannelFd: chanFilePath = /dev/coe-chan-0 .....
[error] [gxf_wrapper.cpp:69] Exception occurred when starting operator: 'fusa_coe_capture_0' - Failed to open NvFusa CoE Handler (error = 6, handle = 0x0)
[warning] [entity_executor.cpp:674] Failed to start entity [fusa_coe_capture_0]
[warning] [greedy_scheduler.cpp:243] Error while executing entity 7 named 'fusa_coe_capture_0': GXF_FAILURE
[error] [entity_executor.cpp:789] Entity [fusa_coe_capture_0] must be in Started, Tick Pending, Ticking or Idle stage before stopping. Current state is StartPending
[info] [greedy_scheduler.cpp:401] Scheduler finished.
[error] [program.cpp:580] wait failed. Deactivating...
[error] [runtime.cpp:1655] Graph wait failed with error: GXF_FAILURE
[warning] [gxf_executor.cpp:2548] GXF call GxfGraphWait(context) in line 2548 of file /workspace/holoscan-sdk/src/core/executors/gxf/gxf_executor.cpp failed with 'GXF_FAILURE' (1)
[info] [gxf_executor.cpp:2563] Graph execution finished.
[error] [gxf_executor.cpp:2571] Graph execution error: GXF_FAILURE
[info] [gxf_executor.cpp:432] Destroying context
[error] [fusa_coe_imx274_player.cpp:284] Application failed: Failed to open NvFusa CoE Handler (error = 6, handle = 0x0)
I am forced to reboot to restore normal operation. This is very inconvenient because it happens when I am developing my driver and when the capture fails.
It seems that the CoE ends up in an improperly managed state.
I have not found a solution to make it work again without rebooting the Thor.
I have recorded a video of the issue (quality is not good because of the GIF and compression under 10MB):
On the right terminal, I test that I can ping the HSB and then I start the capture.
In the top-left terminal, I kill the capture process.
In the bottom-left terminal, I display the dmesg output continuously.
As seen, when it fail to launch capture I show that I cannot ping the HSB anymore.
Then I unload and reload the tegra_capture_coe module and the ping is working again but I cannot capture due to GXF_FAILURE error.
I have also noticed that sometimes rebooting the Thor is not sufficient, a reboot of the HSB is also required.
You will also find the complete dmesg attached:
dmesg_COE_fail.txt
Do you have a solution to restore capture functionality once this issue occurs?
It seem to be a bug in the state of capture module
Regards
Hello,
I am developing a driver that uses the Lattice Sensor Bridge, and I am encountering issues during the driver and camera register debugging phase.
I lose the connection to the HSB when I debug and stop the capture program with the debugger, or when the program crashes due to a failed capture caused by incorrect camera settings. The problem is that a reboot is required to restore full connection with the HSB.
I can reproduce this issue with your hardware. I have set up the following configuration:
Hardware
Versions
To reproduce the issue, I kill the capture program via the command line once the capture has started:
./build/GCC/examples/fusa_coe_imx274_player --sensor=0 --camera-mode=0sudo killall -9 fusa_coe_imx274_playerI repeat steps 1) and 2) until step 1) no longer works (usually need between 0 and 10 tries).
I can see that I am no longer able to ping the HSB.
If I try to reboot the HSB (by unplugging the power supply), it does not help, I still cannot ping the HSB.
I can ping the HSB again if I unload and reload the
tegra_capture_coemodule:However, even though I can ping the HSB again, the capture still does not work. I get the following error:
I am forced to reboot to restore normal operation. This is very inconvenient because it happens when I am developing my driver and when the capture fails.
It seems that the CoE ends up in an improperly managed state.
I have not found a solution to make it work again without rebooting the Thor.
I have recorded a video of the issue (quality is not good because of the GIF and compression under 10MB):
On the right terminal, I test that I can ping the HSB and then I start the capture.
In the top-left terminal, I kill the capture process.
In the bottom-left terminal, I display the dmesg output continuously.
As seen, when it fail to launch capture I show that I cannot ping the HSB anymore.
Then I unload and reload the
tegra_capture_coemodule and the ping is working again but I cannot capture due to GXF_FAILURE error.I have also noticed that sometimes rebooting the Thor is not sufficient, a reboot of the HSB is also required.
You will also find the complete dmesg attached:
dmesg_COE_fail.txt
Do you have a solution to restore capture functionality once this issue occurs?
It seem to be a bug in the state of capture module
Regards