[BUG] MPP decoder passes unsafe PPS scaling-list data to RKVDEC2, leading to kernel Oops in task finalize (RK3588)

[chrisbeeze]

Hardware: RK3588
Kernel: rockchip-linux/kernel 5.10.160 (also seen on 6.1/6.6 lines)
GStreamer: 1.18.5

Related kernel issue (driver side): rockchip-linux/kernel#362

Summary

During H.264/H.265 playback a reproducible kernel crash is triggered in the RKVDEC path
(mpp_dma_release → mpp_task_finalize → rkvdec*_free_task). The stream contains PPS with scaling-list fields that appear out-of-range / malformed.
Note: The analysis below reflects our idea/hypothesis; please verify feasibility and correctness on the MPP side.

Suspected userspace (MPP) problems

1. Decoder parser does not fully validate PPS/SPS scaling-list lengths/offsets before
   building buffers/offsets that are later mapped into the kernel. This can allow out-of-bounds
   reads/writes when the driver copies PPS data (see kernel issue above for details).

2. Task lifecycle robustness: on error paths, MPP may submit or finalize a "partially initialized"
   task (e.g., dma/cmd buffers not fully prepared), which then causes invalid resource releases
   in the driver.

Why MPP is likely involved

• MPP parses parameter sets and forms the PPS/SPS structures and offsets consumed by the kernel.
  If malformed fields are not clamped/validated in userspace, the kernel can still be crashed even if it adds checks.
• MPP release notes show nearby fixes (e.g., h264e_pps: add pic_scaling_matrix_present check,
  h2645d_sei: fix read byte overflow error), suggesting similar bounds checks may be needed on the decoder PPS/ScalingList path.

Minimal reproducer

GStreamer:
H.265:
gst-launch-1.0 -v rtspsrc location="" protocols=tcp latency=200 !
rtph265depay ! h265parse ! mppvideodec ! fakesink sync=false
H.264:
gst-launch-1.0 -v rtspsrc location="" protocols=tcp latency=200 !
rtph264depay ! h264parse ! mppvideodec ! fakesink sync=false

Observed behavior

• Kernel Oops in mutex_lock → mpp_dma_release → mpp_task_finalize → rkvdec*_free_task.
• Occurs shortly after PPS is parsed / task prepared.

What would help to fix in MPP

• Add strict validation for PPS/SPS scaling-list presence, lengths, and derived offsets in
  h264d/h265d parser code before any buffer/offset is exposed to the kernel.
• Guard task finalize paths to tolerate partially constructed tasks (no unconditional finalize).
• Optionally add a contract check to refuse tasks that would map pointers beyond PPS buffers
  (defensive programming on the userspace side, mirroring kernel checks).

This reflects our current hypothesis; please verify feasibility and correctness on the MPP side. Thank you.

@HermanChen @JeffyCN

[FumasterLin]

The rk3588 decode flow do not go to mpp_rkvdec.c, but use mpp_rkvdec2.c and mpp_rkvdec2_link.c。
So the fill_scaling_list_pps() func in mpp_rkvdec.c cannot be called in rk3588 decoding flow.

[chrisbeeze]

My mistake: I originally titled this as “rkvdec,” but the same failure mode is present with drivers/video/rockchip/mpp/mpp_rkvdec2.c (RK3588 / vdpu34x).

Symptom
With bitstreams that contain malformed or out-of-range PPS/SPS scaling-list fields, the pipeline eventually crashes in the task teardown sequence (DMA unmap → task finalize → free). This matches a bounds/consistency problem in the parameter buffers prepared by userspace (MPP) and trusted by the kernel.

Likely root cause

• Userspace (MPP): does not fully validate scaling-list counts, lengths, and offsets when building the parameter (“register”) blobs.
• Kernel (mpp_rkvdec2): accepts those blobs and proceeds to map/use them; if sizes/offsets are inconsistent, the driver can hit invalid pointers/sizes and oops in the error/teardown path.

Proposed fix (two layers)

1. Userspace (MPP):

   • Hard-reject invalid scaling-list data before submitting to the kernel.
   • Verify per-size list lengths (H.264 4×4=16, 8×8=64; HEVC 4/8/16/32 sizes with defined counts).
   • Check all offsets + lengths stay within the allocated param blob; cap total_coeff_bytes to a codec/profile-specific maximum.
   • On failure: return error to the app (do not submit a task).

2. Kernel (mpp_rkvdec2.c):

   • Add sanity checks on buffer sizes against fixed maxima per codec before DMA mapping.
   • In all error paths/finalize, tolerate partially initialized tasks (NULL/size checks; zero after free) to avoid double-free or reset storms.

kernel: mutex_lock+0x28/0x70
kernel: mpp_dma_release+0x28/0xac
kernel: mpp_task_finalize+0x60/Oxb4
kernel: rkvdec2_free_task+0x1c/0:
kernel: mpp_free_task+0x50/0xfO
kernel: rkvdec2_Soft_ccu_worker+0x254/Oxcf4
kernel: kthread_worker_fn+0x12c/0x280
kernel: kthread+0x12c/0x130
kernel: ret_from_fork+0x10/0x24

In addition, I am using GStreamer in my pipeline, which relies on libgstrockchipmpp.

[FumasterLin]

Could you please provide these so we can reproduce and analyze them?

1. The error streams that reproduce the issue.
2. mpp lib version in userspace.
   strings librockchip_mpp.so.0 | grep author
3. mpp driver codes in kernel.
   drivers/video/rockchip/mpp

[chrisbeeze]

You can reproduce this issue by pulling multiple streams (for example, 16 RTSP streams from IP cameras) for decoding or transcoding.
The crash usually appears faster when using UDP transport, or even in a congested network environment when using TCP.

Of course, in UDP mode some packet loss is expected, and losing keyframes can cause temporary green screens, which is normal behavior.
However, the driver crash that occurs in such cases is much more serious, it can break 24/7 operation on decoding NVRs or transcoding devices.

Ideally, the MPP userspace library should validate PPS/SPS data so that unsafe or malformed parameter sets never reach the kernel driver.
But even if such data slips through, the driver itself should fail safely and not crash.

As far as we can tell, the current MPP library version does not include such guards, and the problem seems unrelated to a specific library release, it’s more about missing boundary checks in both MPP and the kernel driver.