Add color correction and dvr-reencode (#115)

* drm colortrans

* upside down

* correct stranger things mode

* add librga dependency

* fix wrong color correct

* drop cpu colortrans

* fix second recording start

* drop LutCache

* fix h265

* add frame pacer

* cleanup

* rec colortrans

* record osd

* fix res change segfault

* gsmenu integration

* readme update

* fix actions

* fix pipeline 2

* fix wrong framerate while toggleing re-enocdeing

* fix dvr in mixed code scenarios

* fix reenc jitter

* debug log frame pacer grace period

* strict encoding modes

* rework dvr options

* add dvr split

* change dvr_reenc_bitrate defaults

Author: henkwiedig

CMakeLists.txt

@@ -146,8 +146,15 @@ set(LIB_SOURCE_FILES
         src/osd.cpp
         src/os_mon.hpp
         src/os_mon.cpp
+        src/osd_gl.cpp
         src/dvr.h
         src/dvr.cpp
+        src/mpp_encoder.h
+        src/mpp_encoder.cpp
+        src/frame_processor.h
+        src/frame_processor.cpp
+        src/frame_colorcorrect.h
+        src/frame_colorcorrect.cpp
         src/mavlink.h
         src/mavlink.c
         src/wfbcli.hpp
@@ -177,6 +184,9 @@ else()
   find_package(yaml-cpp REQUIRED)
   pkg_check_modules(LIBGPIOD REQUIRED libgpiod)
   include_directories(${LIBGPIOD_INCLUDE_DIR})
+
+  find_path(RGA_INCLUDE_DIRS NAMES rga.h rga/rga.h)
+  find_library(RGA_LIBRARIES NAMES rga librga)
 
   # GStreamer setup
   find_package(PkgConfig REQUIRED)
@@ -196,6 +206,7 @@ else()
       ${PROJECT_SOURCE_DIR}
       ${PROJECT_SOURCE_DIR}/lvgl
       ${LIBDRM_INCLUDE_DIRS}
+      ${RGA_INCLUDE_DIRS}
   )
 
   target_link_libraries(${PROJECT_NAME} 
@@ -216,6 +227,8 @@ else()
       PkgConfig::gstreamer-app
       PkgConfig::gstnet
       PkgConfig::gio
+      ${RGA_LIBRARIES}
+      EGL GLESv2 gbm
   )
 endif()
 
@@ -263,7 +276,7 @@ if(BUILD_TESTS)
     pkg_search_module(gstreamer-app REQUIRED IMPORTED_TARGET gstreamer-app-1.0>=1.4)
     pkg_search_module(gstnet REQUIRED IMPORTED_TARGET gstreamer-net-1.0>=1.4)
     pkg_search_module(gio REQUIRED IMPORTED_TARGET gio-2.0)
-    target_link_libraries(pixelpilot_tests PkgConfig::gstreamer PkgConfig::gstreamer-app PkgConfig::gstnet PkgConfig::gio)
+    target_link_libraries(pixelpilot_tests PkgConfig::gstreamer PkgConfig::gstreamer-app PkgConfig::gstnet PkgConfig::gio ${RGA_LIBRARIES} EGL GLESv2 gbm)
 
     target_link_options(pixelpilot_tests
       BEFORE PUBLIC -fsanitize=undefined PUBLIC -fsanitize=address

README.md

@@ -9,6 +9,13 @@
 WFB-ng client (Video Decoder) for Rockchip platform powered by the [Rockchip MPP library](https://github.com/rockchip-linux/mpp).
 It also displays a simple cairo based OSD that shows the bandwidth, decoding latency, and framerate of the decoded video, and wfb-ng link statistics.
 
+Additional features:
+- **GPU color correction** — live linear color transform applied to video and OSD via EGL/GLES2 (configurable gain/offset, e.g. for FPV color grading)
+- **DVR re-encoding with OSD overlay** — records video via Rockchip MPP hardware encoder with the OSD blended in; supports live bitrate, FPS and codec changes
+- **Frame pacer** — feeds the re-encoder at a steady target FPS, dropping excess frames or repeating the last frame as needed
+- **Image flip** — upside-down display support via DRM
+- **GSMenu** — on-screen ground station control menu for live air-unit and link settings
+
 This project is based on a unique frozen development [FPVue_rk](https://github.com/gehee/FPVue_rk) by [Gee He](https://github.com/gehee).
 
 Tested on RK3566 (Radxa Zero 3W) and RK3588s (Orange Pi 5).
@@ -78,6 +85,12 @@ Build on the Rockchip linux system directly.
 sudo apt install libdrm-dev libcairo-dev librockchip-mpp-dev libspdlog-dev nlohmann-json3-dev libmsgpack-dev libgpiod-dev libyaml-cpp-dev
 ```
 
+- EGL/GLES2 and RGA (required for GPU color correction and DVR re-encoding)
+
+```
+sudo apt install libegl-dev libgles2-mesa-dev libgbm-dev librga-dev
+```
+
 - gstreamer
 
 ```
@@ -341,6 +354,24 @@ Specific widgets expect quite concrete facts as input:
 * `{"type": "IconSelectorWidget", "ranges_and_icons": [{"range": [0, 10], "icon_path": "0_10.png"}, {"range": [11, 20], ...}]}` - shows
   different icon depending on the range where the value lands to.
 
+## Color Correction
+
+PixelPilot supports a real-time GPU-accelerated color transform applied to the live video feed.
+The transform formula is:
+
+```
+output = clamp((input + offset) * gain, 0, 1)
+```
+
+Default values are `offset = -0.15` and `gain = 2.5`, which produce a high-contrast FPV look.
+The transform is applied via an EGL/GLES2 shader directly on the DRM buffer (NV12), so there is no CPU copy overhead.
+
+When color correction is active, the OSD overlay is also GPU-processed with the **inverse** transform so that OSD elements remain visually correct.
+
+If DVR re-encoding is enabled alongside color correction, the recorded video uses the same color-corrected frames.
+
+Parameters can be adjusted live through the GSMenu.
+
 ## GSMenu
 
 The gsmenu provides a ui to modify air and ground settings.
@@ -391,8 +422,11 @@ It uses [Direct Rendering Manager (DRM)](https://en.wikipedia.org/wiki/Direct_Re
 display video on the screen, see `drm.c`.
 It uses `mavlink` decoder to read Mavlink telemetry from telemetry UDP (if enabled), see `mavlink.c`
 It uses `cairo` library to draw OSD elements (if enabled), see `osd.c`.
-It uses `lvgl`to draw the gsmenu.
+It uses `lvgl` to draw the gsmenu.
 It writes non-decoded MPEG stream to file as DVR (if enabled) using `minimp4.h` library.
+It uses EGL/GLES2 (via `frame_colorcorrect.cpp`) to apply a GPU color transform to decoded frames before display and recording.
+It uses Rockchip MPP hardware encoder (`mpp_encoder.cpp`) to optionally re-encode the color-corrected video with OSD blended in for DVR recording.
+The `EncoderPacer` (`encoder_pacer.cpp`) decouples the decoder from the re-encoder, ensuring the encoder receives frames at a steady target FPS.
 
 Pixelpilot starts several threads:
 
@@ -425,6 +459,12 @@ Pixelpilot starts several threads:
   periodically draws widgets on a buffer inside `output_list` using Cairo library.
   There exists legacy OSD, is based on `osd_vars`, draws using Cairo library, to be removed.
   The loop yields on queue's mutex with timeout (timeout in order to re-draw OSD at fixed intervals).
+* ENCODER_PACER_THREAD (if DVR re-encoding is enabled):
+  wakes at the target FPS interval and submits the most recent decoded frame to the MPP re-encoder.
+  Drops frames when the source is faster than the target FPS; repeats the last frame when slower.
+* MPP_ENCODER_THREAD (if DVR re-encoding is enabled):
+  receives frames via an RPC queue from the pacer and encodes them with the Rockchip MPP hardware
+  encoder. Handles live bitrate, FPS, and codec changes without full re-initialisation where possible.
 
 ## Release
 

gsmenu.sh

@@ -871,6 +871,51 @@ case "$@" in
             : #noop
         fi
         ;;
+    "values gs system dvr_mode"*)
+        echo -n -e "raw\nreencode\nboth"
+        ;;
+    "get gs system dvr_mode"*)
+        echo "raw"
+        ;;
+    "set gs system dvr_mode"*)
+        : # noop
+        ;;
+    "values gs system dvr_max_size"*)
+        echo -n "1 40" # will be multiplied by 100
+        ;;
+    "get gs system dvr_max_size"*)
+        echo -n "40" # will be multiplied by 100
+        ;;
+    "set gs system dvr_max_size"*)
+        : # noop needs division by 100
+        ;;
+    "values gs system dvr_reenc_codec"*)
+        echo -n -e "h264\nh265"
+        ;;
+    "values gs system dvr_reenc_resolution"*)
+        echo -n -e "720p\n1080p"
+        ;;
+    "values gs system dvr_reenc_fps"*)
+        echo -n -e "30\n60"
+        ;;
+    "values gs system dvr_reenc_bitrate"*)
+        echo -n -e "5000\n10000\n15000\n20000\n25000\n30000\n35000\n40000\n45000\n50000"
+        ;;
+    "set gs system dvr_reenc_codec"*)
+        : # noop
+        ;;
+    "set gs system dvr_reenc_resolution"*)
+        : # noop
+        ;;
+    "set gs system dvr_reenc_fps"*)
+        : # noop
+        ;;
+    "set gs system dvr_reenc_bitrate"*)
+        : # noop
+        ;;
+    "set gs system dvr_osd"*)
+        : # noop
+        ;;
     "get gs system video_scale")
         grep "^video_scale =" /config/setup.txt | cut -d '=' -f2 | xargs
         ;;

src/drm.c

@@ -17,6 +17,7 @@
 #include <pthread.h>
 #include <rockchip/rk_mpi.h>
 #include <assert.h>
+#include <math.h>
 
 int modeset_open(int *out, const char *node)
 {
@@ -338,6 +339,7 @@ int modeset_create_fb(int fd, struct modeset_buf *buf)
 	struct drm_mode_create_dumb creq;
 	struct drm_mode_destroy_dumb dreq;
 	struct drm_mode_map_dumb mreq;
+	struct drm_prime_handle ph;
 	int ret;
 	uint32_t handles[4] = {0}, pitches[4] = {0}, offsets[4] = {0};
 
@@ -355,6 +357,18 @@ int modeset_create_fb(int fd, struct modeset_buf *buf)
 	buf->size = creq.size;
 	buf->handle = creq.handle;
 
+	/* Export as DMA-buf prime fd for RGA. */
+	buf->prime_fd = -1;
+	buf->gl_fb_id = 0;
+	memset(&ph, 0, sizeof(ph));
+	ph.handle = buf->handle;
+	ph.flags  = DRM_CLOEXEC | DRM_RDWR;
+	if (drmIoctl(fd, DRM_IOCTL_PRIME_HANDLE_TO_FD, &ph) == 0) {
+		buf->prime_fd = ph.fd;
+	} else {
+		fprintf(stderr, "warning: PRIME_HANDLE_TO_FD failed (%m); RGA path unavailable\n");
+	}
+
 	handles[0] = buf->handle;
 	pitches[0] = buf->stride;
 
@@ -393,6 +407,7 @@ int modeset_create_fb(int fd, struct modeset_buf *buf)
 err_fb:
 	drmModeRmFB(fd, buf->fb);
 err_destroy:
+	if (buf->prime_fd >= 0) { close(buf->prime_fd); buf->prime_fd = -1; }
 	memset(&dreq, 0, sizeof(dreq));
 	dreq.handle = buf->handle;
 	drmIoctl(fd, DRM_IOCTL_MODE_DESTROY_DUMB, &dreq);
@@ -404,6 +419,8 @@ void modeset_destroy_fb(int fd, struct modeset_buf *buf)
 {
 	struct drm_mode_destroy_dumb dreq;
 
+	if (buf->prime_fd >= 0) { close(buf->prime_fd); buf->prime_fd = -1; }
+
 	munmap(buf->map, buf->size);
 
 	drmModeRmFB(fd, buf->fb);
@@ -735,3 +752,145 @@ void modeset_cleanup(int fd, struct modeset_output *output_list)
 {
 	modeset_output_destroy(fd, output_list);
 }
+
+// Initialize the controller
+void gamma_lut_controller_init(gamma_lut_controller* ctrl, int drm_fd, struct modeset_output* output_list) {
+    ctrl->drm_fd = drm_fd;
+    ctrl->output_list = output_list;
+    ctrl->gamma_lut_blob_id = 0;
+    ctrl->last_offset = 0.0f;
+    ctrl->last_gain = 1.0f;
+    ctrl->is_enabled = false;
+}
+
+// Get LUT size from DRM properties
+static uint64_t get_lut_size(gamma_lut_controller* ctrl) {
+    uint64_t lut_size = get_property_value(ctrl->drm_fd, ctrl->output_list->crtc.props, "GAMMA_LUT_SIZE");
+    if (lut_size == (uint64_t)-1) {
+        lut_size = get_property_value(ctrl->drm_fd, ctrl->output_list->crtc.props, "gamma_lut_size");
+    }
+    return lut_size;
+}
+
+// Enable gamma LUT with specified offset and gain
+bool gamma_lut_enable(gamma_lut_controller* ctrl, float offset, float gain) {
+    // Store parameters for potential re-enable
+    ctrl->last_offset = offset;
+    ctrl->last_gain = gain;
+
+    // Get LUT size
+    uint64_t lut_size = get_lut_size(ctrl);
+    if (lut_size == 0 || lut_size == (uint64_t)-1) {
+        // Fallback to a reasonable default if size not available
+        lut_size = 4096;  // Common LUT size
+    }
+
+    // Create the LUT data
+    struct drm_color_lut* lut = malloc(lut_size * sizeof(struct drm_color_lut));
+    if (!lut) {
+        return false;
+    }
+
+    for (uint64_t i = 0; i < lut_size; i++) {
+        float x = (lut_size > 1) ? (float)i / (float)(lut_size - 1) : 0.0f;
+        float y = (x + offset) * gain;
+
+        // Clamp to [0, 1]
+        if (y < 0.0f) y = 0.0f;
+        if (y > 1.0f) y = 1.0f;
+
+        uint16_t v = (uint16_t)lroundf(y * 65535.0f);
+        lut[i].red = v;
+        lut[i].green = v;
+        lut[i].blue = v;
+        lut[i].reserved = 0;
+    }
+
+    // Destroy previous blob if exists
+    if (ctrl->gamma_lut_blob_id > 0) {
+        drmModeDestroyPropertyBlob(ctrl->drm_fd, ctrl->gamma_lut_blob_id);
+        ctrl->gamma_lut_blob_id = 0;
+    }
+
+    // Create new property blob
+    int ret = drmModeCreatePropertyBlob(ctrl->drm_fd, lut, 
+                                        lut_size * sizeof(struct drm_color_lut), 
+                                        &ctrl->gamma_lut_blob_id);
+    free(lut);
+
+    if (ret != 0) {
+        return false;
+    }
+
+    // Apply the LUT via atomic commit
+    drmModeAtomicReq *req = drmModeAtomicAlloc();
+    if (!req) {
+        drmModeDestroyPropertyBlob(ctrl->drm_fd, ctrl->gamma_lut_blob_id);
+        ctrl->gamma_lut_blob_id = 0;
+        return false;
+    }
+
+    // Try both property names
+    int set_ret = set_drm_object_property(req, &ctrl->output_list->crtc, "GAMMA_LUT", ctrl->gamma_lut_blob_id);
+    if (set_ret < 0) {
+        set_ret = set_drm_object_property(req, &ctrl->output_list->crtc, "gamma_lut", ctrl->gamma_lut_blob_id);
+    }
+
+    if (set_ret < 0) {
+        drmModeAtomicFree(req);
+        drmModeDestroyPropertyBlob(ctrl->drm_fd, ctrl->gamma_lut_blob_id);
+        ctrl->gamma_lut_blob_id = 0;
+        return false;
+    }
+
+    int commit_ret = drmModeAtomicCommit(ctrl->drm_fd, req, 0, NULL);
+    drmModeAtomicFree(req);
+
+    if (commit_ret != 0) {
+        drmModeDestroyPropertyBlob(ctrl->drm_fd, ctrl->gamma_lut_blob_id);
+        ctrl->gamma_lut_blob_id = 0;
+        return false;
+    }
+
+    ctrl->is_enabled = true;
+    return true;
+}
+
+// Disable gamma LUT (revert to defaults)
+bool gamma_lut_disable(gamma_lut_controller* ctrl) {
+    if (!ctrl->is_enabled && ctrl->gamma_lut_blob_id == 0) {
+        return true;  // Already disabled
+    }
+    return gamma_lut_enable(ctrl, 0, 1);
+}
+
+
+// Re-enable with last used parameters
+bool gamma_lut_reenable(gamma_lut_controller* ctrl) {
+    return gamma_lut_enable(ctrl, ctrl->last_offset, ctrl->last_gain);
+}
+
+// Toggle between enabled and disabled states
+bool gamma_lut_toggle(gamma_lut_controller* ctrl) {
+    if (ctrl->is_enabled) {
+        return gamma_lut_disable(ctrl);
+    } else {
+        return gamma_lut_reenable(ctrl);
+    }
+}
+
+// Check if gamma LUT is currently enabled
+bool gamma_lut_is_enabled(gamma_lut_controller* ctrl) {
+    return ctrl->is_enabled;
+}
+
+// Get current parameters
+void gamma_lut_get_params(gamma_lut_controller* ctrl, float* offset, float* gain) {
+    *offset = ctrl->last_offset;
+    *gain = ctrl->last_gain;
+}
+
+// Clean up and disable
+void gamma_lut_cleanup(gamma_lut_controller* ctrl) {
+    gamma_lut_disable(ctrl);
+}