Is it possible to send mouse interactions to the on-screen-controls via a command?

[SumOys]

I'm currently updating my mpv frontend to work with Wayland via OpenGL rendering (using python-mpv). I have the following QtOpenGLWidget:

https://gitlab.com/djsumdog/mediahug/-/blob/f6eb6cf889450d15c9af9b3e6b67e5682481fce8/mediahug/gui/mpv/gl_player.py

In my X11 version, I could enable on screen controls (osc) and input was passed through via python-mpv directory to the mpv process. Using libmpv and the MpvRenderContext, in the above code, I forward input from the widget via mouseMoveEvent() and mousePressEvent(). The osc does show up if I move towards the bottom of the window. However, I cannot interact with it by sending mouse clicks via commands. I made a very basic lua script to try it outside of Qt/my application:

local mp = require 'mp'

local function display_osd_info()
    info = mp.get_property("osd-dimensions")
    mp.osd_message("OSD:" .. info)
    print("OSD:" .. info)

    local mouse_pos = mp.get_property_native("mouse-pos")

    x = mouse_pos.x
    y = mouse_pos.y
    print("Mouse position - X:", x, "Y:", y)

    mp.commandv("mouse", x, y, 0, "single")
end

mp.add_key_binding("k", "display_osd_info", display_osd_info)

I obviously messed up here and confused OSD with OSC. I tried printing the OSD properties to figure out what's going on, but that's obviously the message in the top left, not the OSC. Is there a way to get OSC debugging information?

Anyway, when I press k, I don't get the equivalent of a left click on the osc.

Is it even possible to interact with the osc via the mpv API? I really like the osc and would rather not try to re-implement it in my Qt application.

Also asked this on StackOverflow if anyone wants the internet points: https://stackoverflow.com/questions/79697620/is-it-possible-to-send-mouse-events-clicks-to-the-on-screen-controls-osc-in-an

[hooke007]

osc is a built-in script.

[SumOys]

Thanks. I've been looking through it and still can't seem to figure out how to send inputs to it from my frontend.

If I start mpv using mpv --msg-level=input=v --input-test=yes --script-opts=osc-visibility=always test.mp4, this displays all input events without processing them. From this, I can see these two for button clicks and movements:

[input] Key MBTN_LEFT is bound to:
[input] 1.  'ignore' in <builtin>:1 (default)
[input] 2.  'script-binding osc/__keybinding5' in section {input} in <api>:1
[input] 3.  'script-binding osc/__keybinding11' in section {window-controls} in <api>:1

[input] Key MOUSE_MOVE is bound to:
[input] 1.  'script-binding osc/__keybinding1' in section {showhide} in <api>:1
[input] 2.  'script-binding osc/__keybinding3' in section {showhide_wc} in <api>:1

I can also call python-mpv with the following to get input-test working in my frontend:

        return MPV(vo="libmpv",
                   log_handler=print,
                   input_default_bindings=True,
                   input_vo_keyboard=False,
                   input_test=True,
                   keep_open='yes',
                   msg_level="all=v",
                   volume=current_volume,
                   osc=show_controls)

If I set input_default_bindings=False , I the the message warn input No key binding found for key 'MBTN_LEFT'. on my console. If I enable those default bindings and have input_test=True, I see the exact same messages for mouse movement and button clicking:

info input Key MBTN_LEFT is bound to:

info input 1.  'ignore' in <builtin>:1 (default)

info input 2.  'script-binding osc/__keybinding5' in section {input} in <api>:1

info input 3.  'script-binding osc/__keybinding11' in section {window-controls} in <api>:1

info input Key MOUSE_MOVE is bound to:

info input 1.  'script-binding osc/__keybinding1' in section {showhide} in <api>:1

info input 2.  'script-binding osc/__keybinding3' in section {showhide_wc} in <api>:1

The way I'm forwarding events from Python/PySideQt6 is using this whole block of code:

    def __qt_to_mpv_mouse_button(self, mouse_button: Qt.MouseButton) -> int:
        if mouse_button == Qt.MouseButton.RightButton:
            return 2
        if mouse_button == Qt.MouseButton.LeftButton:
            return 0
        if mouse_button == Qt.MouseButton.MiddleButton:
            return 1
        if mouse_button == Qt.MouseButton.BackButton:
            return 7
        if mouse_button == Qt.MouseButton.ForwardButton:
            return 8
        return mouse_button.value

    def mouseDoubleClickEvent(self, event):
        self.player.command("mouse",
                            event.x(), event.y(),
                            self.__qt_to_mpv_mouse_button(event.button()),
                            'double')

    def mousePressEvent(self, event):
        self.player.command("mouse",
                            event.x(), event.y(),
                            self.__qt_to_mpv_mouse_button(event.button()),
                            'single')

    def mouseMoveEvent(self, event):
        self.player.command('mouse', event.x(), event.y())

    def wheelEvent(self, event: QWheelEvent):
        delta_y = event.angleDelta().y()
        delta_x = event.angleDelta().x()
        if delta_y > 0:
            self.player.command('osd-msg-bar', 'seek', '10', 'relative+keyframes')
        elif delta_y < 0:
            self.player.command('osd-msg-bar', 'seek', '-10', 'relative+keyframes')
        elif delta_x > 0:
            self.player.command('osd-msg-bar', 'add', 'volume', '-2')
        elif delta_x < 0:
            self.player.command('osd-msg-bar', 'add', 'volume', '2')
        else:
            self.log.warning(f'Unknown wheel event: {event}')
        event.accept()

I even tried this variation to try to send a click and move (to try to get a click and release?):

    def mousePressEvent(self, event):
        self.player.command("mouse",
                            event.x(), event.y(),
                            self.__qt_to_mpv_mouse_button(event.button()),
                            'single')
        self.player.command("mouse",
                            event.x(), event.y())

Is sending a mouse event via the scripting API just not the same as when interacting with the process itself? I'm digging through the other plugins and am still not entirely sure how mp.set_mouse_area(... works. Is the final argument a namespace? Do I have to send events specifically to that namespace somehow? Is there some way, via the API, to trigger 'osc/__keybinding5' on input followed by script-binding osc/__keybinding11 on window-controls?

Do I have to send mouse events to those namespaces and the x/y be adjusted relative to the bounding box? Is there some other way to trigger that osc/__keybinding5 and 11? Going back to my Lua script I tried mp.commandv("script-binding", "osc/__keybinding11") and several other variations and couldn't get anything on the osc to trigger.

Is what I'm trying to do even possible? I'm really lost here.

[hooke007]

It seems I cannot make argument single work as expected neither.

[SumOys]

Alright, I finally got it all working. First of all, I couldn't use the build-in osc as is. I copied the script and added in the following hook:

-- Custom for Media Hug
local function script_mouse_input(up_down)
  process_event("mbtn_left", up_down)
end
mp.register_script_message("osc-mouse-left-button-action", script_mouse_input)
-- End Customizations

I turned off the builtin osc and loaded my custom one:

        osc_script = pkg_resources.files('mediahug') / 'osc-custom.lua'
        self.log.info(f'Loading OSC Script: {osc_script}')
        self.player.command('load-script', osc_script)

    def get_mpv_player_implementation(self, current_volume, show_controls):
        return MPV(vo="libmpv",
                   log_handler=print,
                   input_default_bindings=False,
                   input_vo_keyboard=False,
                   keep_open='yes',
                   msg_level="all=v",
                   volume=current_volume,
                   osc=False)

This still didn't quite work. I did a ton of debugging, printing out the element coordinates, the hit boxes, and lots of other crap. I noticed the virtual hit coordinates were slightly off from the bounding boxes for the osc controls. I remembered my widget is currently scaled for HiDPI, as shown in my paintGL() function:

    def paintGL(self):
        if self._ctx:
            rect = self.size()
            scale = self.devicePixelRatio()

            width = int(rect.width() * scale)
            height = int(rect.height() * scale)
            fbo = int(self.defaultFramebufferObject())

            self._ctx.render(
                flip_y=True,
                opengl_fbo={"w": width, "h": height, "fbo": fbo},
            )

If I turned down the scaling using the environment variable QT_SCALE_FACTOR=1, the controls worked! So I realized I needed to scale the mouse movements I passed through to the mpv process.

    def __scale_mouse_event(self, event: QMouseEvent) -> (int, int):
        scale = self.devicePixelRatio()
        return int(event.x() * scale), int(event.y() * scale)

    def mouseDoubleClickEvent(self, event: QMouseEvent):
        x, y = self.__scale_mouse_event(event)
        self.player.command('mouse', x, y,
                            self.__qt_to_mpv_mouse_button(event.button()),
                            'double')

    def mousePressEvent(self, event: QMouseEvent):
        if event.button() == Qt.MouseButton.LeftButton:
            self.player.command('script-message', 'osc-mouse-left-button-action', 'down')
        else:
            x, y = self.__scale_mouse_event(event)
            self.player.command('mouse', x, y,
                                self.__qt_to_mpv_mouse_button(event.button()),
                                'single')

    def mouseReleaseEvent(self, event):
        if event.button() == Qt.MouseButton.LeftButton:
            self.player.command('script-message', 'osc-mouse-left-button-action', 'up')

    def mouseMoveEvent(self, event: QMouseEvent):
        x, y = self.__scale_mouse_event(event)
        self.player.command('mouse', x, y)

That's for pointing out the lua script source @hooke007 . That helped a lot.

Here's the full commit for anyone trying to figure this out in the future:

https://gitlab.com/djsumdog/mediahug/-/commit/b370ea37d86373ec6a8afc9f8dbf1f2cb3e5615b