VLC fails to rotate vertical video from smartphone

[Botspot]

This is a standard vertical video taken from a smartphone.
https://drive.google.com/file/d/1Czpn4BOzt3GpAwljFzMN83GO2elWYce_/view?usp=sharing
I am fairly sure that these used to play correctly on VLC. It uses the right dimensions of a vertical video, but the video itself was not rotated to match, causing it to be stretched.

Expected playback result: (Playing the same video after converting it to MP4)

Actual playback result:

Command line output:

pi@raspberrypi:~ $ vlc /home/pi/Downloads/IMG_3964.MOV
VLC media player 3.0.20 Vetinari (revision 3.0.20-0-g6f0d0ab126b)
[000000558102e560] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[h264_v4l2m2m @ 0x7f68c4d120] Could not find a valid device
[h264_v4l2m2m @ 0x7f68c4d120] can't configure decoder
[0000007f68c23870] avcodec decoder error: cannot start codec (h264_v4l2m2m)
[0000007f500013f0] wl_dmabuf vout display: <<< Open: I420 1920x1090(1920x1080 @ 0,0 0/0), cfg.display: 1080x1920, source: 1920x1090(1920x1080 @ 0,0 1/1)

To reproduce: Download the file from the Google Drive link. Open it in VLC on PiOS Bookworm. Not sure how long ago this issue has existed. It may have predated the release of Bookworm.

[Botspot]

This now appears to be working.
It may have been an update that fixed this, or as a result of my switching from Wayfire to LabWC.

[jc-kynesim]

Possibly both. I did fix various scaling & rotation issues in the last patch, but they did also require the underlying systems to support the rotation.

[ykharuzhy]

Does drm_vout video driver support rotation? I have met the same issue when playing video without of X Window and it still exists at last VLC.

[popcornmix]

DRM does not have rotation support.

[jc-kynesim]

Or to be slightly more accurate (though no more useful), DRM (the API) does support rotation and so does drm_vout however our HVS DRM kernel driver doesn't support 90 or 270 degree rotation (it will do 180). This is due to the HVS not supporting it directly. MMAL/OMX did support 90 degree rotation but the extra h/w block required to do that isn't available to the HVS DRM driver.

[ykharuzhy]

As I remember, ffplay plays video in correct orientation. It rotates it by software?

[jc-kynesim]

Yes

[Geektoolkit]

If I understand this correctly, if I'm using VLC then I can't rotate video 90 or 270 after the Bullseye update? I'm running into video rotation issues when using VLC fullscreen after rotating the video in x11 using xrandr (OS is at 90, video plays at orientation 0 as if it is ignoring the rotation). I saw this in Bullseye and thought it was just bugs being worked out, but now still seeing it on Bookworm. We built a digital photo frame on top of a raspberry pi and have been stuck on Raspbian Buster due to this.

[jekhor]

@Geektoolkit rotation works in X11/Wayland session in bookworm.

[Geektoolkit]

That's not what I'm seeing since Bullseye. If you rotate the screen in x11 using xrandr so the Operating system operates in portrait, then launch a video in vlc in a window, it orients ok. But if you full screen it...it breaks rotation and the video plays at whatever would be 'normal' according to xrandr. I just did sudo apt upgrade/update and sudo apt upgrade vlc. It says I'm on the latest

vlc is already the newest version (1:3.0.21-0+rpt2+deb12u1).

Am I using it wrong, or is there a setting or something I can change there?

[Botspot]

Try arandr instead of xranrdr.

[jekhor]

That's not what I'm seeing since Bullseye. If you rotate the screen in x11 using xrandr so the Operating system operates in portrait, then launch a video in vlc in a window, it orients ok. But if you full screen it...it breaks rotation and the video plays at whatever would be 'normal' according to xrandr. I just did sudo apt upgrade/update and sudo apt upgrade vlc. It says I'm on the latest

vlc is already the newest version (1:3.0.21-0+rpt2+deb12u1).

Am I using it wrong, or is there a setting or something I can change there?

I use VLC to play videos in fullscreen mode but without of rotating entire screen by xrandr. I didn't try to set rotation explicitly but vertical-recorded videos from phone are played normally (they encoded as horizontal but have rotation set in the metadata).

[Geektoolkit]

re: Botspot - Arandr is xrandr via a gui. I get the same behavior.
re: jekhor - There's two rotations I spose, which is where the confusion comes in. I was referring to Popcornmix's comment about DRM not supporting rotation.

Here's the repro that is busted and has been since Bullseye from what I can tell:
In x11, Change the screen orientation of the OS to portrait. You can do this via xrandr.
Play a video in a window...the video should be the correct orientation
Double click on it / set it to fullscreen

Expected:
Video should maintain the correct orientatoin
Actual:
Video rotates incorrectly and the orientation faces the direction that would be used if the monitor was in landscape mode.

This is why I was asking about popcornmix's comment..is full screen using DRM and if so, is this entire scenario not possible on a pi?

[jc-kynesim]

The answer is a bit involved
As it stands DRM doesn't support h/w 90 degree rotation. Wayland (labwc/wayfire) does/should support 90 degree rotation which it will do via EGL; in fullscreen mode wayland may opt to act as a direct passthrough to DRM if it thinks that DRM can handle the required transformations directly (support for this is currently a bit patchy). In the old world MMAL did support direct 90 degree rotation (it did this by having a separate transpose pass hidden from the user) - MMAL is no more.
If you want 4kp60 output DRM is your only choice for smooth playback. DRM is also currently the only output device that support HDR.
In the current world under X VLC will attempt to use DRM for fullscreen (it used to use MMAL) as that works better if it works at all. Under Wayland it cannot make this work so fullscreen is whatever Wayland does.

So 90 degree under Wayland should work and I think it does if the video (rather than display) is rotated - I'll have a look at what happens if the display is rotated - if it only fails fullscreen then probably Wayland is forgetting that the screen is rotated when it enters that mode.