Motivation and Context

The XSHM display source has been unable to handle screen resolution changes on the fly for over 11 years (see commit 76d2bf2): when screen resolution changes while such a source is active, the source fails to resize accordingly, resulting in incorrect output.

While this rarely affects the most typical workflows, as users seldom change physical display resolutions while recording, it becomes a more significant issue with virtual displays and multi-monitor setups. For example, virtual screens created by remote desktop servers like xrdp may be frequently resized during client window drag and resize operations.

I have identified and tried several workarounds, but none are ideal:

• The third-party obs-gstreamer plugin can delegate screen capturing to GStreamer's ximagesrc element. However, this incurs higher CPU usage than the native XSHM source.
• Manually changing any source configuration parameter triggers xshm_update, restarting the capture with the new screen geometry. But manual solutions are inconvenient, and may cause visible artifacts while being applied.
• A custom OBS plugin or WebSocket client could listen for resolution changes and automate the above workaround, but this solution is inelegant and beyond most users' reach.

Description

These changes fix the problem described above at its root by following an approach inspired by other sources and GStreamer's ximagesrc: checking for screen geometry changes on each video tick. When a change is detected, the OBS and XSHM textures are resized to match, avoiding unnecessary reinitialization steps for better performance. I didn't observe any material performance impact in the happy path of a constant screen resolution.

How Has This Been Tested?

I've built OBS 32.0.4 from source on a Debian Unstable X11 environment, both with and without this patch. Then I manually tested the two builds side by side, and confirmed that this patch fixes the intended issue without introducing any regressions. I conducted my tests under the xorgxrdp X server.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Tweak (non-breaking change to improve existing functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.