If you're having trouble getting an X11 app to render correctly on Wayland and you're desperate for it to work. It might be worth considering running the application using Xwayland "rootful" mode.
You're probably more to running Xwayland "rootless" - it normally happens transparently when you open an X11 app from Wayland.
But it's possible to open a virtual X display under Wayland that can run its own X11 window manager and render your apps as if they were running on X11.
To start, create a new display on port 10
by running the following command:
Xwayland :10
Now that you have an Xwayland display opened, you'll need to start a window manager and an application on the new display. I like to do this using Openbox.
Here I've told Openbox to open on :10
and start up the Xeyes application:
DISPLAY=:10 openbox --startup xeyes
If all went well you should see that the application has opened like so:
I'm currently using this technique to run Reaper in Wayland in a bit more of a stable way. I've been running into loads of bugs with menus in audio plugins running with Wine as well as some graphical issues with the Reaper UI.
Running using a stable X11 window manager solves these problems for me. The only real tradeoff being I can't move windows to different monitors.
In the following example, I use an Openbox configuration file to make sure that Reaper is maximized with no window decoration so it doesn't feel like a window running inside a window. Check out the start-reaper script in my dotfiles for an up-to-date version.
#!/usr/bin/env bash
# Create a random file for the Openbox config to live in
file=$(mktemp)
# Choose a random port number for the X11 display to open on
display=$RANDOM
# Dump some Openbox config so that the Reaper application is maximized and doesn't have a window border when it is opened.
cat << EOT >> $file
<openbox_config>
<applications>
<application name="REAPER" title="REAPER v* - Registered to *">
<decor>no</decor>
<fullscreen>yes</fullscreen>
<maximized>yes</maximized>
</application>
</applications>
</openbox_config>
EOT
# Start the Xwayland process on the random display port
Xwayland :$display &
# Start the Jack audio server for better mic input
start-jack
# Start Reaper in Openbox under Xwayland with the provided config
DISPLAY=:$display openbox --config-file $file --startup reaper