You can run any wlroots- (and possibly smithay-) based Wayland compositor nested inside an X11, but it won't be rootless; you'll get a separate window that all the Wayland apps will be drawn into.

There's really no reason why someone couldn't write a "WX11" or something of that nature, but I'm not sure it would be worth the effort.