GeckoView is a wrapper around Gecko to make it easy to embed it into applications. I heard it's API is similar to WebView, but it's not a complete drop-in replacement.
The royalties aren't generally a big deal, being something like 2%. It's other issues, like complexity or startup fees or flat out being unable to get a license.
VLIW in its pure form is old enough that anybody should be legally able to build an ISA based on that concept.
The Mill folks are heavily into patents for their tech: https://millcomputing.com/patents/. So besides Mill's general feasibility it depends on how they work those patents.
If you want to know for sure if Webrender is enabled you can go to about:config and check the Graphics section. If Compositing is 'Webrender' it's turned on.
Webrender (the new renderer maxyme was talking about) will move to gfx-rs later on [1]. That allows the renderer to run on Vulkan, DirectX 11 and 12, Metal and OpenGL.
It only works in Firefox Nightly. The flag exists in Beta and Stable, but since Stylo isn't compiled with these versions the flag doesn't actually do anything.
From what I can tell, Beta 56 has it (but you have to enable it), Nightly 57 has it (https://bugzilla.mozilla.org/show_bug.cgi?id=1330412 is tracking when it'll be enabled by default), and Stable 55 does not have it at all (although the pref might be there).