In some ways, this is similar to use of a Control Net. I've been doing this same technique for a while, using only SVGs as the base image. Works well.