You don't need to restrict yourself to mathematically-nice algebras of operations or whatever if you're doing 3d graphics though. If you want to represent a translation of x by vector v in 3D, just do x+v.
Computable functions have that same property, and that's the space that programs work in. It doesn't matter if the composition elements are linear, only that they can be computed on your hardware within some time budget.
