It all adds up. All those instructions you don't have to execute, especially memory access and cache misses from jumps, pipeline stalls from conditionals, not just from this optimization.