I think you are mistaken on the "Holding the phone isn't the problem;" (mental distraction seems correct though) I suggest a quick test for you: 1) Grab the edge of your display like it's the steering wheel 2) with your other hand hold your phone to your ear 3) try to check both blind spots without moving your hands

The freedom of movement is impaired with your elbow in the awkward position. Also when I do it, my elbow and hand blocks my view to the blind spot on that side.