Milk from lower yield cows doesn’t inherently taste different, there’s a wider range of breeds but you can get identical tasting milk from low yield cows.

There’s several other ways milk tends to be worse but the primary issue is simply age. It can be a long time between when a cow is milked and when you’re adding milk to your morning cereal. Which then requires various compromises in the final product.