All to do with cost. Fuel indirectly - going faster for shorter hops requires flying lower, with geometrically increasing air resistance and thus fuel expenditure. Going higher takes time on takeoff and landing, and requires pressurized cabin, oxygen etc. Making the plane heavier and costing fuel again.

So we're at a carefully calculated sweet spot. Until physics changes, we'll stay here.