That's funny because for some shows with a lot of filler I do watch them at higher speeds, not movies though. I disagree fiction audiobooks at high speeds are the same as movies at high speeds though, once you acclimate to the narrator it doesn't matter whether it's fiction or not, there's no timing like there is in movies.
I've never felt the difference for books at least to be honest. As for shows, you'd be surprised, some actually get better the higher speed you watch them at, such as sitcoms with laugh tracks, where the higher speed means no pauses for the audience to then laugh, it feels more like a modern show.