I've also switched to Instant Space Switcher, it is soo good! Previously I used BetterMouse for only this feature but they made the space switching worse in later versions (slower, on-par with the default macOS speed).

Here's the link if anyone is curious: https://github.com/jurplel/InstantSpaceSwitcher

I also used to use BetterTouch tool just for this feature, no idea what they have been thinking over at Apple with this delay.