I worry about hyperthreading hurting worst-case latency (since a thread might be assigned to run on a virtual core which does not work as fast as expected).

I've looked for this and I see little evidence that worries me (this on Linux). The kernel seems to schedule first on the real primaries and then on the secondaries within a pair, and the kernel is also not all that shy about moving a thread to a different CPU.