It's done via hashes which is why it might need some time that contacts on your phone are recognized that they are on Signal.

Afaik, Signal servers have no information who your contacts are.