> problems if you link code built with different versions
But that has nothing to do with semver.
Semver gives you information about when when you can replace one version with another version. It doesn't promise that you can mix multiple versions together.
Maybe I misunderstood "built with", because I thought you were talking about the compiler version there. I know semver is just intent, but the intent doesn't even touch mixing internal data from multiple versions.
If linking against a different version of the code breaks like that, that sounds like someone did semver wrong. If that happens a lot to you, then oh, I'm sorry about that happening.
But that has nothing to do with semver.
Semver gives you information about when when you can replace one version with another version. It doesn't promise that you can mix multiple versions together.