There is an opportunity cost for society from every piece of land we cannot use for something new. That's reflected by the value of the land. If the value to the owner is much lower than the value to someone else, we aren't using the land efficiently.
It's also unearned income. It's ideal to tax unearned income because you don't create negative incentive for desired activities. It's in essence a tax that comes at no cost to society.
You are right, to some degree this would be taxing unrealized gains which IMO is generally bad. The important part though is that it's taxing a negative externality (the opportunity cost for society).
There is an opportunity cost for society from every piece of land we cannot use for something new. That's reflected by the value of the land. If the value to the owner is much lower than the value to someone else, we aren't using the land efficiently.
It's also unearned income. It's ideal to tax unearned income because you don't create negative incentive for desired activities. It's in essence a tax that comes at no cost to society.