Cast iron is a material, it's good for what it's good for but its material properties don't really come from the casting process, just the ratio of iron to carbon.

Casting has problems with thermal expansion, plenty of materials shrink significantly as they cool and complex parts cool unevenly which can cause them to break or deform.

Casting has problems with microstructure, plenty of materials, especially steels develop complex crystal structures with multiple phases of materials as they cool from liquids and even extensively in hot solid phases. It's hard to control this in a cast part.

Casting has problems with precision. The molds just can't be all that precise when in machining, a thousandth of an inch can be a relatively large distance.

However casting gets a bad reputation because most of the time you see it it's because it's actually very cheap, cheap materials, cheap process, minimal post processing. Higher cost things don't necessarily realize the savings from casting as much so they don't use it. And also a lot of higher quality materials have higher melting points which require more advanced tools to melt and handle.

Plenty of things though are cast and then machined, you notice this if you look.