Data is data and cannot make value judgments so not sure how it can be racist. If the data is how racist people label things it still is not racist data it is data is perfectly valid for what it is racists. Removing the images from ImageNet seems absurd.
In practice, "data" means a set of observations collected by humans- who have inherent biases that influence the collection.
I'm not talking specifically about cultural biases like racial stereotypes etc. Confirmation bias is a thing, there's nothing stopping a researcher from making those observations that confirm their favoured theory and contradict all others.
Then of course there is sampling error. Just because you have a set of data that you collected "at random" doesn't mean that this dataset is representative of the population you are interested in. Let alone the fact that it's very hard to collect a truly random set of observations about processes that we don't understand to begin with.
The kind of data you're describing is an ideal, a principle that we all aspire to. It's far from the reality in practice.
I mean, it's very common colloquially to describe non sentient things as racist because they're based on either purposely or obliviously racist ideas and stereotypes.
Do you hear yourself ?! ImageNet was not based on either purposely or obliviously racist ideas and stereotypes. If it was, sure, I would have some patience for the claim it was racist. But it was not.
