>Moral reasoning is categorically different. It is necessarily subjective because it relies not just on an individual’s intellectual response to a problem but also their emotional one, and that emotional response is grounded in a lifetime of subjective experience. It requires having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made. Without such a history, an LLM can only rephrase expressions of moral reasoning found in its training data.

So it means we need a better harness for the agent, will it not get solved when we get to improve the simulation capability/environments?