We already have a standardized system - robots.txt - and AI bots are already ignoring it. Why would more standardized headers matter? Bots will ignore them just as they do today, pretend to be regular users, and get content without paying.
(A secondary thing is that AI bots have basically zero benefit for most websites, so unless you are some low-cost crappy content farm, it'll be in your interest to raise the prices to the max so the bots are simply locked out. Which will bring us back to point 1, bots ignoring the headers)
Also robots.txt is a suggestion but hashcash is enforced server side. I agree it's a tragedy people have started to completely ignore it but you can't ignore server side behavior.
Being indexed in search engines is very useful, but does not need any filtering - Google and all other major search engines respect robots.txt, use well-known UA and even publish IP address.
AI bots are not search engines, and they have no benefits for the website owners. This can be very clearly seen because they ignore robots.txt, pretend to be regular browsers, and use multiple proxies to avoid IP bans.
(A secondary thing is that AI bots have basically zero benefit for most websites, so unless you are some low-cost crappy content farm, it'll be in your interest to raise the prices to the max so the bots are simply locked out. Which will bring us back to point 1, bots ignoring the headers)