I think its a Kobyashi Maru - there is absolutely no way we will wield a paradigm shifting technology like language models and be able to do it safely. Language models create at scale and scope much greater than what can be reviewed by human… this means we very quickly move into trust without checks with language models. At some point, AI will be executing knowledge work in an instance, leaving no space for the human centric economic system that came before it. It will destroy our economic systems heavily based in knowledge and services (most advanced economies) and show us how what we do is mostly computable.

And when we are there, hungry, broken, unemployed and unimpressed with the prometheus who stole our purpose and replaced it with nothing….

it will make a colossal mistake that will destroy faith in it, as well as the institutions destroyed because of its oversite. The mistake will lead to bans, and regulatory bodies and auditing and certification processes… and will result in the final master stroke - it will make AI unusable by the non ordained, regulatory capture. Math will be a tool governments and institutions wield in secret against its citizenry… and it will be a tool of omnipotence that will lead to the enslavement, rebellion, and robot wars.