Don't put the world state into the model. Use the model as a renderer of whatever objects the "engine" throws at it.
Use the CPU and RAM for world state, then pass it off to the model to render.
Regardless of how this is done, Unreal Engine with all of its bells and whistles is toast. That C++ pile of engineering won't outdo something this flexible.
How many watts and how much capital does it take to run this model? How many watts and how much capital does it take to run unity or unreal? I suspect there's a huge discrepancy here, among other things.
& you're basically seeing GPT-3 and saying it will never be used in any serious application.. the rate of improvement in their model is insane