I wouldn't call that size a small power bank. That chip is in the same ballpark as gaming GPUs, and based on the VRMs in the picture it probably draws about as much power.

But as you said, the next generations are very likely to shrink (especially with them saying they want to do top of the line models in 2 generations), and with architecture improvements it could probably get much smaller.

Top of the line models will need more weights and more transistors, so the shrinking factors will be competing with growing factors, I'd expect them to keep maxing out the ASIC sizes to whatever is economically feasible.

Naturally they'll always have a big expensive SKU, but the existence of a Threadripper doesn't automatically obsolete the Ryzen 3

I’m old enough to remember your typical computer filling warehouse-sized buildings.

Nowadays, your average cellphone has more computing power than those behemoths.

I have a micro SD card with 256GB capacity, and I think they are up to 2TB. On a device the size of a fingernail.

That is all definitely amazing, but data storage is a fundamentally different process with far fewer constraints than continuous computation.

It all uses the same miniaturization techniques, though.