A cluster of many $8000+ gpus. You're looking at around 350GB of vram, so 30 12gb gpus - a 3090 will cost around $1800, so $54k on the gpus, probably another $15k in power, cooling, and infrastructure, $5k in network, and probably another $20k in other costs to bootstrap it.
Or wait 10 years, if gpu capacity scales with Moore's law, consumer hardware should be able to run a ~400GB model locally.
One could use $4.5k RTX A6000 48Gb instead.
They can be joined in pairs of 96Gb common memory pool with NVlink.
That’s 7x$4.5=$31.5k in GPUs to get 336Gb of memory.
Or 8x$4.5=$36k in GPUs to get 384Gb of memory.
Add say $3k per GPU pair for surrounding computer (MB,CPU,RAM,PSU) 4x$3k=$12k.
This is not true. On prem is extremely common for things like this because after ~6 months you'll have paid more in cloud costs than it would have cost to purchase the GPUs. And you don't need to purchase new GPUs every 6 months.
AWS would cost $50-100k/mo for something comparable.
Or wait 10 years, if gpu capacity scales with Moore's law, consumer hardware should be able to run a ~400GB model locally.