I know from practice that it takes a really really long time to train even a small nn (thousands of params) , so you'll need a lot more hardware to train one with billions... But, it's expensive to buy the hardware, not necessarily to use it. If you, for some reason, have a few hundred GPU lying around, it might be "cheap" to do the necessary training.
Now, that's not your point - cost != price. But, still...
I can't think of anyone having a few hundred GPUs around unless:
- They were into Ethereum mining and quit.
- They've already built a cluster with them (e.g. in an academic setting).
- They live in a datacenter.
- They are a total psychopath.
But even assuming one magically has all those GPUs available and ready to train, I don't want to calculate the power cost of it anyway. Unless one has access to free or extremely cheap electricity it would still be very expensive.
I know from practice that it takes a really really long time to train even a small nn (thousands of params) , so you'll need a lot more hardware to train one with billions... But, it's expensive to buy the hardware, not necessarily to use it. If you, for some reason, have a few hundred GPU lying around, it might be "cheap" to do the necessary training.
Now, that's not your point - cost != price. But, still...