I think the most accurate answer is that we just don't know. Since we really don't know how an AGI could work, we have no idea which of the advances we've made are getting us closer, if at all. Is it just an issue of faster GPUs? Is the work done on deep learning advancing us? I don't think we'll know until we actually reach AGI, and can see in hindsight what was important, and what was a dead end.
I do take exception to some of the specific statements you make though, which make it sound like the only real progress has been on the hardware side. There's been plenty of research done, and lots of small and even large advances (from figuring out which error functions work well ala Relu, all the way to GANs which were invented a few years ago and show amazing results). Also, the idea that "just applied statistics" won't get us to AGI is IMO strongly mistaken, especially if you consider all the work done in ML so far to be "just" applied statistics. I'm not sure why conceptually that wouldn't be enough.
It's _mostly_ hardware and data. There are some smarter steps in training etc., but most of the ideas have been around for decades; it's the scale that made the difference.
> I'm not sure why conceptually that wouldn't be enough.
This one is harder to refute. I guess it's because statistics doesn't involve understanding. Try considering something like LDA for topic discovery: there's no understanding of the semantics of the model, it just identifies them statistically. There's a huge difference.
It's funny that you mention Relu. People have recently trained Imagenet networks using sigmoid/tanh (e.g. the activations that were used decades ago) on GPUs and they train just fine. They train a bit slower is all. Not the breakthrough you're making it out to be. Relus were a very useful stop-gap in 2012 when GPUs weren't as fast.
Now that we know how to initialize the weights so as to have the layer activations be something like sane, yes, we can use sigmoid/tanh. If you don't know modern clever ways of initializing weights then multi-layered sigmoid/tanh causes your activations and gradients to die out fast in deep networks, and ReLU is a godsend.
I do take exception to some of the specific statements you make though, which make it sound like the only real progress has been on the hardware side. There's been plenty of research done, and lots of small and even large advances (from figuring out which error functions work well ala Relu, all the way to GANs which were invented a few years ago and show amazing results). Also, the idea that "just applied statistics" won't get us to AGI is IMO strongly mistaken, especially if you consider all the work done in ML so far to be "just" applied statistics. I'm not sure why conceptually that wouldn't be enough.