Btw, every time I read about "we are just at the beginning, and we already think we know all about that" I have a strange feeling. Basically nobody can tell. The contrary happened a lot of times, think about AI, it basically stopped to produce significant results for decades. The only way to know is to invest more work in the field.
And if we want to remember the advice in "you and your research" what really matters is to understand in time fields where it's worth to spend time and investigate more.
Maybe it's obvious just to me? There will be full AI. We will create artificial brains. We will fully understand how the parallel system in the brain works. We will create brains more powerful than ours.
The only claim I strongly disagree with is that "we will fully understand how the parallel system in the brains works". How can you be sure? If understanding is made of abstraction, what if the brain is computationally irreducible? Our understanding will increase of course, but at some point the "explanation" of some component will be equivalent to describing all the neurons in that component and their connections.
Obviously we're discussing faith and intuition here, as none of us has the facts. I don't think there is anything computationally irreducible in the brain.
About some component not being expressible as the result of simpler computations: that has to happen at some point, by definition. But that point is probably within the reach of our understanding.
All in all, I don't think there is much magic, just lack of knowledge on our part. Of course, it's just my intuition.
