@odenskrigare,
Back to the topic of artificial intelligence, is not the entire phenomenon of consciousness opaque from a mathematical perspective? It certainly seems possible to mathematically define the physical state (including brain functions) which attains when I burn my finger, but is the final product, the predictive mathematical construct, at all relevant to the conscious experience, itself?
In order for sensations to be modelled mathematically, they must be reducible - there must be a common denominator somewhere. But are they? We are reminded of Thomas Nagel's famous question, "What is it like to be a bat?" The bat's sensory process is surely reducible to mathematical formulae, but what would a final predictive model actually tell us about the conscious experience of seeing in sonar?
Of course, this doesn't mean that artificial intelligence, or even consciousness, is difficult (nevermind impossible) to create. Consciousness is surely an emergent property of matter and if we can create consciousness with our penises, surely we can do it with our science, as well, right? But then the question becomes, even if we created a conscious machine - one with a "point of view" and experiences qualitatively like our own - how would we know that we had done so? How would we distinguish the "conscious" machine from the calculator on my desk?
The biggest obstacle to creating artificial intelligence is almost certainly not the "artificial" part of the term.