> I was referring to AIXI as the perfect mathematical AI.
I will have to Google AIXI. A big point about
being mathematical is that that is about the only
solid way we can evaluate candidate work before running
software and, say, something like a Turing test.
Some math is most of why we know, well before any
software is written, that (1) heap sort will run
in n ln(n), (2) AVL trees find leaves in ln(n),
and (3) our calculations for navigating a space
craft to the outer planets will work. More
generally, the math is 'deductive' in a
severe and powerful sense and, thus, about the
only tool we have to know well in advance of, say,
writing a lot of software.
But math does not have 'truth' and, instead,
needs hypotheses. So, for hypotheses, for some
design for some software for AI, we need some.
Enough hypotheses are going to be a bit tough
to find. And math gives only some mathematical
conclusions, and we will need to know that these
are sufficient for AI; for that we will want,
likely need, a sufficiently clear definition
of AI, that is, something better than just an
empirical test such as a Turing test or
doing well on and IQ test. Tough challenge.
Instead of such usage of math, about all we have
in AI for a 'methodology' is, (1) here I have
some intuitive ideas I like, (2) with a lot
of money I can write the code and, maybe,
get it to run, and (3) trust me, that program
can read a plane geometry book with all the
proofs cut out and, then, fill in all the
proofs or some such. So, steps (1) and (2)
are, in the opinion of anyone else, say, DARPA,
'long shots', and (3)
will be heavily in the eye of the beholder.
The challenges of (1), (2), and (3) already
make AI an unpromising direction.
> The main recent advancement in machine
learning is deep learning.
It's advanced the state of the art
in machine vision and speech recognition quite a bit.
AI has been talking about 'deep knowledge' for a
long time. That was, say, in a program that
could diagnose car problems, 'knowledge' that
the engine connected to the transmission
connected to the drive shaft connected to the
differential connected to the rear wheels or
some such and, then, be able to use this
'knowledge' in 'reasoning' to diagnose
problems. E.g., a vibration could be
caused by worn U-joints.
When I was in AI, when I worked
in the field, there were plenty of people who
saw the importance of such 'deep knowledge'
but had next to nothing on really how to
make it real.
For 'deep learning', the last I heard, that was
tweaking the parameters 'deep' in some big
'neural network', basically a case of nonlinear
curve fitting. Somehow I just don't accept
that such a 'neural network' is nearly all that
makes a human brain work; that is, I'd expect
to see some promising 'organization'
at a higher level than just the little
elements for the nonlinear curve fitting.
E.g., for speech recognition, I believe an important
part of how humans do it is to take what they
heard, which is often quite noisy and by itself
just not nearly enough, and
compare it with what they know about the
subject under discussion and, then, based
on that 'background knowledge', correct the
noisy parts of what they heard. E.g., if
the subject is a cake recipe for a party
for six people, then it's not "a cup of salt"
but maybe a cup or two or three of flour.
If the subject is the history of US
presidents and war, then "I'll be j..."
may be LBJ and "sson" maybe "Nixon". Here
the speech recognition is heavily
from a base of 'subject understanding'.
An issue will be, how the heck does the
human brain sometimes make such
'corrections' so darned fast.
For image recognition, the situation has
to be in part similar but more so: I doubt that
we have even a shot at image recognition
without a 'prior library' of 'object
possibilities': That is, if we are
looking at an image, say, from
satellite, of some woods and looking
for a Russian tank hidden there,
then we need to know what a Russian
tank looks like so that we can guess
what a hidden Russian tank would look
like on the image so that we can, then,
look for that on the image. Here we
have to understand lighting, shadows,
what a Russian tank looks like from
various directions, etc. So, we are
using some real 'human knowledge'
of the real thing, the tank, we
are looking for.
E.g., my kitty cat has a food
tray. He knows well the difference
between that tray and everything
else that might be around it --
jug of detergent, toaster, bottle
of soda pop, decorative vase,
a kitchen timer. Then I can move
his food tray, and he doesn't get
confused at all. Net, what he is
doing with image recognition is
not just simplistic and, instead,
has within it a 'concept' of his
food tray, a concept that he
created. He's not stupid you know!
So, I begin to conclude that for
speech and image recognition, e.g.,
handwriting recognition, we need
a large 'base' of 'prior human
knowledge' about the 'subject area',
e.g., with 'concepts', etc.,
before we start. That is, we need
close to 'full, real AI' just to,
say, do well reading any handwriting.
From such considerations, I believe
we have a very long way to go.
Broadly one of my first cut guesses about
how to proceed would be to roll back
to something simpler in two respects.
First, start with brains smaller, hopefully
simpler, than those of humans.
Maybe start with a worm and work up to
a frog, bird, ..., in a few centuries, a
kitty cat! Second, start with the baby
animal and see how it learns once it
starts to as an egg, once it's born,
what it gets from its mother, etc.
So, eventually work up to software that
could start learning with just "Ma ma'
and proceed from there. But can't just
start with humans and "Ma ma" because
a human just born likely already has
somehow built in a lot that is crucial
we just don't have a clue about.
So, start with worms, frogs, birds,
etc.
Another idea for how to proceed is to
try for just simple 'cognition'
with just text and image input
and just text output. E.g., start
with something that can diagram
English sentences and move from there
to some 'understanding', e.g., have
made progress enough with
'meaning' that, e.g., know
when two sentences with quite different
words and grammar really mean essentially
the same thing and when they don't mean
the same thing report why not and be
able to revise one of the sentences so that
the two do mean the same thing.
So, here we are essentially assuming
that AI has to stand on some capabilities
with language -- do kitty cats have
an 'internal language'? Hmm ...! If
kitty cats don't have such an 'internal
language', then I am totally stuck!
Then with some text and
image input, the thing should be able
to cook up a good proof of the
Pythagorean theorem.
I can believe that
some software can diagram
English sentences or come close
to it, but that is just a tiny
start on what I am suggesting.
The real challenge, as I am guessing,
is to have the software keep track of
and manipulate 'meaning', whatever
the heck that is.
And I would anticipate
a 'bootstrap' approach: Postulate and
program something for doing such things
with meaning, 'teach' it, and then
look at the 'connections'
it has built internally, say, between
words and meaning, and also observe
that the thing appears to work well.
So, it's a 'bootstrap' because it
works without our having any
very good prior idea just why;
that is, we could not prove in
advance that it could work.
So, for kitty cat knowledge,
have it understand its environment
in terms of 'concepts' (part of 'meaning') hard, soft, strong, weak,
hot, cold, and, then, know when
it can use its claws to hold on to
a soft, strong, not too hot or too
cold surface, push out of the way
a hard, weak obstacle, etc.
Maybe some such research direction
could be made to work.
But I'm not holding my breath waiting.
Keep in mind evolution managed to make strong AI, us, through pretty much blind, random mutations, and inefficient selection.
The thing about deep learning is that it's not just nonlinear curve fitting. It learns increasingly high level features and representations of the input. Recurrent neural networks have the power of a Turing machine. And stuff like dropout are really efficient at generalization. My favorite example is word2vec. Creating a representation for every English word. Subtracting "man" from "king" and adding "woman" gives the representation for "queen".
Speech recognition is moving that way. It outputs a probability distribution of possible words, and a good language model can use that to figure out what is most likely. But even a raw deep learning net should eventually learn those relationships. Same with image recognition. I think you'd be surprised at what is currently possible.
> It learns increasingly high level features and representations of the input.
In the words of Darth Vader, impressive. In
my words, astounding. Perhaps beyond belief.
I'm thrilled if what you say is true, but I'm
tempted to offer you a great, once in a life time
deal on a bridge over the East River.
> The future looks bright.
From 'The Music Man', "I'm reticent. Yes, I'm
reticent." Might want to make sure
no one added some funny stuff to the Kool Aid!
On AI, my 'deep learning' had a good 'training
set', the world of 'expert systems'. My first
cut view was that it was 99 44/100% hype and
half of the rest polluted water. What was left
was some somewhat clever software, say, the Forgy
RETE algorithm. My views
after my first cut view was that my first
cut view was quite generous, that expert systems
filled a much need gap in the literature and would
be illuminating if ignited.
So, from
my 'training set' my Bayesian 'prior probability'
is that nearly anything about AI is at least
99 44/100% hype.
That a bunch of neural network nodes can somehow
in effect develop internally just via adjustments
in the 'weights' or whatever 'parameters' it has
just from analysis of a 'training set' images of
a Russian tank (no doubt complete with
skill at 'spacial relations' where it is claimed
that boys are better than girls) instead of somehow
just 'storing' the data on the tank separately
looks like rewiring the Intel processor
when download a new PDF file instead of just
putting the PDF file in storage. But, maybe
putting the 'recognition means' somehow
'with' the storage means is how it is actually done.
The old Darwinian guess I made was that
early on it was darned important to understand
three dimensions and paths through three dimensions.
So, going after a little animal, and it goes behind
a rock. So, there's a lot of advantage to
understanding the rock as a concept and that
can go the other way around the rock and get
the animal. But it seems that the concept of
a rock stays even outside the context of
chasing prey. So, somehow intelligence
works with concepts such a rocks and also
uses that concept for chasing prey,
turning the rock over and looking under it,
knowing that a rock is hard and dense,
etc.
Net, my view is that AI is darned hard,
so hard that MLE, neural nets, decision
theory, etc. are hardly up to the level
of even baby talk. Just my not very well
informed, intuitive, largely out of date
opinion. But, I have a good track record:
I was correct early on that expert systems are
a junk approach to AI.
There is a degree of hype. They are really good at pattern recognition, maybe even superhuman on some problems and with enough training and data. But certainly they can't "think" in a normal sense or are a magical solution to the AI problem. And like everything in AI, once you understand how it actually works, it may not seem as impressive as it did at first.
>instead of somehow
just 'storing' the data on the tank separately
looks like rewiring the Intel processor
when download a new PDF file instead of just
putting the PDF file in storage.
Good analogy, but how would you even do that? One picture of a tank isn't enough to generalize. Is a tank any image colored green? Is it any object painted camouflage? Is it any vehicle that has a tube protruding from it?
In order to learn, you need a lot of examples, and you need to test a lot of different hypotheses about what a tank is. That's a really difficult problem.
I will have to Google AIXI. A big point about being mathematical is that that is about the only solid way we can evaluate candidate work before running software and, say, something like a Turing test.
Some math is most of why we know, well before any software is written, that (1) heap sort will run in n ln(n), (2) AVL trees find leaves in ln(n), and (3) our calculations for navigating a space craft to the outer planets will work. More generally, the math is 'deductive' in a severe and powerful sense and, thus, about the only tool we have to know well in advance of, say, writing a lot of software.
But math does not have 'truth' and, instead, needs hypotheses. So, for hypotheses, for some design for some software for AI, we need some. Enough hypotheses are going to be a bit tough to find. And math gives only some mathematical conclusions, and we will need to know that these are sufficient for AI; for that we will want, likely need, a sufficiently clear definition of AI, that is, something better than just an empirical test such as a Turing test or doing well on and IQ test. Tough challenge.
Instead of such usage of math, about all we have in AI for a 'methodology' is, (1) here I have some intuitive ideas I like, (2) with a lot of money I can write the code and, maybe, get it to run, and (3) trust me, that program can read a plane geometry book with all the proofs cut out and, then, fill in all the proofs or some such. So, steps (1) and (2) are, in the opinion of anyone else, say, DARPA, 'long shots', and (3) will be heavily in the eye of the beholder. The challenges of (1), (2), and (3) already make AI an unpromising direction.
> The main recent advancement in machine learning is deep learning. It's advanced the state of the art in machine vision and speech recognition quite a bit.
AI has been talking about 'deep knowledge' for a long time. That was, say, in a program that could diagnose car problems, 'knowledge' that the engine connected to the transmission connected to the drive shaft connected to the differential connected to the rear wheels or some such and, then, be able to use this 'knowledge' in 'reasoning' to diagnose problems. E.g., a vibration could be caused by worn U-joints. When I was in AI, when I worked in the field, there were plenty of people who saw the importance of such 'deep knowledge' but had next to nothing on really how to make it real.
For 'deep learning', the last I heard, that was tweaking the parameters 'deep' in some big 'neural network', basically a case of nonlinear curve fitting. Somehow I just don't accept that such a 'neural network' is nearly all that makes a human brain work; that is, I'd expect to see some promising 'organization' at a higher level than just the little elements for the nonlinear curve fitting.
E.g., for speech recognition, I believe an important part of how humans do it is to take what they heard, which is often quite noisy and by itself just not nearly enough, and compare it with what they know about the subject under discussion and, then, based on that 'background knowledge', correct the noisy parts of what they heard. E.g., if the subject is a cake recipe for a party for six people, then it's not "a cup of salt" but maybe a cup or two or three of flour. If the subject is the history of US presidents and war, then "I'll be j..." may be LBJ and "sson" maybe "Nixon". Here the speech recognition is heavily from a base of 'subject understanding'. An issue will be, how the heck does the human brain sometimes make such 'corrections' so darned fast.
For image recognition, the situation has to be in part similar but more so: I doubt that we have even a shot at image recognition without a 'prior library' of 'object possibilities': That is, if we are looking at an image, say, from satellite, of some woods and looking for a Russian tank hidden there, then we need to know what a Russian tank looks like so that we can guess what a hidden Russian tank would look like on the image so that we can, then, look for that on the image. Here we have to understand lighting, shadows, what a Russian tank looks like from various directions, etc. So, we are using some real 'human knowledge' of the real thing, the tank, we are looking for.
E.g., my kitty cat has a food tray. He knows well the difference between that tray and everything else that might be around it -- jug of detergent, toaster, bottle of soda pop, decorative vase, a kitchen timer. Then I can move his food tray, and he doesn't get confused at all. Net, what he is doing with image recognition is not just simplistic and, instead, has within it a 'concept' of his food tray, a concept that he created. He's not stupid you know!
So, I begin to conclude that for speech and image recognition, e.g., handwriting recognition, we need a large 'base' of 'prior human knowledge' about the 'subject area', e.g., with 'concepts', etc., before we start. That is, we need close to 'full, real AI' just to, say, do well reading any handwriting. From such considerations, I believe we have a very long way to go.
Broadly one of my first cut guesses about how to proceed would be to roll back to something simpler in two respects. First, start with brains smaller, hopefully simpler, than those of humans. Maybe start with a worm and work up to a frog, bird, ..., in a few centuries, a kitty cat! Second, start with the baby animal and see how it learns once it starts to as an egg, once it's born, what it gets from its mother, etc. So, eventually work up to software that could start learning with just "Ma ma' and proceed from there. But can't just start with humans and "Ma ma" because a human just born likely already has somehow built in a lot that is crucial we just don't have a clue about. So, start with worms, frogs, birds, etc.
Another idea for how to proceed is to try for just simple 'cognition' with just text and image input and just text output. E.g., start with something that can diagram English sentences and move from there to some 'understanding', e.g., have made progress enough with 'meaning' that, e.g., know when two sentences with quite different words and grammar really mean essentially the same thing and when they don't mean the same thing report why not and be able to revise one of the sentences so that the two do mean the same thing. So, here we are essentially assuming that AI has to stand on some capabilities with language -- do kitty cats have an 'internal language'? Hmm ...! If kitty cats don't have such an 'internal language', then I am totally stuck!
Then with some text and image input, the thing should be able to cook up a good proof of the Pythagorean theorem.
I can believe that some software can diagram English sentences or come close to it, but that is just a tiny start on what I am suggesting. The real challenge, as I am guessing, is to have the software keep track of and manipulate 'meaning', whatever the heck that is.
And I would anticipate a 'bootstrap' approach: Postulate and program something for doing such things with meaning, 'teach' it, and then look at the 'connections' it has built internally, say, between words and meaning, and also observe that the thing appears to work well. So, it's a 'bootstrap' because it works without our having any very good prior idea just why; that is, we could not prove in advance that it could work.
So, for kitty cat knowledge, have it understand its environment in terms of 'concepts' (part of 'meaning') hard, soft, strong, weak, hot, cold, and, then, know when it can use its claws to hold on to a soft, strong, not too hot or too cold surface, push out of the way a hard, weak obstacle, etc.
Maybe some such research direction could be made to work. But I'm not holding my breath waiting.