> Anyone know what percent of questions the AI answered, vs redirected?

That's actually a very good question. The article doesn't give many details other than: "There are many questions Jill can’t handle."

For comparison, here's are the highlights of a study from 2014 on the mobile assistants (Siri, Cortana, Google): http://searchengineland.com/google-now-beats-siri-cortana-di...

Google was able to offer "enhanced" answers (i.e. more than a web search) 58% of the time. Siri was at 29% and Cortana was at 20%. Siri improved some with iOS9, but not enough to beat google: http://www.recode.net/2015/9/20/11618696/how-intelligent-is-...

From the actual study: ... we took 3086 different queries and compared them across all three platforms. These were not random queries. In fact, they were picked because we felt they were likely to trigger a knowledge panel. ( https://www.stonetemple.com/great-knowledge-box-showdown/ )

> That's actually a very good question. The article doesn't give many details other than: "There are many questions Jill can’t handle."

I emailed and asked. If I get a reply I'll post it here.

It would be nice if Jill Watson could respond to your email.

I really appreciate the enhanced answers.. "OK Google" has become my go to tool for a lot of things... "OK Google... what is 5 ounces in grams", or the inverse. It's almost invaluable, and could definitely make certain classes of applications (such as a calorie tracking app) far easier to use.

So, I'm guessing right now Jill performs about as well as a human TA who is pretty new on the job.

Nothing wrong with that, of course.

> Knowing when you don't know the answer is something that has been lacking

Definitely, and I'm not sure why that space isn't getting more attention. For RL there's KWIK (knows what it knows) http://www.research.rutgers.edu/~lihong/pub/Li08Knows.pdf [pdf]. A large class of problems (the robo TA being one of them) lend themselves very well to AI with error bounds, for lack of a better term.

That constraint alone makes the AI approx. 92.3% more wise then most humans.

Anytime I see anything AI related it seems context is the key otherwise answers are all over the place. It's the same for everyday life context seems to be the driver of humans.