# Watson the Computer on Jeopardy



## Hcour (Dec 24, 2007)

A computer named Watson will be a contestant on Jeopardy this week for three nights, Mon - Wed. This was the subject of a recent Nova and the computer actually had to audition just like any potential contestant. Watson's fellow contestants will be all-time champs Ken Jennings and Brad Rutter.


----------



## drumorgan (Jan 11, 2003)

Pretty cool.
http://www.eetimes.com/electronics-news/4212844/IBM-s-Watson-computer-beats-humans-at-Jeopardy

Now, show me the computer that can win at Wipeout.


----------



## Turtleboy (Mar 24, 2001)

Here is a bit of the practice match.


----------



## drumorgan (Jan 11, 2003)

Does it have all the data stored or is it accessing the Internet?


----------



## Hcour (Dec 24, 2007)

drumorgan said:


> Does it have all the data stored or is it accessing the Internet?


Stored, no internet access.


----------



## Turtleboy (Mar 24, 2001)

drumorgan said:


> Does it have all the data stored or is it accessing the Internet?


It has all of the data. It has to compete and buzz in too. I think that the questions are pre-loaded on a segregated portion of the program (as it can't see or hear), and it gets access to the questions at the same time the host starts reading it.

It has to figure out the answer in about 3-4 seconds, and buzz in first.


----------



## drumorgan (Jan 11, 2003)

That was my next question. I didn't know if it used OCR or voice recognition.


----------



## Combat Medic (Sep 6, 2001)

Turtleboy said:


> It has all of the data. It has to compete and buzz in too. I think that the questions are pre-loaded on a segregated portion of the program (as it can't see or hear), and it gets access to the questions at the same time the host starts reading it.
> 
> It has to figure out the answer in about 3-4 seconds, and buzz in first.


As I understand it, it is doing speech recognition as well as figuring out the puns in the questions on the fly.


----------



## Turtleboy (Mar 24, 2001)

It's not doing speech recognition. It's being fed the questions electronically.


----------



## Jesda (Feb 12, 2005)

So its basically a giant google. meh.


----------



## Ereth (Jun 16, 2000)

Jesda said:


> So its basically a giant google. meh.


The big deal is the natural language. It has to understand the format of the Jeopardy answer and parse that into a search.

Type a Jeopardy answer into Google and unless that episode has aired, and thus has been indexed, you won't get a direct question for it.


----------



## drumorgan (Jan 11, 2003)

drumorgan said:


> What is Google?


I see what you did there.


----------



## waruwaru (Mar 2, 2002)

There was a Nova special on this
http://www.pbs.org/wgbh/nova/tech/smartest-machine-on-earth.html


----------



## Neenahboy (Apr 8, 2004)

I've been looking forward to this since it was announced. Can't wait.


----------



## busyba (Feb 5, 2003)

I wonder if they programmed in a slight delay for the "buzz in" procedure to simulate a human's reaction time to the light that let's the players know when their clickers go active. If not, that would be a pretty big advantage over typical human players, since Watson could buzz in the millisecond the clickers went active and beat the other players to the punch.

During Ken Jennings run, one of the things that started to work additionally to his favor was the fact that after so many games, he had developed a very good feel for the rythym and timing of the clicker activation, so he was able to buzz in more effectively than his opponents.


----------



## andyw715 (Jul 31, 2007)

One contributing factor to ken's success is that he took the chance and buzzed in as soon as he could, before he could even process the answer.


----------



## omnibus (Sep 25, 2001)

I believe the clickers are active only after the moderator finishes reading the jeopardy answer. true?


----------



## Doh (May 18, 2001)

busyba said:


> I wonder if they programmed in a slight delay for the "buzz in" procedure to simulate a human's reaction time to the light that let's the players know when their clickers go active. If not, that would be a pretty big advantage over typical human players, since Watson could buzz in the millisecond the clickers went active and beat the other players to the punch.


I have been wondering this too because I agree it would be a big advantage.

IIRC not only can you not buzz in before the question is done, but if you buzz in a little early you are locked out from buzzing in again for a second or two.


----------



## Turtleboy (Mar 24, 2001)

Doh said:


> I have been wondering this too because I agree it would be a big advantage.
> 
> IIRC not only can you not buzz in before the question is done, but if you buzz in a little early you are locked out from buzzing in again for a second or two.


I saw a video in which it showed the computer doing a complex calcuation as to whether or not to buzz in at all. It has to be "certain" of the answer by a certain percent before buzzing in. Plus, that number will change based upon its relative position in the game. If it's way ahead, then it will need to be more certain to buzz in, because it's not necessary to risk the money. If it's behind, it will be less certain because it needs to catch up.

Watson not only knows how to find the answer, but to engage in strategy.


----------



## wedgecon (Dec 28, 2002)

Let me be the first to welcome Watson as our new Over...woops wrong forum...


----------



## pjenkins (Mar 8, 1999)

Turtleboy said:


> I saw a video in which it showed the computer doing a complex calcuation as to whether or not to buzz in at all. It has to be "certain" of the answer by a certain percent before buzzing in. Plus, that number will change based upon its relative position in the game. If it's way ahead, then it will need to be more certain to buzz in, because it's not necessary to risk the money. If it's behind, it will be less certain because it needs to catch up.
> 
> Watson not only knows how to find the answer, but to engage in strategy.


truthfully, that's probably the easiest of the code they had to write for Watson. The ability to understand natural language / approach to answering the question, access that much data, and do it fast enough to answer is a massively complex problem. Strategy as you outline is pretty trivial, in comparison...


----------



## Mike_TV (Jan 10, 2002)

The code for Watson contains hundred of different algorithms to figure out the Jeopardy questions. Each question or "answer" is scored based on probability and then Watson selects the best one in a few seconds. Also this is running Linux and has 15TB of RAM while the data set that Watson is fed is under 1TB. You can read more / watch videos here...

http://ibm.com/watson

and a little more technical details here...

https://www.ibm.com/developerworks/...hat_is_the_smartest_machine_on_earth7?lang=en

Happy 100th Birthday to IBM (this June)! Not many technology companies are a hundred years old.


----------



## Turtleboy (Mar 24, 2001)

Lots of good info here.

http://www-03.ibm.com/innovation/us/watson/what-is-watson/index.html


----------



## Michael S (Jan 12, 2004)

Now can Watson get a question wrong? It would sound boring if he got every question right.


----------



## LoadStar (Jul 24, 2001)

Michael S said:


> Now can Watson get a question wrong? It would sound boring if he got every question right.


I would imagine it could, if it misinterpreted the question.


----------



## MikeMar (Jan 7, 2005)

Or as Turtleboy said about the "certainty" part, it could calculate it has a 90&#37; chance of getting it right and it's worth the risk. So could get the 10% wrong there.


----------



## pjenkins (Mar 8, 1999)

LoadStar said:


> I would imagine it could, if it misinterpreted the question.


or, if it were behind and been programmed to lower the thresholds and/or buzz in while still formulating the answer... imagine if it buzzed in then went blank like many constestats! "um, watson??". "sorry Alex, I dunno!"..


----------



## Turtleboy (Mar 24, 2001)

Michael S said:


> Now can Watson get a question wrong? It would sound boring if he got every question right.


I'm sure it can. I hope they throw a lot of tricky wordplay categories at it, like "Before and After," where an correct answer is something like "Who is Babe Ruth Bader Ginsberg?"


----------



## Turtleboy (Mar 24, 2001)

Michael S said:


> Now can Watson get a question wrong? It would sound boring if he got every question right.


I saw an older video in which the clue was, "In REM's _It's the End of the World as We Know It_ two of the three people named with the initials L.B." and Watson said, "What is 'I feel fine.'"


----------



## madscientist (Nov 12, 2003)

I've seen video of test rounds where Watson was really, really wrong. So yes, it can get things wrong. I agree with the above: the ability to parse the question (even without speech recognition, Watson still has to understand the exact same question as the other contestants) is really amazing. It's funny because for people the hard part is knowing the answer. For Watson, it knows the answers always since it knows so many facts, but the hard part is the interpretation of the question. Just shows you how very different computers are than people, and how complementary they should be to each other.


----------



## busyba (Feb 5, 2003)

omnibus said:


> I believe the clickers are active only after the moderator finishes reading the jeopardy answer. true?


Yes. There's actually a light on each contestant's podium that turns on to let them know when they can buzz in, and as Doh said, buzzing early results in a brief lock-out penalty.

Typically, for a human, there's a minimum amount of time that simply must pass between the person seeing the light go on and reacting to it and buzzing in.

Absent a forced delay, that amount of time is virtually zero for a computer.


----------



## tlrowley (Jun 10, 2004)

waruwaru said:


> There was a Nova special on this


Thanks for this, I enjoyed it.


----------



## Fool Me Twice (Jul 6, 2004)

Do they use LEDs on the contestant panels now? I suppose those would be much faster. Would using some sort of optical coupler slow Watson down enough to be relevant? Instinctually I doubt it, but I don't know the timing of that sort of thing.

And would it be pedantic to point out that Watson doesn't understand or know anything? And you can't even call it right or wrong, except in the way you might say a light switch is wrong if the light doesn't go on when you flip it up. It's probably beside the point, but I felt like saying it on behalf of our species.


----------



## trainman (Jan 29, 2001)

busyba said:


> Yes. There's actually a light on each contestant's podium that turns on to let them know when they can buzz in, and as Doh said, buzzing early results in a brief lock-out penalty.
> 
> Typically, for a human, there's a minimum amount of time that simply must pass between the person seeing the light go on and reacting to it and buzzing in.


The "okay to buzz in" lights are actually in a strip around the game board, not on the contestants' podiums. It has been seen on TV, but it's very rare -- it takes a very short clue that doesn't give the director very much time to switch from the clue to the 3-shot of the contestants.

Most good contestants, as I understand it, don't really pay attention to the lights; instead, they go off of Alex's voice and "anticipate" the instant the system will be unlocked when he finishes reading a clue. (The unlocking is done manually by a production assistant.)


----------



## DancnDude (Feb 7, 2001)

I watched a bunch of those videos posted and I find this completely amazing. Even if Watson doesn't win, it's still a very impressive machine. I'm looking forward to watching the match.


----------



## LoadStar (Jul 24, 2001)

Turtleboy said:


> Here is a bit of the practice match.


And here's Engadget's video of the same match, including a couple of shots showing a little bit of how Watson "thinks":


----------



## d-dub (Mar 8, 2005)

pjenkins said:


> truthfully, that's probably the easiest of the code they had to write for Watson. The ability to understand natural language / approach to answering the question, access that much data, and do it fast enough to answer is a massively complex problem. Strategy as you outline is pretty trivial, in comparison...


It's not just understanding the "answer" that's hard. Some of them rely on things like sarcasm, double entendre, etc. And the "questions" that the contestant must supply sometimes must rhyme, or requires some other specific language construct.

Jeopardy was chosen because it is hard, and requires the computer to do more than simply regurgitate factual data.


----------



## Neenahboy (Apr 8, 2004)

LoadStar said:


> And here's Engadget's video of the same match, including a couple of shots showing a little bit of how Watson "thinks":


I really wish they had focused on the Watson analysis screen for the duration; it seemed to struggle in the "'M.C.' 5" category, and I'm curious whether that was due to buzzing late or an algorithmic flaw that couldn't push it past 50% certainty.


----------



## stalemate (Aug 21, 2005)

In the NOVA special, it was neat to see Watson learn within one category. The category was something where the contestants had to answer with a month. For the first question, Watson answered "what is holiday" or something like that, but as the category played out it learned from the other contestants that the correct answer should be a month and got the last one right.


----------



## stalemate (Aug 21, 2005)

Neenahboy said:


> I really wish they had focused on the Watson analysis screen for the duration; it seemed to struggle in the "'M.C.' 5" category, and I'm curious whether that was due to buzzing late or an algorithmic flaw that couldn't push it past 50% certainty.


It buzzed in and answered two of the questions and on another one it had the right answer with pretty high certainty but Ken beat it to the buzz.


----------



## madscientist (Nov 12, 2003)

It looked like Watson had the right answer for just about every question (at least those where they showed its decision screen), but decided to not buzz in on answers it was less sure of. I'm sure that's related to the strategy algorithm we already discussed: at the time it was winning by almost double against the nearest human. Why risk it on a 60&#37; probability?

The interesting thing about the "M.C." category is that Watson seemed to understand that it should only answer names that matched those initials... all the answers it considered, even the ones that were very low probability, were "M.C." answers. That's pretty impressive, IMO. Obviously it is knowledge gleaned from lots of old Jeopardy shows but to me that's completely legitimate. Think how lost people new to crossword puzzles are until they learn the "code" that so many puzzle creators use.


----------



## DancnDude (Feb 7, 2001)

Based on one of the other videos, it looks like the vertical bar on the analysis screen shows the threshold for Watson buzzing in. If Watson is at least as confident about the answer as that bar, then it will buzz in. When Watson is winning, the bar moves higher so it will only buzz in when it is more confident. Not sure why it didn't buzz in on some of those unless it was just beat by one of the other guys.

It's interesting to me that they are talking about using Watson for things like medical diagnoses. The fact that it could read every journal available and compile that data as well as xrays, doctor's notes and other things to give advice would be a huge advice. It could make sure it was very confident in any answer or diagnosis before giving it, and then doctors could use that to supplement their own diagnosis or other things to look for. And that is learns and gets better as it gets more data.


----------



## Worf (Sep 15, 2000)

Aww, no one's said it yet?

We're watching the birth of Skynet, Feb 14-16, 2011.

After all, if Watson gets the nuances of speech (which Jeopardy plays on a lot), we're all doomed!

Yes, it's impressive, but also a bit scary when you realize the computer is so close to actually recognizing natural language with all its idioms and metaphors and "get it" in a very emotionless way.

Still cool, though, and I have the NOVA episode KUID and the Jeopardy ones also.


----------



## pdhenry (Feb 28, 2005)

madscientist said:


> The interesting thing about the "M.C." category is that Watson seemed to understand that it should only answer names that matched those initials... all the answers it considered, even the ones that were very low probability, were "M.C." answers. That's pretty impressive, IMO. Obviously it is knowledge gleaned from lots of old Jeopardy shows but to me that's completely legitimate. Think how lost people new to crossword puzzles are until they learn the "code" that so many puzzle creators use.


It's a basic rule of Jeopardy that the quoted word/phrase/letters in a category title must be part of the answer. Since Watson has been programmed to play Jeopardy it doesn't seem all that extraordinary.


----------



## Legion (Aug 24, 2005)

busyba said:


> Yes. There's actually a light on each contestant's podium that turns on to let them know when they can buzz in, and as Doh said, buzzing early results in a brief lock-out penalty.


True, except when I was on the show the light was around the monitor of the active question.

Hardest part was syncing up the brain, to the eye, to the hand. In that order.


----------



## Turtleboy (Mar 24, 2001)

I agree that Watson's biggest advantage is going to be its ability to buzz in. It never gets tired. It never mis-presses the buzzer. I assume it's timing is almost perfect.


----------



## d-dub (Mar 8, 2005)

madscientist said:


> I've seen video of test rounds where Watson was really, really wrong. So yes, it can get things wrong. I agree with the above: the ability to parse the question (even without speech recognition, Watson still has to understand the exact same question as the other contestants) is really amazing. It's funny because for people the hard part is knowing the answer. For Watson, it knows the answers always since it knows so many facts, but the hard part is the interpretation of the question. Just shows you how very different computers are than people, and how complementary they should be to each other.


Watson doesn't always "know" the answer, since that would imply that every fact known to mankind has been entered into his data store. In fact, Watson only knows the data that has been programmed in.

On top of that, answering questions on Jeopardy (more correctly, posing the question to the given answer) requires inferring connections between different facts in an indirect manner, as well as forming the question in a particular way that may not be related to the question at all, but in a way that may be implied by the category name.


----------



## mmilton80 (Jul 28, 2005)

This is how the robot apocalypse begins. First they come for our game shows, next our reality shows, and pretty soon our day time soaps. 

I am rooting for the two humans because they are our first line of defense.


----------



## andyw715 (Jul 31, 2007)

pjenkins said:


> or, if it were behind and been programmed to lower the thresholds and/or buzz in while still formulating the answer... imagine if it buzzed in then went blank like many constestats! "um, watson??". "sorry Alex, I dunno!"..


This is exactly what Ken was describing. He buzzes in even before knowing the answer, relying on himself to know the correct question to ask.

Many people would hear the answer, formulate the question, then buzz in. He takes the chance and buzzes in first. Of course this depends on category knowledge as well.


----------



## WhiskeyTango (Sep 20, 2006)

I saw Ken on the news this morning. He said in three 'practice games', Watson and each of the humans each won one game a piece. He also said Watson confused Jamie Fox with Beethoven because it didn't understand the answer.


----------



## Rob Helmerichs (Oct 17, 2000)

WhiskeyTango said:


> He also said Watson confused Jamie Fox with Beethoven because it didn't understand the answer.


I wonder if Jamie Fox has ever been confused with Beethoven before?


----------



## Gregor (Feb 18, 2002)

Spoilered - hasn't aired anywhere but the eastern US...



Spoiler



A very 60's vibe to the whole thing...reminded me of when I was a kid and the technology shows were on sometimes on Sunday night.

Interesting problem, seems to work well most the time...


----------



## spikedavis (Nov 23, 2003)

I, for one, welcome our new robot overlords.


----------



## MikeAndrews (Jan 17, 2002)

andyw715 said:


> This is exactly what Ken was describing. He buzzes in even before knowing the answer, relying on himself to know the correct question to ask.
> 
> Many people would hear the answer, formulate the question, then buzz in. He takes the chance and buzzes in first. Of course this depends on category knowledge as well.


You could see that Ken did that once, starting out his question with "Let's see...."


Gregor said:


> Spoilered - hasn't aired anywhere but the eastern US...
> 
> 
> 
> ...





Spoiler



Having spent the last few days in frustration trying to figure out why my Mac stalls....Watson looked unstoppable until the end of the second segment when it went loopy. Trebeck even said it's known to do that.

I wonder if it overheated. 

Another thing is that Watson *can't hear,* so one fail was when Watson gave the same wrong answer that Ken did. Not that it would be fast enough but they don't have a real-time closed captioning feed for Watson!


I'm very impressed with the technology (I buy IBM P6 servers as part of my day job. They showed that Watson is a compute cluster of P7 servers.)

I was rooting for the humans!

They have the nerve to play regular commercials when the whole show is an infomercial for IBM?


----------



## Gregor (Feb 18, 2002)

Spoiler



I was pretty stunned that Watson built a big lead up early, then gave some of it back later. I would be very curious to see how Watson picks the next question. You could see some of it when Watson went immediately for an $800 question, which turned out to be the Daily Double.


----------



## heyitscory (Apr 6, 2004)

Anywhere I could find this episode if I missed it?


----------



## mattack (Apr 9, 2001)

pjenkins said:


> or, if it were behind and been programmed to lower the thresholds and/or buzz in while still formulating the answer... imagine if it buzzed in then went blank like many constestats! "um, watson??". "sorry Alex, I dunno!"..


On the Nova episode, they did ask for more detail on one question, and Watson said something like:

I'm sorry, the I only know "What is whatever?"


----------



## busyba (Feb 5, 2003)

Interesting counter-strategy (or at least I assume that's what he was doing) by Ken...



Spoiler



One of Watson's paradigms is to use information gleaned from previous questions in a category to refine its knowledge base for use in future questions of the same category. Ken started going after the $1000 questions in a category first, which I guess was his way of getting the most money into play before Watson has gotten around to increasing its advantage with the extra data. Not unlike maximizing your bet in blackjack when the count is in your favor.


----------



## Unbeliever (Feb 3, 2001)

I'm actually less impressed with the NLP than I am with the database performance, of which I am quite impressed with.

--Carlos V.


----------



## Thom (Jun 5, 2000)

The NOVA show on Watson's game testing during development showed a human giving the wrong answer, then Watson grabbing the second chance and giving the same wrong answer. Watson's developers then put in a fix to give Watson the known incorrect prior answer if they occur. Subsequent game testing showed the fix worked. In a situation where Watson would have given the same wrong answer, but rang in too slowly, Watson rang in on the second chance and used its secondary answer, which was correct. This was shown on the NOVA broadcast. I was surprised this didn't happen on the actual game broadcast. Either they pulled this feature, or they didn't have enough time to enter the wrong answer prior to the second chance occurring. I noticed that there was very little time between the first answer being declared wrong and Watson immediately ringing in to grab the second chance and giving the same wrong answer. I hope they fix/allow this feature before continuing with these games. Allowing the computer to know any prior wrong answers puts it on an equal footing with human players.


----------



## Wil (Sep 27, 2002)

Turtleboy said:


> I agree that Watson's biggest advantage is going to be its ability to buzz in. It never gets tired. It never mis-presses the buzzer.


That, and knowing the lyrics to every popular song in history, and all famous quotes as well; the answer is either going to be the work title or the person, pretty much an electronic slam dunk as far as understanding which the "answer" is aimed at. I think Mr. IBM-founder-named will rarely lose on one of those.


----------



## TriBruin (Dec 10, 2003)

Thom said:


> The NOVA show on Watson's game testing during development showed a human giving the wrong answer, then Watson grabbing the second chance and giving the same wrong answer. Watson's developers then put in a fix to give Watson the known incorrect prior answer if they occur. Subsequent game testing showed the fix worked. In a situation where Watson would have given the same wrong answer, but rang in too slowly, Watson rang in on the second chance and used its secondary answer, which was correct. This was shown on the NOVA broadcast. I was surprised this didn't happen on the actual game broadcast. Either they pulled this feature, or they didn't have enough time to enter the wrong answer prior to the second chance occurring. I noticed that there was very little time between the first answer being declared wrong and Watson immediately ringing in to grab the second chance and giving the same wrong answer. I hope they fix/allow this feature before continuing with these games. Allowing the computer to know any prior wrong answers puts it on an equal footing with human players.


I just watched the NOVA episode on Sunday and I don't think they said that. The one big change that I heard was that the programmers were input the CORRECT answers after each question and Watson was able to learn from that. (The example was that the questions are were months and Watson finally figured that out by the last question in the category.) I don't see how anyone would have time to type the wrong answer before Watson would need to ring in.

There are still some questions I have about the whole process. They mentioned that watch has a physical mechanism to push the button, but what is triggering the timing on his button? Is it connected to the light that goes on when the contestents can ring in?

Also, I wonder what the thought process for the producers was when picking the categories and questions? Obviously they can't pick all simple fact questions, Watson would blow through those too easily. But it doesn't seem like they went too much for the "tricky" questions either.


----------



## TiVo'Brien (Feb 8, 2002)

I felt strangely threatened by Watson's success last night.


----------



## wmcbrine (Aug 2, 2003)

Fascinating so far. Watson's doing well, but it makes mistakes that no human ever would. I came into this thinking that, if a computer could win Jeopardy, it would be far more impressive than beating the world champion at chess. But now I see that Watson is still rather stupid narrowly focused.


----------



## MikeAndrews (Jan 17, 2002)

Thom said:


> The NOVA show on Watson's game testing during development showed a human giving the wrong answer, then Watson grabbing the second chance and giving the same wrong answer. Watson's developers then put in a fix to give Watson the known incorrect prior answer if they occur. Subsequent game testing showed the fix worked. ....Either they pulled this feature, or they didn't have enough time to enter the wrong answer prior to the second chance occurring. ....Allowing the computer to know any prior wrong answers puts it on an equal footing with human players.


Since Watson can't parse spoken English (and I wonder why not when Google can do it on an iPhone) it would require an operator to be a very fast typist to give the prior answer. It may be that the Jeopardy producers didn't want that channel of input open to avoid any hint that one or more smart humans were helping with answers.

I have to watch the Nova show. I wondered how Watson knows it has buzzed in first. As above I guess it can "see" the light.


----------



## Turtleboy (Mar 24, 2001)

I think any human meddling at all during the show, other than feeding it the clues at the appropriate time, would be "cheating."


----------



## wmcbrine (Aug 2, 2003)

My favorite was the one where the correct answer was "Voldemort". Watson had the answer in its "thinking box", but it was below "Harry Potter" (and it wasn't confident enough to buzz in, either way). It recognized the right overall context, but missed so much.

Before the start, I wondered how Watson might do in one of those horrible categories where you have to essentially combine two answers into a new phrase. I assumed it wouldn't do well, since this requires a kind of creativity that's different from the skill it takes to answer in most categories. I'm pretty sure now that it would be slaughtered. (I still hope one of these will come up.)


----------



## Turtleboy (Mar 24, 2001)

wmcbrine said:


> Before the start, I wondered how Watson might do in one of those horrible categories where you have to essentially combine two answers into a new phrase.


This TCF Poster starred in _The Goonies_ and _Stand By Me._


----------



## TiVo'Brien (Feb 8, 2002)

I wonder if they programmed Watson to find the Daily Double. He seemed to go right for one of the statistically most likely squares.

I like that Ken beat Watson to the clicker twice.


----------



## Bob Coxner (Dec 1, 2004)

heyitscory said:


> Anywhere I could find this episode if I missed it?


----------



## MikeMar (Jan 7, 2005)

TiVo'Brien said:


> I wonder if they programmed Watson to find the Daily Double. He seemed to go right for one of the statistically most likely squares.
> 
> I like that Ken beat Watson to the clicker twice.


That was crazy, but he did also zig-zag across categories. I wonder since Watson is a computer, it doesn't need a "rhythm" and can jump around just as easy as go down a column. It could do this to throw off the other two while giving no disadvantage to itself.


----------



## Jon J (Aug 23, 2000)

Thom said:


> The NOVA show on Watson's game testing during development showed a human giving the wrong answer, then Watson grabbing the second chance and giving the same wrong answer. Watson's developers then put in a fix to give Watson the known incorrect prior answer if they occur. Subsequent game testing showed the fix worked.


Evidently not. It happened in the first round. Watson repeated an incorrect answer.


----------



## MikeMar (Jan 7, 2005)

I can't remember the question/answer but there was one where it was pretty much a 50/50 shot at getting it

Ken guessed wrong, Watson guessed the same wrong, no clue why Brad didn't get it (I just remember it being easy)


----------



## stalemate (Aug 21, 2005)

Jon J said:


> Evidently not. It happened in the first round. Watson repeated an incorrect answer.


Or, just maybe, as stated in the part you clipped out, they didn't use the feature for some reason. The person you quoted clearly knew Watson repeated an incorrect answer, that was the whole point of the post.


----------



## jlb (Dec 13, 2001)

One question of note Watson would have gotten wrong, but didn't buzz in on time was the villian in the Potterverse. Watson was going to answer Harry Potter and not Voldemort....


----------



## That Don Guy (Mar 13, 2003)

TiVo'Brien said:


> I wonder if they programmed Watson to find the Daily Double. He seemed to go right for one of the statistically most likely squares.





Turtleboy said:


> I think any human meddling at all during the show, other than feeding it the clues at the appropriate time, would be "cheating."


I can't remember exactly where, but "I read somewhere that" (pause for everybody to reset their Urban Legend Detector alarms):
(a) whoever programmed Watson's strategy did, in fact, include an algorithm to find the most likely locations for Daily Doubles historically, and
(b) they specifically did not include the ability for Watson to be fed incorrect answers during the show; Watson has to "stand by itself" during the game. (It was mentioned that Watson could neither "hear" nor "see", so there is no way for it to gain additional information during the show, except possibly that if someone else buzzes in first and gets it wrong, Watson may decide to switch to its second choice response.)

-- Don


----------



## MikeAndrews (Jan 17, 2002)

TriBruin said:


> ...
> 
> There are still some questions I have about the whole process. They mentioned that watch has a physical mechanism to push the button, but what is triggering the timing on his button? Is it connected to the light that goes on when the contestents can ring in?
> ...


Remember that Watson gets the question in text. Watson can buzz in the millisecond it sees the NULL at the end of the text file, if it has parsed the answer by then.

The light tells it it buzzed in first so it can give the answer.


----------



## Mike_TV (Jan 10, 2002)

Ah, it was a good first round. Does anything think that Watson is throttled, so he doesn't crush the humans in the first round?


----------



## pdhenry (Feb 28, 2005)

netringer said:


> Remember that Watson gets the question in text. Watson can buzz in the millisecond it sees the NULL at the end of the text file, if it has parsed the answer by then.
> 
> The light tells it it buzzed in first so it can give the answer.


Like any player, Watson has to wait until Alex has finished reading the question. If you buzz in too early you're locked out for some period (1 or 2 seconds, IIRC). Probably what they do is wait until the Jeopardy staffer activates the visual signal that tells the players they can buzz in. I assume Watson has a voltage sensor to know when the light gets power.


----------



## Azlen (Nov 25, 2002)

TiVo'Brien said:


> I wonder if they programmed Watson to find the Daily Double. He seemed to go right for one of the statistically most likely squares.


It was done too early in the game to have much benefit. Watson only had $400 at the time and was limited to the $1000 wager. It would have been more beneficial to go for the daily double when it had built up some money first.


----------



## MNoelH (Mar 17, 2005)

Azlen said:


> It was done too early in the game to have much benefit. Watson only had $400 at the time and was limited to the $1000 wager. It would have been more beneficial to go for the daily double when it had built up some money first.


I like the strategy of burning it early to take it out of an opponent's arsenal. I often do this in Words with Friends and the TW spot. I try to use it my advantage, but if it's available and I don't have a "good" word, I still burn it with a lousy word so my opponent doesn't score big against me.


----------



## Hunter Green (Feb 22, 2002)

Turtleboy said:


> I saw an older video in which the clue was, "In REM's _It's the End of the World as We Know It_ two of the three people named with the initials L.B." and Watson said, "What is 'I feel fine.'"


...aren't there four, not three?


----------



## Turtleboy (Mar 24, 2001)

Hunter Green said:


> ...aren't there four, not three?


Yes, I guess I misremembered.


----------



## mcb08 (Mar 10, 2006)

MikeMar said:


> I can't remember the question/answer but there was one where it was pretty much a 50/50 shot at getting it
> 
> Ken guessed wrong, Watson guessed the same wrong, no clue why Brad didn't get it (I just remember it being easy)


I thought that Brad answered it correctly ("What are the 1910's?"; actually, he said "What are the 19-teens?", but that's another story )


----------



## MikeMar (Jan 7, 2005)

mcb08 said:


> I thought that Brad answered it correctly ("What are the 1910's?"; actually, he said "What are the 19-teens?", but that's another story )


it was a different category, I forget what it was though


----------



## stalemate (Aug 21, 2005)

I think that one was 1900s and he said 19-aughts or however you spell it.


----------



## mcb08 (Mar 10, 2006)

MikeMar said:


> it was a different category, I forget what it was though


The one that I'm referring to was "Name The Decade". The question was "The first modern crossword puzzle is published, and oreo cookies are introduced". Ken answered "What are the 1920's?", Watson gave the same (incorrect) answer (Alex says "No, Ken said that"), and Brad then answered correctly ("19 Teens").


----------



## mcb08 (Mar 10, 2006)

stalemate said:


> I think that one was 1900s and he said 19-aughts or however you spell it.


Same category as the other question ("Name The Decade"). The answer was "The first flight takes place at KittyHawk and baseball's first World Series is played". Brad buzzes in first and answers correctly ("What is the 19-aughts, or the 1900's?).


----------



## Legion (Aug 24, 2005)

Here is the question set from round one. It also shows who answered correctly/incorrectly for each question. Round two will be up shortly I imagine.

http://www.j-archive.com/showgame.php?game_id=3575


----------



## MikeMar (Jan 7, 2005)

Legion said:


> Here is the question set from game #1. It also shows who answered what for each question.
> 
> http://www.j-archive.com/showgame.php?game_id=3575


It was the LEG one I was thinking!!!!


Spoiler



Ken said "he only had one hand"
Watson "leg"

I thought it was painfully obvious that it was "MISSING a leg"


----------



## mcb08 (Mar 10, 2006)

That was a tough one for Watson. I imagine that it's difficult for the algorithm to distinguish between "missing a leg", and "his leg" as anatomical oddites. I think that there's an argument to be made that either one is correct. What is more surprising is that Brad didn't jump in with the correct answer!


----------



## MikeMar (Jan 7, 2005)

mcb08 said:


> That was a tough one for Watson. I imagine that it's difficult for the algorithm to distinguish between "missing a leg", and "his leg" as anatomical oddites. I think that there's an argument to be made that either one is correct. What is more surprising is that Brad didn't jump in with the correct answer!


yes, that's what I have been saying! And this was the question I was referring to, not the decade ones  I knew it wasn't that category


----------



## plateau10 (Dec 11, 2007)

Am I alone in thinking that they should have given Watson that one (and that they would have for a human)?


----------



## MikeMar (Jan 7, 2005)

plateau10 said:


> Am I alone in thinking that they should have given Watson that one (and that they would have for a human)?


no, the question was what oddity, not what was missing. could have been an extra leg, or 8 toes or something


----------



## Waldorf (Oct 4, 2002)

Huh... doesn't seem anyone has linked this yet.

https://spreadsheets.google.com/lv?hl=en&key=tth_jhM8vyBAuogqHllHmHQ

Very cool presentation of the data from the first show.

Oh and this regarding the "leg" answer...

Alex had stated Watson's answer as correct assuming he was continuing Ken's answer. Production was stopped and the points were removed. The editing of this correction was impressively seamless. This is the reason you don't see Brad attempt to answer.

Finally, Watson does not know how Jennings or Rutter answer a clue. So, Watson cannot use their responses in its accuracy assessment or to change a response it may be considering. This is why it guessed "What were the 1920s" after Ken guessed "What were the 20s".

Finally, you may note that Watson "sniped" the daily double right away.

If Watson gets to choose a category and clue, its first priority is finding any remaining of the three Daily Doubles in a game. These clues allow a contestant to wager a specific dollar amount on the clue without worry of the other two contestants buzzing in. Jennings, Rutter and Watson have a high chance to answer these correctly, so Daily Doubles provide three opportunities for a critical score boost.

The Watson Research team studied the historical distribution of Daily Doubles and found they appear most-frequently in the three bottom rows, with the fourth being the most common. Daily Doubles also most frequently appear in the first column. Watson also makes use of even more statistics to dynamically predict their location based on what has been exposed so far in a game.

Once the Daily Doubles are off the board, Watson looks for the lowest clue value in a category, for which there are still a significant number of high value clues. Lower value clues help it get the gist of a category with less risk, so that it has a better shot at the high value clues to come.

If you're more interested in how the optimal wagering algorithms work for daily doubles and final jeopardy, I have that info also.


----------



## plateau10 (Dec 11, 2007)

Waldorf said:


> Alex had stated Watson's answer as correct assuming he was continuing Ken's answer. Production was stopped and the points were removed. The editing of this correction was impressively seamless. This is the reason you don't see Brad attempt to answer.


Exactly my point. We know Watson wasn't continuing Ken's answer only because we know Watson doesn't even know Ken is there. A human giving the same answer would probably have been assumed to be doing exactly that. At the very least, they would have been given a "be more specific" prompt (which I have no idea what Watson would do with). Watson was robbed on that one.


----------



## Idearat (Nov 26, 2000)

There's a Jeopardy forum where there's a lively thread for each night's game. Here's the one from last night:
http://boards.sonypictures.com/boards/showthread.php?t=49377

A couple things explained:

The "leg" question was initially ruled OK by Alex. The judges ruled later that it was not correct and they re-shot the exchange. Brad couldn't have chimed in initially as at that time Watson had been credited with a correct response.

Watson does get "correct" answers when another player gets them, but does not get incorrect. Makes sense as the correct answer can be queued up and sent right away but an incorrect one would need to be typed or somehow added on the fly. That explains the issue with re-trying one of Ken's incorrect answer. I guess they needed to add a "don't bother because the response you're going to try first is already ruled incorrect" function.


----------



## donnoh (Mar 7, 2008)

For those of you that read Time magazine there is a time line of the predicted advancement of computers. In 2045 computers are predicted to have surpassed the entire computing power of the human population as a whole. It's a scary thought, but I won't be alive so I wish my kids well.


----------



## Rob Helmerichs (Oct 17, 2000)

donnoh said:


> For those of you that read Time magazine there is a time line of the predicted advancement of computers. In 2045 computers are predicted to have surpassed the entire computing power of the human population as a whole. It's a scary thought, but I won't be alive so I wish my kids well.


If it's any consolation, that entire article is an uncritical presentation of the rantings of that deranged idiot, Ray Kurzweil.


----------



## spear (Oct 11, 2006)

plateau10 said:


> [...]At the very least, they would have been given a "be more specific" prompt (which I have no idea what Watson would do with). Watson was robbed on that one.


During the practice games in the NOVA special, they showed a number of instances where Watson was asked to be "more specific". In all but one, Watson was able to do so. (I don't think they explained how Watson was asked -- maybe another text message or a specific signal?)


----------



## Bananfish (May 16, 2002)

plateau10 said:


> Exactly my point. We know Watson wasn't continuing Ken's answer only because we know Watson doesn't even know Ken is there. A human giving the same answer would probably have been assumed to be doing exactly that. At the very least, they would have been given a "be more specific" prompt (which I have no idea what Watson would do with). Watson was robbed on that one.


I think the way Alex would typically handle this with a human is simply to pause and give the answerer a chance to provide clarifying information (probably while glancing over at the judges) - I'm not convinced he would ask for them to be more specific, but he certainly might.


----------



## Bananfish (May 16, 2002)

Waldorf said:


> Huh... doesn't seem anyone has linked this yet.
> 
> https://spreadsheets.google.com/lv?hl=en&key=tth_jhM8vyBAuogqHllHmHQ
> 
> ...


That all makes a lot of sense.

Daily doubles are also typically in the most cerebral topics - if you're looking for a daily double and your topic choices are down to "Madonna songs" or "Molecular Biology," you can be darn sure it's in Molecular Biology. I'd be interested in knowing if Watson can somehow glean the "cerebralness" (cerebrality?) of a topic and use that information in selecting categories when looking for the daily double.

They also (almost?) never put the daily double in any of their specialty categories (e.g., Before and After), and never in any category where they give you a letter or part of the answer - you can tell these latter categories because something in the category name is in quotes (e.g., ACT YOUR "AGE" or ENDS IN "X"). I would assume that Watson knows about this because it's easy to program in - just look for a double quote or "BEFORE AND AFTER" and put that at the end of the queue.

And I think there's only been one time in the thousands of episodes of Jeopardy I've watched have I seen the daily double in the last category, because it's usually a "light" - i.e., non-cerebral category. That one time was only a few months ago, and it was pretty jarring when I saw that they had done it. I would guess Watson always tries the last category last when daily double hunting.


----------



## Hunter Green (Feb 22, 2002)

Turtleboy said:


> Yes, I guess I misremembered.


I didn't mean to be picking on you, I just wondered if Jeopardy got it wrong.


----------



## Turtleboy (Mar 24, 2001)

Hunter Green said:


> I didn't mean to be picking on you, I just wondered if Jeopardy got it wrong.


Not at all. I always forget about Lester Bangs.


----------



## Rickvz (Sep 5, 2000)

Watson certainly whipped the humans on the second night up until the Final Jeopardy question. Some odd Daily Double wagers as well. How in the world could Watson choose Toronto when the category was U.S. Cities?


----------



## Rob Helmerichs (Oct 17, 2000)

Rickvz said:


> How in the world could Watson choose Toronto when the category was U.S. Cities?


Forgetting that Canada is not a part of the US is common enough among humans...


----------



## MikeMar (Jan 7, 2005)

I wish for the daily doubles they made Watson round his bets to the nearest $100 since most humans do this.

But I guess it was the statistically the best bet


----------



## Turtleboy (Mar 24, 2001)

Interesting discussion on the final jeopardy mess up.

http://thenumerati.net/index.cfm?postID=726


----------



## busyba (Feb 5, 2003)

At the end of the second night, Alex said "we'll add these totals to the results of tomorrow's game to see who wins the million dollars."

Million dollars??? At the beginning of the first night, Alex called it an "exhibition match" of Jeopardy. I took that to mean that they weren't playing for money.

So what makes it an "exhibition" then?

And what's a computer going to do with a million dollars anyway?


----------



## MikeMar (Jan 7, 2005)

Maybe it's a million dollars for charity?


----------



## jsmeeker (Apr 2, 2001)

busyba said:


> At the end of the second night, Alex said "we'll add these totals to the results of tomorrow's game to see who wins the million dollars."
> 
> Million dollars??? At the beginning of the first night, Alex called it an "exhibition match" of Jeopardy. I took that to mean that they weren't playing for money.
> 
> ...


Buy some RAM?


----------



## busyba (Feb 5, 2003)

If Watson has an unsurmountable lead by the end of the second game, I hope he responds to Final Jeopardy, no matter what the clue, with "who are three men who have never been in my kitchen."


----------



## MikeAndrews (Jan 17, 2002)

busyba said:


> ...And what's a computer going to do with a million dollars anyway?


Pay off about 3 months of the lease on his P7-750 servers.


----------



## wmcbrine (Aug 2, 2003)

Rob Helmerichs said:


> If it's any consolation, that entire article is an uncritical presentation of the rantings of that deranged idiot, Ray Kurzweil.


Come on, he's at least a deranged genius.


----------



## gossamer88 (Jul 27, 2005)

As impressive as Watson is, he's pretty boring!


----------



## Rob Helmerichs (Oct 17, 2000)

wmcbrine said:


> Come on, he's at least a deranged genius.


Well, HE certainly thinks so.

But everything he says is idiotic. When I see a bird that walks like an idiot and swims like an idiot and quacks like an idiot, I call that bird an idiot.


----------



## Tivo_60 (Jun 13, 2003)

Sorry, but just appears to be an big adverisement for IBM to me.


----------



## mattack (Apr 9, 2001)

MikeMar said:


> Maybe it's a million dollars for charity?


IIRC, for some bizarre (ILLOGICAL) reason, the hu-mans are donating half to charity, but IBM's keeping it all if they win..


----------



## LoadStar (Jul 24, 2001)

mattack said:


> IIRC, for some bizarre (ILLOGICAL) reason, the hu-mans are donating half to charity, but IBM's keeping it all if they win..


No, IBM is donating the full amount to charity if they win. The other two will donate half to charity if they win.


----------



## Idearat (Nov 26, 2000)

30-second skip is my friend these first two days. Tomorrow will hopefully be a game with normal pacing and less infomercials.

While it's impressive that Watson is getting the answers, though with few "tricky" categories so far, it really does look like the button work is more of a deciding factor. There were some fairly easy questions I knew the answer to less then 1/4 way through Alex reading them. We know for sure that during his long run Ken would ring in before he'd figured out the answer, figuring that the odds were good he'd figure it out in time. 

With Watson getting the jump on the buzzer so often it's more about whether a solenoid can fire more quickly when given a signal than a human can press a button when they see a light.


----------



## kdmorse (Jan 29, 2001)

mattack said:


> IIRC, for some bizarre (ILLOGICAL) reason, the hu-mans are donating half to charity, but IBM's keeping it all if they win..


I do not believe that to be correct. IBM has pledged to donate all it's winnings to two charitable organizations, World Vision (Starving Children) and World Community Grid.

-Ken


----------



## Stephen Tu (May 10, 1999)

It seems like Watson has a buzzer timing advantage on Ken & Brad; I would think the humans knew most all the answers also. Maybe Jeopardy would be fairer, if all buzzes within x milli-seconds were considered simultaneous and a random contestant was picked to give their "question".


----------



## ThreeSoFar (May 24, 2002)

Or maybe make it a national level competition. You know, U.S. computing power against maybe Chinese and Russian.

Or wait....that would be an even bigger blowout. Never mind.


----------



## stalemate (Aug 21, 2005)

One of my friends complained saying that Watson's buzzer advantage is unfair.

I don't think this competition is about fairness and it shouldn't be, IMO. It is about demonstrating an achievement or a breakthrough or whatever you want to call it. The idea that this needs to be "fair" doesn't matter to me in the least.


----------



## Unbeliever (Feb 3, 2001)

Jennings was visibly frustrated on the buzzer.

They also seemed to forget rounding off to the nearest 100/1000 for their wager formula.

--Carlos V.


----------



## kdmorse (Jan 29, 2001)

stalemate said:


> One of my friends complained saying that Watson's buzzer advantage is unfair.


In a way, that itself is an achievement. We've gone from "Can it understand the question? Can it come up with the right answer? Can it possibly compete?" - straight to "It's too good at it, it needs to be handicapped on the buzzer or it's unfair".

(Am I evil for really, really hoping one of the categories tomorrow is a return of "Fark.com Headlines?")

-Ken


----------



## pjenkins (Mar 8, 1999)

Unbeliever said:


> Jennings was visibly frustrated on the buzzer.
> 
> They also seemed to forget rounding off to the nearest 100/1000 for their wager formula.
> 
> --Carlos V.


forget? that's a bug


----------



## murgatroyd (Jan 6, 2002)

Chat with Jennings:

http://live.washingtonpost.com/jeopardy-ken-jennings.html?hpid=talkbox1

Ken is a funny guy. I like his sense of humor.


----------



## pjenkins (Mar 8, 1999)

rating bump for Jeopardy 

http://voices.washingtonpost.com/tvblog/2011/02/jeopardy-debuts-computer-conte.html



> One night into his gig, Watson the Computer has handed "JEOPARDY!" it's best single-day rating in four years. Nearly 9 percent of the country's TV homes tuned in to watch the IBM computer take on Alex Trebek's questions, according to preliminary, metered-market stats. Watson's TV debut was the No. 2 rated program on ALL of television Monday, behind only CBS's red-letter "Two and a Half Men" episode, marking the network's very last original episode of that show before shutting down production while star Charlie Sheen went into rehab.


----------



## Worf (Sep 15, 2000)

What I want to know is the reasoning behind the wagers...

Alex seemed somewhat stunned at the wagers... and I don't believe there has been a rounding issue with the bets - you can, after all, bet $1 if you're not confident. 

And props to the guy that programmed the voice and the Jeopardy interface for Watson. It's almost freaky when he guessed and said "I'm not sure, but what is...". We may have hit the uncanny valley with Watson's Jeopardy playing skills - it's getting creepy.


----------



## kdmorse (Jan 29, 2001)

Worf said:


> What I want to know is the reasoning behind the wagers...


And for the final question... Did it's low wagering represent a extreme lack of confidence in the category? Or some sense that it was well enough ahead and didn't need any more money.

The former would surprise me, as I would think something as concrete as "US Cities" would be confidence inspiring. And regarding it's dollar total, is it playing to win the match (one game), or playing with a two day total in mind (in which case might it behave differently tomorrow).

-Ken


----------



## murgatroyd (Jan 6, 2002)

Watson is impressive. But I'll be really impressed when Watson can play _Only Connect_.


----------



## busyba (Feb 5, 2003)

I'm sure that Watson's strategic choices were tailored to win the two-game total.

As for the odd Daily Double wagers, there's actually some fairly thoroughly thought out game theory algorithms on optimal betting values. I think someone linked to them somewhere upthread.

Now, even if a human is familiar with the equations, they're still unlikely to do the math in their head and calculate the value down to the dollar. An approximation to the nearest hundred dollars is "optimal enough" and easier to figure out on the fly.

Watson, on the other hand, is calculating it to the dollar.


----------



## TheMerk (Feb 26, 2001)

busyba said:


> If Watson has an unsurmountable lead by the end of the second game, I hope he responds to Final Jeopardy, no matter what the clue, with "who are three men who have never been in my kitchen."


That made me chuckle. One of my all time favorite episodes of any sitcom ever.


----------



## wmcbrine (Aug 2, 2003)

stalemate said:


> One of my friends complained saying that Watson's buzzer advantage is unfair.


Watson was beaten to the buzzer several times (someone else buzzing in before it, on questions where it knew the right answer with high confidence, and would've buzzed in). It may have an advantage, but it's beatable (at least on individual questions).



Rob Helmerichs said:


> But everything [Kurzweil] says is idiotic.


Everything? Really?

Dis Singularitarianism all you want; it's a tiny part of his legacy. It's like saying Linus Pauling was a moron because of the Vitamin C thing.


----------



## Bananfish (May 16, 2002)

Unbeliever said:


> Jennings was visibly frustrated on the buzzer.


How ironic - he frustrated about 147 opponents with his ability to buzz in ahead of them. I remember one game during his streak where this other contestant kept getting more and more perturbed as Ken beat him to the punch on question after question to the point where the poor guy was just abusing that poor buzzer trying to get in ahead.

Regardless of who wins, these episodes are basically three infomercials for IBM. Jeopardy was apparently happy to allow them to get all that free advertising since they knew their ratings would go through the roof. Still, it's a little sickening.

Could IBM's lead researcher (David Ferrucci) look any more like Carlos from Desperate Housewives?!?!? Good Lord, even the way his mouth moves when he talks is identical - those guys must have been separated at birth.

I found it interesting that Watson missed such a painfully easy question/answer on Final Jeopardy.

Any reasonable human could quickly narrow down the possible answers to U.S. cities with more than one airport - which reasonably is only New York, Chicago, Washington DC and Los Angeles. (I say "reasonably" because there are a few others that could be credited with more than one, but the extra airport(s) are either not generally associated with the city or simply so unknown that they just wouldn't be the subject of a Jeopardy question.) From there it was an easy step to get Chicago.

I think Watson just didn't understand certain fundamental aspects of the "answer" - that it had to be a U.S. city and that the clue contained information about two separate airports within that city. I think Watson probably lumped all the information about those two airports into one thing it was looking for, so it didn't realize it had to look for a city with 2 separate airports that had a relationship with World War II. Given its answer of Toronto (which has one airport with a name related to a World War, and that was World War I), Watson certainly didn't understand that the information had to be related to World War II rather than World War I, which is really quite surprising, because there was really nothing tricky at all about the way the "answer" was phrased.

I would have to say that I would be pretty darn uncomfortable with a machine that displays that kind of faulty reasoning about what question it is being asked being used to make critical medical decisions, which is the use that was strongly espoused in one of the infomercial portions of the episode.


----------



## kdmorse (Jan 29, 2001)

Bananfish said:


> How ironic - he frustrated about 147 opponents with his ability to buzz in ahead of them. I remember one game during his streak where this other contestant kept getting more and more perturbed as Ken beat him to the punch on question after question to the point where the poor guy was just abusing that poor buzzer trying to get in ahead.


He specifically addressed that in the Q&A listed above, it's an amusing read.



> Q:So... Now do you know how people felt when they were competing against you?
> 
> A:This is very perceptive. Knowing lots of answers but being a millisecond slow on the buzzer is indeed very frustrating. To the 149 Ken Jennings losers back in 2004: if you are cheering for Watson right now, I forgive you.
> 
> Karma is a *****. Unless you can't say "*****" in the Washington Post, in which case karma is, uh, a fickle strumpet or something.


----------



## Dan203 (Apr 17, 2000)

Bananfish said:


> I would have to say that I would be pretty darn uncomfortable with a machine that displays that kind of faulty reasoning about what question it is being asked being used to make critical medical decisions, which is the use that was strongly espoused in one of the infomercial portions of the episode.


I don't think a computer like this would ever have the final say in any medical decision. Instead they will be used to take a list of symptoms, scans, x-rays, etc... and use them to compile a list of possible diagnoses/treatments, with percentage based confidence ratings, that a human doctor can then use to make a more informed decision. Used in this way Watson would be a medical breakthrough, offering possible diagnoses or treatments that the human doctor may have never even heard of.

Contrary to the way they're portraying it on the show, Watson isn't actually doing any real "thinking". It's simply doing high speed statistical analysis using natural language parsing. It's never going to replace your doctor, but it could be an incredible tool for him/her to use to give you better health care.

Personally I'm extremely impressed with what Watson is capable of and I think we're seeing an early glimpse of how we're all going to interface with, and use, computers in the not to distant future. :up:

Dan


----------



## Hunter Green (Feb 22, 2002)

The difference between being able to understand the puns and wordplay in a Jeopardy question, and being able to analyze symptoms and make diagnoses, is so tremendous that the mistake is comparing them at all. But if you must... the latter is, computationaly, millions of times easier.


----------



## harrinpj (Oct 29, 2004)

Hunter Green said:


> The difference between being able to understand the puns and wordplay in a Jeopardy question, and being able to analyze symptoms and make diagnoses, is so tremendous that the mistake is comparing them at all. But if you must... the latter is, computationaly, millions of times easier.


Certainly. When I think of Watson as a doctor I think of the TV show House. He could take all the symptoms and compute the most reasonable diagnosis. No interaction with the patient required. He could even use probability to determine if the patient is lying about a symptom or something else related. He could do this without any personal feelings getting in the way.


----------



## Rob Helmerichs (Oct 17, 2000)

wmcbrine said:


> Everything? Really?
> 
> Dis Singularitarianism all you want; it's a tiny part of his legacy. It's like saying Linus Pauling was a moron because of the Vitamin C thing.


Yes, really. You'll notice I said "says"; it's been many years since he's said anything borderline sane. And yes, Linus Pauling was a moron because of the Vitamin C thing. Being brilliant earlier in your life doesn't give you a pass for being a moron later.


----------



## wmcbrine (Aug 2, 2003)

Dan203 said:


> Contrary to the way they're portraying it on the show, Watson isn't actually doing any real "thinking".


I don't know... do we even know enough about how to define "thinking" to say that? It's clear that it doesn't think the way a human does, but that's as far as I'd go.


----------



## TriBruin (Dec 10, 2003)

Bananfish said:


> Regardless of who wins, these episodes are basically three infomercials for IBM. Jeopardy was apparently happy to allow them to get all that free advertising since they knew their ratings would go through the roof. Still, it's a little sickening.


I guess, I say so what. I bet that a majority of the Jeopardy audience have no idea how far computing power and algorthms have come. I doubt anyone will buy an IBM server just because they saw it on Jeopardy.



> I found it interesting that Watson missed such a painfully easy question/answer on Final Jeopardy.
> 
> Any reasonable human could quickly narrow down the possible answers to U.S. cities with more than one airport - which reasonably is only New York, Chicago, Washington DC and Los Angeles. (I say "reasonably" because there are a few others that could be credited with more than one, but the extra airport(s) are either not generally associated with the city or simply so unknown that they just wouldn't be the subject of a Jeopardy question.) From there it was an easy step to get Chicago.
> 
> ...


As soon as I saw the answer given, I knew that Watson would have a tough time with it. If you watched the Nova special, they gave a very basic overview of how Watson "thinks". It tries to parse each answer and look for specific words or phrases and then find the link between those.

This Final Jeopardy answer was difficult for Watson, becuase it had 4 different parsing to do, plus then get the correct association. Watson had to figure out the following:

World War II Hero
Airport named after above
WWII Battle
Airport named after above

And then, it had to use the Category to make the assoication of what city both airports are in.

For humans, if you know that Chicago has Midway (which most people have probably heard of the Battle of Midway) and O'Hare. Even if you don't know how O'Hare was, it wouldn't be to big of a guess to assume he is a WWII hero.


----------



## DancnDude (Feb 7, 2001)

Bananfish said:


> Could IBM's lead researcher (David Ferrucci) look any more like Carlos from Desperate Housewives?!?!? Good Lord, even the way his mouth moves when he talks is identical - those guys must have been separated at birth.


I think Brad Rudder looks a lot like Wil Wheaton. 



Bananfish said:


> I found it interesting that Watson missed such a painfully easy question/answer on Final Jeopardy.
> 
> Any reasonable human could quickly narrow down the possible answers to U.S. cities with more than one airport - which reasonably is only New York, Chicago, Washington DC and Los Angeles. (I say "reasonably" because there are a few others that could be credited with more than one, but the extra airport(s) are either not generally associated with the city or simply so unknown that they just wouldn't be the subject of a Jeopardy question.) From there it was an easy step to get Chicago.
> 
> I think Watson just didn't understand certain fundamental aspects of the "answer" - that it had to be a U.S. city and that the clue contained information about two separate airports within that city. I think Watson probably lumped all the information about those two airports into one thing it was looking for, so it didn't realize it had to look for a city with 2 separate airports that had a relationship with World War II. Given its answer of Toronto (which has one airport with a name related to a World War, and that was World War I), Watson certainly didn't understand that the information had to be related to World War II rather than World War I, which is really quite surprising, because there was really nothing tricky at all about the way the "answer" was phrased.


The link posted above gives the reasons why. I'm quoting it here because their website appears to be hit really hard:


> 1) Watson can never be sure of anything. Is it possible that the old rock star Alice Cooper is a man? If Watson finds enough evidence, it will bet on it--even though the name "Alice" is sure to create a lot of doubt. This flexibility in its thinking can save Watson from gaffes--but also lead to a few.
> 
> 2) Category titles cannot be trusted. I blogged about this earlier, in a post How Watson Thinks. It has learned through exhaustive statistical analysis that many clues do not jibe with categories. A category about US novelists, for example, can ask about J.D. Salinger's masterpiece. Catcher in the Rye is a novel, not a novelist! These things happen time and again, and Watson notices. So it pays scant attention to the categories.
> 
> ...





Bananfish said:


> I would have to say that I would be pretty darn uncomfortable with a machine that displays that kind of faulty reasoning about what question it is being asked being used to make critical medical decisions, which is the use that was strongly espoused in one of the infomercial portions of the episode.


Watson was not very confident in it's answer (only 14%) but this is the way the show works. It needs to come up with an answer for Final Jeopardy and this is why which is why it was programmed to put all those question marks after the answer. In the other rounds it would have just not buzzed in.

In medical decisions, it would only be used to help the doctor find possible diagnoses. It would have said it was only 14% confident, but the doctor could then research the condition it came up with and see if it applies. There is so much research these days that doctors just can't keep up with it all. Therefore it'd be great to have this to use as a tool in order to point doctors toward possible answers.


----------



## murgatroyd (Jan 6, 2002)

TriBruin said:


> For humans, if you know that Chicago has Midway (which most people have probably heard of the Battle of Midway) and O'Hare. Even if you don't know how O'Hare was, it wouldn't be to big of a guess to assume he is a WWII hero.


I was talking about this with a couple of people who don't fly very much, and they didn't know the answer.

To me it was trivial -- WII Battle --> Midway -- > must be Chicago. I am willing to ignore the fact that I don't know for sure if O'Hare matches the other part of the answer, and go with the answer "Chicago" because of my basic knowledge that many airports are named after people, or the place where the airport is, and that an airport named after a battle that took place somewhere else is unusual.

Watson's programming doesn't (yet) know how to weigh some bits of the answer more heavily than others, the way a human Jeopardy player could.


----------



## Bananfish (May 16, 2002)

harrinpj said:


> Certainly. When I think of Watson as a doctor I think of the TV show House. He could take all the symptoms and compute the most reasonable diagnosis. No interaction with the patient required. He could even use probability to determine if the patient is lying about a symptom or something else related. He could do this without any personal feelings getting in the way.


I'm very skeptical - Watson has been optimized for Jeopardy, which is really worlds away from medical diagnostics.

With Jeopardy, the biggest challenge is figuring out the call of the question - from there, actually answering the question is not that difficult for a computer with a monstrously large database ... just see what things are associated in your database with the call of the question and pick the thing that is most like/most often associated with what is being asked.

On the other hand, with diagnoses, the call of the question is inherent - what is the best fit medically for symptoms X, Y and Z and events A, B and C. While a database with all kinds of case histories might help answer that type of problem, the critical thinking part of the problem (could there be two problems? are these old symptoms related to the problem? could a problem with organ X manifest itself in this way?) does not seem to fit well with Watson's strengths.

Also, from what I've seen, there's little reason to believe that Watson would be good at determining whether a patient is lying about a symptom. That requires an understanding of an extraordinarily intricate subject - human motivation and behavior. That is exactly the type of skill that Watson has shown to be far far inferior to than even, say, a 5th grader.


----------



## mcb08 (Mar 10, 2006)

busyba said:


> If Watson has an unsurmountable lead by the end of the second game, I hope he responds to Final Jeopardy, no matter what the clue, with "who are three men who have never been in my kitchen."


That would be the funniest TV moment ever.


----------



## mcb08 (Mar 10, 2006)

Bananfish said:


> /snip
> Also, from what I've seen, there's little reason to believe that Watson would be good at determining whether a patient is lying about a symptom. That requires an understanding of an extraordinarily intricate subject - human motivation and behavior. That is exactly the type of skill that Watson has shown to be far far inferior to than even, say, a 5th grader.


I think we can all agree that Watson would be best utilized as an aid to a real, live doctor.


----------



## Bananfish (May 16, 2002)

DancnDude said:


> Watson was not very confident in it's answer (only 14%) but this is the way the show works. It needs to come up with an answer for Final Jeopardy and this is why which is why it was programmed to put all those question marks after the answer. In the other rounds it would have just not buzzed in.


I saw that ... but all that really says is "here's why Watson is stupid sometimes." And it's stupid in ways that to me are surprising. For instance, that article reveals that Watson understands that category titles cannot be trusted ... and rightfully so. But the truth is that sometimes category titles can be almost completely trusted, and sometimes they can't be trusted at all, and there are fairly simple ways that human contestants parse the category titles and figure out which is which. Watson doesn't seem to be able to do that at all.

For instance, anyone who has watched the show regularly understands that there is at least a 95% probability that a final jeopardy category with the title "U.S. Cities" will have an answer that is a city in the U.S. (or maybe more than one). And that probability probably goes up to at least 99% if there is no U.S. city name in the clue. It's pretty clear that Watson doesn't even try to figure out which categories can be trusted from the category name. (From the descriptions I've read, it can figure out what a category is asking for from other answers in the category, but not the category name itself. E.g., it apparently learned to a certain degree that one category called for the name of a decade once other contestants had been correct in answering that way.)



DancnDude said:


> In medical decisions, it would only be used to help the doctor find possible diagnoses. It would have said it was only 14% confident, but the doctor could then research the condition it came up with and see if it applies. There is so much research these days that doctors just can't keep up with it all. Therefore it'd be great to have this to use as a tool in order to point doctors toward possible answers.


Certainly if Watson could do that, it would be useful in medical diagnostics. But I'm not convinced that Watson wouldn't, in a difficult diagnosis situation, come up with 200 possible diagnoses, each less than 1%, requiring the doctor to research 200 different conditions. Or come up with several that are between 5 and 15%, none of which is right, and leave the one that is right down at 0.2%, thereby pushing a doctor in the wrong direction.


----------



## murgatroyd (Jan 6, 2002)

Yes, but you're forgetting that Watson is not going to be making a diagnosis alone. Watson would be a 'community memory' showing possible diagnoses with percentages of confidence for each answer. Presumably in a medical 'expert system' the computer could be prompted to display a list of journal articles which were used in coming up with the answer, or reveal other information about why it has x% confidence in its answer.

One of my hobbies is genealogy, and I have a subscription to a couple of services that allow you to search documents online (e.g. Ancestry and Footnote). Assume you can put in basic data about a person -- name, birth year, death year, place the person lived, and so on. Ancestry will give you ranked search results, to which it assigns "stars", a five-star being the best match. However, once you have the answer, you aren't given any explanation of why the five-star matches are allegedly better than the three-star matches. Since the results are only as good as the items indexed in each data collection, and there is not much transparency about what is actually indexed for each collection, the system can miss out on the records you want while presenting you with results that might be of interest (a son or grandson of the person you're looking for), are clearly not good candidates for the current search (e.g. people alive after the death date you gave).

If, on the other hand, you present the search results in groups which say "n results from the city _Chicago_ have the surname _Jones_ and the street name _Main_", and can add or take away elements and see the number of results change explicitly, it makes a huge difference in how the searcher can understand how the search process works.

Anybody who looks at how ordinary people use Ancestry, versus the more experienced users, can see that the newbies don't understand how the search works. The point of Watson is for the searcher to be able to use a more natural language in doing the search, instead of being required to game the system to get the right answers to come out.

In the difficult diagnosis situation, doctors' human experience would allow them to instantly reject most of the 1% alternatives, and winnow down the search results.


----------



## busyba (Feb 5, 2003)

Bananfish said:


> I'm very skeptical - Watson has been optimized for Jeopardy, which is really worlds away from medical diagnostics.
> 
> With Jeopardy, the biggest challenge is figuring out the call of the question - from there, actually answering the question is not that difficult for a computer with a monstrously large database ... just see what things are associated in your database with the call of the question and pick the thing that is most like/most often associated with what is being asked.
> 
> ...


I think you're missing the larger point.

The primary goal behind the Watson project is to develop computer-based natural language processing. Playing Jeopardy is merely one application of that ability.

Think of the Watson on Jeopardy as three black boxes that are pipelined together. One black box is a natural language processor. One black box is a massive database of trivia. One black box is a machine designed to play Jeopardy that uses the other two black boxes to understand the clues and find the answers.

That first black box, the natural language processor, knows nothing about Jeopardy, it just parses language.

So now, when you want to make a machine that diagnoses patients instead of playing Jeopardy, you still need to make two new black boxes from scratch, one that contains all the medical journals and info, and one that is designed to perform diagnoses. But for the third black box, the natural language processor, you just lift the black bix from the Jeopardy system and plug it into your other two new boxes.

See, one of the primary challenges to making a "robot doctor" is getting it to understand the vast multitudes of medical information that is out there that is not in a format that machines can easily understand. That's the challenge that Watson is designed to address.


----------



## Dan203 (Apr 17, 2000)

Exactly! Watson is basically a sophisticated search engine capable of better understanding natural language, both in the question asked and the material it's searching. The question side of a medical system may not require the same natural language skills as Jeopardy, but the material side would since most sources of information (i.e. journals, texts, etc...) are written for humans to understand, not computers. The ability to parse information written for humans into something it reliably can search is pretty amazing.

And lets not forget that this version of Watson is coming up with it's answers in just a few seconds from a preloaded set of data. Given a few minutes and access to the internet it might be able to find enough information to make all of it's answers 99&#37;.

Dan


----------



## MikeAndrews (Jan 17, 2002)

Idearat said:


> ...
> While it's impressive that Watson is getting the answers, though with few "tricky" categories so far, it really does look like the button work is more of a deciding factor. There were some fairly easy questions I knew the answer to less then 1/4 way through Alex reading them. We know for sure that during his long run Ken would ring in before he'd figured out the answer, figuring that the odds were good he'd figure it out in time.
> 
> With Watson getting the jump on the buzzer so often it's more about whether a solenoid can fire more quickly when given a signal than a human can press a button when they see a light.





Stephen Tu said:


> It seems like Watson has a buzzer timing advantage on Ken & Brad; I would think the humans knew most all the answers also. Maybe Jeopardy would be fairer, if all buzzes within x milli-seconds were considered simultaneous and a random contestant was picked to give their "question".


Which brings up how the "Don't buzz in until Alex is finished reading the question" can possibly work. It requires that Alex or some crew member pushes a button to indicate that he's done talking. They would have needed that even without Watson to prompt.


----------



## vman41 (Jun 18, 2002)

netringer said:


> Which brings up how the "Don't buzz in until Alex is finished reading the question" can possibly work. It requires that Alex or some crew member pushes a button to indicate that he's done talking. They would have needed that even without Watson to prompt.


I believe in the studio there is a light signal to indicate then the question is 'done', so someone is pushing a button.


----------



## vman41 (Jun 18, 2002)

I think Watson guessed Toronto because it's analysis showed every game has a Canada-related answer and there hadn't yet been one to that point.


----------



## Waldorf (Oct 4, 2002)

busyba said:


> I think you're missing the larger point.
> 
> The primary goal behind the Watson project is to develop computer-based natural language processing. Playing Jeopardy is merely one application of that ability.
> 
> ...


Not to mention the whole AI/neural network learning part of the system which learns from its mistakes.

So one could feed it hundreds of years of historical cases and let it attempt the diagnosis, then learn what the diagnosis really was and where it went wrong (if it misdiagnosed) so it can learn from those mistakes.

As a C.S. major, I found Neural Networks was one of the most fascinating classes and my Master's project was a system that intelligently composed transitions between two music pieces, so I've really been enjoying this aspect of the research.


----------



## jbernardis (Oct 22, 2003)

netringer said:


> Which brings up how the "Don't buzz in until Alex is finished reading the question" can possibly work. It requires that Alex or some crew member pushes a button to indicate that he's done talking. They would have needed that even without Watson to prompt.


Watson is fed the question as a text message - he's not listening to Alex read. I don't think he gets the message until Alex is done reading the question. Someone turns on that light - that is the same stimulus for releasing the message. The receipt of the message is the cue that it's OK to buzz in; no need to wait for anything else.

At least that's how I'd design it.


----------



## Bananfish (May 16, 2002)

busyba said:


> I think you're missing the larger point.
> 
> The primary goal behind the Watson project is to develop computer-based natural language processing. Playing Jeopardy is merely one application of that ability.
> 
> ...


I do get all that. But.

There seem to me to be two natural language problems they've had to try to solve to play Jeopardy: (1) interpreting all the information in Watson's database -- the first black box you've described which can be plugged into other applications -- and (2) interpreting Jeopardy clues. Those problems strike me as being highly interrelated - it's difficult to understand what answers to look for in a database if you don't understand the question you're being asked.

From what I've seen so far, Watson is generally effective but certainly has some real hiccups and challenges (final jeopardy last night being a good example) at problem (2). Since the two natural language problems seem to me to be interrelated, I interpret that to mean that Watson still has some hiccups and challenges remaining at problem (1) as well. I could be talked out of that, I suppose, by a cogent explanation by someone with some inside knowledge of how Watson does what it does, but they've been working toward this particular Jeopardy challenge for over six years now, with an extremely large sample size of Jeopardy games to work from in a highly structured and repetitive data form, so that person would have to be pretty convincing.

The problem that Watson is truly a champion at is speed - getting it to scour its vast database, parsing all the natural language sources, and provide an answer with a confidence measurement in the time it takes to buzz in is a major challenge. BUT, in almost all real life cases where Watson might be used, speed isn't such a big deal. E.g., if a computer takes hours (or even days in most cases) to diagnose a patient rather than a few seconds, it's rarely going to be an issue.

(FWIW, I do have a bit of Artificial Intelligence background myself, though it's been a while since I was in that field.)


----------



## ThreeSoFar (May 24, 2002)

murgatroyd said:


> I was talking about this with a couple of people who don't fly very much, and they didn't know the answer.
> 
> To me it was trivial -- WII Battle --> Midway -- > must be Chicago. I am willing to ignore the fact that I don't know for sure if O'Hare matches the other part of the answer, and go with the answer "Chicago" because of my basic knowledge that many airports are named after people, or the place where the airport is, and that an airport named after a battle that took place somewhere else is unusual.
> 
> Watson's programming doesn't (yet) know how to weigh some bits of the answer more heavily than others, the way a human Jeopardy player could.


I couldn't even THINK of a second airport in Chicago and I was ready to say that.

Humans are WAY better at pulling an answer out of their ass...


----------



## trainman (Jan 29, 2001)

Worf said:


> Alex seemed somewhat stunned at the wagers... and I don't believe there has been a rounding issue with the bets - you can, after all, bet *$1* if you're not confident.


Actually, contestants have to bet at least $5 on Daily Doubles (but Final Jeopardy! bets can of course be as low as $0).

Apparently the Daily Double minimum is $5 because that was half the value of the lowest clues on the board on the original Art Fleming-hosted version of "Jeopardy!" -- they "should" have raised it to $50 and then $100 for the current version, but haven't bothered to change the rules, probably because it comes up so seldom. Seems like it's once in a great while that a contestant jokes about wanting to wager $0 on a Daily Double, and Alex reminds them about the $5 minimum.


----------



## Hcour (Dec 24, 2007)

Curious - since they've had a Jeopardy round and a Double Jeopardy round, what is tonight's show going to be - Triple Jeopardy?


----------



## ThreeSoFar (May 24, 2002)

Hcour said:


> Curious - since they've had a Jeopardy round and a Double Jeopardy round, what is tonight's show going to be - Triple Jeopardy?


It's the second of a two show series. As stated on the air, and here as well, several times.

Are you even listening?


----------



## MikeMar (Jan 7, 2005)

I think they won't be doing any of the IBM background stuff, probably JUST the game, or least almost all game.


----------



## pdhenry (Feb 28, 2005)

ThreeSoFar said:


> It's the second of a two show series. As stated on the air, and here as well, several times.
> 
> Are you even listening?


Um, it was on Monday and Tuesday. Tonight's the third show.

Perhaps you meant the second game.


----------



## kdmorse (Jan 29, 2001)

Todays news flurry was that despite it's seamless performance on air, Watson crashed repeatedly during the filming, dragging the game out to over four hours.

However, this is a misstatement. As has been since corrected in most news articles: _"It was not Watson, but the system that was the interface between Watson and the Jeopardy computer, completely separate from Watson, that crashed during the taping.""_

(Although it does make me wonder, how long does Watson take to IPL?)

-Ken


----------



## pjenkins (Mar 8, 1999)

MikeMar said:


> I think they won't be doing any of the IBM background stuff, probably JUST the game, or least almost all game.


first 2 shows were the background/IBM ads w/the game spread over it. tonight is just like a real game. full game in the 1/2 hour. the "winner" is the contestant with the highest combined $ amount. i'm not 100% sure, but i believe the results from yesterday are "locked in" and they start over at $0 for today, so Watson, for example, can't get a daily double and have $38k to risk right of the bat.


----------



## pjenkins (Mar 8, 1999)

kdmorse said:


> Todays news flurry was that despite it's seamless performance on air, Watson crashed repeatedly during the filming, dragging the game out to over four hours.
> 
> However, this is a misstatement. As has been since corrected in most news articles: _"It was not Watson, but the system that was the interface between Watson and the Jeopardy computer, completely separate from Watson, that crashed during the taping.""_


um, imo Watson is the complete system, up to and including the avatar and buzzer pressing thingy madoodad ((c) 2011 pjenkins).


----------



## kdmorse (Jan 29, 2001)

pjenkins said:


> um, imo Watson is the complete system, up to and including the avatar and buzzer pressing thingy madoodad ((c) 2011 pjenkins).


Both of which functioned perfectly.

The computer that was supposed to be sending Watson text messages however, is a different matter.

-Ken


----------



## Hcour (Dec 24, 2007)

ThreeSoFar said:


> It's the second of a two show series. As stated on the air, and here as well, several times.
> 
> Are you even listening?


Why, is there going to be a quiz? I'll try to pay closer attention next time.


----------



## LoadStar (Jul 24, 2001)

ThreeSoFar said:


> It's the second of a two show series. As stated on the air, and here as well, several times.
> 
> Are you even listening?


It's actually a 3 show series, but a set of two games. The first game was on Monday and Tuesday, the second game is today/tonight.


----------



## MikeAndrews (Jan 17, 2002)

jbernardis said:


> Watson is fed the question as a text message - he's not listening to Alex read. I don't think he gets the message until Alex is done reading the question. Someone turns on that light - that is the same stimulus for releasing the message. The receipt of the message is the cue that it's OK to buzz in; no need to wait for anything else.
> 
> At least that's how I'd design it.


It's not fair to Watson if it has to wait a millisecond after the question is read in its entirety to "see" the question while the humans have seen and heard it. On the other hand it wouldn't be fair to the humans that Watson could "read" and parse the whole question within milliseconds of it being unveiled while the humans have to read and hear it.


----------



## MikeAndrews (Jan 17, 2002)

pjenkins said:


> rating bump for Jeopardy
> 
> http://voices.washingtonpost.com/tvblog/2011/02/jeopardy-debuts-computer-conte.html


It occurred to me when when we didn't see "Promotional consideration by...IBM" in the credits that IBM and the producers must have made the deal where IBM's investment of $millions and providing the facility and hospitality and compensates for the buzz(!) and ratings boost that Jeopardy would get and they threw in the free infomercials for IBM.


----------



## MikeMar (Jan 7, 2005)

This game was a lot better, many questions that Watson wasn't sure on.

Only shot Ken had was to get the 2 daily doubles in double jeopardy, but he didn't, so was screwed.


----------



## jbernardis (Oct 22, 2003)

netringer said:


> It's not fair to Watson if it has to wait a millisecond after the question is read in its entirety to "see" the question while the humans have seen and heard it. On the other hand it wouldn't be fair to the humans that Watson could "read" and parse the whole question within milliseconds of it being unveiled while the humans have to read and hear it.


I'm sure watson has parsed the text message and is actively searching before the humans even realize the light is on


----------



## kdmorse (Jan 29, 2001)

Just one question that amused me - spoilered since it hasn't aired in all timezones.



Spoiler



It's a poor workman who blames these?

First Thought: Tools
Second thought: *Yogi Berra*
Initial Third thought: Charles Dickens


----------



## ElJay (Apr 6, 2005)

I laughed for a minute at Watson's alternate responses to this one:



Spoiler


----------



## busyba (Feb 5, 2003)

I think Watson gets sent the text as soon as the clue is revealed, and he spends the time between that moment and when the buzz-in light turns on figuring out the answer.

It was interesting to see that in the category about actor/directors, where the clue was mostly just a movie title, Watson eventually had the right answer, but because the interval between the reveal and the buzz-in moment was so short, he wouldn't get the answer in time and Ken would beat Watson to the buzzer.


----------



## Worf (Sep 15, 2000)

Jeopardy has always been about the buzzer discipline - knowing when to hold off so you can buzz in quickly.

Watson may get the message instantly, but it has to react *after* the light goes on. The humans can see and parse the question visually first, and anticipate when Alex finishes and when the light goes on, so they can buzz in quicker. It's just like drag racers - they actually "go" on the yellow, as the interval of the yellow-to-green transition is well-known, so they tune their reflexes to take that duration and have practically zero green-to-throttle time.

Makes the production assistant's job actually quite stressful because they have to turn on the light in time with Alex finishing the question because all the contestants are expecting to buzz in when Alex finishes the last syllable.

And yes, players know they have 5 seconds from buzzing in to answer. Watson buzzes in when it knows the answer. Human players buzz in before and hope they can get the answer before the 5 seconds is up. That's why they often take 3 or 4 seconds to come up with an answer.

In the second game, it seems Ken and Brad have managed to anticipate and beat Watson. Though I don't thing it's quite fair that Ken has to sit closest to Watson's "thumb". That clicking would be quite annoying.

It's all about the buzzer.


----------



## Dan203 (Apr 17, 2000)

busyba said:


> I think Watson gets sent the text as soon as the clue is revealed, and he spends the time between that moment and when the buzz-in light turns on figuring out the answer.
> 
> It was interesting to see that in the category about actor/directors, where the clue was mostly just a movie title, Watson eventually had the right answer, but because the interval between the reveal and the buzz-in moment was so short, he wouldn't get the answer in time and Ken would beat Watson to the buzzer.


I believe you're right, and the shorter questions tonight seemed to reveal that. On one question you could even see that all 3 possible answers were red and then the top one jumped to green while one of the other contestants was answering it. So even though he gets the full text of the question immediately when Alex starts reading it still takes him a few seconds to actually find the answer.

Overall I really enjoyed the show. It really played to my inner computer geek. For me it was similar to when I saw Asimo as CES a few years ago. It wasn't perfect, but it was still an amazing thing to see.

Dan


----------



## TiVo'Brien (Feb 8, 2002)

Loved Ken's little note beneath his final Jeopardy response.  :up:


----------



## LordKronos (Dec 28, 2003)

Worf said:


> Watson may get the message instantly, but it has to react *after* the light goes on. The humans can see and parse the question visually first, and anticipate when Alex finishes and when the light goes on, so they can buzz in quicker. It's just like drag racers - they actually "go" on the yellow, as the interval of the yellow-to-green transition is well-known, so they tune their reflexes to take that duration and have practically zero green-to-throttle time.
> 
> Makes the production assistant's job actually quite stressful because they have to turn on the light in time with Alex finishing the question because all the contestants are expecting to buzz in when Alex finishes the last syllable.


I really don't think it's the human's buzzing in anticipation that give them an edge over Watson. The way it works is (as you alluded to), the buzzer will not work before the production assistant clicks the button to enable the buzzers (which he does as soon as Alex finishes reading). For Watson, as soon as he clicks the button to enable the buzzers, the system sends a message to Watson telling it that it can now buzz in. Watson can then immediately activate the buzzer. The whole communication delay for watson can't be more than a millisecond, even considering that it has to mechanically trigger the buzzer.

I think with a window that small, the chances of someone being able to beat watson to the buzzer is extremely remote, and not likely to happen multiple times in the same game. I'd suspect it's more likely that the reason they appear to beat Watson to the buzzer is because, in those cases, Watson takes just a tiny bit too long to reach a conclusion and doesn't have it's answer ready until just slightly after the buzzers become available. This provides a small window of opportunity for the human players. Yeah, the human players still probably need to anticipate the timing to get in before watson, but it's not that they are beating Watson to the buzzer...they are managing to buzz in before Watson has come up with an answer.


----------



## tiassa (Jul 2, 2008)

ThreeSoFar said:


> I couldn't even THINK of a second airport in Chicago and I was ready to say that.
> 
> Humans are WAY better at pulling an answer out of their ass...


Of course I'm probably one of the few airport geeks that knows that Butch O'Hare was a WWII fighter pilot.


----------



## pdhenry (Feb 28, 2005)

LordKronos said:


> I'd suspect it's more likely that the reason they appear to beat Watson to the buzzer is because, in those cases, Watson takes just a tiny bit too long to reach a conclusion and doesn't have it's answer ready until just slightly after the buzzers become available. This provides a small window of opportunity for the human players. Yeah, the human players still probably need to anticipate the timing to get in before watson, but it's not that they are beating Watson to the buzzer...they are managing to buzz in before Watson has come up with an answer.


There's an article in the NY Times today that says the speed at which Watson buzzes in is a function of its confidence in the answer. If it's not very sure it lags a bit so as to be less likely to buzz in with an incorrect response.


----------



## Jon J (Aug 23, 2000)

TiVo'Brien said:


> Loved Ken's little note beneath his final Jeopardy response.  :up:


His online Washington Post Q&A showed how great a sense of humor he has.


----------



## Magnolia88 (Jul 1, 2005)

Ken is a fun guy. I met him once and he's exactly the same in person.

I'm impressed that both Ken and Brad seemed to beat Watson to the buzzer a few times. I would think that would be nearly impossible.

Most of the Watson computer talk sailed way over my head, but I enjoyed this exhibition or whatever it was.

Watson is so "smart" but it's funny how he came up with wrong answers that no human would have come up with: Toronto and Dorothy Parker are the ones that really made me scratch my head. I'm still not clear on how he could mess up with "U.S. city" and think that Toronto qualified. Or how Dorothy Parker could be the response when the answer was about a book. Or so I thought.


----------



## TiVo'Brien (Feb 8, 2002)

Jon J said:


> His online Washington Post Q&A showed how great a sense of humor he has.


Enjoyable chat transcript w/Ken about the whole experience.


----------



## Fool Me Twice (Jul 6, 2004)

I wonder if Ken and Brad count this as a win for Ken against Brad. Are they even now?


----------



## Legion (Aug 24, 2005)

Slate article by Jennings.

http://www.slate.com/id/2284721/


----------



## astrohip (Jan 7, 2003)

Worf said:


> Though I don't thing it's quite fair that Ken has to sit closest to Watson's "thumb". That clicking would be quite annoying.


From the Washington Post blogerview:

Q:Good morning Mr. Jennings, thank you very much for doing this chat. You are my favorite Jeopardy Champion (with the possible exception of Babu Srinivasan, who has one of the coolest names ever). My question is this: with all the Jeopardy appearances you've had, how did this one compare as far as pressure? I can imagine that such a high-profile game could potentially be nerve-wracking. Or do you instead approach every game with the same mindset, regardless of opponent?​
A.Ken Jennings :

Babu! I'm sure he appreciates the shoutout.

I tried to think of the computer the same as any other opponent, but in practice that turned out to be pretty hard, given the creepy insectoid clicking of its mechanical thumb buzzing relentlessly just to my left. I did feel like the underdog this time, which isn't a position I'm used to being in on Jeopardy.
 February 15, 2011 11:24 AM​


----------



## cwerdna (Feb 22, 2001)

It'd be interesting if IBM ends up posting a walk thru of all/most of the cases where the other 2 lower probability choices were (or all 3) were just really wacky/way off. Same goes for the Toronto answer.

I'd be know the nature of the bugs, esp. if it were in parsing the "question" and what it interpreted it to be and what pieces if threw out or weighted less.


----------



## MikeAndrews (Jan 17, 2002)

Legion said:


> Slate article by Jennings.
> 
> http://www.slate.com/id/2284721/





Ken Jennings said:


> Watson has lots in common with a top-ranked human Jeopardy! player: It's very smart, very fast, speaks in an uneven monotone, and has never known the touch of a woman.


 

This was the first time I saw Ken Jennings in action. I'm liking the guy.


----------



## pdhenry (Feb 28, 2005)

netringer said:


> This was the first time I saw Ken Jennings in action. I'm liking the guy.


He has a blog (naturally; who doesn't?) and there's a weekly quiz mailing list as well.


----------



## mattack (Apr 9, 2001)

Idearat said:


> 30-second skip is my friend these first two days. Tomorrow will hopefully be a game with normal pacing and less infomercials.


That's funny, I can't remember the last time I watched every non-(normal-paid)-commercial part of an episode of Jeopardy. Usually I watch them in ~18 minutes at most (not sure exactly), on the treadmill.. since I skip over the talking to contestants the vast majority of the time.

That, and the various bits about Watson were fascinating.


----------



## mattack (Apr 9, 2001)

Worf said:


> And props to the guy that programmed the voice and the Jeopardy interface for Watson. It's almost freaky when he guessed and said "I'm not sure, but what is...". We may have hit the uncanny valley with Watson's Jeopardy playing skills - it's getting creepy.


What's freaky about it? On a Daily Double, you lose money even if you don't say anything.. so it only makes sense to ALWAYS guess. If you don't know (if it's not above Watson's threshold line), don't ring in.. if you're on a daily double, always 'guess'.


----------



## mattack (Apr 9, 2001)

Weird that they didn't actually start with Ken, since he was on the left in the 'returning champion' position. Other times, they put whoever wins the coin toss there. IIRC, Brad started (at least one game), so it wasn't simply "Watson ['s avatar] was set up to be in the middle, they can't move it"


----------



## stalemate (Aug 21, 2005)

mattack said:


> What's freaky about it? On a Daily Double, you lose money even if you don't say anything.. so it only makes sense to ALWAYS guess. If you don't know (if it's not above Watson's threshold line), don't ring in.. if you're on a daily double, always 'guess'.


I think he means the voice and the spoken wording, not he fact that Watson made a guess.

Putting some of that wording in the voice makes Watson seem more human and I can see how that seems kinda freaky.


----------



## ElJay (Apr 6, 2005)

```
if (confidence < 0.5 && forcedToAnswer) 
  response = "I'm not sure, but " + response;
```
It's not that freaky, some human wrote that code...


----------



## stalemate (Aug 21, 2005)

ElJay said:


> ```
> if (confidence < 0.5 && forcedToAnswer)
> response = "I'm not sure, but " + response;
> ```
> It's not that freaky, some human wrote that code...


So if humans do it, it isn't freaky?


----------



## mostman (Jul 16, 2000)

Funny bit about my watching experience. 

I, with a Computer Science background watched these three hours with my mouth basically wide open, amazed at Watson's ability to figure some of those clues out. 

My wife, without that background, watched and said to me "I don't understand" - this looks easy. After explaining some of the challenges to her, she started to get it. 

You almost have to be in a position where you have worked on something close to this to really understand what a massive accomplishment this was. Especially when we have things like Google that provide immediate response to our questions. It's almost as if we expect that these problems are easy. 

I encourage everyone who hasn't already, go pick up Pinker's "The Language Instinct" and give it a read.


----------



## ThreeSoFar (May 24, 2002)

mostman said:


> Funny bit about my watching experience.
> 
> I, with a Computer Science background watched these three hours with my mouth basically wide open, amazed at Watson's ability to figure some of those clues out.
> 
> ...


:up:THIS!


----------



## Flop (Dec 2, 2005)

Exactly. I have been writing various software systems for the past 15 years, and found this absolutely fascinating. I was talking to a friend about it, and he said, "Big deal. It's just Google that can press a buzzer." It seems if you understand how computers work and the current limitations of them, Watson is huge. If you don't have that background, it's just Google playing Jeopardy with an IBM promotional overlay.


----------



## TriBruin (Dec 10, 2003)

mostman said:


> Funny bit about my watching experience.
> 
> I, with a Computer Science background watched these three hours with my mouth basically wide open, amazed at Watson's ability to figure some of those clues out.
> 
> ...


This exactly. After watching NOVA, and the (basic) explanation of how Watson was parsing the questions, I was trying to figure out how Watson was parsing each question and then trying to guess whether Watson would be able to answer them. Considering he was right at least 90% (even if Brad or Ken beat him to the buzzer), it was amazing how well he could disect the questions.


----------



## pjenkins (Mar 8, 1999)

TriBruin said:


> This exactly. After watching NOVA, and the (basic) explanation of how Watson was parsing the questions, I was trying to figure out how Watson was parsing each question and then trying to guess whether Watson would be able to answer them. Considering he was right at least 90% (even if Brad or Ken beat him to the buzzer), it was amazing how well he could disect the questions.


IIRC, 90% was the minimum level needed to compete with the Jeopardy champions in a match.


----------



## murgatroyd (Jan 6, 2002)

mostman said:


> Funny bit about my watching experience.
> 
> I, with a Computer Science background watched these three hours with my mouth basically wide open, amazed at Watson's ability to figure some of those clues out.
> 
> My wife, without that background, watched and said to me "I don't understand" - this looks easy.


Ditto. Fill in "linguist" and "friends" in my case.

I remember getting into an argument with my teacher in our introductory syntax class, where we were making up rules to describe what was going on in language. I argued that some of my classmates' rules did not produce the outcome they claimed. They were fudging the outcome of their rules with that pesky human 'stuff we don't know we know' information.

I said that if we had a computer, and could run the rules through as written, without taking the context into account, they would see that their rule was ill-formed.

(This was when I was told as a native English speaker, I didn't have the same insight into how my language worked than a non-native speaker did.  )

I could have used a couple of "What is Toronto??????" style answers to drive home the point.


----------



## vman41 (Jun 18, 2002)

TriBruin said:


> This exactly. After watching NOVA, and the (basic) explanation of how Watson was parsing the questions, I was trying to figure out how Watson was parsing each question and then trying to guess whether Watson would be able to answer them. Considering he was right at least 90% (even if Brad or Ken beat him to the buzzer), it was amazing how well he could disect the questions.


You have to wonder, though, whether the constrictive nature of the show format made things easier for Watson. The writers have to construct the clues so that in a few words there is only one right question. There's less extraneous information to be information to be discarded that in normal human conversation.


----------



## mattack (Apr 9, 2001)

I would say the opposite, as others have done -- with fewer words, there's less for Watson to use to 'target on' the right answer. Plus, there's the puns and such, which make it extremely hard to parse.

(Also, the writers don't always succeed in "construct[ing] the clues so that in a few words there is only one right question", since they have to go back and give someone money when there's another acceptable answer.... Unfortunately, I wish they would actually go back and 'rewind' the whole thing and redo everything after the error.. but they really only fix the outcome of that one clue..)


----------



## murgatroyd (Jan 6, 2002)

Okay, show of hands. 

Who else has been watching Jeopardy since the IBM Challenge, and thinking "I wonder what Watson would have done with this category?"

Or wishing we could have a bar showing all the three top answers and the percentages for the human contestants as well?


----------



## Wil (Sep 27, 2002)

murgatroyd said:


> wishing we could have a bar showing all the three top answers and the percentages for the human contestants as well?


It would take an interesting adapter cable.

People often speak of pulling the answer out of their ...


----------



## Worf (Sep 15, 2000)

Yeah. It would be nice if we had "Watson Guesses" segment on Jeopardy now...

They should arrange with IBM to make it happen. It doesn't have to be real-time and hey, Watson can have internet.


----------



## jeepair (Apr 22, 2004)

Worf said:


> Yeah. It would be nice if we had "Watson Guesses" segment on Jeopardy now...
> 
> They should arrange with IBM to make it happen. It doesn't have to be real-time and hey, Watson can have internet.


Just design something like Closed Caption for gameshows where you can select to turn that feature on or off.


----------



## ThreeSoFar (May 24, 2002)

murgatroyd said:


> Okay, show of hands.
> 
> Who else has been watching Jeopardy since the IBM Challenge, and thinking "I wonder what Watson would have done with this category?"


And more important (to the producers and the show), and likely a big part of why they did this whole thing, how many of you now watching (more) regularly hadn't been doing so prior to the Watson run?


----------



## rifleman69 (Jan 6, 2005)

murgatroyd said:


> Okay, show of hands.
> 
> Who else has been watching Jeopardy since the IBM Challenge


Not I, don't like the teen/kids tournaments.


----------



## plateau10 (Dec 11, 2007)

Here's what is probably all but inevitable: someone will put a Watson-like device live on the Internet for the whole world to play with. And of course it will be "learning" from each and every query.


----------



## MikeAndrews (Jan 17, 2002)

ThreeSoFar said:


> And more important (to the producers and the show), and likely a big part of why they did this whole thing, how many of you now watching (more) regularly hadn't been doing so prior to the Watson run?


I set a Season Pass to get the Watson episodes. After watching a few of the teen tournament ones, I deleted the SP.

After a while guessing the question stops being amusing and I have plenty of unwatched content to not get to.


----------



## Boot (Mar 15, 2000)

busyba said:


> At the end of the second night, Alex said "we'll add these totals to the results of tomorrow's game to see who wins the million dollars."
> 
> Million dollars??? At the beginning of the first night, Alex called it an "exhibition match" of Jeopardy. I took that to mean that they weren't playing for money.
> 
> ...


It's probably considered an exhibition as a consolation to Ken and Brad, so that neither of them have their win percentages affected by going on the show. And probably also because Jeopardy! has some restrictions on players returning to play again.


----------



## jbernardis (Oct 22, 2003)

plateau10 said:


> Here's what is probably all but inevitable: someone will put a Watson-like device live on the Internet for the whole world to play with. And of course it will be "learning" from each and every query.


It's called Google


----------



## plateau10 (Dec 11, 2007)

jbernardis said:


> It's called Google


There's a big difference between "here's a bunch of web pages that might have the answer you're looking for somewere in them" and "I'm 95% sure the answer is 'X'". But understanding that is the key difference between people who watched Watson and said "wow" versus those who said "big deal, I've got Google".


----------



## Boot (Mar 15, 2000)

Washington Post interview said:


> Q
> Did Watson's voice throw you off your game at all?
> 
> A.
> ...


----------



## jsmeeker (Apr 2, 2001)

plateau10 said:


> There's a big difference between "here's a bunch of web pages that might have the answer you're looking for somewere in them" and "I'm 95% sure the answer is 'X'". But understanding that is the key difference between people who watched Watson and said "wow" versus those who said "big deal, I've got Google".


Indeed. The people who say "I have Google" also have a human brain. Watson does not.


----------



## n548gxg (Mar 7, 2003)




----------



## mattack (Apr 9, 2001)

DOES NOT COMPUTE.

fsck is a file system check, not a format command.


----------



## wmcbrine (Aug 2, 2003)

Yeah, should be "mkfs". But people like saying "fsck" better.


----------



## murgatroyd (Jan 6, 2002)

ThreeSoFar said:


> And more important (to the producers and the show), and likely a big part of why they did this whole thing, how many of you now watching (more) regularly hadn't been doing so prior to the Watson run?


We had been watching it live, off and on, in one of our local restaurants when we stopped in for dinner on the way home from work -- or watching it live if we happened to get home early enough.

But once we heard about the IBM challenge, we set up a Season Pass. So yes, we're watching more regularly now.

On the other hand, we are watching via TiVo -- so do we count?


----------



## sieglinde (Aug 11, 2002)

LOL at the Unix joke. I have only watched the first episode and saw that it was a tad bit kludgy but I was impressed at where AI is right now. I hate talking to the computer things on the phone for customer service so I just say "agent' but something like this if it was fast, would probably work. 
It does not pass the Turing test yet. Will some team at IBM eventually get a Nobel prize for this? (I don't think there is a catagory for this subject but this is really an amazing achievement.)
When the machine "cares" about winning then I will begin worrying.


----------



## pdhenry (Feb 28, 2005)

sieglinde said:


> When the machine "cares" about winning then I will begin worrying.


What if Watson was deliberately holding back so as not to panic us? 

Sort of like when you play checkers against a 4-year-old...


----------



## jbernardis (Oct 22, 2003)

rifleman69 said:


> Not I, don't like the teen/kids tournaments.


Of course the REAL reason to not like the teen tournament is that it's more embarrassing when you don't know the answers


----------



## Regina (Mar 30, 2003)

jbernardis said:


> Of course the REAL reason to not like the teen tournament is that it's more embarrassing when you don't know the answers


What is really embarrassing is that the teen tournament's questions are more difficult than the college tournament's questions!


----------



## Azlen (Nov 25, 2002)

Watson is now competing on American Idol. We are doomed.


----------



## MikeAndrews (Jan 17, 2002)

wmcbrine said:


> Yeah, should be "mkfs". But people like saying "fsck" better.


Should have been "rm -rf /"


----------



## wmcbrine (Aug 2, 2003)

No, they said "format", not "remove all files".


----------



## vman41 (Jun 18, 2002)

I've had a KAM 1 season pass for a long time. It's a great show for TiVoing because you can skip the 'meet the contestants' segment after the first break and the commercial break before final jeopardy is exceptionally long.


----------



## rifleman69 (Jan 6, 2005)

jbernardis said:


> Of course the REAL reason to not like the teen tournament is that it's more embarrassing when you don't know the answers


Well at least you admit it.


----------



## sieglinde (Aug 11, 2002)

More embarrassing is finding out just how old you are. What band was Paul McCartney in? What is Wings? ARRRGGGGHHHH!!!!


----------



## cwerdna (Feb 22, 2001)

I just finished watching a the Nova ep on Watson that people were referring to (had to wait for the rerun, hate watching stuff online). It was quite good and insightful.


----------



## sieglinde (Aug 11, 2002)

Could not find the repeat. My PBS has finally been restored. I live in an area that gets KCET but my cable company put something up called HD08 which is PBS. I didn't know about it until yesterday.


----------



## MikeMar (Jan 7, 2005)

I love the teen tournaments because I know a vast majority of the answers 

Some are obviously stupidly dump/simple questions that almost ANY teen or anyone would get right, I hate those ones


----------



## cwerdna (Feb 22, 2001)

(Thread resurrection...)
I'm in the middle of watching this interview:[media]http://www.youtube.com/watch?v=8e8G-jJa0KI&list=PLFE64B625AB4A8FC7&index=6&feature=plpp_video[/media].

It's an ep of the show http://www.kqed.org/tv/programs/index.jsp?pgmid=20444.

Interesting stuff... (I never watched the stuff on IBM's site about Watson). He mentioned that Watson had 2880 cores. The raw data that it had processed (books, the Bible, encyclopedias, dictionaries, etc.) was only ~100 gigs. But, Watson had 15 TB of RAM.


----------



## sieglinde (Aug 11, 2002)

Impressive. It will be more impressive if they have made some sort of holographic memory or database, more like how a human brain works. We don't look through the entire part of the Bible that we have memorized or happen to remember, we use word association when someone says something familiar and we recognize it and may be able to quote more. Associative memory. Computers have a hard time with that.


----------

