DoC Professor disputes whether computer 'Eugene Goostman' passed Turing Test

by Royston Ingram

11 June 2014

National news reporting that a computer programme pretending to be a 13-year-old Ukrainian boy passed the Turing test is disputed by DoC Professor.

A computer program named Eugene Goostman which imitates a Ukrainian teenager has won an artificial intelligence competition at the Royal Society in London.

The program convinced 10 out of 30 judges at the nation's most prestigious scientific institution that it was a real person in a series of online chats lasting five minutes each.

The event's organisers claimed Eugene had made history by passing the Turing Test, a significant goal in the field of artificial intelligence. However experts in artificial intelligence, including our own Professor Murray Shanahan here in the Department of Computing challenge this assumption.

Professor Murray Shannahan Professor Shanahan in interviews with Buzzfeed, BBC's "Click" program and articles in the Telegraph, Metro and The International Business Times stated that it was ‘a great shame’ the test was reported as passed.

He states:

Of course the Turing Test hasn’t been passed. I think its a great shame it has been reported that way, because it reduces the worth of serious AI research. We are still a very long way from achieving human-level AI, and it trivialises Turing’s thought experiment (which is fraught with problems anyway) to suggest otherwise.

For starters the 30% pass boundary was never set by Turing, who just said the test would be passed if ‘the interrogator decides wrongly as often when the game is played between a computer and a human as he does when the game is played by a man and a woman.’

The 30% figure was his belief of what the pass rate would be in the year 2000, not the pass boundary.

Turing also didn’t say a five-minute test would mean success achieving human-level AI; for that, he would require much longer conversations.

And the supercomputer is actually not a supercomputer, but a chatbot, which can just scan for keywords and pull stock responses from a database – whether they are artificial intelligence is debatable.

Even if this test were valid, it would not be the first time it was passed under these criteria.

In 1972, a chatbot called PARRY fooled 48% of psychiatrists into believing it was a person suffering from schizophrenia.

And the designers bent the rules from the beginning, creating a 13-year-old Ukrainian, which allowed for limited communication ability.

This is not really what Turing had in mind.