HON 207 - Introduction to Cognitive Science

Lab Assignment 1
Due at next class meeting

Note: you can do this assignment in groups of up to two people

  1. In this assignment you are to chat with two different online chatbots and report your findings. The goal is not to determine if they can pass the Turing test. However, you should attempt to identify portions of your conversation that "appear" to be intelligent behavior on the part of the chatbot and portions of the conversation that clearly indicate non-human responses. Your work on this problem will be evaluated based on the creativity of your questions that help identify these aspects of the chatbot behavior, as well as your analysis of the results. In each case your should try to use multiple questions or comments and carry-on the conversation for at least 5-10 minutes. Be sure to log your questions and the chatbot's responses, however, you do not need to turn in the logs of all of your conversations. You should select portions of the conversations that provide good examples of intelligent and non-intelligent responses and only submit those (copy and paste these portions of the conversations into an MS Word or a text file which you will submit).

    The two chatbots are available from the following links:

    • A.L.I.C.E. - Artificial Intelligence Foundation. ALICE was the winner of the 2004 Loebner Prize.
    • Teachable Web Hal - This version of Hal learns from your conversation, as well as from previous conversations with other users.
       

  2. Find out about the Loebner Prize and write a short summary (2-3 paragraphs) of what it is and why it came into existence.

  3. Both ALICE and Teachable Web Hal use linguistic rules and limited knowledge bases in order to carry on conversations. The winner of the 2005 Loebner Prize was another chatbot called Jabberwacky. This chatbot is very different from the others in that it does not use linguistic rules. It stores everything everyone has ever said, and finds the most appropriate thing to say using contextual pattern matching techniques. In speaking to you it uses only learnt material. With no hard-coded rules, it relies entirely on the principles of feedback.
     
    1. Read about Jabberwacky on its Web site and also look at some sample conversations. You may also want to look at the FAQ on the icogno Web site (the parent company of Jabberwacky). The summarize what you have learned about this chatbot and write what you think is significant in 2-3 paragraphs.
      2.

    2. Carry on a similar chat with Jabberwacky, as you did with the other chatbots and again report your findings. Did you notice any significant difference in the way these chatbots behave?

  4. Use the Research Methods Tutor (RMT) and evaluate the appropriateness of its dialog for the task it is attempting to accomplish. At the "Welcome Guest" page, select "text only" and "interactive tutoring." (But feel free to try out the animated agent on your computer at home some time if you like, just for fun.) Start with the "introduction" module that tells you a little bit about how to interact with the tutor, then go back to the "welcome" page and pick any of the other modules to try out. Note that you will need to go back to http://alarm.cti.depaul.edu/rmt to start each module - If you try to follow the link the tutor gives you at the end of a module or after the introduction, you will end up at a different class's login page.

    Write a paragraph or two evaluating RMT's ability to understand and respond appropriately to what you typed in the session. Include examples of dialog turns in which things went well, and examples in which RMT did not do so well. Were there things you said that you were surprised RMT could "understand"? Were there things you were surprised it did not get right? Were you able to draw any conclusions about the strengths and limitations of RMT's approach to natural language understanding? Note that RMT's behavior depends on several components:

    • A natural language understanding component (used to check whether what you typed in is the answer RMT is looking for)
    • A curriculum script of questions, expected answers, and hints to give when the expected answer is not forthcoming
    • A set of rules for what to do in response to correct and incorrect answers, such as when to give a hint, when to ask another question, and when to just state the answer and move on

    Try to identify which of these three components is responsible for any problems you see when you use the tutor.