Watson! - Deducing IBM's Answer to Jeopardy

Watson, an IBM built Question Answering computer, is most popular for its extraordinary performance in the popular TV show Jeopardy. Most recently, it has been in the news for its application in financial services and healthcare. Watson was built as part of the DeepQA Project, and an overview of the project was published in a 2010 paper. This paper was part of an AI course that I took in Fall 2013. Below is a quick summary of how WATSON solves Jeopardy questions that I wrote (rather hastily) as part of an assignment.

Building Watson: An overview of the QA project

Question Answering technology can be useful to professionals in timely and sound decision making in the areas of compliance, legal, healthcare, etc. With this technology in mind, Ferrucci and et al. started working on a project to build Watson, a QA computer system that would compete in Jeopardy. Jeopardy is a TV competition that asks natural language questions from a broad domain to the competitors. To answer a question and get points, the competitor should be both accurate, confident and fast. To compete at human level, Watson would have to be able to answer around 70% of the questions asked in less than 3 second each with 80% precision.

To meet its goal, Watson incorporates QA technologies such as parsing, question classification, knowledge representation, etc. The decision for Watson to press or not press the buzzer in order to answer a question would depend on its confidence on its answer generated. Since most questions in Jeopardy are complex, there are different components of the questions that need to be evaluated separately. Hence, to calculate the confidence for an answer, each component's confidences is combined to generate the answer's combined confidence. Confidence estimation is an important part of QA technology.

An answer for a Jeopardy question might not be generated from the whole question. The subclues could be generated from two different parts of the question sentence, or the subclues could be hidden within another subclue in the sentence. Similarly, puzzles in the question can be categorized for special processing.

For Watson, certain kinds of Jeopardy questions were excluded. Questions with audio visual components have been excluded as they are outside the scope.

The domain: A Lexical Answer Type (LAT) is a word in the clue that indicates the type of the answer. In 20,000 questions, 2500 distinct LATs were discovered. It is worth mentioning that about 12% of the questions did not have any distinct LAT. The most frequent 200 LATs made less than 50% of the total information. Since figuring out the distinct LAT for a question is a complex process, a system like Watson must have a sound Natural Language Processing and Knowledge Representation and Reasoning technology.

The metrics: To play Jeopardy, factors such as betting strategy and luck of the draw is important as well. But for building Watson, the main factors being investigated are speed, confidence and correctness. Precision is the number of correct answers out of the number of questions Watson chose to answer. Whether Watson chooses to answer a question is based on the confidence threshold. Any confidence score above the confidence threshold would mean Watson would press the buzzer in order to answer the question. This threshold controls the tradeoff between precision and percent answered. Data collected show that human winners usually have 85%-95% precision with 40% - 50% of the questions answered.

Watson uses the DeepQA technology, which is a massively parallel, probabilistic evidence based architecture. More than 100 techniques are used for analyzing natural language, collecting sources, finding and generating hypotheses, finding and ranking hypotheses. The main principals implemented are: massive parallelism for multiple interpretation, combination of probabilistic questions from a broad domain, contribution of all components of a question towards its confidence generation, use of strict and shallow semantics.

Content Acquisition: This is the process of collecting the content to be used for answer generation. There are a manual process and an automatic one. First, it is important to analyze the question to find out what type of answer is required (manual), and if possible the category of the question, possibly characterized by a LAT (automatic). The sources of content are literary works, dictionaries, encyclopedias, etc. After question evaluation, the process of content evaluation begins. Four high level steps take place: identify seed source documents from the web, extract the text nuggets from the documents, give scores to each text nugget depending on their informative nature with respect to their source document, merge the text nuggets to give one result to be evaluated.

Question Analysis: The process of Question Analysis uses many techniques, but they mainly consist of Question classification, LAT detection, Relation Detection and Decomposition. Question classification is the process of using words or phrases to identify the type of the question by considering the phrases' syntactic, semantic or rhetorical functionality. Some example of question types are puzzles, math question, definition question, etc. LAT detection involves determining whether the answer of a question can be an instance of a LAT. This can be done by replacing a component of the question with the LAT and then gathering evidence for the correctness of the given LAT candidate. Similarly, many questions in Jeopardy contain relations such as syntactic subject-verb-object predicates or semantics relationships. Watson uses relation detecting in many of its QA process, from LAT detection to answer scoring. Because of the large knowledge domain in Jeopardy, detecting most frequent relations can be helpful in only about 25% of the Jeopardy questions.

Hypotheses generation: After question analysis, the QA system generates candidate answers by searching the system's sources and generating answer sized snippets. This stage consists of two processes: Primary search and candidate answer generation. The goal of Primary search is to find as many answer-bearing contents as possible. Many search techniques are used, mainly: multiple text-search engines, document search as well as passage search, knowledge base search using SPARQL.

Soft filtering: After candidate generation, there are algorithms that are light-weight (less resource intensive) that prune the larger set of candidates to a smaller set. There is usually a threshold for the number of candidates in the final set. Candidates that are pruned are taken to the final merging step.

Hypothesis and evidence sorting: After pruning, the candidate answers go through a stronger evaluation process. This step consists of two main processes: Evidence Retrieval and Scoring. Evidence Retrieval is the process in which contents that provide evidence for a candidate answer. Search techniques like passage search using a question component with the candidate answer as query are used. Scoring: After evidence gathering, each answer-candiate is given confidence scores based on different factors. Watson itself employs more than 50 scoring components that range from formal probabilities to categorical features. These scorers use logical, geospatial, relational, logical reasoning. After these scores have been generated, they are merged for each candidate answer. The merged scores are then ranked by a system that has already been run over a training questions whose answers are known.