Experiments: Enriching indirect answers

1. Overview
2. Experiment 1: A deterministic strategy
   1. The one-word cases
      1. WordNet enrichment
      2. Review enrichment
      3. Combining resources
   2. Multi-word cases
   3. Discussion
3. Experiment 2: A probabilistic (MaxEnt) model
   1. Background
   2. Features
      1. Subset and superset features
      2. Negation feature
      3. Review score feature
      4. Review score inference
      5. WordNet relations
      6. WordNet inferences
   3. The experiment
   4. Results
   5. Discussion
4. Exercises

Overview

In this section, we return to the IQAP corpus, bringing together the information we extracted from WordNet and the IMDB reviews to try to enrich the IQAP answers. I begin with a deterministic enrichment strategy and then move to a probabilistic model.

Throughout this section, I work with a binary version of the ratings: 'definite-yes' and 'probable-yes' collapse to 'yes', and 'definite-no' and 'probable-no' collapse to 'no'. This makes the experiments conceptually simpler, and it side-steps my current lack of understanding of how the 'probable' categories were used.

In iqap.Item, the methods response_counts, response_dist, and max_label all have an optional keyword argument make_binary which peforms this category collapse if set to True.

In R, one can achieve the collapse by adding column values:

1. iqap = read.csv('iqap-data.csv')
2. iqap$yes = iqap$definite.yes + iqap$probable.yes
3. iqap$no = iqap$definite.no + iqap$probable.no

Code and data:

• iqap-data.csv.zip: the corpus file
• iqap.py: Python classes for working with iqap-data.csv
• iqap_classifier.py: maxent classifier interface for iqap.py
• iqap_experiments.py: the experiment file
• imdb-words-assess.csv.zip: word scores produced by this method
• imdb-bigrams.csv.zip: bigram data in a format comparable to imdb-words.csv (relevant for exercise BIGRAMS below)
• wnscores_inquirer.csv.zip: an alternative to the IMDB data
• wnscores.py: WordNet score propagation; implements the approach of Blair-Goldensohn et al. 2008
• wnscores.readme.txt: description of how wnscores_inquirer.csv was created

To rerun the experiments, head to the bottom of the iqap_experiments.py, where you will see options for running either the series of deterministic experiments or the probabilistic one.

Experiment 1: A deterministic strategy

The first experiment is deterministic in the following sense: we work towards defining a single function that predicts 'yes' or 'no' as the label for each Item.

To prevent mistyping, and to allow flexibility, I define two global Python variables for specifying predictions:

1. YES = 'yes'
2. NO = 'yes'

The one-word cases

To begin, we look at cases where both the question and the answer have just one word in their '-CONTRAST' sets, which means that we can compare the two words directly using the resources we've developed.

The following function allows us to move quickly to just these 'one-worders'; using this will simplify a lot of our code.

1. def is_one_worder(item):
2. """
3. The argument is an iqap.Item instance. If the -CONTRAST tree sets
4. for both the question and the answer contain exactly one terminal,
5. return those terminals, else return False.
6. """
7. q_words = item.question_contrast_pred_pos()
8. a_words = item.answer_contrast_pred_pos()
9. if len(q_words) == 1 and len(a_words) == 1:
10. return (q_words[0], a_words[0])
11. else:
12. return False

WordNet enrichment

The following is a WordNet prediction function. It uses wordnet_relations to get all the relations that hold between its arguments word1 and word2, and then it uses the following heuristics to guess an answer:

1. If the question is inconsistent with the answer, return NO. (Entailment.)
2. If the question word is stronger than the answer, return NO. (Upper-bounding implicature.)
3. If the question word is weaker than the answer, return YES. (Entailment.)
4. If the question word is synonymous with the answer, return YES. (Entailment.)
5. If no relation can be found, return None. (Ultimately, we will treat such lack of evidence as positive evidence that the two predicates are unrelated, which implicates NO.)

1. def wordnet_word_predict(word1, word2):
2. """
3. Uses wordnet_functions.wordnet_relations to try to determine the
4. rich enrichment for an IQAP answer, based on both logical and
5. pragmatic principles.
6. """
7. rels = wordnet_functions.wordnet_relations(word1, word2)
8. # Where WordNet can't relate the two, opt out:
9. if not rels:
10. return None
11. else:
12. # Where the question radical and answer are incompatible:
13. if set(['antonyms']) & rels:
14. return NO
15. # Where word1 entails word2 (Great? / Good.)
16. elif set(['entailments', 'hypernyms', 'member_meronyms', 'substance_meronyms', 'part_meronyms']) & rels:
17. return NO
18. # Where word2 entails word1 (Good? / Great!)
19. elif set(['causes', 'hyponyms', 'member_holonyms', 'substance_holonyms', 'part_holonyms']) & rels:
20. return YES
21. # Where word1 and word2 seem roughly synonymous:
22. elif set(['also_sees', 'similar_tos']) & rels:
23. return YES
24. # In the event of (only) a different relation, opt out:
25. else:
26. return None

To assess this function as well as the others defined below, I define an assessment function, which keeps track of the overall coverage and prints out an effectiveness summary for the items that the function does make predictions about:

1. def assess_one_worder_performance(prediction_function, print_errors=False):
2. """
3. See how many of the one_worder cases are captured by WordNet, and
4. also assess whether it makes correct predictions where it makes
5. predictions at all. Prints the assessment to standard outut.
6. """
7. coverage = defaultdict(int)
8. cm = defaultdict(lambda : defaultdict(int))
9. corpus = IqapReader('iqap-data.csv')
10. for item in corpus.dev_set():
11. status = is_one_worder(item)
12. if status:
13. q_word, a_word = status
14. predicted = prediction_function(q_word, a_word)
15. # Check for a value at all:
16. if predicted:
17. coverage['related'] += 1
18. actual = item.max_label(make_binary=True)
19. cm[actual][predicted] += 1
20. # View incorrect predictions:
21. if print_errors and actual != predicted:
22. print q_word, a_word
23. else:
24. coverage['unrelated'] += 1
25. # View the confusion matrix:
26. print_effectiveness(cm)
27. # View the coverage:
28. print "coverage:", coverage['related'], "/", sum(coverage.values()), "=", round(coverage['related'] / float(sum(coverage.values())), 2)

The assessment is done by effectiveness(), which does standard precision, recall, and accuracy calculations for each class:

1. def print_effectiveness(d):
2. """
3. Prints an effectiveness report to standard output.
5. Argument: d -- a two-dimensional defaultdict in which the initial
6. keys are the actual values and the secondary keys are the
7. predicted values
9. The first step is creating a confusion matrix where the rows are
10. the actual values and the predictions are the column values. One
11. that is created, the precision and recall calculations are done
12. for each class.
13. """
14. # Create a proper matrix to make the calculations easier:
15. cm = numpy.zeros((len(d), len(d)))
16. keys = sorted(d.keys())
17. for i, actual in enumerate(keys):
18. for j, predicted in enumerate(keys):
19. cm[i, j] = d[actual][predicted]
20. # Precision and recall by category:
21. precisions = []
22. recalls = []
23. for i, key in enumerate(keys):
24. # Precision: tp / (tp + fn).
25. precision = cm[i,i] / numpy.sum(cm[: , i])
26. precisions.append(precision)
27. print key, 'precision:', round(precision, 2)
28. # Recall: tp / (tp + fp).
29. recall = cm[i,i] / numpy.sum(cm[i, : ])
30. recalls.append(recall)
31. print key, 'recall:', round(recall, 2)
32. # Micro-averaging:
33. print 'micro-averaged precision', round(numpy.mean(precisions), 2)
34. print 'micro-averaged recall', round(numpy.mean(recalls), 2)
35. # Accuracy (percentage correct):
36. print 'accuracy:', round(numpy.sum(cm.diagonal()) / numpy.sum(cm), 2)

The assessment:

1. assess_one_worder_performance(wordnet_word_predict, print_errors=False)
2. no precision: 1.0
3. no recall: 0.75
4. yes precision: 0.82
5. yes recall: 1.0
6. micro-averaged precision 0.91
7. micro-averaged recall 0.88
8. accuracy: 0.88
9. coverage: 17 / 82 = 0.21

Where WordNet makes a guess, it is highly accurate, missing just two items. This is encouraging. However, the overall coverage is very slight, so we will need to recruit other information.

Review enrichment

Next we recruit the IMDB review data. My expectation is that this will have better coverage but prove less accurate than WordNet.

If you followed along with the IMDB discussion through to the section on building scales, then you probably created a file called imdb-words-assess.csv. If not, you can get my versio here:

1. imdb-words-assess.csv.zip

The function review_functions.get_all_imdb_scores, reads in this file and turns it into a Python dictionary mapping word-tag pairs to dictionaries of values {'Word':word, 'Tag':tag, 'ER':float, 'Coef':float: 'P':float}:

1. SCORER = review_functions.get_all_imdb_scores('imdb-words-assess.csv')

The following function uses p-values to restrict attention to just the linear coefficient values that we are willing to call reliable. If the p-value threshold isn't met, it returns None.

1. def coef_score(word, p=0.1):
2. """
3. Return the linear coefficient for word according to scorer, if word's
4. p-value is below the p-value defined threshold:
5. Arguments:
6. word -- a (string, pos) pair
7. p -- a p-value threshold (default: 0.1; use 1 for no threshold)
8. Value: the coefficient for word, or None if the word is not in the dictionary
9. or the p value is too high.
10. """
11. val_dict = SCORER.get(word, None)
12. if not val_dict:
13. return None
14. else:
15. if val_dict['P'] <= p:
16. return val_dict['Coef']
17. else:
18. return None

The prediction function compares coeffecient values. The first comparison checks to see whether they have the same sign. Differing signs are assumed to correspond to NO; these are pairs like good and bad. Once that comparison is made, we move to absolute values, using their size as a proxy for strength. This is simply an effecient way of making it so that, for example, bad has a "smaller" (less negative) coefficent than terrible.

1. def reviews_word_predict(word1, word2, p=0.1):
2. """
3. Make an IQAP answer prediction based on review coefficient scores:
4. Arguments:
5. word1, word2 -- (string, pos) pairs
6. p -- a p-value threshold (default: 0.1; use 1 for no threshold)
7. Value: YES, NO, or None
8. """
9. word1 = wordnet_functions.wordnet_sanitize(word1)
10. word2 = wordnet_functions.wordnet_sanitize(word2)
11. score1 = coef_score(word1, p=p)
12. score2 = coef_score(word2, p=p)
13. # If a score is missing, opt out:
14. if score1==None or score2==None:
15. return None
16. # Where the signs are different, the words are predicted to be
17. # inconsistent with each other:
18. if sign(score1) != sign(score2):
19. return NO
20. score1 = abs(score1)
21. score2 = abs(score2)
22. # Where word1 is greater than word2 (Great? Good.):
23. if score1 > score2:
24. return NO
25. # Where word1 is less than word2 (Good? Great.):
26. if score1 <= score2:
27. return YES

And finally an assessment of the IMDB data on their own:

1. assess_one_worder_performance(reviews_word_predict, print_errors=False)
2. no precision: 0.77
3. no recall: 0.77
4. yes precision: 0.62
5. yes recall: 0.62
6. micro-averaged precision 0.69
7. micro-averaged recall 0.69
8. accuracy: 0.71
9. coverage: 35 / 82 = 0.43

Combining resources

Still restricting attention to the one-worders, we carefully put the resources together into a single prediction function. In doing this, we favor WordNet, appealing to the reviews only where WordNet provides no information.

In addition, the function includes one additional inference: where neither of our resources makes a prediction, we venture a NO, on the grounds that unrelated predicates deliver this implicature via a Relevance-based calculation.

1. def combined_resources_word_predict(word1, word2):
2. """
3. Use WordNet and the reviews to make predictions. The logic:
4. 1. If WordNet makes a prediction, go with it, since that is a high-precision resources.
5. 2. Where Wordnet makes no prediction, use the reviews prediction if there is one.
6. 3. Where the reviews make no prediction, assume that the two predicates are independent,
7. which, by Gricean reasoning, means NO in the IQAP context.
8. """
9. predicted = None
10. wn_predicted = wordnet_word_predict(word1, word2)
11. if wn_predicted:
12. predicted = wn_predicted
13. else:
14. predicted = reviews_word_predict(word1, word2)
15. if not predicted:
16. predicted = NO
17. return predicted

The whole is indeed better than its parts, in the sense that we are now cover the entire data set (though with a small drop in overall performance):

1. assess_one_worder_performance(combined_resources_word_predict, print_errors=False)
2. no precision: 0.68
3. no recall: 0.85
4. yes precision: 0.68
5. yes recall: 0.44
6. micro-averaged precision 0.68
7. micro-averaged recall 0.65
8. accuracy: 0.68
9. coverage: 82 / 82 = 1.0

Multi-word cases

We can put it off no longer: abut 40% of the development set examples are not one-worders, so we need to generalize our approach to the complexities of multi-word expressions.

Both Wordnet and the reviews are comprised of data on single words, so our multi-word strategy needs to make use of those basic resources somehow. Some heuristics:

1. If the question's contrast predication is a subset of the answer's, return YES. The heuristic is that all modification will be restrictive, and thus that the more heavily restricted answer will be stronger as well.
2. Reverse the above reasoning if the answer's contrast prediction is a subset of the question's.
3. For the remaining cases, make predictions about the cross-product of the question vocabulary and the answer vocabulary, and then pick the majority choice where there is one, else choose NO.
4. If either predication includes a negation, reverse the final decision.

The following code implements this set of heuristics, drawing heavily on code we developed for the one-worders:

1. def combined_resources_item_predict(item):
2. """
3. The function makes predictions for all iqap.Item instances,
4. whether one worders or multi-worders. The return value is always
5. YES or NO.
6. """
7. # Get the words:
8. q_words = set(item.question_contrast_pred_pos())
9. a_words = set(item.answer_contrast_pred_pos())
10. # Predictions:
11. predicted = NO
12. # Get this easy identity case out of the way:
13. if q_words == a_words:
14. return YES
15. # Check the subset/superset relationships:
16. if q_words.issubset(a_words):
17. predicted = YES
18. elif a_words.issubset(q_words):
19. predicted = NO
20. else:
21. # Now check the cross-product of q_words and a_words for relevant connections:
22. prediction_dict = defaultdict(int)
23. for q in q_words:
24. for a in a_words:
25. prediction_dict[combined_resources_word_predict(q, a)] += 1
26. # We sort all the predictions by their degree of represetation, and then
27. # pick the best represented, favoring NO in the case of ties:
28. sorted_predictions = sorted(prediction_dict.items(), key=itemgetter(1), reverse=True)
29. predicted = sorted_predictions[0][0]
30. if len(sorted_predictions) > 1 and predicted == sorted_predictions[1][0]:
31. predicted = NO
32. # If the answer is negated, then we reverse the prediction:
33. if a_words & set(["n't", "not", "never"]):
34. if predicted == NO: predicted = YES
35. elif predicted == YES: predicted = NO
36. return predicted

A new assessment method (one that doesn't restrict to one-worders):

1. def assess_performance(print_errors=False):
2. """
3. Assesses a full deterministic experiment and prints a summary to
4. standard output. Use print_errors=True to see the errors above the
5. performance report.
6. """
7. cm = defaultdict(lambda : defaultdict(int))
8. corpus = IqapReader('iqap-data.csv')
9. for item in corpus.dev_set():
10. predicted = combined_resources_item_predict(item)
11. actual = item.max_label(make_binary=True)
12. cm[actual][predicted] += 1
13. # View incorrect predictions for the multi-word cases (we saw the single-word ones already):
14. if print_errors and actual != predicted and \
15. (len(item.question_contrast_pred_pos()) > 1 or len(item.answer_contrast_pred_pos()) > 1):
16. print '======================================================================'
17. print item.question_contrast_pred_pos()
18. print item.answer_contrast_pred_pos()
19. print actual, predicted
20. # View the confusion matrix:
21. print_effectiveness(cm)

Here is the output of assessment:

1. assess_performance(print_errors=False)
2. no precision: 0.66
3. no recall: 0.86
4. yes precision: 0.71
5. yes recall: 0.44
6. micro-averaged precision 0.68
7. micro-averaged recall 0.65
8. accuracy: 0.67

exercise ERROR, exercise FEATURES, exercise BIGRAMS

Discussion

The performance of the determinisic model is pretty good, on the whole. We are well above chance on accuracy, and the precision and recall numbers show that we are making solid, educated guesses.

One unsatisfying thing about the model is that it is so hand-crafted. We decide ahead of time how to prioritize the information, and then we reduce it all to a single judgment. In doing this, we are likely hiding, ignoring, or obscuring many important factors.

The next model we develop addresses all these concerns. In my view, it allows us to be guided by specific scentific insights, but it also learns from the data we provide it. In general, its chief advantage is that it synthetizes a wide variety of information into a single predictive function, and it is also attentive to the data, in the sense that it is trained on, and therefore learns from, its details.

Experiment 2: A probabilistic (MaxEnt) model

Background

Maximum Entropy (MaxEnt) classification is a version of multinominal logistic regression. Since we'll deal with just two classes ('yes' and 'no') for our experiments, the model is effectively the same as a basic logistic regression model of the sort that we built with the review data.

One of the great strengths of MaxEnt models is that they are able to deal with lots of different kinds of features, and they deal reasonably well with correlations between predictors that can lead other models astray. What this means for us as linguists is that we are relatively free to define a lot of features that make sense to us scientifically, relying on the model to sort them out.

To get a feel for how the model works, it's useful to consider a simple model in which there is just one predictor. For this, let's use the difference in the cumulative coefficient scores between the question and the answer. The function for calcuating the cumulative score for each example is review_score_feature below. We'll take the diffence of the two values to be the sole predictor.

What we do at this point is reduce each item in the data to its feature vector. Here, our feature vectors have just one element in them — the float-valued difference between the scores. (For the models we develop below, we will map each item to a long and diverse feature vector.)

The model is then trained on some percentage of the data. For standard classification, this means is that we feed it pairs consisting of the correct label for the example and the associated feature vector.

However, there is one twist: each of our examples has, not one label, but rather 30 of them. We could take the majority label, but this would obscure the truly mixed message sent by some of our annotations. Thus, when training our model, we include each item 30 times, once for each label it has:

The model then learns a weight associated with each of the predictors, based on the associations in the training data. For the actual example here, the weight it learns for the feature review_score_diff is 2.923 for the correct class being 'no'. (This weight will vary somewhat depending on the composition of the train/test split used.)

The prediction step then involves taking each feature vector and plugging it into the model equation. For example, if the feature vector is -0.07, as for Item 4015 above, then we calculate invlogit(3.331 * -0.07) = 0.56. That is, the model says that there is a 56% chance that this example is labeled 'no'. In turn, there is a 1.0-0.56 = 0.44 chance that the correct label is 'yes'.

In a regular classifications setting, we can simply take the larger of the two probabilities to be the predicted class. Since each of our examples has 30 labels, our method is slightly different. We take two basic approaches:

1. Max-label: For example 4015, the max label is 'no', with 17 choices. The max label chosen by our model is 'no', since it assigns that category the higher the two probabilities. Thus, we say that the model classifies correctly here.
2. KL divergence: For this, we turn the annotations into a distribution and compare it to the distribution returned by the model. Example 4015 has the distribution {yes: 13/30, no:17/30} = {yes:0.43, no:0.57}, whereas the model predicts {yes:0.44, no:0.56}. The KL divergence of the predicted distribution from the actual one is 0.0002032661. We calculate this value for each item and then take some kind of average of the values. (I use the mean.)

Each of these methods has advantages and disadvantages. Max-label is very easy to interpret, but it ignores a lot of the structure in our annotations. KL divergence embraces the uncertainty inherent in our annotations and predicted by our model, but it is hard to interpret in isolation, since the KL values don't mean much. In what follows, I use both methods together, comparing to baseline models to firm up our intuitions about what the numbers mean.

Figure FITS

A graphical illustration of how to use a fitted model (the one from the prose) to classify examples.

Features

I now define a bunch of feature functions that we will use to reduce our items to feature vectors.

Subset and superset features

The deterministic model returns a definitive answer based on these simple relationships between the question and the answer. This often works well (Good? / Very good!), but it can also fail (Good? / Somewhat good.). With a MaxEnt model, we can include these features, expecting them to be balanced against other considerations.

1. def subset_feature(q_words, a_words):
2. """Return True if q_words is a subset of a_words, else False."""
3. if q_words.issubset(a_words):
4. return True
5. else:
6. return False
7.  
8. def superset_feature(q_words, a_words):
9. """Return True if a_words is a subset of q_words, else False."""
10. if a_words.issubset(q_words):
11. return True
12. else:
13. return False

Negation feature

We'll use negation in a variety of ways. This is the basis for those uses, and we'll also include it as a feature:

1. def negation_feature(words):
2. """Return True if words contains a negation, else False."""
3. if words & set(["n't", "not", "never"]):
4. return True
5. else:
6. return False

Figure NEGIMPACT helps to convey why the negation feature is important: encouraged is weaker than optimistic, suggesting a 'yes' reply, but the negation in the answer reverses this.

Figure NEGIMPACT

Negation reverses the ordering expected of the adjectives.

Figure NEGIMPACT was generated by the following code:

1. source('review_functions.R')
2. imdb = read.csv('imdb-words.csv')
3. par(mfrow=c(1,2), oma=c(2,0,0,0))
4. WordDisplay(imdb, 'encouraged', 'a')
5. WordDisplay(imdb, 'optimistic', 'a')
6. mtext("Are you encouraged by the new developments?; I'm not so optimistic for the time being.", side=1, outer=TRUE)

Review score feature

This function builds a total score for the words provided, based on the review data. It reverse the sign of the final score if negation_feature == True. The intended use of this argument is that we get the result of negation_feature(words) and use it here.

1. def review_score_feature(words, negation_feature):
2. """
3. Returns the cumulative review score for words. If negation_feature
4. is True (this should be determined by negation_feature), then flip
5. the sign of the final score.
6. """
7. score = 0.0
8. for w in words:
9. coef = coef_score(w)
10. if coef:
11. score += coef
12. if negation_feature:
13. score = -score
14. return score

The reversal triggered where negation_feature == True is motivated by interactions like that of figure NEGIMPACT.

Review score inference

In addition to using the raw review scores, we also use this careful, pragmatically-informed comparison of them:

1. def review_score_inference(q_score, a_score):
2. """
3. This function employs the same logic as reviews_word_predict,
4. except now dealing with the cumulative score as given by
5. review_score_feature. (Both q_score and a_score are presumed to
6. have come from that function.)
7. """
8. if sign(q_score) != sign(a_score):
9. return False
10. q_score = abs(q_score)
11. a_score = abs(a_score)
12. if q_score < a_score:
13. return True
14. else:
15. return False

WordNet relations

1. def wordnet_relation_features(q_words, a_words, features):
2. """
3. This function creates a number of features, on that it adds a
4. mapping relname -> True to features for each relation that it
5. finds between question and answer words. In addition, it reverses
6. the scalar relations in the presence of differing negation values.
7. """
8. for q in q_words:
9. for a in a_words:
10. for rel in wordnet_functions.wordnet_relations(q, a):
11. if features['q_negated'] != features['a_negated']:
12. rel = rel.replace('hypernym', 'hyponym').replace('hyponym', 'hypernym')
13. rel = rel.replace('meronym', 'holonym').replace('holonym', 'meronym')
14. features[rel] = True
15. return features

WordNet inferences

This function essentially acts as though each (question_word, answer_word) pair were the sole basis for prediction, using wordnet_word_predict from the deterministic experiment to make a YES/NO inference about each pairing:

1. def wordnet_inferences(q_words, a_words, features):
2. """
3. This function uses wordnet_word_predict to create predictions, but
4. it does that for each pair of words (q, a), adding to features a
5. feature that reflects the presence of this feature.
6. """
7. for q in q_words:
8. for a in a_words:
9. features['wn_'+ wordnet_word_predict(q, a)] = True
10. return features

The experiment

The feature function puts all of the above pieces together:

1. def combined_resources_item_features(item):
2. """
3. Feature function for iqap.Item instances. The intent is to feed
4. this into iqap_classifier.IqapClassifier as its feature function,
5. for maxent classification.
7. Argument: item -- any iqap.Item instance
8. Value: features -- a default dict with a variety of different value types
9. """
10. features = defaultdict(int)
11. # Get the words as sets:
12. q_words = set(item.question_contrast_pred_pos())
13. a_words = set(item.answer_contrast_pred_pos())
14. # Subset and superset features (boolean valued):
15. features['subset'] = subset_feature(q_words, a_words)
16. features['superset'] = superset_feature(q_words, a_words)
17. # Negation features (boolean valued):
18. features['q_negated'] = negation_feature(q_words)
19. features['a_negated'] = negation_feature(a_words)
20. # Score features (float valued):
21. features['q_score'] = review_score_feature(q_words, features['q_negated'])
22. features['a_score'] = review_score_feature(a_words, features['a_negated'])
23. # Relational score feature (boolean valued):
24. features['review_score_cmp'] = review_score_inference(features['q_score'], features['a_score'])
25. # WordNet relational features (count valued):
26. features = wordnet_relation_features(q_words, a_words, features)
27. # Wordnet relation features (count valued):
28. features = wordnet_inferences(q_words, a_words, features)
29. return features

Results

Table RESULTS gives a set of results for the model:

  Run   Log-lik.    Max-label acc.      Mean KL div.   Micro-avg prec.  Micro-avg recall    Train acc.
    1      -0.48               0.8              0.54              0.88              0.73          0.81
    2      -0.48              0.77              0.59              0.77              0.76          0.78
    3       -0.4              0.83              0.43              0.85              0.83          0.78
    4      -0.57              0.67              0.83              0.68              0.66          0.82
    5      -0.49               0.7              0.65              0.71              0.69          0.81
    6      -0.52              0.83              0.66              0.85              0.79           0.8
    7      -0.39               0.8              0.49              0.83              0.79          0.79
    8      -0.44              0.77              0.59              0.82              0.81          0.81
    9      -0.51              0.73              0.55              0.78               0.7          0.82
   10       -0.5              0.83               0.5              0.85              0.79           0.8
Means      -0.48              0.77              0.58               0.8              0.76           0.8

exercise ERRORCMP

Discussion

An examination of the model's feature weights suggests that it is behaving sensibly. Table POS and table NEG provide information about the top indicators of 'yes' and 'no', respectively.

It is somewhat hard to think about the performance numbers in isolation. Some considerations:

1. Performance is about the same on both the training and testing sets, suggesting that we are not over-fitting.
2. The model certainly does better than the deterministic one. Here are the deterministic results again:
   1. assess_performance(print_errors=False)
   2. no precision: 0.66
   3. no recall: 0.86
   4. yes precision: 0.71
   5. yes recall: 0.44
   6. micro-averaged precision 0.68
   7. micro-averaged recall 0.65
   8. accuracy: 0.67

To gain another baseline, I fit a model that simply uses word-counts inside the -CONTRAST predicates as features. (This experiment is run if iqap_classifier.py is run from the command-line.) This is a standard sort of baseline in natural language processing. The results are given in table UNIGRAMS:

1. The overall accuracy and KL-divergence numbers are about the same as for our model, except this one is more volatile in its performance
2. The model has so many features that it simply memorizes the training data, producing perfect performance on it each time. This suggests that it won't generalize well.

  Run    Log-lik.    Max-label acc.      Mean KL div.   Micro-avg prec.  Micro-avg recall    Train acc.
    1        -0.3              0.83              0.57              0.83              0.83           1.0
    2        -0.5               0.7              0.52              0.66              0.63           1.0
    3       -0.39              0.77              0.57              0.77              0.76           1.0
    4       -0.52               0.7              0.64              0.66               0.6           1.0
    5       -0.51              0.63              0.78              0.58              0.57           1.0
    6       -0.53              0.63              0.75              0.61              0.58           1.0
    7       -0.47               0.7              0.64               0.7              0.65           1.0
    8       -0.55               0.6              0.76              0.61              0.58           1.0
    9       -0.43              0.73              0.59              0.75              0.71           1.0
   10       -0.43              0.67              0.68              0.73              0.68           1.0
Means       -0.46               0.7              0.65              0.69              0.66           1.0

I conclude that we have identified a number of important predictors, and they are working well together. Of course, there are lots of other possibilities to try. That's what most of the exercises are about, and this area is also ripe for projects.

exercise SENTI, exercise PROPAGATE, exercise CLASS, exercise PER

Exercises

Table ADVADJ

Adverb–adjective pairs in the bigrams data. The adverbs' effects seem highly predictable from their own distributions and those of their complements.