Analyzing the Game of Survivor -- Analyzing Sentiment of Reddit Posts (3)¶

Welcome back, to the third installment of the Survivor analysis series! My last post began to investigate the [relationship between Reddit mentions and particular contestants]. The next step is to dive a bit deeper into the Reddit comments themselves, using some basic NLP techniques.

In this third installment of the series, I will continue to digging into some of the Reddit data that I had collected via the Pushift.io API. For more information on how this data was collected, please check out the [first article in this series], where I describe the ETL process for the Pushift (as well as other!) data.

Introduction to NLP and Sentiment Analysis¶

We will be using a similar query to the first post on this issue, so I will skip over any explanations here.

A quick explanation of NLP -- and a disclaimer: I have done some work in this field, but am by no means an expert. The next few paragraphs gives a brief overview of some of the topics in this area. You are encouraged to dive deeper into this yourself. The following isn't necessary to understand this analysis, but I wanted to give a quick overview of some of the challenges in this field and some of the potential shortcoming of this analysis before I dive in.

Sentiment analysis is a type of analysis in the realm of Natural Language Processing which investigates the sentiment of particular words and sentences. There is quite a lot of research on this topic, but essentially an annotated list of sentences is used to generate a model to predict the sentiment of the sentence using the attributes of the text itself.

Extracting these attributes is an article in and to itself, and can be done in quite a few ways. The simplest, and most widely known, is using a bag of words approach. In this approach, each word is first tokenized to represent its meaning, and then across a variety of examples is vectorized based on the counts of each words. This assumes that essentially, everything interesting about a sentence can be broken down into its components (words). That is, the whole is the sum of the parts.

However, we know this isn't the case. Words have contextual meaning as well, and there are correlations with other words in a sentence. In this space, there are some contextual encoders that try to handle this issue, like ELMO and BERT. That is a topic for another time, however.

NLP is a rich field, and there are many topics of interest in the field. For this analysis, we will mostly be glossing over the most important portions of this to use pretrained models for sentiment analysis.

Often times in these kinds of tasks, if the problem is generalizable enough, using pretrained models from a general corpus (like Wikipedia, or reviews across industries, for example) provides a sufficient model for your use-case. Of course, there may be domain specific meaning to particular words or phrases. For instance, a sentence:

I hate Russell.

is semantically very similar to:

I hate Frosted Flakes.

but

Tony got the third immunity idol.

has a Survivor-specific meaning in terms the word "immunity". In other contexts, like:

I am immune to the Chicken Pox, since I've already had it.

the word immunity has an essentially different meaning. In fact, in no cases (in general) will the word "Immunity" mean exactly the same thing as in a Survivor context. This is important to recognize, as it shows us the limitations of using a generalized model.

For this reason, there is often a desire to fine-tune these generalized models to a specific use-case.

However, while this is true, the word fine-tune is quite intentional. The generalized model handles the bulk of the work (usually a neural network in the case of embeddings) -- the fine-tuning has a smaller, incremental benefit as compared to the generalizable model. The general model gets you a lot of the way there.

While the topics above (contextual embeddings and fine-tuning models to domain-specific applications) are certainly of interest, and could potentially improve the models, I will leave this (and all model fitting) to a later date. For now, we will just use publically available, out-of-the-box solutions to get a quick view on the sentiment. Then, if the results seem worthwhile to dig deeper into, I will investigate in future articles building contextual models to gain more insight into the semantics and context of the sentences.

TextBlob¶

To analyze the sentiment of the comments, we will be using textblob, a NLP package that builds on top of NLTK and pattern, two very popular NLP libraries. They have a very simple API which will allow us to do alot of interesting analysis right out of the box!

We will be using their pre-trained sentiment analysis to extract the sentiment from a word.

To give a sense of what this looks like, let's look at a dummy example:

In each of these cases, we will get a polarity which represents the actual sentiment (best being 1, worst being -1) and subjectivity, which tries to analyze how subjective the sentence itself it (0 being "objective", 1 being subjective.)

We will take a look at both in this analysis.

Let's look at a particular comment and see what we will want to look at.

The first thing we can do is just look at the sentiment of this entire paragraph. That's straightforward enough:

But there's a few thing with this. First off, we notices there are two different names in the above sentence, with different sentiments towards both. How do we associate the sentiment to the correct person?

This is a hard problem. Essentially, we would have to find a way to match each noun to a verb or adjective. This isn't a trivial task, and assumes a lot -- that we can identify the Part of Speech (not a given at all), and then we can unambiguously match them. In short, this is not easy to do and, as far as I can see, there's no way to handle this out the box with TextBlob.

There is, however, a somewhat "close enough" way to handle this problem. For now, since we're just looking at things on aggregate, this is what we will do.

First, we can extract all of the relevant contestants from the string. Remember, each comment is replicated for each possible subject, if there are multiple names. In this case, we have:

It's Will. Looking at the above sentences, it looks like this person doesn't like Will too much. And that Tyson made the "absolutely best strategic move". So let's first extract only the sentences that have Will's name in it, and look at the average sentiment of these sentences.

Let's compare this with the other potential subject, on the next row:

We see that this meets our expectations, rather than just using the overall polarity.

Note that this method is far from perfect, but short of being able to associate each potential subject-adjective pair, it will perform well enough for our purposes.

To give an example of how difficult this problem can be, take a look at this comment:

Look how many names we have there! By my count there is:

1. David
2. Tony
3. Mike
4. Kristie
5. Zeke
6. Yul
7. Jonathan
8. Jessica
9. Jay
10. Jeff (Probst!)
11. Cochran
12. Michelle

Somehow, this person managed to jam that many names into so few words! (Impressive!)

In this case, using the above method doesn't work so well.

We'll stop here for now, but potentially there could be some interesting venues to go down associating subjects/objects with verbs and adjectives.

Let's take a look at one of the comments to get a sense of what this looks like!

It's important to remember that there seems to be a lot of discussion about which contestants could win or should win or whether they made the right decision or not. This is, of course, a separate question from sentiment -- there are plenty of people who make good strategic moves but are not the most well liked, by fans or by other contestants. Strong words in either direction ("hate", "love", etc.)

Out of curiosity, let's take a look at the overall feelings of sentiment in these comments.

The first thing that stands out is that there are a large number of 0 sentiments for both the sentence and overall polarity.

To only look at cases where there is some sentiment, let's look at a histogram of non-zero values:

We see that in both cases, it seems that most comments are positive (a slight skew to the left). This is interesting -- anyone who says redditors are inherently negative would probably disagree! To break this down a bit more, let's tae a look at the different seasons:

Box plots by each season don't immediately reveal too much -- it appears that the distributions are somewhat similar for all of the seasons.

Next, we will look at some of the individual contestant episode combinations to see which have the highest (and lowest) skewing distributions.

Interestingly, we see some of the most and least popular players/episodes in this graph.

It appears that the players who were liked (for instance, Malcolm in Kill or Be Killed and John Cody in Blood is Thicker than Anything) seemed to have a smaller variance -- meaning they are more unanimously liked.

While others who were disliked (like Colton Cumbie in One World Is Out the Window and Ozzy from Double Agent) had a larger range of values for the 1st and 3rd quantile, indicating that the sentiment towards them are are more divided among Redditors.

Interestingly, even some of the least "liked" contestant episode combinations (like Sophie Clarke in Cult Like) are above zero, which reflects the average positive sentiment for most of these comments.

Disliked Player -- Examples¶

"Worst" things people said about Ozzy:¶

"Best" things people said about Ozzy¶

"Worst" Things People Said About Colton¶

"Best" things people said about Colton¶

Liked Player -- Examples¶

"Best" things people said about Malcolm:¶

"Worst" things people said about Malcolm:¶

"Best" things people said about John Cody:¶

Worst things people said about John Cody:¶

For the overall sentiment, it seems like we are getting some leakage from the other potential contestants. This is still interesting in some ways -- for instance, if you compare the distribution of Colton Cumbie in One World is Out The Window between the two graphs, you can see that Colton had a larger distribution with the sentence polarity as compared to the overall polarity.

This seems to indicate that comments concerning Colton are more divided than comments mentioning Colton, which indicates his controversiality. Also, comments concerning Colton has a slightly lower median (-.125, -.0625).

Additionally, and somewhat subjectively, we notice a few characters who were associated or allied with people who were liked (or disliked) more than they themselves were noticed or liked. For instance, Ken McNickle and Woo Hwang with David and Tony, respectively, from their seasons.

You can see below that there are some relationships between the person who is speaking and general characters in the game. For instance, Caleb Bankston is mentioned often with Colton, because they are enggaged. However, Caleb appears to be generally well liked -- especially in his relation to other characters.

Looking at the subjectivity of the sentences, we see some interesting results. In particular, there are some examples of popular, well-known male characters (Ozzy, Jonas, and John) recieving less subjective evaluations than some female players. We investigate this in the next graphs, and it appears there os a slight increase in subjectivity at the extremes (the players with the highest/lowest subjectivity scores) but not in general for female vs. male contestants.

It is also interesting that Ozzy Lusth, Jonathan Penner and Russell Swan have the lower subjectivity scores. To me, this seems to indicate that they are being judged based on their merits in the game. All three of these players have some level of strategic gameplay (or in Ozzys case, great at challenges) that makes them seem like objectively good Survivor players. At the same time, they are controversial players and may not be the most socially well-liked. This isn't a thorough investigation, and we've seemed to gloss over subjectivity so far (and will not dive deeper), but it may be an interesting area for further analysis.

Conversely, at the bottom of the list, you see players like Malcolm Freberg and Denise Stapley. I'm focusing on these two because they're the most immediately familiar to me, but they are characters that tend to be well liked, and were underdogs (their tribes lost a lot at the start of the game). Perhaps this has led them to being ranked higher here. (Also of note, of course, are the episodes that these occur in. For Malcolm, it was actually an episode on Caramoan, which was the returning season. Malcolm was well liked previously, and perhaps was one of the few returning "favorites" who was actually liked. Perhaps this made people less objective about him?)

Of course, these are all hypotheses and would require some further research to investigate, but these are interesting results nevertheless.

Breakdowns by Gender¶

After the subjectivity analysis, as well as some data I have read concerning attitudes towards different genders of survivor players, I wanted to take a look at the subjectivity and polarity based on the contestants sex. The following graphs only consider the most (and least) subjective players for each sex.

The results are inconclusive -- it does not appear, based on this information alone, that the distributions are substantially different from one another. This statement is supported by the simple linear regression model later, which investigates what is predictive of the sentence sentiment, where Sex is removed as an insignificant feature.

Now we've dug into the sentiment analysis a bit, the next step will be to build a linear model for sentiment and see which variables are significant. In the [next post], we will fit a linear model to the sentence_polarity variable and take a look at some of the coefficients to determine a statistically significant effect. This will be an inferential model, in the sense that we are not aiming for predictive accuracy but to explain changes in the dependent variable.