Skip to content

Commit

Permalink
add reference
Browse files Browse the repository at this point in the history
  • Loading branch information
@jieyima
jieyima committed Mar 13, 2018
1 parent 358288a commit eb15fcc
Showing 1 changed file with 363 additions and 0 deletions.
363 changes: 363 additions & 0 deletions NLP_AWS_SouthPark.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,363 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Sentiment Analysis with Amazon Comprehend\n",
" -- In Application with South Park Review "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this notebook, I used Amazon comprehend to implement sentiment analysis on one review of the episode of South Park from Dani Di Placido Forbes **#OnTV** section. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](http://imagecurl.com/images/04065054009291592566_thumb.jpg)"
]
},
{
"cell_type": "markdown",
"metadata": {
"toc": true
},
"source": [
"<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
"<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Plot-Excerpt\" data-toc-modified-id=\"Plot-Excerpt-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Plot Excerpt</a></span></li><li><span><a href=\"#Setting-up-Environment\" data-toc-modified-id=\"Setting-up-Environment-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Setting up Environment</a></span></li><li><span><a href=\"#Sentiment-Analysis-with-Amazon-Comprehend\" data-toc-modified-id=\"Sentiment-Analysis-with-Amazon-Comprehend-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Sentiment Analysis with Amazon Comprehend</a></span><ul class=\"toc-item\"><li><span><a href=\"#One-text-line\" data-toc-modified-id=\"One-text-line-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>One text line</a></span></li><li><span><a href=\"#One-One-line\" data-toc-modified-id=\"One-One-line-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>One One line</a></span></li><li><span><a href=\"#On-one-movie-review-document\" data-toc-modified-id=\"On-one-movie-review-document-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>On one movie review document</a></span></li></ul></li><li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Conclusion</a></span></li><li><span><a href=\"#References\" data-toc-modified-id=\"References-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>References</a></span></li></ul></div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plot Excerpt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- First thing first, let's look at the plot excerpt of this episode 'Doubling Down': "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cartman's abusive treatment of Heidi Turner is straining their relationship. He blames his anger issues on his poor diet, so Heidi convinces Cartman to try a vegan diet with her in an attempt to reconcile. The next day at lunch, Cartman continues to degrade Heidi to the rest of the boys, but treats her kindly when she shows up. When Kyle talks to Heidi after school about her relationship with Cartman, she gets defensive and avoids the subject. At Heidi's home, Cartman states that he now enjoys Beyond Meat and presents her with a bucket of KFC to try, claiming that it is actually \"Beyond KFC\" and vegan. Back at school, she feels bloated and ill afterwards while Cartman insults Heidi behind her back and tells the boys that she is getting fat. Kyle goes to a girls' volleyball practice to ask them to stop mocking Heidi, and they question if Kyle is doing so because he has feelings for Heidi himself. He decides to back off his attempts to break up Cartman and Heidi, but later talks to Heidi in the school gymnasium. The two bond as Kyle tells Heidi that Cartman is an expert at playing the victim and will never change."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setting up Environment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In AWS account:\n",
"- create a [new IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_cliwpsapi)\n",
"- note down 1) **YOUR_ACCESS_KEY** 2) **YOUR_SECRET_KEY** once the IAM user sets up\n",
"\n",
"In terminal:\n",
"- pip install awscli\n",
"- aws configure\n",
" - aws_access_key_id = **YOUR_ACCESS_KEY**\n",
" - aws_secret_access_key = **YOUR_SECRET_KEY**\n",
" - Default region name = **us-east-1**\n",
" - Default output format = **json**"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import pandas as pd\n",
"import boto3\n",
"import json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sentiment Analysis with Amazon Comprehend "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### One text line "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"comprehend = boto3.client(service_name='comprehend', region_name='us-east-1')"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Calling DetectSentiment\n",
"{\n",
" \"ResponseMetadata\": {\n",
" \"HTTPHeaders\": {\n",
" \"connection\": \"keep-alive\",\n",
" \"content-length\": \"163\",\n",
" \"content-type\": \"application/x-amz-json-1.1\",\n",
" \"date\": \"Tue, 13 Mar 2018 03:18:50 GMT\",\n",
" \"x-amzn-requestid\": \"3ff2887f-266d-11e8-b48a-9f02d80b9bc8\"\n",
" },\n",
" \"HTTPStatusCode\": 200,\n",
" \"RequestId\": \"3ff2887f-266d-11e8-b48a-9f02d80b9bc8\",\n",
" \"RetryAttempts\": 0\n",
" },\n",
" \"Sentiment\": \"NEGATIVE\",\n",
" \"SentimentScore\": {\n",
" \"Mixed\": 0.04560389369726181,\n",
" \"Negative\": 0.5313727259635925,\n",
" \"Neutral\": 0.41348445415496826,\n",
" \"Positive\": 0.009538856334984303\n",
" }\n",
"}\n",
"End of DetectSentiment\n",
"\n"
]
}
],
"source": [
"text = \"GarrisonTrump’s solution is to violently rape all three, as he nurses a black eye and defends Trump’s actions, is painfully accurate. The parallel between Heidi and Cartman’s poisonous, yet stubborn relationship is extended to Trump’s supporters, who watch Trump’s latest gaffe on tv, and look away, unwilling to criticize him for fear of the left’s rabid gloating. This is unfortunately, exactly what’s happening.\" \n",
"\n",
"print('Calling DetectSentiment')\n",
"print(json.dumps(comprehend.detect_sentiment(Text=text, LanguageCode='en'), sort_keys=True, indent=4))\n",
"print('End of DetectSentiment\\n')"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"![Garrison](https://hips.hearstapps.com/esq.h-cdn.co/assets/16/42/980x544/gallery-1476975335-south-park.png?resize=768:*)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### One One line"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"path = \"/Users/Jessica/Desktop/BAX452/Natural_Language_Processing/SouthParkReview.txt\"\n",
"doc1 = open(path, \"r\")\n",
"output = doc1.readlines()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" -- Note: there is a text size limit for Detectsemtiment analysis in AWS of 5000 bytes. So I have to manually cut down the text file and make the size fit. "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'The episode focuses on my favorite subplot of the season, Heidi and Cartman’s relationship. \\n'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"output[1]"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"ResponseMetadata\": {\n",
" \"HTTPHeaders\": {\n",
" \"connection\": \"keep-alive\",\n",
" \"content-length\": \"163\",\n",
" \"content-type\": \"application/x-amz-json-1.1\",\n",
" \"date\": \"Tue, 13 Mar 2018 03:18:50 GMT\",\n",
" \"x-amzn-requestid\": \"400d631f-266d-11e8-9d52-a71b4437c023\"\n",
" },\n",
" \"HTTPStatusCode\": 200,\n",
" \"RequestId\": \"400d631f-266d-11e8-9d52-a71b4437c023\",\n",
" \"RetryAttempts\": 0\n",
" },\n",
" \"Sentiment\": \"NEUTRAL\",\n",
" \"SentimentScore\": {\n",
" \"Mixed\": 0.0037473251577466726,\n",
" \"Negative\": 0.008039911277592182,\n",
" \"Neutral\": 0.6919484734535217,\n",
" \"Positive\": 0.2962643802165985\n",
" }\n",
"}\n"
]
}
],
"source": [
"print(json.dumps(comprehend.detect_sentiment(Text=output[1], LanguageCode='en'), sort_keys=True, indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Heidi and Cartman](https://i.ytimg.com/vi/EhF0VR1PFRE/maxresdefault.jpg)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### On one movie review document"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"ResponseMetadata\": {\n",
" \"HTTPHeaders\": {\n",
" \"connection\": \"keep-alive\",\n",
" \"content-length\": \"161\",\n",
" \"content-type\": \"application/x-amz-json-1.1\",\n",
" \"date\": \"Tue, 13 Mar 2018 03:18:50 GMT\",\n",
" \"x-amzn-requestid\": \"402642ce-266d-11e8-a22e-41f36acf1541\"\n",
" },\n",
" \"HTTPStatusCode\": 200,\n",
" \"RequestId\": \"402642ce-266d-11e8-a22e-41f36acf1541\",\n",
" \"RetryAttempts\": 0\n",
" },\n",
" \"Sentiment\": \"NEUTRAL\",\n",
" \"SentimentScore\": {\n",
" \"Mixed\": 0.08717764168977737,\n",
" \"Negative\": 0.13139696419239044,\n",
" \"Neutral\": 0.6841796636581421,\n",
" \"Positive\": 0.09724566340446472\n",
" }\n",
"}\n"
]
}
],
"source": [
"whole_doc = ', '.join(map(str, output))\n",
"print(json.dumps(comprehend.detect_sentiment(Text=whole_doc, LanguageCode='en'), sort_keys=True, indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The author's attitude towards the episode \"Doubling down\" of South Park is netural (score: 0.68) to somewhat negative (score: 0.13). "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References\n",
"\n",
"- ['South Park' Review: 'Doubling Down' Is The Most Insightful Episode In Years](https://www.forbes.com/sites/danidiplacido/2017/11/09/south-park-review-doubling-down-is-the-most-insightful-episode-in-years/#171864e17684)\n",
"- <a href=\"https://en.wikipedia.org/wiki/Doubling_Down_(South_Park)#Plot\">'South Park' Season 21 Episode 7 'Doubling Down'</a>"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
},
"toc": {
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": true,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": true,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
"nbformat_minor": 2
}

0 comments on commit eb15fcc

Please sign in to comment.