111Scraping_Workthrough.html

<!DOCTYPE html>

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta charset="utf-8" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="pandoc" />


<title>111Scraping_Workthrough.utf8</title>

<script src="libs/jquery-1.11.3/jquery.min.js"></script>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link href="libs/bootstrap-3.3.5/css/bootstrap.min.css" rel="stylesheet" />
<script src="libs/bootstrap-3.3.5/js/bootstrap.min.js"></script>
<script src="libs/bootstrap-3.3.5/shim/html5shiv.min.js"></script>
<script src="libs/bootstrap-3.3.5/shim/respond.min.js"></script>
<script src="libs/navigation-1.1/tabsets.js"></script>
<link href="libs/highlightjs-9.12.0/default.css" rel="stylesheet" />
<script src="libs/highlightjs-9.12.0/highlight.js"></script>
<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-68219208-1', 'auto');
  ga('send', 'pageview');

</script>

<style type="text/css">code{white-space: pre;}</style>
<style type="text/css">
  pre:not([class]) {
    background-color: white;
  }
</style>
<script type="text/javascript">
if (window.hljs) {
  hljs.configure({languages: []});
  hljs.initHighlightingOnLoad();
  if (document.readyState && document.readyState === "complete") {
    window.setTimeout(function() { hljs.initHighlighting(); }, 0);
  }
}
</script>


<style type="text/css">
h1 {
  font-size: 34px;
}
h1.title {
  font-size: 38px;
}
h2 {
  font-size: 30px;
}
h3 {
  font-size: 24px;
}
h4 {
  font-size: 18px;
}
h5 {
  font-size: 16px;
}
h6 {
  font-size: 12px;
}
.table th:not([align]) {
  text-align: left;
}
</style>

<link rel="stylesheet" href="libs/local/main.css" />
<link rel="stylesheet" href="libs/local/nav.css" />
<link rel="stylesheet" href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" />

</head>

<body>

<style type = "text/css">
.main-container {
  max-width: 940px;
  margin-left: auto;
  margin-right: auto;
}
code {
  color: inherit;
  background-color: rgba(0, 0, 0, 0.04);
}
img {
  max-width:100%;
  height: auto;
}
.tabbed-pane {
  padding-top: 12px;
}
button.code-folding-btn:focus {
  outline: none;
}
</style>


<div class="container-fluid main-container">

<!-- tabsets -->
<script>
$(document).ready(function () {
  window.buildTabsets("TOC");
});
</script>

<!-- code folding -->


<header>
  <div class="nav">
    <a class="nav-logo" href="index.html">
      <img src="static/img/stat545-logo-s.png" width="70px" height="70px"/>
    </a>
    <ul>
      <li class="home"><a href="index.html">Home</a></li>
      <li class="faq"><a href="faq.html">FAQ</a></li>
      <li class="syllabus"><a href="syllabus.html">Syllabus</a></li>
      <li class="topics"><a href="topics.html">Topics</a></li>
      <li class="people"><a href="people.html">People</a></li>
    </ul>
  </div>
</header>

<div class="fluid-row" id="header">


</div>

<div id="TOC">
<ul>
<li><a href="#packages">Packages</a></li>
<li><a href="#the-lyrics">The lyrics</a></li>
<li><a href="#titles">Titles</a></li>
<li><a href="#and-purrr">And <code>purrr</code>!</a></li>
<li><a href="#the-flow">The flow</a></li>
<li><a href="#another-artist">Another Artist</a><ul>
<li><a href="#bonus-sentiment-analysis">Bonus: sentiment analysis</a></li>
</ul></li>
<li><a href="#what-about-the-rest">What about the rest?</a><ul>
<li><a href="#challenge">Challenge</a></li>
</ul></li>
</ul>
</div>

<p><img src="https://i.ytimg.com/vi/8_5TCqHVEW8/maxresdefault.jpg" /></p>
<p>We are starting in this lecture, and end in the next one.</p>
<p>The goal is to build a little collection of songs from our own preferred artist. Let’s say, it’s <em>Straight Line Stitch</em> (they are great!). A little kicker for the <a href="https://www.youtube.com/watch?v=4_5VAKdHMek">morning</a>.</p>
<p>The <strong>highly suggested</strong> browser (or, at least, the one that I’ll be using) is <a href="https://www.mozilla.org/en-US/firefox/developer/">Firefox</a>, the developer edition.</p>
<section id="packages" class="level2">
<h2>Packages</h2>
<blockquote>
<p>Don’t be afraid of the dark you’re still held up by the stars</p>
</blockquote>
<p>We are going to use a bunch of the usual packages:</p>
<pre class="r"><code>library(tidyverse)
library(magrittr)
library(purrr)
library(glue)
library(stringr)</code></pre>
<p>and introduce a new one:</p>
<pre class="r"><code>library(rvest)
library(xml2)</code></pre>
<p>which is meant explicitly to scrape stuff from a webpage. We are going to use a couple more in the bonus section, if we get there.</p>
</section>
<section id="the-lyrics" class="level2">
<h2>The lyrics</h2>
<p>We are going to extract the lyrics from here: <a href="https://www.musixmatch.com/" class="uri">https://www.musixmatch.com/</a> . Chose it because it’s rather consistent, and it’s from Bologna, Italy (yeah!).</p>
<p>The webiste offers the first 15 lyrics up front. That will do for the moment (and fixing that is not that easy). Let’s take a look <a href="https://www.musixmatch.com/artist/Straight-Line-Stitch#">here</a>.</p>
</section>
<section id="titles" class="level2">
<h2>Titles</h2>
<p>First thing first, we would like to get a list of those title. Let’s see how.</p>
<pre class="r"><code>url_titles &lt;- &quot;https://www.musixmatch.com/artist/Straight-Line-Stitch#&quot;

page_title &lt;- read_html(url_titles)</code></pre>
<p>Now, what is this <code>page_title</code> object?</p>
<p>let’s see:</p>
<pre class="r"><code>page_title</code></pre>
<pre><code>## {xml_document}
## &lt;html xmlns:og=&quot;http://ogp.me/ns#&quot; class=&quot;artist-page-page&quot;&gt;
## [1] &lt;head prefix=&quot;og: http://ogp.me/ns# fb: http://ogp.me/ns/fb# musixma ...
## [2] &lt;body spellcheck=&quot;false&quot; class=&quot;&quot;&gt;\n  &lt;div id=&quot;fb-root&quot;&gt;&lt;/div&gt;\n  &lt;d ...</code></pre>
<p>OK. It’s a document. Thanks. And it’s an XML document. That’s sort of html. We’ll handle it with <code>xml2</code> and <code>rvest</code>. Let’s see a bit more of that page.</p>
<pre class="r"><code>page_title %&gt;% html_structure()</code></pre>
<pre><code>## &lt;html.artist-page-page [xmlns:og]&gt;
##   &lt;head [prefix]&gt;
##     &lt;style&gt;
##       {cdata}
##     &lt;script&gt;
##       {cdata}
##     &lt;script&gt;
##       {cdata}
##     &lt;script&gt;
##       {cdata}
##     &lt;script&gt;
##       {cdata}
##     &lt;script [async, src]&gt;
##     &lt;script [type]&gt;
##       {cdata}
##     &lt;link [rel, type, href]&gt;
##     &lt;link [rel, type, href]&gt;
##     &lt;script&gt;
##       {cdata}
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;title&gt;
##       {text}
##     &lt;meta [charset]&gt;
##     &lt;meta [http-equiv, content]&gt;
##     &lt;meta [http-equiv, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;link [href, rel]&gt;
##     &lt;link [href, rel]&gt;
##     &lt;link [href, rel]&gt;
##     &lt;link [href, rel]&gt;
##     &lt;link [rel, type, href, title]&gt;
##     &lt;link [rel, href]&gt;
##     &lt;link [rel, href]&gt;
##     &lt;link [rel, href]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;link [rel, href, hreflang]&gt;
##     &lt;link [rel, href, hreflang]&gt;
##     &lt;link [rel, href, hreflang]&gt;
##     &lt;link [rel, href, hreflang]&gt;
##     &lt;link [rel, href, hreflang]&gt;
##     &lt;link [rel, href, hreflang]&gt;
##     &lt;link [rel, href, hreflang]&gt;
##     &lt;link [rel, href, hreflang]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;link [rel, sizes, href]&gt;
##     &lt;link [rel, sizes, href]&gt;
##     &lt;link [rel, href]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [name, content]&gt;
##     &lt;link [rel, href]&gt;
##     &lt;link [rel, href]&gt;
##     &lt;link [rel, href]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [property, content]&gt;
##     &lt;meta [http-equiv, content]&gt;
##     &lt;script&gt;
##       {cdata}
##     &lt;script&gt;
##       {cdata}
##     &lt;script [type]&gt;
##       {cdata}
##     &lt;script [type]&gt;
##       {cdata}
##   &lt;body. [spellcheck]&gt;
##     {text}
##     &lt;div#fb-root&gt;
##     {text}
##     &lt;div#site&gt;
##       &lt;div#artist-page . [data-reactroot]&gt;
##         &lt;div#header .navbar [role]&gt;
##           &lt;nav.top-bar.desktop-bar [data-topbar]&gt;
##             &lt;section.top-bar-section&gt;
##               &lt;ul.title-area&gt;
##                 &lt;li.name&gt;
##                   &lt;h1&gt;
##                     &lt;a#logo [href]&gt;
##                       &lt;img [alt, width, src]&gt;
##               &lt;ul.search-nav [role]&gt;
##                 &lt;li.search.has-form&gt;
##                   &lt;form&gt;
##                     &lt;div#search-bar&gt;
##                       &lt;input#search-input [type, placeholder, value, autocomplete]&gt;
##                       &lt;button#search-button .button-icon&gt;
##                         &lt;i.icon.icon-search&gt;
##               &lt;div.top-bar-right&gt;
##                 &lt;ul.site-nav [role]&gt;
##                   &lt;li&gt;
##                     &lt;a [href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [href]&gt;
##                       {text}
##                 &lt;ul.user-nav&gt;
##                   &lt;li&gt;
##                     &lt;ul.user-nav-items.loggedOut&gt;
##                       &lt;li#sign-in&gt;
##                         &lt;a [data-external-link, href]&gt;
##                           {text}
##                       &lt;li#sign-up&gt;
##                         &lt;a [data-external-link, href]&gt;
##                           {text}
##           &lt;nav.top-bar.mobile-top-bar&gt;
##             &lt;ul.title-area&gt;
##               &lt;li.offcanvas-toggler&gt;
##                 &lt;a [href]&gt;
##                   &lt;i.icon.icon-list&gt;
##               &lt;li.name&gt;
##                 &lt;h1&gt;
##                   &lt;a [href]&gt;
##                     &lt;img [alt, src]&gt;
##               &lt;li.toggle-search-bar&gt;
##                 &lt;a [href]&gt;
##                   &lt;span&gt;
##                     &lt;i.icon.icon-search&gt;
##               &lt;li.search.has-form&gt;
##                 &lt;form&gt;
##                   &lt;div#search-bar&gt;
##                     &lt;input#search-input [type, placeholder, value, autocomplete]&gt;
##                     &lt;button#search-button .button-icon&gt;
##                       &lt;i.icon.icon-search&gt;
##                   &lt;button.button.button-link.search-bar-cancel&gt;
##                     &lt;i.icon.icon-cross&gt;
##           &lt;noscript&gt;
##         &lt;div#content [role]&gt;
##           &lt;div.artist-page.main-wrapper [itemtype]&gt;
##             &lt;meta [content, itemprop]&gt;
##             &lt;div.profile-header.artist-header&gt;
##               &lt;div.main-wrapper&gt;
##                 &lt;div.noCoverPlaceholder&gt;
##                 &lt;div.bg&gt;
##                 &lt;div.profile-info&gt;
##                   &lt;div.profile-head-link&gt;
##                     &lt;div.profile-image&gt;
##                       &lt;img.profile-avatar.large [itemprop, src, alt]&gt;
##                     &lt;div.profile-data&gt;
##                       &lt;meta [itemprop, content]&gt;
##                       &lt;meta [itemprop, content]&gt;
##                       &lt;span.lyricsLabel&gt;
##                         {text}
##                       &lt;h1. [title, itemprop]&gt;
##                         {text}
##                       &lt;ul.genres&gt;
##                         &lt;li&gt;
##                           &lt;a [href]&gt;
##                             {text}
##                     &lt;a.verify-link [href]&gt;
##                       {text}
##                   &lt;button.button-primary.contribute-button&gt;
##                     {text}
##                     {comment}
##                     {text}
##                   &lt;ul.profile-head-info&gt;
##                   &lt;span&gt;
##                   &lt;span&gt;
##             &lt;div.profile-content&gt;
##               &lt;div&gt;
##                 &lt;div.section-header&gt;
##                   &lt;ul.tabs&gt;
##                     &lt;li#overview .selected&gt;
##                       &lt;a [href]&gt;
##                         {text}
##                     &lt;li#community .&gt;
##                       &lt;a [href]&gt;
##                         {text}
##                     &lt;li#albums .&gt;
##                       &lt;a [href]&gt;
##                         {text}
##                 &lt;div.tab-content&gt;
##                   &lt;div.main-wrapper&gt;
##                     &lt;div.main&gt;
##                       &lt;div.row&gt;
##                         &lt;div.main-panel&gt;
##                           &lt;div#top-songs .box.box-style-plain&gt;
##                             &lt;div.box-header&gt;
##                               &lt;h3.box-title&gt;
##                                 {text}
##                                 {comment}
##                                 {text}
##                               &lt;div.actions&gt;
##                             &lt;div.box-content&gt;
##                               &lt;div&gt;
##                                 &lt;div&gt;
##                                   &lt;ul.tracks.list&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                     &lt;li.showPosition.showCoverart&gt;
##                                       &lt;span.position&gt;
##                                         {text}
##                                       &lt;div.track-card.media-card.has-picture&gt;
##                                         &lt;div.media-card-picture&gt;
##                                           &lt;img [srcset, alt]&gt;
##                                         &lt;div.media-card-body&gt;
##                                           &lt;div.media-card-text&gt;
##                                             &lt;h2.media-card-title&gt;
##                                               &lt;a.title [href]&gt;
##                                                 &lt;span&gt;
##                                                   {text}
##                                             &lt;h3.media-card-subtitle&gt;
##                                               &lt;span.artist-field&gt;
##                                                 &lt;span&gt;
##                                                   &lt;a.artist [href]&gt;
##                                                     {text}
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                         &lt;meta [content, itemprop]&gt;
##                                 &lt;div.load-more-pager&gt;
##                                   &lt;a.button.page-load-more [href, data-page]&gt;
##                                     {text}
##                         &lt;div.side-panel&gt;
##                           &lt;div.box.box-style-plain&gt;
##                             &lt;div.box-content&gt;
##                               &lt;div. [style]&gt;
##                                 &lt;div#div_gpt_ad_336x280_musixmatch_artist_sidebar_top [style]&gt;
##                                   &lt;script [type]&gt;
##                                     {cdata}
##                           &lt;div#albums .box.box-style-plain&gt;
##                             &lt;div.box-header&gt;
##                               &lt;h3.box-title&gt;
##                                 {text}
##                                 {comment}
##                                 {text}
##                               &lt;div.actions&gt;
##                             &lt;div.box-content&gt;
##                               &lt;ul.song-writers&gt;
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                                 &lt;li&gt;
##                                   &lt;a [href]&gt;
##                                     {text}
##                           &lt;div#albums .box.box-style-plain&gt;
##                             &lt;div.box-header&gt;
##                               &lt;h3.box-title&gt;
##                                 {text}
##                                 {comment}
##                                 {text}
##                               &lt;div.actions&gt;
##                                 &lt;a.see-all-action [href]&gt;
##                                   {text}
##                             &lt;div.box-content&gt;
##                               &lt;ul.albums.list&gt;
##                                 &lt;li&gt;
##                                   &lt;div.album-card.media-card.has-picture&gt;
##                                     &lt;div.media-card-picture&gt;
##                                       &lt;img [srcset, alt]&gt;
##                                     &lt;div.media-card-body&gt;
##                                       &lt;div.media-card-text&gt;
##                                         &lt;h2.media-card-title&gt;
##                                           &lt;a [href]&gt;
##                                             {text}
##                                         &lt;h3.media-card-subtitle&gt;
##                                           {text}
##                                     &lt;meta [content, itemprop]&gt;
##                                     &lt;meta [content, itemprop]&gt;
##                                 &lt;li&gt;
##                                   &lt;div.album-card.media-card.has-picture&gt;
##                                     &lt;div.media-card-picture&gt;
##                                       &lt;img [srcset, alt]&gt;
##                                     &lt;div.media-card-body&gt;
##                                       &lt;div.media-card-text&gt;
##                                         &lt;h2.media-card-title&gt;
##                                           &lt;a [href]&gt;
##                                             {text}
##                                         &lt;h3.media-card-subtitle&gt;
##                                           {text}
##                                     &lt;meta [content, itemprop]&gt;
##                                     &lt;meta [content, itemprop]&gt;
##                                 &lt;li&gt;
##                                   &lt;div.album-card.media-card.has-picture&gt;
##                                     &lt;div.media-card-picture&gt;
##                                       &lt;img [srcset, alt]&gt;
##                                     &lt;div.media-card-body&gt;
##                                       &lt;div.media-card-text&gt;
##                                         &lt;h2.media-card-title&gt;
##                                           &lt;a [href]&gt;
##                                             {text}
##                                         &lt;h3.media-card-subtitle&gt;
##                                           {text}
##                                     &lt;meta [content, itemprop]&gt;
##                                     &lt;meta [content, itemprop]&gt;
##                                 &lt;li&gt;
##                                   &lt;div.album-card.media-card.has-picture&gt;
##                                     &lt;div.media-card-picture&gt;
##                                       &lt;img [srcset, alt]&gt;
##                                     &lt;div.media-card-body&gt;
##                                       &lt;div.media-card-text&gt;
##                                         &lt;h2.media-card-title&gt;
##                                           &lt;a [href]&gt;
##                                             {text}
##                                         &lt;h3.media-card-subtitle&gt;
##                                           {text}
##                                     &lt;meta [content, itemprop]&gt;
##                                     &lt;meta [content, itemprop]&gt;
##                                 &lt;li&gt;
##                                   &lt;div.album-card.media-card.has-picture&gt;
##                                     &lt;div.media-card-picture&gt;
##                                       &lt;img [srcset, alt]&gt;
##                                     &lt;div.media-card-body&gt;
##                                       &lt;div.media-card-text&gt;
##                                         &lt;h2.media-card-title&gt;
##                                           &lt;a [href]&gt;
##                                             {text}
##                                         &lt;h3.media-card-subtitle&gt;
##                                           {text}
##                                     &lt;meta [content, itemprop]&gt;
##                                     &lt;meta [content, itemprop]&gt;
##         &lt;span&gt;
##         &lt;div#footer&gt;
##           &lt;div.main-wrapper&gt;
##             &lt;div.footer-container&gt;
##               &lt;div.mxm-footer__sitemap&gt;
##                 &lt;div.mxm-footer__sitemap-logo&gt;
##                   &lt;div.mui-logo&gt;
##                     &lt;a.mui-logo__link [href]&gt;
##                       &lt;img.mui-logo__img [src, alt, width]&gt;
##                   &lt;div.mxm-footer__language&gt;
##                     &lt;button#language-menu .small.button.dropdown&gt;
##                       {text}
##               &lt;div.mxm-footer__sitemap-links&gt;
##                 &lt;h4&gt;
##                   {text}
##                 &lt;ul&gt;
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##               &lt;div.mxm-footer__sitemap-links&gt;
##                 &lt;h4&gt;
##                   {text}
##                 &lt;ul&gt;
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##               &lt;div.mxm-footer__sitemap-links&gt;
##                 &lt;h4&gt;
##                   {text}
##                 &lt;ul&gt;
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##                       {comment}
##                       {text}
##                       &lt;span.label.label-primary.text-uppercase&gt;
##                         {text}
##                   &lt;li&gt;
##                     &lt;a [href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, href]&gt;
##                       {text}
##                   &lt;li&gt;
##                     &lt;a [data-external-link, target, href]&gt;
##                       {text}
##             &lt;div.mxm-footer__footer-language&gt;
##               &lt;button#language-menu .small.button.dropdown&gt;
##                 {text}
##             &lt;div.mxm-footer__footnote&gt;
##               &lt;div.copyright&gt;
##                 &lt;span&gt;
##                   {text}
##                 &lt;span.mxm-footer__footnote-dash&gt;
##                   {text}
##                 &lt;ul [itemtype]&gt;
##                   &lt;link [itemprop, href]&gt;
##                   &lt;li&gt;
##                     &lt;a.social-icon.fb-icon [target, href, itemprop, type]&gt;
##                       &lt;i.icon.icon-facebook2&gt;
##                   &lt;li&gt;
##                     &lt;a.social-icon.tw-icon [target, href, itemprop, type]&gt;
##                       &lt;i.icon.icon-twitter&gt;
##                   &lt;li&gt;
##                     &lt;a.social-icon.googleplus-icon [target, href, itemprop, type]&gt;
##                       &lt;i.icon.icon-googleplus&gt;
##                   &lt;li&gt;
##                     &lt;a.social-icon.youtube-icon [target, href, itemprop, type]&gt;
##                       &lt;i.icon.icon-youtube2&gt;
##                   &lt;li&gt;
##                     &lt;a.social-icon.pinterest-icon [target, href, itemprop, type]&gt;
##                       &lt;i.icon.icon-pinterest&gt;
##                   &lt;li&gt;
##                     &lt;a.social-icon.instagram-icon [target, href, itemprop, type]&gt;
##                       &lt;i.icon.icon-instagram&gt;
##               &lt;p&gt;
##                 {text}
##         &lt;span&gt;
##     {text}
##     &lt;script&gt;
##       {cdata}
##     &lt;script [async, src]&gt;
##     &lt;script&gt;
##       {cdata}</code></pre>
<p>Wait, whaaaaaat?</p>
<p><img src="https://media.giphy.com/media/ZkEXisGbMawMg/giphy.gif" /></p>
<p>To the browser! Look at that “class” tags: they are <em>css selectors</em>, and we will use them as handles to navigate into the extremely complex list that we get from a web page.</p>
<p>Sometimes, we can be lucky. For example, the css selector for the titles are in the class “.title”. Let’s see.</p>
<pre class="r"><code>page_title %&gt;%
  html_nodes(&quot;.title&quot;)</code></pre>
<pre><code>## {xml_nodeset (15)}
##  [1] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Laughing-In-The ...
##  [2] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Black-Veil&quot;&gt;&lt;sp ...
##  [3] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Never-Surrender ...
##  [4] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/What-You-Do-to- ...
##  [5] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Taste-of-Ashes&quot; ...
##  [6] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Conversion&quot;&gt;&lt;sp ...
##  [7] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Promise-Me&quot;&gt;&lt;sp ...
##  [8] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Bar-Room-Brawl&quot; ...
##  [9] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/One-Reason&quot;&gt;&lt;sp ...
## [10] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Yesterday-s-Gon ...
## [11] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Eucharist&quot;&gt;&lt;spa ...
## [12] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Adult-Cinema&quot;&gt;&lt; ...
## [13] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Living-Dead&quot;&gt;&lt;s ...
## [14] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Bleeding-Heart- ...
## [15] &lt;a class=&quot;title&quot; href=&quot;/lyrics/Straight-Line-Stitch/Cold-Front&quot;&gt;&lt;sp ...</code></pre>
<p>That’s still quite a mess: we have too much stuff, such as some links (called “href”) and more text than we need. Let’s clean it up with <code>html_text()</code></p>
<pre class="r"><code>page_title %&gt;%
  html_nodes(&quot;.title&quot;) %&gt;%
  html_text()</code></pre>
<pre><code>##  [1] &quot;Laughing In The Rearview&quot; &quot;Black Veil&quot;              
##  [3] &quot;Never Surrender&quot;          &quot;What You Do to Me&quot;       
##  [5] &quot;Taste of Ashes&quot;           &quot;Conversion&quot;              
##  [7] &quot;Promise Me&quot;               &quot;Bar Room Brawl&quot;          
##  [9] &quot;One Reason&quot;               &quot;Yesterday&#39;s Gone&quot;        
## [11] &quot;Eucharist&quot;                &quot;Adult Cinema&quot;            
## [13] &quot;Living Dead&quot;              &quot;Bleeding Heart Theory&quot;   
## [15] &quot;Cold Front&quot;</code></pre>
<p>Wundebar! Now we have 15 song titles. But we want the lyrics! Let’s do better.</p>
<pre class="r"><code>SLS_df &lt;- data_frame(Band = &quot;Straight Line Stitch&quot;,
                     Title = page_title %&gt;%
                       html_nodes(&quot;.title&quot;) %&gt;%
                       html_text())</code></pre>
<p>Now we are going to use a bit of string magic</p>
<pre class="r"><code>SLS_lyrics &lt;- SLS_df %&gt;% mutate(Link = glue(&#39;https://www.musixmatch.com/lyrics/{Band}/{Title}&#39;) %&gt;%
                           str_replace_all(&quot; &quot;,&quot;-&quot;))</code></pre>
<p>It seems it works.</p>
<p>There is a better trick to do this job. If we look again at what we get when we select the <code>.title</code> you may see that the <em>actual</em> link is there, coded as <code>href</code>. Can we extract that? Yes we can!</p>
<pre class="r"><code>page_title %&gt;%
  html_nodes(&quot;.title&quot;) %&gt;%
  html_attrs() %&gt;%
  glimpse()</code></pre>
<pre><code>## List of 15
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Laughing-In-The-Rearview&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Black-Veil&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Never-Surrender&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/What-You-Do-to-Me&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Taste-of-Ashes&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Conversion&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Promise-Me&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Bar-Room-Brawl&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/One-Reason&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Yesterday-s-Gone&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Eucharist&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Adult-Cinema&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Living-Dead&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Bleeding-Heart-Theory&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;
##  $ : Named chr [1:2] &quot;title&quot; &quot;/lyrics/Straight-Line-Stitch/Cold-Front&quot;
##   ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;class&quot; &quot;href&quot;</code></pre>
<p>In particular, we want the element called <code>href</code>. Hey, we can get that with <code>map</code>!</p>
<pre class="r"><code>page_title %&gt;%
  html_nodes(&quot;.title&quot;) %&gt;%
  html_attrs() %&gt;%
  map_chr(&quot;href&quot;)</code></pre>
<pre><code>##  [1] &quot;/lyrics/Straight-Line-Stitch/Laughing-In-The-Rearview&quot;
##  [2] &quot;/lyrics/Straight-Line-Stitch/Black-Veil&quot;              
##  [3] &quot;/lyrics/Straight-Line-Stitch/Never-Surrender&quot;         
##  [4] &quot;/lyrics/Straight-Line-Stitch/What-You-Do-to-Me&quot;       
##  [5] &quot;/lyrics/Straight-Line-Stitch/Taste-of-Ashes&quot;          
##  [6] &quot;/lyrics/Straight-Line-Stitch/Conversion&quot;              
##  [7] &quot;/lyrics/Straight-Line-Stitch/Promise-Me&quot;              
##  [8] &quot;/lyrics/Straight-Line-Stitch/Bar-Room-Brawl&quot;          
##  [9] &quot;/lyrics/Straight-Line-Stitch/One-Reason&quot;              
## [10] &quot;/lyrics/Straight-Line-Stitch/Yesterday-s-Gone&quot;        
## [11] &quot;/lyrics/Straight-Line-Stitch/Eucharist&quot;               
## [12] &quot;/lyrics/Straight-Line-Stitch/Adult-Cinema&quot;            
## [13] &quot;/lyrics/Straight-Line-Stitch/Living-Dead&quot;             
## [14] &quot;/lyrics/Straight-Line-Stitch/Bleeding-Heart-Theory&quot;   
## [15] &quot;/lyrics/Straight-Line-Stitch/Cold-Front&quot;</code></pre>
<p>Or, even better, by letting <code>rves</code> do the job for us:</p>
<pre class="r"><code>page_title %&gt;%
  html_nodes(&quot;.title&quot;) %&gt;%
  html_attr(&quot;href&quot;)</code></pre>
<pre><code>##  [1] &quot;/lyrics/Straight-Line-Stitch/Laughing-In-The-Rearview&quot;
##  [2] &quot;/lyrics/Straight-Line-Stitch/Black-Veil&quot;              
##  [3] &quot;/lyrics/Straight-Line-Stitch/Never-Surrender&quot;         
##  [4] &quot;/lyrics/Straight-Line-Stitch/What-You-Do-to-Me&quot;       
##  [5] &quot;/lyrics/Straight-Line-Stitch/Taste-of-Ashes&quot;          
##  [6] &quot;/lyrics/Straight-Line-Stitch/Conversion&quot;              
##  [7] &quot;/lyrics/Straight-Line-Stitch/Promise-Me&quot;              
##  [8] &quot;/lyrics/Straight-Line-Stitch/Bar-Room-Brawl&quot;          
##  [9] &quot;/lyrics/Straight-Line-Stitch/One-Reason&quot;              
## [10] &quot;/lyrics/Straight-Line-Stitch/Yesterday-s-Gone&quot;        
## [11] &quot;/lyrics/Straight-Line-Stitch/Eucharist&quot;               
## [12] &quot;/lyrics/Straight-Line-Stitch/Adult-Cinema&quot;            
## [13] &quot;/lyrics/Straight-Line-Stitch/Living-Dead&quot;             
## [14] &quot;/lyrics/Straight-Line-Stitch/Bleeding-Heart-Theory&quot;   
## [15] &quot;/lyrics/Straight-Line-Stitch/Cold-Front&quot;</code></pre>
<pre class="r"><code>SLS_df %&lt;&gt;%
  mutate(Link = page_title %&gt;%
  html_nodes(&quot;.title&quot;) %&gt;%
  html_attr(&quot;href&quot;))</code></pre>
<p>Cool, we don’t gain much in terms of line of code, but it will be usefull later!</p>
</section>
<section id="and-purrr" class="level2">
<h2>And <code>purrr</code>!</h2>
<p>Cool, now we want to put grab all lyrics. Let’s start with one at a time. What is the url we want?</p>
<pre class="r"><code>url_song &lt;- glue(&quot;https://www.musixmatch.com{SLS_df$Link[1]}&quot;)

url_song</code></pre>
<pre><code>## https://www.musixmatch.com/lyrics/Straight-Line-Stitch/Laughing-In-The-Rearview</code></pre>
<p>And let’s grab the lyrics for that song. The content is marked by a css selector called “p.mxm-lyrics__content“. That stands for”p“, an object of class paragraph, plus”mxm-lyrics__content“, the specific class for the lyrics.</p>
<pre class="r"><code>url_song %&gt;%
  read_html() %&gt;%
  html_nodes(&quot;.mxm-lyrics__content&quot;) %&gt;%
  html_text()</code></pre>
<pre><code>## [1] &quot;You never say what you mean, mean what you say\n\nPick up the pieces&quot;                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
## [2] &quot;And heal the pain of your cruelty\nMaybe one day you&#39;ll be able to say\nThat you&#39;re truly free but there&#39;s no guarantee\n\nIt&#39;s always the same everywhere you go\nWelcome to the slide-show\nRight up front for the whole world to see your display\nYou will never get away\n\nYou never say what you mean, mean what you say\nI&#39;ll blow you away, blow you away\nShot down in a blaze now you&#39;re lost\nWithout a trace, without a trace\n\nBut don&#39;t get caught\nOut by the undertow, out by the undertow\nBecause today is the day that you&#39;ll never\nNever escape, never escape\n\nI hope you see my face as clear as day\nAnd it continues to haunt you\nMaybe someday you&#39;ll be able to breakthrough\nBut you don&#39;t have that kind of virtue\n\nIt&#39;s always the same every time I move\nLike you got something to prove, something to prove\nYou think I&#39;d lose my edge over the third degree\nExcuse me if I disagree\n\nYou never say what you mean, mean what you say\nI&#39;ll blow you away, blow you away\nShot down in a blaze now you&#39;re lost\nWithout a trace, without a trace\n\nBut don&#39;t get caught\nOut by the undertow, out by the undertow\nBecause today is the day that you&#39;ll never\nNever escape, never escape\n\nYou&#39;ll never escape\nYou turned your back on me\nYou turned your back on me\n\nYou&#39;ll never, never escape\nYou&#39;ll never, never escape\nYou&#39;ll never, never escape\nYou&#39;ll never, never escape&quot;</code></pre>
<p>Ach, notice that it comes in different blocks: one for each section of text, broken by the advertisment. Well, we can just <code>collapse()</code> them together with <code>glue</code>. As we are doing this, let’s turn that flow into a function:</p>
<pre class="r"><code>get_lyrics &lt;- function(link){
  
  lyrics_chunks &lt;- glue(&quot;https://www.musixmatch.com{link}#&quot;) %&gt;%
   read_html() %&gt;%
   html_nodes(&quot;.mxm-lyrics__content&quot;)
  
  # we do a sanity check to see that there&#39;s something inside the lyrics!
  stopifnot(length(lyrics_chunks) &gt; 0)
  
  lyrics &lt;- lyrics_chunks %&gt;%
   html_text() %&gt;%
   collapse(sep = &quot;\n&quot;)
  
  return(lyrics)
}</code></pre>
<p>Let’s test it!</p>
<pre class="r"><code>SLS_df$Link[3] %&gt;%
  get_lyrics() %&gt;%
  glue() # we paste into glue to get the nice formatting</code></pre>
<pre><code>## Out from the darkness into the light
## I opened up my eyes, awakened by the sight
## Of the walls that are closing in, it&#39;s happening again
## Just another black cloud that&#39;s become my only friend
## 
## Bring it up, brother, no one said it&#39;d be easy
## And you don&#39;t have to believe me when I tell you
## It&#39;s not over, it&#39;s not over
## 
## I&#39;ve gone too far to back down now
## Gotta finish what I started
## It&#39;s not over, no, it&#39;s not over
## 
## So hold on, said we&#39;ll never surrender
## If there&#39;s a chance of losing more then let it go
## So hold on, said we&#39;ll never surrender
## If there&#39;s a chance of losing more then let it go
## 
## Another sleepless night alone and I can&#39;t do this on my own
## I&#39;m reaching out to touch the one just like me
## But no one seems to hear my voice
## It echoes across the void, shattering with the noise
## 
## Bring it on, this is the day for redemption
## You don&#39;t have to wake him
## Whether we&#39;re wrong or if we&#39;re right
## Whether we&#39;re wrong or if we&#39;re right
## 
## I&#39;ve gone too far to back down now
## Gotta finish what I started
## It&#39;s not over, no, it&#39;s not over
## 
## So hold on, said we&#39;ll never surrender
## If there&#39;s a chance of losing more then let it go
## So hold on, said we&#39;ll never surrender
## If there&#39;s a chance of losing more then let it go
## So hold on
## 
## This is the fight of our lives
## So hold on
## With all the [Incomprehensible]
## Hold on, hold on, hold on, hold on
## 
## So hold on, said we&#39;ll never surrender
## If there&#39;s a chance of losing more then let it go
## So hold on, said we&#39;ll never surrender
## If there&#39;s a chance of losing more then let it go
## 
## Hold on, hold on</code></pre>
<p>Now we can use purrr to map that function over our dataframe!</p>
<pre class="r"><code>SLS_df %&lt;&gt;%
  mutate(Lyrics = map_chr(Link, get_lyrics))</code></pre>
<p>Ok, here we were quite lucky, as all the links were right. In general we may want to play safe, and use a <code>possibly</code> wrapper so not to have to stop everything in case something bad happens.</p>
</section>
<section id="the-flow" class="level2">
<h2>The flow</h2>
<p><strong>Explore, try, test, automatize, test.</strong></p>
<p>Scraping data from the web will require a lot of trial and error. In general, I like this flow: I explore the pages that I want to scrape, trying to identify patterns that I can exploit. Then I try, on a smaller subset, and I test if it worked. Then I automatize it, using <code>purrr</code> or something similar. And finally some more testing.</p>
</section>
<section id="another-artist" class="level2">
<h2>Another Artist</h2>
<p>Let’s do this for Angel Haze. Notice that here we <strong>have</strong> to use the attributes from the web page, as the name of the authors of the lyrics is not always the same (the <code>glue</code> approach would fail).</p>
<pre class="r"><code>AH_url &lt;- &quot;https://www.musixmatch.com/artist/Angel-Haze&quot;

AH_lyrics &lt;- data_frame(Band = &quot;Angel Haze&quot;,
                        
                         Title = AH_url %&gt;%
                          read_html() %&gt;%
                           html_nodes(css = &quot;.title&quot;) %&gt;%
                           html_text(),
                        
                         Link = AH_url %&gt;%
                          read_html() %&gt;%
                           html_nodes(css = &quot;.title&quot;) %&gt;%
                          html_attr(&quot;href&quot;),
                        
                        Lyrics = map_chr(Link,get_lyrics))</code></pre>
<section id="bonus-sentiment-analysis" class="level3">
<h3>Bonus: sentiment analysis</h3>
<p>The idea is to attribute to each word a score, expressing wether it’s more negative and positive, and then to sum up. To do this, we are going to use Julia Silge’s and David Robinson’s great <a href="https://github.com/juliasilge/tidytext"><em>Tidytext</em></a> library and a <em>vocabulary</em> of words for which we have the scores (there are different options, we are using “afinn”).</p>
<pre class="r"><code>library(tidytext)
afinn &lt;- get_sentiments(&quot;afinn&quot;)</code></pre>
<p>Now, a bit of massaging: we breaks the lyrics into their words, remove the words that are considered not interesting (they are called “stop words”), stitch the dataframe to the scoress from afinn, and do the math for each song.</p>
<pre class="r"><code>SLS_df %&gt;%
  unnest_tokens(word, Lyrics) %&gt;% #split words
  anti_join(stop_words, by = &quot;word&quot;) %&gt;% #remove dull words
  inner_join(afinn, by = &quot;word&quot;) %&gt;% #stitch scores
  group_by(Title) %&gt;% #and for each song
  summarise(Length = n(), #do the math
    Score = sum(score)/Length) %&gt;%
  arrange(-Score)</code></pre>
<pre><code>## # A tibble: 15 x 3
##                       Title Length       Score
##                       &lt;chr&gt;  &lt;int&gt;       &lt;dbl&gt;
##  1               Promise Me     17  0.23529412
##  2         Yesterday&#39;s Gone     15  0.20000000
##  3        What You Do to Me     17  0.17647059
##  4              Living Dead     26  0.07692308
##  5                Eucharist      9  0.00000000
##  6           Bar Room Brawl     22 -0.04545455
##  7               Black Veil     13 -0.07692308
##  8          Never Surrender     18 -0.55555556
##  9               Cold Front     12 -0.58333333
## 10           Taste of Ashes     16 -0.68750000
## 11 Laughing In The Rearview     19 -0.78947368
## 12               One Reason     10 -0.90000000
## 13               Conversion     15 -1.46666667
## 14    Bleeding Heart Theory     14 -1.92857143
## 15             Adult Cinema      2 -3.00000000</code></pre>
<p>So, what was the most positive song?</p>
<pre class="r"><code>SLS_df %&gt;%
  filter(Title == &quot;Promise Me&quot;) %$%
  Lyrics %&gt;%
  glue()</code></pre>
<pre><code>## This is the last day
## The last day that I sway
## No longer will you dictate what I do or say
## I&#39;ve taken a leave of absence
## I&#39;ve taken a leave of the senses
## And there&#39;s nothing in the world that could change this
## I can&#39;t change this
## No regrets
## I&#39;m calling on you angels of truth
## Tell me that you&#39;ll never fall through
## Promise me Promise me
## Don&#39;t crumble like paper mache’
## Tell me that you&#39;ll always find your way
## Promise me Promise me
## This is the last day
## The last day that I sway
## No longer will they dictate what I do or say
## Not another dying day
## A part of me that&#39;s wasted away
## And there&#39;s nothing in the world that could change this
## You can&#39;t change this
## No regrets
## I&#39;m calling on you angels of truth
## Tell me that you&#39;ll never fall through
## Promise me Promise me
## Don&#39;t crumble like paper mache’
## Tell me that you&#39;ll always find your way
## Promise me Promise me
## Standing on the sidelines
## Where I&#39;ve been left behind
## Right here on the sidelines
## Waiting for the right time
## And life goes by
## Life goes by
## I&#39;m calling on you angels of truth
## Tell me that you&#39;ll never fall through
## Promise me Promise me
## Don&#39;t crumble like paper mache’
## Tell me that you&#39;ll always find your way
## Promise me Promise me
## This is the last day
## And life goes by</code></pre>
<p>And we can easily do the same with Angela Haze:</p>
<pre class="r"><code>AH_lyrics %&gt;%
  unnest_tokens(word, Lyrics) %&gt;% #split words
  anti_join(stop_words, by = &quot;word&quot;) %&gt;% #remove dull words
  inner_join(afinn, by = &quot;word&quot;) %&gt;% #stitch scores
  group_by(Title) %&gt;% #and for each song
  summarise(Length = n(), #do the math
    Score = sum(score)/Length) %&gt;%
  arrange(-Score)</code></pre>
<pre><code>## # A tibble: 15 x 3
##                                                      Title Length
##                                                      &lt;chr&gt;  &lt;int&gt;
##  1                                      Pleasure This Pain     50
##  2                                              I Love You     15
##  3                                              Shit, Man!     12
##  4                                                    NSFW     35
##  5                                              Battle Cry     56
##  6                                            Black Dahlia     76
##  7                                       Hell Could Freeze     40
##  8                                           Sing About Me     28
##  9                                                 Ribcage      9
## 10                                                 On Fire     41
## 11 Weapon (Bastille vs. Angel Haze vs. F*U*G*Z vs. Braque)     25
## 12                                                  Weapon     22
## 13 Weapon (Bastille Vs. Angel Haze Vs. F*U*G*Z Vs. Braque)     22
## 14                                          22 Jump Street     25
## 15                                                   D-Day     18
## # ... with 1 more variables: Score &lt;dbl&gt;</code></pre>
<p>More resources about Sentiment Analysis (with Tidytext) are available <a href="http://varianceexplained.org/r/yelp-sentiment/">here</a> and <a href="http://www.jakubglinka.com/2017-03-01-text_mining_part1/">here</a>.</p>
</section>
</section>
<section id="what-about-the-rest" class="level2">
<h2>What about the rest?</h2>
<p>We want to do it also for other artists. Best things is to turn some of those scripts into functions. Let’s try with <em>Billie Holiday</em> and <em>A Tribe Called Red</em> (I picked them ’cause they are great, and also because they will show some limitations of the code I’m interested to tackle).</p>
<p>When we are about to do something over over, it’s better to write functions. So, let’s do it!</p>
<pre class="r"><code>get_words &lt;- function(band_name){

  # remove white space from band name
  collapsed_name &lt;- str_replace_all(band_name, &quot; &quot;, &quot;-&quot;) 
  
  # define url to get the title and links
  url &lt;- glue(&quot;https://www.musixmatch.com/artist/{collapsed_name}&quot;)
  
  # read title page and extract the title chunks 
  title_page &lt;- url %&gt;%
    read_html() %&gt;%
    html_nodes(css = &quot;.title&quot;)
  
  # and build the dataframe
  lyrics &lt;- data_frame(Band = band_name,
                       # extract text title
                       Title = title_page %&gt;%
                           html_text(),
                       # extract title link
                       Link =  title_page %&gt;%
                          html_attr(&quot;href&quot;),
                       # map to get lyrics
                       Lyrics = map_chr(Link,get_lyrics))
  
  return(lyrics)
}</code></pre>
<p>And the sentiment analysis:</p>
<pre class="r"><code>get_soul &lt;- function(Lyrics_df) {
  Lyrics_df %&gt;%
    unnest_tokens(word, Lyrics) %&gt;% #split words
    #anti_join(stop_words, by = &quot;word&quot;) %&gt;% #remove dull words
    inner_join(afinn, by = &quot;word&quot;) %&gt;% #stitch scores
    group_by(Title) %&gt;% #and for each song
    summarise(Length = n(), #do the math
      Score = sum(score)/Length) %&gt;%
    arrange(-Score) %&gt;%
    return()
}</code></pre>
<p>Let’s see if it works:</p>
<pre class="r"><code>Billie_words &lt;- &quot;Billie Holiday&quot; %&gt;% get_words()
Billie_sentiment &lt;- Billie_words %&gt;% get_soul()</code></pre>
<p>Most positive song:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/HxG6K59FUGQ" frameborder="0" allowfullscreen>
</iframe>
<p>and most negative <iframe width="560" height="315" src="https://www.youtube.com/embed/EIgVCU19pjg" frameborder="0" allowfullscreen></iframe></p>
<p>It works for me! Let’s finally with <em>A Tribe Called Red</em>.</p>
<pre class="r"><code>ATCR_words &lt;- &quot;A Tribe Called Red&quot; %&gt;% get_words()</code></pre>
<p>Uuuh, we get an error now! What the problem? Where, let’s try to see. Some of the songs do not have lyrics, yet! So, when we try to scrape this <a href="https://www.musixmatch.com/lyrics/A-Tribe-Called-Red-ft-Black-Bear/Stadium-Pow-Wow">page</a> we get an error, as the text is not there.</p>
<p>This is a rather common situation when scraping, as often what we are looking for is not there. Thus, we need a more safe approach. We can either write ad hoc <code>if ...else ...</code> statements, to control for the presence/absence of things, or (and it is better to do it anyhow) wrap our function into <code>purrr::possibly()</code> construct. We can do it by modifying just slightly our workflow. Notice that now we use <code>get_lyrics_safe()</code> inside the mapping instead of <code>get_lyrics()</code>.</p>
<pre class="r"><code>get_lyrics_safe &lt;- purrr::possibly(get_lyrics,NA_character_)

get_words &lt;- function(band_name){

  # remove white space from band name
  collapsed_name &lt;- str_replace_all(band_name, &quot; &quot;, &quot;-&quot;) 
  
  # define url to get the title and links
  url &lt;- glue(&quot;https://www.musixmatch.com/artist/{collapsed_name}&quot;)
  
  # read title page and extract the title chunks 
  title_page &lt;- url %&gt;%
    read_html() %&gt;%
    html_nodes(css = &quot;.title&quot;)
  
  # and build the dataframe
  lyrics &lt;- data_frame(Band = band_name,
                       # extract text title
                       Title = title_page %&gt;%
                           html_text(),
                       # extract title link
                       Link =  title_page %&gt;%
                          html_attr(&quot;href&quot;),
                       # map to get lyrics
                       Lyrics = map_chr(Link,get_lyrics_safe))
  
  return(lyrics)
}</code></pre>
<p>Let’s try again:</p>
<pre class="r"><code>ATCR_words &lt;- &quot;A Tribe Called Red&quot; %&gt;% get_words()</code></pre>
<p>Much better.</p>
<pre class="r"><code>ATCR_sentiment &lt;- ATCR_words %&gt;% get_soul()</code></pre>
<section id="challenge" class="level3">
<h3>Challenge</h3>
<p>Another singer you should, should, should listen to is <em>Militia Vox</em>. Try to replicate our work with her lyrics. What’s the problem? (If you think you get the answer, please discuss with me :-) )</p>
<p><strong>note</strong>: this workthrough is loosely inspired by Max Humber’s <a href="https://www.r-bloggers.com/fantasy-hockey-with-rvest-and-purrr/">post</a> and David Laing’s post <a href="https://laingdk.github.io/kendrick-lamar-data-science/">here</a>. Great things are from them, errors are mine.</p>
</section>
</section>

<div class="footer">
This work is licensed under the  <a href="http://creativecommons.org/licenses/by-nc/3.0/">CC BY-NC 3.0 Creative Commons License</a>.
</div>


</div>

<script>

// add bootstrap table styles to pandoc tables
function bootstrapStylePandocTables() {
  $('tr.header').parent('thead').parent('table').addClass('table table-condensed');
}
$(document).ready(function () {
  bootstrapStylePandocTables();
});


</script>

<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
  (function () {
    var script = document.createElement("script");
    script.type = "text/javascript";
    script.src  = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML";
    document.getElementsByTagName("head")[0].appendChild(script);
  })();
</script>

</body>
</html>