Retrieving Reading Levels with R

For those that don't work in education or aren't aware, there is a measurement for a child's reading level called a Lexile ® Level.  There are ways in which this level can be retrieved using different reading assessments.  The measurement can be used to match a child's reading level to books that are the same level.  So the company maintains a database of books that they have assigned levels.  There are other assessments that measure how well a child is reading, but I'm not aware of any systems that assign books as comprehensively.  The library at the school I work at lacked Lexile Levels for their books.  This is unfortunate because of the fact that teachers are not able provide students with books for their respective Lexile Level.  Fortunately though the library had a list of ISBNs for the books.

On the Lexile website there is a way to search for books using ISBN numbers to retrieve their Lexile ® Level if the book is available in their database.  Entering every ISBN number available is a task fit for something not human.

rvest to the rescue.

Below is the script to retrieve the Lexile Levels of books if a list of ISBNs is available.  This was an incredible time save provided by some R code and hopefully someone else out there could use it.
 

	library(rvest) library(httr)
	library(htmltools)
	library(dplyr)
	##Prep for things used later
	url<-"https://www.lexile.com/fab/results/?keyword="
	url2<-"https://lexile.com/book/details/"
	##CSV file with ISBN numbers
	dat1<-read.csv("~isbns.csv",header=FALSE)
	##dat1<-data.frame(dat1[203:634,])
	dat<-as.character(dat1[,1])%>%trimws()
	##dat<-dat[41:51]
	blank<-as.character("NA")
	blank1<-as.character("NA")
	##blank2<-as.character("NA")
	##blank3<-as.character("NA")
	all<-data.frame("A","B","C")
	colnames(all)<-c("name","lexiledat","num")
	all<-data.frame(all[-1,])
	for(i in dat) {
	sites<-paste(url,i,sep="")
	x <- GET(sites, add_headers('user-agent' = 'r'))
	webpath<-x$url%>%includeHTML%>%read_html()
	##Book Name
	name<-webpath%>%html_nodes(xpath="///div[2]/div/div[2]/h4/a")%>%html_text()%>%trimws()
	##Lexile Range
	lexile<-webpath%>%html_nodes(xpath="///div[2]/div/div[3]/div[1]")%>%html_text()%>%trimws()%>%as.character()
	##CSS change sometimes
	lexiledat<-ifelse(is.na(lexile[2])==TRUE,lexile,lexile[2])
	test1<-data.frame(lexiledat,NA)
	##Breaks every now and then when adding Author/Pages
	##Author Name
	##author<-webpath%>%html_nodes(xpath='///div[2]/div/div[2]/span')%>%html_text()%>%as.character()%>%trimws()
	##author<-sub("by: ","",author)
	##Pages
	##pages<-webpath%>%html_nodes(xpath='///div[2]/div/div[2]/div/div[1]')%>%html_text()%>%as.character()%>%trimws()
	##pages<-sub("Pages: ","",pages)
	##Some books not found, this excludes them and replaces with NA values
	df<-if(is.na(test1)) data.frame(blank,blank1) else data.frame(name,lexiledat,stringsAsFactors = FALSE)
	colnames(df)<-c("name","lexiledat")
	df$num <- i
	all<-bind_rows(all,df)
	}
	master<-rbind(all1,all)

   
Link to code

State of the Union Speeches and Data

I've done a couple posts on the SOTU speeches.  In the past these dealt with word count, approval, and the vague notion that the applause the president receives has a relationship with his approval rating at that time (which had a lower correlation this year in fact).

Wired had a good article highlighting the sentiment in the current and previous State of the Union (SOTU) speeches.  They went through the speech for several of the past years, highlighted the events that occurred each year, and gave the corresponding frequency or usage of terms in the speech that communicated the impact of those events.  This blog post is not duplicating the article.  I did see the graph though and wanted to see if I got a similar sentiment score for the speeches.  I used the 'syuzhet' library in R to conduct the analysis (big thanks to Matthew Jockers for the package).

The graph is similar to the one in the Wired article, but not entirely.  Some smoothing was involved and perhaps a different sentiment analysis technique.  We do see a similar finding in the most recent SOTU speech:  it ended with the highest sentiment score out of all the speeches.  Several of the speeches in my analysis showed a curving up toward the end, which would in general go along with "ending on a positive note".  Additionally, one can see the "valleys" or lower sentiment values occurring between the 50 and 75 time intervals.  This isn't too surprising in that the same speech writer is being used and that the SOTU has perhaps a more standard sentiment form (another analysis perhaps?).  

This same library has a function which scores certain words to emotional categories.  These 10 categories include a positive/negative categorization.  Along with these, I added in the applause count for each speech and the approval rating for each year for the time period of the speech.  The matrix below depicts the correlation values of each category with corresponding color.  Additionally, I added in a p-value scoring for each relationship, those >.1 were given bubbles.

There's a lot here in terms of what could be said about the speeches but I'll only say a few things that I thought were interesting.  The applause/approval rating correlation showed a weaker value than last year (-.5), which isn't too surprising since this is probably spurious anyways.  Negative word categorization and applause had a higher correlation than positive word categorization and applause.  Meaning, when comparing applause and negative word use across speeches, these counts varied in a similar way (applause count higher - negative word count higher and vice versa).  Speeches with words categorized as "anger" or "fear" had a weak correlation to the applause count.  Conversely, speeches with words categorized in emotions like "joy", "surprise", and "trust" portray a stronger correlation with applause count in those same speeches.  So perhaps to get more applause in general, certain positive words are better than others?  Yoda's advice about fear would make sense here in that words associated with fear tend to vary similarly to words associated with anger.

We also see a decent amount of correlation among more positive emotions as well as within more negative emotions.  This refers back to the common "curve" that these speeches may have.  In that the sentiment used year over year tend to be similar, or at least the emotional categorization of words follow similar patterns.

Thanks to Matthew Jockers, Taiyun Wei, and Hadley Wickam for their work on the 'syuzhet', 'corrplot', and 'ggplot' packages respectively.  Code for the above analysis is on my github page.

Presidential Approval and Applause

Some may have seen a twitter post about spurious correlations that myself and others mentioned on twitter.  Basically this was a joke about how correlation can be found in many things that certainly have no influence over each other.  I mention this because this post may or may not be in that category ;-)

About this time last year I looked at the two most recent State of the Union speeches and talked about the political priorities ostensibly shown in each.  For those that don't know, The State of the Union is the speech that the President of the United States delivers at the beginning of each year to a joint session of Congress (that is, both House and Senate).  For the most part or at least traditionally the aim of this speech is to outline the priorities for the next year for the President's office and to give a bit of an idea where the United States is at in general, or the "state of the union".

One of the more nuanced parts of the speech is that there are periods where the President is either interrupted with applause by members who feel what he is saying is good, or where he pauses to allow for applause (typically from his party).  The speeches are fairly lengthy.  The past several years these speeches have averaged about an hour.  Turns out applause is definitely a big part of the speech (it's polite afterall).  For President Obama's terms in his speeches, the word "applause" appears more than any other word (outside articles).  If applause lasts about 10 seconds on average, we're looking somewhere around 12-13 minutes of total applause during his speeches.  This doesn't take account the length of the applause times as in the text of the speeches it is only shown as "(applause)".

So what's the point other than that's a lot of clapping?  I wanted to look at if there was any similarities between the applause being given and the President's approval rating.  Appropriately, I'll be using a popular graph theme from the political analysis etc. site fivethirtyeight to display this brief analysis.  The theme was actually put together in R here, by Austin Clemens (thanks!).

So just by looking at the two lines, one indicating the number of times applause occurs during the speech, the other indicating the % approval (though the scale on the left not in % terms), we can see that it doesn't change a lot.  Except for two years, 2010 and 2014.  In 2010 he received 50% more applause than in the other years and about 30% more in 2014.

The question becomes, is the applause tactical to show support for the president by the party in a period of lessening approval?  The correlation coefficient was -.50, but as with spurious correlations, this very well could speak nothing of the influence of approval on the amount of applause.  In general just looking at the graph, the percentage change isn't the same for Applause and Approval however we can see that in general the change year over year between Applause and Approval certainly has an inverse relationship.   Meaning the line moves up for Applause between 2009 and 2010 then the line moves down for Approval rating for the same period.

Showing support and unity for a party leader by applauding is certainly reasonable, especially when support may be lacking from the general public.  Guessing as to whether this is considered before in response to the approval rating is more difficult.  Then again, it's a bit more fun to think that members of Congress would tactically use this:

  Code for this will appear on my Github page.

MVP and Speech

Kevin Durant delivered an amazing MVP acceptance speech Tuesday in Oklahoma City.  The award clearly meant much to "Mr. Reliable" as he's now called in multiple media outlets.  He went player by player through his teammates discussing some of their attributes and how they have contributed to him winning the award and making him a better player.   In general, the MVP speech isn't something that I would think in general there is a lot of expectation on.  Meaning, the public feels like you can accept the award, thank a few people, and get back to business.  That's why KDs speech was so unusual.  We can compare it to Lebron's speech this last year.  The sheer difference in length is the first difference you may notice between the two MVPs.

It is interesting to note what words were used.  I chose only the top 11 words from each player to get a sense of how they focused their speech.  Both players mentioned "guys" a lot as well as "team" which gives a sense that they were both giving credit to their teammates.  Graphed against each other, the two speeches show a bit more interesting parts in terms of the frequency of words.

"Love" for instance was a big part of Durant's speech, his communicating the way he felt about his team and those around him was really important.  His speech was about thanking people, showing his appreciation for those around him.  We can also see Durant was also concerned about being "better" and can infer from the text that those around him were the people that made him better since appreciation, love, and team were mentioned so much.  If we scale the graph to only show those words mentioned 13 times or less, we can see where some words were used several times by either player and not by the other.

Here we see "team" mentioned over 10 times by Kevin Durant and only 1 time by Lebron.  Lebron referred to his team as "guys" and we can't discern just from these words that his speech wasn't about his team.  But, it is clear just from looking at word usage that Kevin's speech was clearly more about team than Lebron's without having to listen to the speeches themselves.  Those of you who watched/listened Durant's speech understand that going player by player was a clear objective for his accepting the award and that communicating the importance of his team in the journey was a priority.

As mentioned before, this speech isn't something to have high expectations for, which is why what Durant said and how he said it was so remarkable.  

State of the Union Speech Comparison

The State of the Union is historically a speech used to set the tone for the year...at least I think that's what it is.  The process of writing these speeches has been discussed before and first-hand detail is available in this article.  I was interested in seeing the difference in language between the 2013 and 2014 State of the Union address in terms of what was said.  A lot was said in each speech so I narrowed it down to words that were mentioned 20 times total in the two speeches.  Below is a wordcloud showing these words, with the larger words having a higher frequency.

You can see just from this wordcloud how these speeches seem to take a more emphatic/determined/action-oriented tone with the word "will" having the highest frequency (aside from another word we will see below).  Also, you can get an idea of some of the policy issues he discussed.

As noted before, I wanted to compare the two speeches to see which words were being used more/less when comparing the two.  The number of times a word is mentioned in the 2013 State of the Union address is on the X-axis and the 2014 address is on the Y-axis.

Probably one of the odder yet clearer takeaways from this graph is the applause given in 2014 vs 2013 (in the speech text this appears when there is pause for applause also for cheers).  The President went into this speech having a very low approval rating.  That being said, was the applause a function his party knowing his lack-luster rating?  Or was it just that the speech was better and elicited much more applause than last year?  It is interesting that this year's speech received more than 10 additional pauses for applause than 2013.  "Jobs" was used more last year in the speech than this year whereas "help" was used more this year.  This may reflect some of the policy decisions with the Affordable Healthcare Act having been past and other programs intended to provide "help" for Americans.   Last year after the election jobs and employment were a large part of what worried many Americans (and for many still does).  "Energy" was mentioned much less in this year's SOTU.  Perhaps the controversy over the Keystone XL pipeline influenced the topic of energy as a priority in the speech this year.  Many of the words were also very linear in their use (seen clustered around 10).

Context is certainly important in speeches and this technique of comparing words by no means completely depicts what is important or how things were communicated.  However, the words that are chosen for these speeches are strategic and outline (if not in function than in form) the year for the president and his priorities.  

Lyric Analytics

I was messing around with the text mining (tm) package in R and was thinking of something I could comb through.  I looked through some other blogs and websites to see how they were using it:  mining through presidential speeches debates being one of the more notable uses.  I was thinking about it in terms of where we focus mostly on how things are said and really not what is actually said...ie music.  For some people, what is said is important and I would argue for most artists who, "make it" eventually it comes around to be important to an artist's being catapulted into fame :-)

One of the greater(est) rock and roll icons who began making albums slightly before my ears were ready for it is "The Boss".  Lots of albums, lots of words.  Although Bruce Springsteen was definitely communicating a lot of powerful ideas in his music, ultimately it's the passion he sings with, the way he is miraculously able to have a sax weaved into most of his songs, or just having come up with the nickname "The Boss" that makes him awesome.  Anyways, I thought about looking at the album Born to Run as a primer in what I'll call, "lyric mining".

Below is a graph showing the most frequent words used in the album across every song that are mentioned at least 5 times.  For those of us familiar with the album, if you just look at these words you can hear the music.

Number of times words are used in the album Born to Run

Below is a heat map showing words and their corresponding albums (lighter the colors = more usage).  I chose only words that were mentioned at least 5 times in the album, otherwise it was too large.  On the axis not labeled are dendograms (basically a graphical way to show how things are associated or how similar they are - more on how to read them here).  Each cell in the heat map has a bar graph showing the relative usage of the word...highest being 10 times in one song...that being the word "one" in the song "She's the One".

Born to Run album Heatmap

In terms of what is said you'll notice that the song "Night" is associated with "Jungleland"(height of the lines and being in the same "clade" on the dendogram) in terms of the words used, at least those that are used at least 5 times.  Here's how this looks when they are graphed against each other:

Night lyrics graphed against Jungleland lyrics for words mentioned 5=< times

Alternatively "10 Avenue Freeze Out" is an outlier in terms of word usage among the other songs as you can see it sits relatively unconnected from other songs in the dendogram on the x-axis.

On the y-axis you can see the associations across different words and their usage.  "Night" and "One" even though they are used a lot are not distributed the same (different "clades")...meaning when "The Boss" is belting it out, he's using these words in different places - different songs.

Which brings up an interesting point about great albums (in my opinion):  their distribution of themes.  "Born to Run" definitely has some great themes in it and while I won't interpret the meaning of each song, we can see it through the distribution of words in the lyrics and songs in the album.  While we know the song themes are strong in this album, we can also see (through the word distribution) that the great themes are distributed across the album and it's not just one song that makes this album great.

What Twitter Says About A&E and Duck Dynasty

I was looking on twitter and noticed a lot of people talking about Duck Dynasty and then talking about the consequences to A&E.  I was curious on what was being said.  I mined Twitter for #A&E and #duckdynasty and got the following wordclouds:

#A&E

#duckdynasty amp is ampersand

These were for words mentioned at least 20 times over a few thousand tweets.  Clearly, people have more variety to say about A&E for their move in regards to Phil Robertson.  Whereas people hashtagging Duck Dynasty itsefl are just memorializing/supporting Phil.  One hashtag is an A&E bash whereas the other seems like a support network.  The two hashtags are definitely connected, but are communicating two different things it seems across the twittersphere and are being used by maybe the same people but two different emotions.  No sentiment analysis here...maybe at a later date.

Happy Thanksgivukkah!

The first post of this blog will commemorate the extremely rare event occurring this Thursday:  Thanksgivukkah.

For those of you who don't know, this thanksgiving (2013) we will experience the convergence of the first day of the Jewish holiday of Hannkkah and the american holiday of Thanksgiving.  This is a rare event because we are dealing with two different calendars.  The rareness of the event is attributed to its not occurring again for another 75,000 years.  

Social media is a nice outlet to discern how people reflect on events and holidays.  As such, I built a wordcloud by mining Twitter for "#Thanksgivukkah".  This hashtag was actually mentioned about 1500 times which is impressive.  

This was done using R.  In addition I had some useful instruction from an blogger in terms of setting up the API for R to search through hashtags.  

##search twitter
>thanksgivukkah<-searchTwitter("#Thanksgivukkah",n=1500,cainfo="cacert.pem")
>thanksgivukkah_text<-sapply(thanksgivukkah,function(x) x$getText())
##Build Corpus
>thanksgivukkah_text_corpus<-Corpus(VectorSource(thanksgivukkah_text))
##Remove punctuation, stopwords, and lowercase
>thanksgivukkah_text_corpus<-tm_map(thanksgivukkah_text_corpus,tolower)
>thanksgivukkah_text_corpus<-tm_map(thanksgivukkah_text_corpus,removePunctuation)
>thanksgivukkah_text_corpus<-tm_map(thanksgivukkah_text_corpus,function(x)removeWords(x,stopwords()))
##Build wordcloud
>wordcloud(thanksgivukkah_text_corpus,scale=c(8,.3),min.freq=20,vfont=c("sans serif","plain"))

Pretty encouraging overall...seems like people are happy...which is great.  Happy Hanukkah and Happy Thanksgiving!