README.rmd

---
output:
  md_document:
    variant: markdown_github
---

# Purpose

This README documents Zander Prinsloo's (20065124) answers to the Data Science test on Monday 24 May 2021. It describes how I go about answering the three questions of the test, and also gives those answers in the sections below. The sections are appropriately named *Question 1*, *Question 2*, and *Question 3*. 

Note, however, that there are also sub-folders for each question. These also have their own projects. Within those folders you will find, amongst other things, a) pdfs with the answers to the relevant questions, b) a code folder that contains specific code and functions used in answering that question. The latter are sourced into this README. 

#### How I Got Started
Here is the code for how I created this folder, specifically the *20065124.Rproj* and corresponding folders, as well as the sub-directories for each question.
```{r}
# Create the project - This is all commented out now. 
# LOCATION <- c("/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test")
# fmxdat::make_project(FilePath = LOCATION, 
#                      ProjNam = "20065124", 
#                      Mac = T)
# # Create Projects for each question including Texevier template
# Texevier::create_template(directory = paste0(LOCATION, "/20065124"), 
#                           template_name = "Question1")
# Texevier::create_template(directory = paste0(LOCATION, "/20065124"), 
#                           template_name = "Question2")
# Texevier::create_template(directory = paste0(LOCATION, "/20065124"), 
#                           template_name = "Question3")
```


# Question 1


```{r, echo = FALSE, message = FALSE, warning = FALSE}
knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE, fig.width = 6, fig.height = 5)
options(dplyr.summarise.inform=F)

#Load Packages
if(!require(pacman))(install.packages("pacman")) # in case someone has not installed `pacman`
library(pacman)
p_load(tidyverse, lubridate, xtable, purrr, ggsci)

# Notice that as you are working in a .Rproj file (I am assuming you are) - the relative paths of your directories start at your specified root.
# This means that when working in a .Rproj file, you never need to use getwd() - it is assumed as your base root automatically.

# Source Functions
list.files('Question1/code/', full.names = T, recursive = T) %>% 
    as.list() %>% 
    walk(~source(.))

# Data

```

```{r}
list.files('code/', full.names = T, recursive = T) %>% as.list() %>% walk(~source(.))
# Source Question 1's code too ()for plot_order function) - note ..// here means go up one level from current folder
list.files('..//Question1/code/', full.names = T, recursive = T) %>% as.list() %>% walk(~source(.))
# Data loading:

# Import Data
Loc <- "/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/data/Movies/"
df_movies <- read_csv(paste0(Loc, "Movies.csv"),
                      col_names = T,
                      col_types = cols())

```




This section provides the answers for Question 1. Before answering the question I need to a) do garbage collection, b) load important packages, c) source relevant code from the `Question1/code`. This is done the separate chunks below. 

Garbage Collection:
```{r}
# Garbage Collection
gc()

# Load Packages
if(!require(pacman))(install.packages("pacman")) # in case someone has not installed `pacman`
library(pacman)
p_load(tidyverse, lubridate, forcats)

# Source Functions
list.files('Question1/code/', full.names = T, recursive = T) %>% 
    as.list() %>% 
    walk(~source(.))
```

## Task

I will seek to prove that movie critics were not a near perfect prediction of a film'a popularity and profitability amongst audiences in the mid-2000s. In order to do that I use the **Movies.csv** data set given to me by my friend at *Mr Video*. I will try to disprove her claims using plots throughout. 

## Claim 1 

She claims that: 
*:if a movie had a rating of more than 80% on Rotten Tomatoes, audiences would rate it above 85% every time."*

The barplot below clearly shows that my friend's claim is false. For the films that Rotten Tomatoes rated above 80, an overwhelming majority was not rated above 85 by audiences. 

```{r Figure1,  warning =  FALSE, fig.align = 'center', fig.cap = "Caption Here \\label{Figure1}", fig.ext = 'png', fig.height = 6, fig.width = 7}
g1 <- Barplot1_Q1(df = df_movies, 
            title = "Rotten Tomatoes is Rotten", 
            subtitle = "RT is a bad predictor of what audiences love", 
            caption = "Thank you to Mr Video for the data", 
            Xlab = "Rotten Tomatoes Rating", 
            Ylab = "Number of Films in Each Class", 
            xaxis_size = 10)

g1
```


## Claim 2

The second claim is that *"Disney films may not have the highest grossing numbers, but they’ve always been the most profitable of all the leading studios."*

I will show that they have not always been the most profitable out of the leading studios. 


There is one film called `r df_movies %>% filter(Profitability>60) %>% select(Film)` that has a very high profitability of `r df_movies %>% filter(Profitability>60) %>% select(Profitability)`. It is an outlier that will be excluded from the plot along with any missing values of profitability or studios. 
```{r}
df_movies %>% 
    filter(Profitability>60)
```
In \ref{Figure2} we see that Disney is not clearly the most profitable film. While it's median is very close to Fox, Sony, and Summit, it's upper quintile is substantially lower than Summit. Note also that Disney is being pulled up by an outlier, `r df_movies %>% filter(Profitability>20 & Profitability <60) %>% select(Film)`

```{r Figure2,  warning =  FALSE, fig.align = 'center', fig.cap = "Boxplot Comparing Disney's Profitability \\label{Figure2}", fig.ext = 'png', fig.height = 6, fig.width = 7}

g2 <- Boxplot_Q1(df = df_movies, 
                 Title = "Profitability of Leading Studios", 
                 Subtitle = "Disney vs Others", 
                 Caption = "Note: \n Thank you to Mr Video for the data", 
                 Ylab = "Studios", 
                 Xlab = "Profitability of Films")
g2

```
If we remove this outlier, we get \ref{Figure3}. It shows that Disney is an average performer in terms of Profitability. It's median is lwoer than Fox, and its upper quintile appears lower than Fox, Sony, Summit, and Universal Studios.
```{r Figure3,  warning =  FALSE, fig.align = 'center', fig.cap = "Boxplot Comparing Disney's Profitability \\label{Figure3}", fig.ext = 'png', fig.height = 6, fig.width = 7}

df_g2 <- g2$data
g2 %+%
    subset(df_g2, Profitability<20)

```

Therefore, it is not clear that Disney is indeed more profitable. The plots indicate that there are a number of studios that are more profitable and more consistent. 



## Claim 3

The third claim is that *"Audiences are always drawn to the highest grossing films"*. She goes further to claim that the correlation between world wwide grossing numbers and audience scores would be near 80%. 

<!-- The correlation between these two variables is `r as.matrix(df_movies %>% select("Audience  score %", "Worldwide Gross")) ` -->



```{r}
df_use <- bind_rows(df_movies %>% 
    select("Audience  score %", "Worldwide Gross") %>% 
        mutate(Type = "Audience and Gross Relationship") %>% 
    summarise(AudienceScore = as.numeric(`Audience  score %`), 
              WorldWideGross = as.numeric(`Worldwide Gross`), 
              Type), 
    tibble(AudienceScore = seq(from = 30, to = 90, length = nrow(df_movies)), 
              WorldWideGross = seq(from = 100, to = 600, length = nrow(df_movies)))) %>% 
    mutate(Type = ifelse(is.na(Type), "Linear Relationship", Type))

```

Note that \ref{Figure4} gives the relationship between world wide gross and audience rating. The linear line is shows what a strong correlation would look like. However, the actual line plot of the relationship between world wide gross and audience scores is much more jagged and sporadic. There is not a clear linear relationship and correlation. This means that the third claim also appears false.

```{r Figure4,  warning =  FALSE, fig.align = 'center', fig.cap = "Jagged Line Plot of Gross and Rating \\label{Figure4}", fig.ext = 'png', fig.height = 6, fig.width = 7}

g3 <- Lineplot_Q1(df = df_movies, 
                  Title = "Line Plot - Relationship Between World Wide Gross and Audience Scores",
                  Subtitle = "Compared to a strong correlation", 
                  Caption = "Note: \n Thank you to Mr Video for the data", 
                  Ylab = "World Wide Gross", 
                  Xlab = "Audience Ratings")
g3
```

# Question 2



```{r, echo = FALSE, message = FALSE, warning = FALSE}
options(dplyr.summarise.inform=F)
if(!require("pacman")) install.packages("pacman")
library(pacman)
p_load(tidyverse)

list.files('code/', full.names = T, recursive = T) %>% as.list() %>% walk(~source(.))
list.files('..//Question1/code/', full.names = T, recursive = T) %>% as.list() %>% walk(~source(.))
list.files('..//Question2/code/', full.names = T, recursive = T) %>% as.list() %>% walk(~source(.))

source("/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/Question2/code/fn_Collate_Data.R")
source("/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/Question2/code/plot1_Q2.R")

# Data loading:
df_forbes <- fn_Collate_Data(path = "/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/data/Forbes/", ext = "csv")

```

# Task
Some interesting findings using the Forbes data set


# Plot 1

```{r Figure1, ,  warning =  FALSE, fig.align = 'center', fig.cap = "Names of Rich People \\label{Figure1}", fig.ext = 'png', fig.height = 3, fig.width = 6}

g1 <- plot1_Q2(df = df_forbes, title = "Common Names of Rich People", 
               Xlab = "Names", 
               Ylab = "Frequency of Names")
g1



```

This plot in \ref{Figure1} shows that there is a surprisingly high requency of rich people with ('Common') names John, Mike/Michael, James, or Bill. Similarly, there are quite a number of families that make up the Billionaires on Forbes' list.






# Question 3


```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE, fig.width = 6, fig.height = 5, fig.pos="H", fig.pos = 'H')
# Note: Include = FALSE implies the code is executed, but not printed in your pdf.
# warning and message = FALSE implies ugly messages and warnings are removed from your pdf.
# These should be picked up when you execute the command chunks (code sections below) in your rmd, not printed in your paper!

# Load Packages
if(!require(pacman))(install.packages("pacman")) # in case someone has not installed `pacman`
library(pacman)
p_load(tidyverse, lubridate, xtable)

# Notice that as you are working in a .Rproj file (I am assuming you are) - the relative paths of your directories start at your specified root.
# This means that when working in a .Rproj file, you never need to use getwd() - it is assumed as your base root automatically.

# Source Functions
list.files('Question3/code/', full.names = T, recursive = T) %>%
    as.list() %>%
    walk(~source(.))

source("/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/Question3/code/plot1_q3.R")

# Data
df_country <- read_csv("/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/data/Tweets/Country_list.csv")
country_names <- colnames(df_country)
df_bbc <- readRDS("/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/data/Tweets/tweets_bbc.rds")
df_cnn <- readRDS("/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/data/Tweets/tweets_cnn.rds")
df_eco <- readRDS("/Users/zanderprinsloo/Library/Mobile Documents/com~apple~CloudDocs/Desktop/Desktop – MacBookPro’s MacBook Pro/Academic/Postgraduate/Masters/Modules/Data Science/Test/20065124/data/Tweets/tweets_eco.rds")

```



For this question I will give a barplot to show the composition of tweets that use photos and videos for the different media outlets

```{r}
df_photo <- bind_rows(
  df_bbc %>%
    select(photos, video) %>%
    mutate(photos = ifelse(photos=="[]", 0, 1),
                      Media = ifelse(photos==1, "photo", ifelse(video==1, "video", "none")),
           Outlet = "BBC"),
  df_cnn %>%
    select(photos, video) %>%
    mutate(photos = ifelse(photos=="[]", 0, 1),
           Media = ifelse(photos==1, "photo", ifelse(video==1, "video", "none")),
           Outlet = "CNN"),
  df_eco %>%
    select(photos, video) %>%
    mutate(photos = ifelse(photos=="[]", 0, 1),
                      Media = ifelse(photos==1, "photo", ifelse(video==1, "video", "none")),
           Outlet = "Eco"))
```




```{r Figure5,  warning =  FALSE, fig.align = 'center', fig.cap = "Compare Videos and Photos of Outlets \\label{Figure1}", fig.ext = 'png', fig.height = 3, fig.width = 6}


plot1_q3 <- function(df_bbc, df_cnn, df_eco, title, subtitle, Ylab, Xlab){

    df_use <- bind_rows(
        df_bbc %>%
            select(photos, video) %>%
            mutate(photos = ifelse(photos=="[]", 0, 1),
                   Media = ifelse(photos==1, "photo", ifelse(video==1, "video", "none")),
                   Outlet = "BBC"),
        df_cnn %>%
            select(photos, video) %>%
            mutate(photos = ifelse(photos=="[]", 0, 1),
                   Media = ifelse(photos==1, "photo", ifelse(video==1, "video", "none")),
                   Outlet = "CNN"),
        df_eco %>%
            select(photos, video) %>%
            mutate(photos = ifelse(photos=="[]", 0, 1),
                   Media = ifelse(photos==1, "photo", ifelse(video==1, "video", "none")),
                   Outlet = "Eco"))

    g <- ggplot(df_photo) +
        geom_bar(aes(x = Outlet, colour = Media, fill = Media))+
        labs(title = title,
             subtitle = subtitle,
             y = Ylab,
             x = Xlab)

    g

}


plot1_q3(df_bbc, df_eco, df_cnn,
         title = "Compare Photo/Video Usage of Main Outlets",
         subtitle = "Comparison of Photos and Videos in Tweets",
         Xlab = "Media Outlet",
         Ylab = "Count")


```








# Question 3b

Now I do a similar plot to compare the quantity of hashtags


```{r Figure6, warning =  FALSE, fig.align = 'center', fig.cap = "Caption Here \\label{Figure2}", fig.height = 3, fig.width = 6, dev = 'png'}

pacman::p_load(lubridate)
df_hashtags <- bind_rows(
  df_bbc %>%
    select(hashtags, date) %>%
    mutate(hashtags = ifelse(hashtags=="[]", 0, 1),
           Outlet = c("BBC"),
           date = as.character(date),
           year = str_sub(date, start = 1, end = 4)),
  df_cnn %>%
    select(hashtags, date) %>%
    mutate(hashtags = ifelse(hashtags=="[]", 0, 1),
           Outlet = "CNN",
           date = as.character(date),
           year = str_sub(date, start = 1, end = 4)),
  df_eco %>%
    select(hashtags, date) %>%
    mutate(hashtags = ifelse(hashtags=="[]", 0, 1),
           Outlet = "Eco",
           date = as.character(date),
           year = str_sub(date, start = 1, end = 4))) %>%
  group_by(Outlet, year) %>%
  summarise(TotalHashtags = sum(hashtags))

pacman::p_load(ggpubr)

g2 <- ggplot(data = df_hashtags) +
  geom_bar(aes(x = year, y = TotalHashtags, colour = Outlet, fill = Outlet), stat = "identity") + 
  labs(title = "Hashtags")

g2

```