slide_tidyverse.Rmd

---
title: "Tidy work in Tidyverse"
subtitle: "R Foundation for Life Scientists"
author: "Marcin Kierczak"
keywords: r, rstats, r programming, markdown, tidyverse
output:
  xaringan::moon_reader:
    encoding: 'UTF-8'
    self_contained: false
    chakra: 'assets/remark-latest.min.js'
    css: 'assets/slide.css'
    lib_dir: libs
    include: NULL
    nature:
      ratio: '4:3'
      highlightLanguage: r
      highlightStyle: github
      highlightLines: true
      countIncrementalSlides: false
      slideNumberFormat: "%current%/%total%"
---

exclude: true
count: false

```{r,echo=FALSE,child="assets/header-slide.Rmd"}
```

<!-- ----------------- Only edit title & author above this ----------------- -->

```{r setup,include=FALSE}
knitr::opts_chunk$set(dev="png",results="hold",fig.show="hold",fig.align="left",echo=TRUE,warning=FALSE,message=FALSE)
options(crayon.enabled = FALSE)

# report related
library(knitr)
library(markdown)
library(rmarkdown)
```

```{r,echo=FALSE,message=FALSE,warning=FALSE}
# load the packages you need

library(tidyverse)
library(ggplot2) # static graphics
library(kableExtra)
library(magrittr)
library(emo)
```

---
name: setup_livecode
# Livecode Setup

By typing:

`http://livecode.kierczak.net:7777`

in your browser, you can access the livecode server. 


---
name: learning_outcomes
# Learning Outcomes

<br>

Upon completing this module, you will:

* know what `tidyverse` is and a bit about its history

* be aware of useful packages within `tidyverse`

* be able to use basic pipes (including native R pipe)

* know whether the data you are working with are tidy

* will be able to do basic tidying of your data

---
name: tidyverse_overview
# Tidyverse -- What is it all About?

* [tidyverse](http://www.tidyverse.org) is a collection of &nbsp; `r emo::ji('package')` `r emo::ji('package')`
* created by [Hadley Wickham](http://hadley.nz)
* has become a *de facto* standard in data analyses
* a philosophy of programming or a **programming paradigm**: everything is about the &nbsp;`r emo::ji('water_wave')`  &nbsp; flow of &nbsp; `r emo::ji('broom')` &nbsp; tidy data

.center[
<img src="data/slide_tidyverse/hex-tidyverse.png", style="height:200px;">
<img src="data/slide_tidyverse/Hadley-wickham2016-02-04.jpeg", style="height:200px;">
<img src="data/slide_tidyverse/RforDataScience.jpeg", style="height:200px;">
]
.vsmall[sources of images: www.tidyverse.org, Wikipedia, www.tidyverse.org]

---
name: tidyverse_curse
# ?(Tidyverse OR !Tidyverse)

> `r emo::ji('skull_and_crossbones')` &nbsp;There are still some people out there talking about the tidyverse curse though...&nbsp; `r emo::ji('skull_and_crossbones')`<br>

--

> Navigating the balance between base R and the tidyverse is a challenge to learn.<br>[-Robert A. Muenchen](http://r4stats.com/articles/why-r-is-hard-to-learn/)

--

.center[<img src="data/slide_tidyverse/tidyverse-flow.png", style="height:400px;">]

.vsmall[source: http://www.storybench.org/getting-started-with-tidyverse-in-r/]

---
name: intro_to_pipes
# Pipes or Let my Data Flow &nbsp; `r emo::ji('water_wave')`

.pull-left-50[

.center[<img src="data/slide_tidyverse/pipe_magritte.jpg", style="width:300px;">]

.vsmall[Rene Magritt, *La trahison des images*, [Wikimedia Commons](https://en.wikipedia.org/wiki/The_Treachery_of_Images#/media/File:MagrittePipe.jpg)]

.center[<img src="data/slide_tidyverse/magrittr.png", style="width:150px;">]
]

--

.pull-right-50[

* Let the data flow.
* *Ceci n'est pas une pipe* -- `magrittr`
* The `%>%` pipe:
  + `x %>% f` $\equiv$ `f(x)`
  + `x %>% f(y)` $\equiv$ `f(x, y)`
  + `x %>% f %>% g %>% h` $\equiv$ `h(g(f(x)))`

]

--

.pull-right-50[

instead of writing this:

```{r magrittr0, eval=F}
data <- iris
data <- head(data, n=3)
```

]

--

.pull-right-50[

write this:

```{r magrittr1}
iris %>% head(n=3)
```

]

---
name: native_r_pipe
# Native R Pipe

From R 4.1.0, we have a native pipe operator `|>` that is a bit faster than the `magrittr` pipe `%>%`.
It, however, differs from the `magrittr` pipe in some aspects, e.g., it does not allow for the use of the dot `.` as a placeholder (it has a simple `_` placeholder though).

```{r native_pipe1}
c(1:5) |> mean()
```

```{r native_pipe2}
c(1:5) %>% mean()
```


---
name: tibble_intro

# Tibbles

.pull-left-50[

.center[<img src="data/slide_tidyverse/tibble_tweet.jpg">]
]

.pull-right-50[

* `tibble` is one of the unifying features of tidyverse,
* it is a *better* `data.frame` realization,
* objects `data.frame` can be coerced to `tibble` using `as_tibble()`
]

---
name: convert_to_tibble
# Convert `data.frame` to `tibble`
```{r}
as_tibble(iris)
```

---
name: tibble_from_scratch
# Tibbles from scratch with `tibble()`

```{r tibble_from_scratch, eval=FALSE}
tibble(
  x = 1,          # recycling
  y = runif(4),
  z = x + y^2,
  outcome = rnorm(4)
)
```

--

```{r tibble_from_scratch_eval, echo = F, eval=TRUE}
tibble(
  x = 1,          # recycling
  y = runif(4),
  z = x + y^2,
  outcome = rnorm(4)
)
```

---
name: more_on_tibbles
# More on Tibbles

* When you print a `tibble`:
  + all columns that fit the screen are shown,
  + first 10 rows are shown,
  + data type for each column is shown.

```{r tibble_printing}
as_tibble(cars)
```

---
name: tibble_printing_options
# Tibble Printing Options

* `my_tibble %>% print(n = 50, width = Inf)`,
* `options(tibble.print_min = 15, tibble.print_max = 25)`,
* `options(dplyr.print_min = Inf)`,
* `options(tibble.width = Inf)`

---
name: subsetting_tibbles
# Subsetting Tibbles

```{r tibble_subs}
vehicles <- as_tibble(cars[1:5,])
vehicles %>% print(n = 5)
```
  
  
--

We can subset tibbles in a number of ways:

```{r tibble_subs1}
vehicles[['speed']] # try also vehicles['speed']
vehicles[[1]]
vehicles$speed
```
  
  
--

> **Note!** Not all old R functions work with tibbles, than you have to use `as.data.frame(my_tibble)`.

---
name: tibbles_partial_matching

# Tibbles are Stricter than `data.frames`

```{r tibble_strict, warning=T, echo=F}
cars <- cars[1:5,]
```
```{r tibble_strict1, warning=T}
cars$spe      # partial matching
```
```{r tibble_strict2, warning=T}
vehicles$spe  # no partial matching
```
```{r tibble_strict3, warning=T}
cars$gear
```
```{r tibble_strict4, warning=T}
vehicles$gear
```

---
name: loading_data

# Loading Data

In `tidyverse` you import data using `readr` package that provides a number of useful data import functions:
* `read_delim()` a generic function for reading *-delimited files. There are a number of convenience wrappers:
  + `read_csv()` used to read comma-delimited files,
  + `read_csv2()` reads semicolon-delimited files,
  `read_tsv()` that reads tab-delimited files.
* `read_fwf` for reading fixed-width files with its wrappers:
  + fwf_widths() for width-based reading,
  + fwf_positions() for positions-based reading and
  + read_table() for reading white space-delimited fixed-width files.
* `read_log()` for reading Apache-style logs.

--

>The most commonly used `read_csv()` has some familiar arguments like:
* `skip` -- to specify the number of rows to skip (headers),
* `col_names` -- to supply a vector of column names,
* `comment` -- to specify what character designates a comment,
* `na` -- to specify how missing values are represented.

---
name: readr_writing

# Writing to a File

The `readr` package also provides functions useful for writing tibbled data into a file:

* `write_csv()`
* `write_tsv()`
* `write_excel_csv()`

They **always** save:

* text in UTF-8,
* dates in ISO8601

But saving in csv (or tsv) does mean you loose information about the type of data in particular columns. You can avoid this by using:

* `write_rds()` and `read_rds()` to read/write objects in R binary rds format,
* use `write_feather()` and `read_feather()` from package `feather` to read/write objects in a fast binary format that other programming languages can access.

---
name: basic_data_transformations

# Basic Data Transformations with `dplyr`

Let us create a tibble:

```{r}
bijou <- as_tibble(diamonds) %>% head()
bijou[1:5, ]
```

.center[ <img src="data/slide_tidyverse/diamonds.png", style="height:200px"> ]

---
name: filter

# Picking Observations using `filter()`

```{r}
bijou %>% filter(cut == 'Ideal' | cut == 'Premium', carat >= 0.23) %>% head(n = 4)
```
  

-- 

>`r emo::ji('boat')` &nbsp; Be careful with floating point comparisons! <br>
`r emo::ji('pirate')` &nbsp; Also, rows with comparison resulting in `NA` are skipped by default!

```{r, echo=T, eval=F}
bijou %>% filter(near(0.23, carat) | is.na(carat)) %>% head(n = 4)
```
  
---
name: arrange

# Rearranging Observations using `arrange()`

```{r, echo=T, eval=FALSE}
bijou %>% arrange(cut, carat, desc(price))
```
  
--

```{r, echo=FALSE, eval=TRUE}
bijou %>% arrange(cut, carat, desc(price))
```
  
--

>The `NA`s always end up at the end of the rearranged `tibble`!

---
name: select

# Selecting Variables with `select()`

Simple `select` with a range:

```{r}
bijou %>% select(color, clarity, x:z) %>% head(n = 4)
```

--

Exclusive `select`:
```{r}
bijou %>% select(-(x:z)) %>% head(n = 4)
```

---
name: rename
# Renaming Variables

>`rename` is a variant of `select`, here used with `everything()` to move `x` to the beginning and rename it to `var_x`

```{r, eval=FALSE, echo=TRUE}
bijou %>% rename(var_x = x) %>% head(n = 5)
```
  
--

```{r, eval=T, echo=F}
bijou %>% rename(var_x = x) %>% head(n = 5)
```
  
---
name: bring_to_front
# Bring columns to front

>use `everything()` to bring some columns to the front:

```{r, echo=TRUE, eval=FALSE}
bijou %>% select(x:z, everything()) %>% head(n = 4)
```
  
--

```{r, echo=FALSE, eval=TRUE}
bijou %>% select(x:z, everything()) %>% head(n = 4)
```

---
name: mutate

# Create/alter new Variables with `mutate`

```{r, echo=T, eval=F}
bijou %>% mutate(p = x + z, q = p + y) %>% 
  select(-(depth:price)) %>% 
  head(n = 5)
```
  
  
--

```{r, echo=F, eval=T}
bijou %>% mutate(p = x + z, q = p + y) %>% 
  select(-(depth:price)) %>% 
  head(n = 5)
```

---
name: transmute
# Create/alter new Variables with `transmute` `r emo::ji('wizard')`

>Only the transformed variables will be retained.

```{r}
bijou %>% transmute(carat, cut, sum = x + y + z) %>% head(n = 5)
```
  

---
name: grouped_summaries
# Group and Summarize

```{r}
bijou %>% group_by(cut) %>% summarize(max_price = max(price),
                                      mean_price = mean(price),
                                      min_price = min(price))
```
    

--

```{r}
bijou %>% group_by(cut, color) %>%
  summarize(max_price = max(price),
            mean_price = mean(price),
            min_price = min(price)) %>% head(n = 4)
```

---
name: other_data_manipulations

# Other data manipulation tips

```{r}
bijou %>% group_by(cut) %>% summarize(count = n())
```

--

When you need to regroup within the same pipe, use `ungroup()`.

---
name: concept_of_tidy_data

# The Concept of Tidy Data

Data are tidy *sensu Wickham* if:
* each and every observation is represented as exactly one row,
* each and every variable is represented by exactly one column,
* thus each data table cell contains only one value.
`r knitr::include_graphics("data/slide_tidyverse/tidy_data.png")`

Usually data are untidy in only one way. However, if you are unlucky, they are really untidy and thus a pain to work with...

---
name: tidy_data

# Tidy Data

<img src="data/slide_tidyverse/tidy_data.png" style="height:100px">

--

.center[**Are these data tidy?**]

.pull-left-70[
```{r tidy_iris1, echo=FALSE}
data("iris")
iris %>% head(n=3) %>% kable("html",escape=F,align="c") %>%
  kable_styling(bootstrap_options=c("striped","hover","responsive","condensed"),
                position="left",full_width = F)
```
]

--

.pull-right-30[

```{r tidy_iris2, echo=FALSE}
iris2 <- iris %>%
  gather(key=variable, value=value, -Species)
iris2 %>%
  head(n=3) %>%
  kable("html",escape=F,align="c") %>%
  kable_styling(bootstrap_options=c("striped","hover","responsive","condensed"),
                position="left",full_width = F)
```

]

<br>&nbsp;<hr><br>

--
.pull-left-50[

```{r tidy_iris3, echo=FALSE}
iris3 <-
  iris %>%
  unite(Sepal.L.W, Sepal.Length, Sepal.Width, sep = "/") %>%
  unite(Petal.L.W, Petal.Length, Petal.Width, sep = "/")
iris3 %>%
  head(n = 3) %>%
  kable("html",escape=F,align="c") %>%
  kable_styling(bootstrap_options=c("striped","hover","responsive","condensed"),
                position="left",full_width = F)
```

]

--

.pull-right-50[
```{r tidy_iris4, echo=FALSE}
iris4 <- t(iris)
iris4[,1:4] %>%
  kable("html",escape=F,align="c") %>%
  kable_styling(bootstrap_options=c("striped","hover","responsive","condensed"),
                position="left",full_width = F)
```

]

---
name: tidying_data_pivot_longer

# Tidying Data with `tidyr::pivot_longer`

If some of your column names are actually values of a variable, use `pivot_longer` (replaces `gather`):

```{r include=FALSE}
bijou %>%
  mutate(`2008` = price) %>%
  select(-price) %>%
  mutate(`2009` = `2008` + sample(rnorm(100, mean = 0.01 * mean(`2008`)),
                                size = 1,
                                replace=T
                                )
         ) %>%
  select(cut, `2008`, `2009`) -> bijou2
```
```{r bijou2}
bijou2 %>% head(n = 5)
```
```{r}
bijou2 %>%
  pivot_longer(c(`2008`, `2009`), names_to = 'year', values_to = 'price') %>%
  head(n = 5)
```

---
name: tidying_data_pivot_wider

# Tidying Data with `tidyr::pivot_wider`

If some of your observations are scattered across many rows, use `pivot_wider` (replaces `gather`):

```{r include=FALSE}
bijou %>% head(n = 3) %>% select(cut, price, clarity, x, y, z) %>% gather(x,y,z, key='dimension', value='measurement') -> bijou3
```

```{r bijou3}
bijou3
```

```{r}
bijou3 %>%
  pivot_wider(names_from=dimension, values_from=measurement) %>%
  head(n = 4)
```

---
name: tidying_data_separate

# Tidying Data with `separate`

If some of your columns contain more than one value, use `separate`:

```{r include=FALSE}
bijou %>% head(n = 5) %>% select(cut, price, clarity, x, y, z) %>% unite(dim, x, y, z, sep='/') -> bijou4
```
```{r bijou4}
bijou4
```
```{r}
bijou4 %>%
  separate(dim, into = c("x", "y", "z"), sep = "/", convert = T)
```

---
name: tidying_data_unite
# Tidying Data with `unite`

If some of your columns contain more than one value, use `separate`:

```{r include=FALSE}
bijou %>% head(n = 5) %>% select(cut, price, clarity, x, y, z) %>% separate(clarity, into = c('clarity_prefix', 'clarity_suffix'), sep = 2) -> bijou5
```
```{r bijou5}
bijou5
```
```{r}
bijou5 %>% unite(clarity, clarity_prefix, clarity_suffix, sep='')
```

---
name: missing_complete

# Completing Missing Values Using `complete`

```{r eval=FALSE, include=FALSE}
bijou %>%
  head(n = 10) %>%
  select(cut, clarity, price) %>%
  mutate(cut, cut2=replace(cut, sample(1:10, 4, F), NA)) -> missing_stones
```
```{r}
bijou %>% head(n = 10) %>%
  select(cut, clarity, price) %>%
  mutate(continent = sample(c('AusOce', 'Eur'),
                            size = 6,
                            replace = T)) -> missing_stones
```
```{r}
missing_stones %>% complete(cut, continent)
```

---
name: joins

# Joining Data with `_join`

```{r echo=FALSE, eval=TRUE}
  # create two tibbles that share key column and can illustrate joins
  tibble1 <- tibble(key = c(1, 2, 3, 4, 5), value1 = c('a', 'b', 'c', 'd', 'e'))
  tibble2 <- tibble(key = c(1, 2, 3, 6, 7), value2 = c('A', 'B', 'C', 'F', 'G'))
```

.pull-left-50[
    ```{r echo=FALSE}
    print(tibble1)
    ```
]

.pull-right-50[
    ```{r echo=FALSE}
    print(tibble2)
    ```
]

**Example:**

```{r}
inner_join(tibble1, tibble2, by = 'key')
```
    
`[inner, left, right, full]_join` are available. Try these!

---
name: more_tidyverse

# Some Other Friends

* `stringr` for string manipulation and regular expressions,
* `forcats` for working with factors,
* `lubridate` for working with dates.

---
name: end-slide
class: end-slide

# Thank you. Questions? [More?](https://nbisweden.github.io/raukr-2024/)

```{r,echo=FALSE,child="assets/footer-slide.Rmd"}
```