diff --git a/.gitignore b/.gitignore index cfa16c3..093bed7 100644 --- a/.gitignore +++ b/.gitignore @@ -40,5 +40,4 @@ vignettes/*.pdf # R Environment Variables .Renviron -docs .DS_Store diff --git a/docs/404.html b/docs/404.html new file mode 100644 index 0000000..297c8c8 --- /dev/null +++ b/docs/404.html @@ -0,0 +1,97 @@ + + +
+ + + + +MIT License + +Copyright (c) 2020 Semantic Priming Across Many Languages (SPAML) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. ++ +
ambrosini_vignette.Rmd
Italian Age of Acquisition Norms for a Large Set of Words (ItAoA)
+Data provided by: Ettore Ambrosini
+The age of acquisition (AoA) represents the age at which a word is +learned. This measure has been shown to affect performance in a wide +variety of cognitive tasks (see reviews by Juhasz, 2005; Johnston and +Barry, 2006; Brysbaert and Ellis, 2016), with faster reaction times for +words learned early in life compared to those learned later.
+There are two main approaches to derive AoA data. First, objective +AoA measures can be obtained by analysis of children’s production +(Chalard et al., 2003; Álvarez and Cuetos, 2007; Lotto et al., 2010; +Grigoriev and Oshhepkov, 2013). Within this approach, children +(classified by age) are asked to name the picture of common objects and +activities. The AoA of a given word is computed as the mean age of the +group of children in which at least 75% of them can name the picture +correctly. Alternatively, subjective AoA can be obtained using adult +estimates (Barca et al., 2002; Ferrand et al., 2008; Moors et al., +2013). Here, adult participants are asked to provide AoA ratings on a +Likert scale (Schock et al., 2012; Alonso et al., 2015; Borelli et al., +2018) or directly in years, indicating the number corresponding to the +age they thought they had learned a given word (Stadthagen-Gonzalez and +Davis, 2006; Ferrand et al., 2008; Moors et al., 2013). Compared to the +use of a Likert scale, the latter method is easier for participants to +use and does not restrict artificially the response range, instead +providing more precise information on the AoA of words’ AoA (Ghyselinck +et al., 2000). It has been shown that the AoA estimates obtained from +the two different methods are highly correlated (Morrison et al., 1997; +Ghyselinck et al., 2000; Pind et al., 2000; Lotto et al., 2010; see also +Brysbaert, 2017; Brysbaert and Biemiller, 2017) and this correlation +still remains significant when other variables, such as familiarity, +frequency, and phonological length, are controlled (Bonin et al., +2004).
+Only two sets of Italian norms with objective AoA (Rinaldi et al., +2004) and subjective AoA (Borelli et al., 2018) include abstract and +concrete words and different word classes (adjective, noun, and verb), +but they are limited to a relatively small number of word stimuli (519 +and 512 words, respectively). Unfortunately, the lack of overlap between +AoA (Dell’Acqua et al., 2000; Barca et al., 2002; Barbarotto et al., +2005; Della Rosa et al., 2010; Borelli et al., 2018) and +semantic-affective norms (Zannino et al., 2006; Kremer and Baroni, 2011; +Montefinese et al., 2013b, 2014; Fairfield et al., 2017) for Italian +words has prevented direct comparison of different lexical-semantic +dimensions to establish the extent to which they overlap or complement +each other in word processing. An important motivation of the present +study is to extend previous Italian norms by collecting AoA ratings for +a much larger range of Italian words for which concreteness and +semantic-affective norms are now available, thus ensuring greater +coverage of words varying along these dimensions.
+A total of 507 native Italian speakers were enrolled to participate +in an online study (436 females and 81 males; mean age: 20.82 years, SD += 2.22; mean education: 15.16 years, SD = 1.11). We selected 1,957 +Italian words from our Italian adaptations of the original ANEW +(Montefinese et al., 2014; Fairfield et al., 2017) and from available +Italian semantic norms (Zannino et al., 2006; Kremer and Baroni, 2011; +Montefinese et al., 2013). The set of stimuli included 76% of nouns, 16% +of adjectives, and 8% of verbs. The word stimuli were presented in the +same verbal form as the previous Italian norms (e.g., the verbs were +presented in the infinitive form) to preserve consistency with these +data collections (Montefinese et al., 2014; Fairfield et al., 2017). +Word stimuli were distributed on 20 lists containing 97–98 words each. +To avoid primacy or recency effects, the order in which words appeared +on the list was randomized for each participant separately. All lists +were roughly matched for word length, word frequency, number of +orthographic neighbors, and mean frequency of orthographic neighbors. +For each list, an online form was created using Google modules. +Participants were asked to estimate the age (in years) at which they +thought they had learned the word, specifying that this information +should indicate the age at which, for the first time, they understood +the word when someone else used it in their presence, even when they did +not use the word themselves. These instructions and the examples +provided to the participants closely matched those used in a large +number of previous studies (Ghyselinck et al., 2000; Stadthagen-Gonzalez +and Davis, 2006; Kuperman et al., 2012; Moors et al., 2013; Łuniewska et +al., 2016). The task lasted about 40 min.
+Included with the vignette and Data Location: https://osf.io/rzycf/
+
+DF <- import("data/ambrosini_data.csv.zip")
+
+DF <- DF %>%
+ arrange(Ita_Word) %>% #orders the rows of the data by the target_name column
+ group_by(Ita_Word) %>% #group by the target name
+ transform(items = as.numeric(factor(Ita_Word)))%>% #transform target name into a item
+ select(items, Eng_Word, Ita_Word, everything()
+ ) #select all variables from items and target_name
+
+DF <- DF %>%
+ group_by(Ita_Word) %>%
+ filter (Rating != 'Unknown')
+
+head(DF)
+#> # A tibble: 6 × 5
+#> # Groups: Ita_Word [1]
+#> items Eng_Word Ita_Word SS_ID Rating
+#> <dbl> <chr> <chr> <int> <chr>
+#> 1 1 dazzle abbaglio 282 16
+#> 2 1 dazzle abbaglio 283 10
+#> 3 1 dazzle abbaglio 284 12
+#> 4 1 dazzle abbaglio 285 10
+#> 5 1 dazzle abbaglio 286 8
+#> 6 1 dazzle abbaglio 287 9
Montefinese, M., Vinson, D., Vigliocco, G., & Ambrosini, E. +(2018, November 26). Italian age of acquisition norms for a large set of +words (ItAoA). https://doi.org/10.17605/OSF.IO/3TRG2
+age of acquisition, word, lexicon, Italian language, cross-linguistic +comparison, subjective rating
+
+metadata <- import("data/ambrosini_metadata.xlsx")
+
+flextable(metadata) %>% autofit()
Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
items |
+Item number |
+Numeric |
+
Eng_Word |
+English translation of the item |
+Character |
+
Ita_Word |
+Italian translation of the item |
+Character |
+
SS_ID |
+Subject ID Number |
+Numeric |
+
Rating |
+Age of acquisition rating |
+Numeric |
+
Note that the data are already in long format (each item has one +row), and therefore, we do not need to restructure the data.
+In this dataset, we have 48772 individual words to select from for +our research study. You would obviously not use all of these in one +study. Let’s say we wanted participants to rate 75 pairs of words during +our study (note: this selection is completely arbitrary).
+
+random_items <- unique(DF$items)[sample(unique(DF$items), size = 75)]
+
+DF <- DF %>%
+ filter(items %in% random_items)
+
+# Function for simulation
+var1 <- item_power(data = DF, # name of data frame
+ dv_col = "Rating", # name of DV column as a character
+ item_col = "items", # number of items column as a character
+ nsim = 10,
+ sample_start = 20,
+ sample_stop = 100,
+ sample_increase = 5,
+ decile = .4)
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
What is the usual standard error for the data that could be +considered for our stopping rule using the 40% decile?
+
+# individual SEs
+var1$SE
+#> 24 27 29 99 104 109 111 114
+#> 0.6574699 0.4308132 0.4722993 0.4497407 0.5314132 0.5715476 0.4868949 0.4013311
+#> 118 201 287 343 376 394 439 451
+#> 0.4247352 0.6823489 0.2516611 0.3109126 0.6631239 0.3646002 0.5057008 0.5540156
+#> 458 469 483 500 509 521 553 560
+#> 0.2242023 0.4028234 0.3302524 0.5102940 0.5867424 0.6327717 0.4214262 0.6161169
+#> 621 628 635 677 707 780 809 816
+#> 0.2708013 0.4623130 0.4621688 0.3265986 0.2444040 0.5794250 0.6189238 0.4434712
+#> 822 840 841 898 912 918 924 938
+#> 0.4062840 0.3872983 0.5461380 0.3415650 0.4735680 0.3720215 0.5295281 0.6574699
+#> 985 986 1008 1022 1034 1041 1062 1089
+#> 0.2928026 0.2835489 0.5781580 0.6715157 0.4206344 0.4200000 0.4082483 0.3015515
+#> 1096 1155 1187 1223 1227 1248 1256 1282
+#> 0.2091252 0.3083288 0.2948446 0.6582806 0.3824483 0.6193545 0.2581989 0.2690725
+#> 1341 1359 1395 1429 1435 1467 1469 1515
+#> 0.3282276 0.5224302 0.5540156 0.5326662 0.5605949 0.3316625 0.4541143 0.4504812
+#> 1522 1528 1538 1606 1617 1642 1650 1655
+#> 0.4393935 0.4512206 0.3969887 0.5595236 0.3214550 0.4446722 0.5648599 0.2715388
+#> 1660 1835 1865
+#> 0.3390182 0.3229035 0.3555278
+
+var1$cutoff
+#> 40%
+#> 0.4048997
Using our 40% decile as a guide, we find that 0.405 is our target +standard error for an accurately measured item.
+To estimate the minimum sample size, we should figure out what number +of participants it would take to achieve 80%, 85%, 90%, and 95% of the +SEs for items below our critical score of 0.405?
+
+cutoff <- calculate_cutoff(population = DF,
+ grouping_items = "items",
+ score = "Rating",
+ minimum = as.numeric(min(DF$Rating)),
+ maximum = as.numeric(max(DF$Rating)))
+# showing how this is the same as the person calculated version versus semanticprimeR's function
+cutoff$cutoff
+#> 40%
+#> 0.4048997
+
+final_table <- calculate_correction(
+ proportion_summary = var1$final_sample,
+ pilot_sample_size = DF %>% group_by(items) %>% summarize(n = n()) %>%
+ pull(n) %>% mean() %>% round(),
+ proportion_variability = cutoff$prop_var
+ )
+
+flextable(final_table) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
80.40000 |
+45 |
+43.27286 |
+
88.80000 |
+55 |
+54.58167 |
+
91.86667 |
+60 |
+59.97526 |
+
97.20000 |
+70 |
+70.19716 |
+
Our minimum sample size is small at 80% (n = 43 as the +minimum). We could consider using 90% (n = 60) or 95% +(n = 70).
+While there are many considerations for maximum sample size (time, +effort, resources), if we consider a higher value just for estimation +sake, we could use n = at 98%.
+In any estimate of sample size, you should also consider the +potential for missing data and/or unusable data due to any other +exclusion criteria in your study (i.e., attention checks, speeding, +getting the answer right, etc.). In this study, these values can be +influenced by the other variables that we used to select the stimuli in +the study.
+barzykowski_vignette.Rmd
Cue Word Triggered Memory’s Phenomenological
+Data provided by: Krystian Barzykowski
+Participants participated in a voluntary memory task, where they were +provided with a word cue in response to which they were about to recall +an autobiographical memory. The item set consists of 30 word cues that +were rated/classified by 142 separate participants.
+They briefly described the content of their thoughts recalled in +response to the word-cue and rated it on a 7-point scale: (a) to what +extent the content was accompanied by unexpected physiological +sensations (henceforth, called physiological sensation), (b) to what +extent they had deliberately tried to bring the thought to mind +(henceforth, called effort), (c) clarity (i.e. how clearly and well an +individual remembered a given memory/mental content), (d) how detailed +the content was, (e) how specific and concrete the content was, (f) +intensity of emotions experienced in response to the content, (g) how +surprising the content was, (h) how personal it was, and (i) the +relevance to current life situation (not included).
+Data included within this vignette.
+
+DF <- import("data/barzykowski_data.xlsx") %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 2)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 3)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 4)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 5)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 6)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 7)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 8)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 9)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 10)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 11)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 12)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 13)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 14)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 15)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 16)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 17)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 18)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 19)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 20)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 21)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 22)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 23)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 24)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 25)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 26)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 27)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 28)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 29)) %>%
+ bind_rows(import("data/barzykowski_data.xlsx", sheet = 30))
+
+str(DF)
+#> 'data.frame': 918 obs. of 11 variables:
+#> $ Participant's ID : num 12 13 14 15 16 17 18 19 20 21 ...
+#> $ Cue no : num 1 1 1 1 1 1 1 1 1 1 ...
+#> $ Physiological reaction: num 1 1 1 1 1 1 2 2 4 2 ...
+#> $ Effort : num 4 4 7 4 1 2 4 4 2 3 ...
+#> $ Vividness : num 5 6 1 4 2 7 3 6 3 3 ...
+#> $ Clarity : num 2 4 1 4 6 7 4 5 4 4 ...
+#> $ Detailidness : num 5 5 1 4 4 7 2 3 3 2 ...
+#> $ Concretness : num 2 4 3 5 4 7 5 5 3 2 ...
+#> $ Emotional Intensity : num 3 1 2 2 1 6 2 1 2 1 ...
+#> $ How surprising : num 5 2 1 1 4 4 3 2 5 1 ...
+#> $ Personal nature : num 2 1 4 3 1 5 1 1 2 1 ...
The cues were used in study published here: Barzykowski, K., +Niedźwieńska, A., & Mazzoni, G. (2019). How intention to retrieve a +memory and expectation that it will happen influence retrieval of +autobiographical memories. Consciousness and Cognition, 72, 31-48. DOI: +https://doi.org/10.1016/j.concog.2019.03.011
+Open access with reference to original paper +(Attribution-NonCommercial-ShareAlike CC BY-NC-SA)
+
+metadata <- import("data/barzykowski_metadata.xlsx")
+
+flextable(metadata) %>% autofit()
Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
Participant's ID |
+Participants’ identification number |
+Numeric |
+
Cue no |
+Number of the specific cue they saw |
+Numeric |
+
Physiological reaction |
+To what extent the content was accompanied by unexpected physiological sensations (henceforth, called physiological sensation) |
+Numeric (1 to 7 scale) |
+
Effort |
+To what extent they had deliberately tried to bring the thought to mind (henceforth, called effort) |
+Numeric (1 to 7 scale) |
+
Vividness |
+How vivid the thought was |
+Numeric (1 to 7 scale) |
+
Clarity |
+Clarity (i.e. how clearly and well an individual remembered a given memory/mental content) |
+Numeric (1 to 7 scale) |
+
Detailidness |
+How detailed the content was |
+Numeric (1 to 7 scale) |
+
Concretness |
+How specific and concrete the content was |
+Numeric (1 to 7 scale) |
+
Emotional Intensity |
+Intensity of emotions experienced in response to the content |
+Numeric (1 to 7 scale) |
+
How surprising |
+How surprising the
+ |
+Numeric (1 to 7 scale) |
+
Personal nature |
+How personal it was |
+Numeric (1 to 7 scale) |
+
Note that the data is already in long format (each item has one row), +and therefore, we do not need to restructure the data.
+In this example, we have multiple variables to choose from for our +analysis. We could include several to find the sample size rules for +further study. In this example, we’ll use the variables with the least +and most variability and take the average of the 40% decile as suggested +in our manuscript. This choice is somewhat arbitrary - in a real study, +you could choose to use only the variables you were interested in and +pick the most conservative values or simply average together estimates +from all variables.
+
+apply(DF[ , -c(1,2)], 2, sd)
+#> Physiological reaction Effort Vividness
+#> 1.691892 1.577815 1.651065
+#> Clarity Detailidness Concretness
+#> 1.651066 1.696818 1.646454
+#> Emotional Intensity How surprising Personal nature
+#> 1.753200 1.500973 1.889276
These are Likert type items. The variance within them appears roughly +equal. The lowest variance appears to be How surprising, and the maximum +appears to be Personal nature.
+Run the function proposed in the manuscript:
+
+# set seed
+set.seed(8548)
+# Function for simulation
+var1 <- item_power(data = DF, # name of data frame
+ dv_col = "How surprising", # name of DV column as a character
+ item_col = "Cue no", # number of items column as a character
+ nsim = 10,
+ sample_start = 20,
+ sample_stop = 100,
+ sample_increase = 5,
+ decile = .4)
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+var2 <- item_power(DF, # name of data frame
+ "Personal nature", # name of DV column as a character
+ item_col = "Cue no", # number of items column as a character
+ nsim = 10,
+ sample_start = 20,
+ sample_stop = 100,
+ sample_increase = 5,
+ decile = .4)
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
What the usual standard error for the data that could be considered +for our stopping rule?
+
+# individual SEs for how surprising
+var1$SE
+#> 1 2 3 4 5 6 7 8
+#> 0.2654025 0.1843656 0.2395449 0.2463527 0.2478174 0.2702634 0.2385878 0.1781742
+#> 9 10 11 12 13 14 15 16
+#> 0.2643095 0.2784668 0.2803546 0.2595053 0.2568655 0.2387265 0.3268158 0.2692327
+#> 17 18 19 20 21 22 23 24
+#> 0.3114397 0.3007016 0.2886410 0.2815891 0.2677375 0.1979609 0.2478808 0.2986490
+#> 25 26 27 28 29 30
+#> 0.3589899 0.2982066 0.2921602 0.3035231 0.2643255 0.3249033
+# var 1 cut off
+var1$cutoff
+#> 40%
+#> 0.2643191
+
+# individual SEs for personal nature
+var2$SE
+#> 1 2 3 4 5 6 7 8
+#> 0.2870153 0.2839809 0.2432371 0.3319512 0.3600411 0.4064413 0.3389511 0.3191424
+#> 9 10 11 12 13 14 15 16
+#> 0.2502489 0.3696715 0.3708976 0.2800560 0.2674643 0.2832563 0.4580862 0.2836613
+#> 17 18 19 20 21 22 23 24
+#> 0.3588157 0.3122039 0.3498098 0.3617125 0.3435939 0.3702658 0.3156137 0.2744988
+#> 25 26 27 28 29 30
+#> 0.4125648 0.3298670 0.3760496 0.3258969 0.2797701 0.3615385
+# var 2 cut off
+var2$cutoff
+#> 40%
+#> 0.3177309
+
+# overall cutoff
+cutoff <- mean(var1$cutoff, var2$cutoff)
+cutoff
+#> [1] 0.2643191
The average SE cutoff across both variables is 0.264.
+How large does the sample have to be for 80% to 95% of the items to +be below our stopping SE rule?
+
+cutoff_personal <- calculate_cutoff(population = DF,
+ grouping_items = "Cue no",
+ score = "Personal nature",
+ minimum = as.numeric(min(DF$`Personal nature`)),
+ maximum = as.numeric(max(DF$`Personal nature`)))
+# showing how this is the same as the person calculated version versus semanticprimeR's function
+cutoff_personal$cutoff
+#> 40%
+#> 0.3177309
+
+final_table_personal <- calculate_correction(
+ proportion_summary = var1$final_sample,
+ pilot_sample_size = length(unique(DF$`Participant's ID`)),
+ proportion_variability = cutoff_personal$prop_var
+ )
+
+flextable(final_table_personal) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
83.33333 |
+40 |
+31.06408 |
+
91.66667 |
+45 |
+37.55466 |
+
91.66667 |
+45 |
+37.55466 |
+
97.66667 |
+50 |
+44.05820 |
+
+cutoff_surprising <- calculate_cutoff(population = DF,
+ grouping_items = "Cue no",
+ score = "How surprising",
+ minimum = as.numeric(min(DF$`How surprising`)),
+ maximum = as.numeric(max(DF$`How surprising`)))
+# showing how this is the same as the person calculated version versus semanticprimeR's function
+cutoff_surprising$cutoff
+#> 40%
+#> 0.2643191
+
+final_table_surprising <- calculate_correction(
+ proportion_summary = var2$final_sample,
+ pilot_sample_size = length(unique(DF$`Participant's ID`)),
+ proportion_variability = cutoff_surprising$prop_var
+ )
+
+flextable(final_table_surprising) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
89 |
+45 |
+37.88715 |
+
89 |
+45 |
+37.88715 |
+
97 |
+50 |
+44.40287 |
+
97 |
+50 |
+44.40287 |
+
In this scenario, we could go with the point wherein they both meet +the 80% criterion, which is \(n_{personal}\) = 31 to \(n_{surprising}\) = 38. In these scenarios, +it is probably better to estimate a larger sample.
+If you decide to use 95% power as your criterion, you would see that +items need somewhere between \(n_{personal}\) = 44 to \(n_{surprising}\) = 44 participants for both +variables. In this case, you could choose to make the larger value for +participants your maximum sample size to ensure both variables reach the +criterion.
+You should also consider any potential for missing data and/or +unusable data given the requirements for your study. Given that +participants are likely to see all items in this study, we could use the +minimum, stopping rule, and maximum defined above. However, one should +consider that not all participants will be able to respond to all items +within a memory.
+batres_vignette.Rmd
+knitr::opts_chunk$set(echo = TRUE)
+
+# Set a random seed
+set.seed(5989320)
+
+# Libraries necessary for this vignette
+library(rio)
+library(flextable)
+library(dplyr)
+#>
+#> Attaching package: 'dplyr'
+#> The following objects are masked from 'package:stats':
+#>
+#> filter, lag
+#> The following objects are masked from 'package:base':
+#>
+#> intersect, setdiff, setequal, union
+library(tidyr)
+library(psych)
+library(semanticprimeR)
+
+# Function for simulation
+item_power <- function(data, # name of data frame
+ dv_col, # name of DV column as a character
+ item_col, # number of items column as a character
+ nsim = 10, # small for cran
+ sample_start = 20,
+ sample_stop = 200,
+ sample_increase = 5,
+ decile = .5){
+
+ DF <- cbind.data.frame(
+ "dv" = data[ , dv_col],
+ "items" = data[ , item_col]
+ )
+
+ # just in case
+ colnames(DF) <- c("dv", "items")
+
+ # figure out the "sufficiently narrow" ci value
+ SE <- tapply(DF$dv, DF$items, function (x) { sd(x)/sqrt(length(x)) })
+ cutoff <- quantile(SE, probs = decile)
+
+ # sequence of sample sizes to try
+ samplesize_values <- seq(sample_start, sample_stop, sample_increase)
+
+ # create a blank table for us to save the values in
+ sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(DF$items)))
+
+ # make it a data frame
+ sim_table <- as.data.frame(sim_table)
+
+ # add a place for sample size values
+ sim_table$sample_size <- NA
+
+ iterate <- 1
+
+ for (p in 1:nsim){
+ # loop over sample sizes
+ for (i in 1:length(samplesize_values)){
+
+ # temp that samples and summarizes
+ temp <- DF %>%
+ group_by(items) %>%
+ sample_n(samplesize_values[i], replace = T) %>%
+ summarize(se = sd(dv)/sqrt(length(dv)))
+
+ # dv on items
+ colnames(sim_table)[1:length(unique(DF$items))] <- temp$items
+ sim_table[iterate, 1:length(unique(DF$items))] <- temp$se
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ iterate <- iterate + 1
+ }
+ }
+
+ # figure out cut off
+ final_sample <- sim_table %>%
+ pivot_longer(cols = -c(sample_size, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ group_by(sample_size, nsim) %>%
+ summarize(percent_below = sum(se <= cutoff)/length(unique(DF$items))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+
+ return(list(
+ SE = SE,
+ cutoff = cutoff,
+ DF = DF,
+ sim_table = sim_table,
+ final_sample = final_sample
+ ))
+
+}
This dataset contains 200 participants rating 20 faces on +attractiveness. Ethical approval was received from the Franklin and +Marshall Institutional Review Board and each participant provided +informed consent. All participants were located in the United States. +Participants were instructed that they would be viewing several faces +which were photographed facing forward, under constant camera and +lighting conditions, with neutral expressions, and closed mouths. Each +participant would have to rate the attractiveness of the presented +faces. More specifically, participants were asked “How attractive is +this face?”, where 1 = “Not at all attractive” and 7 = “Very +attractive”. Participants rated each face individually, in random order, +and with no time limit. Upon completion, participants were paid for +participation in the study.
+Included with the vignette.
+
+DF <- import("data/batres_data.sav")
+
+str(DF)
+#> 'data.frame': 200 obs. of 21 variables:
+#> $ Participant_Number: num 1 2 3 4 5 6 7 8 9 10 ...
+#> ..- attr(*, "label")= chr "Unique number assigned to each participant"
+#> ..- attr(*, "format.spss")= chr "F3.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_1 : num 1 2 5 2 3 1 2 2 1 2 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #1"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_2 : num 1 6 5 2 3 1 3 2 2 2 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #2"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_3 : num 3 6 7 7 4 3 5 4 4 4 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #3"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_4 : num 3 7 5 3 4 3 3 3 4 3 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #4"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_5 : num 5 7 7 5 5 6 3 3 3 3 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #5"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_6 : num 5 5 4 5 6 5 4 4 5 3 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #6"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_7 : num 5 7 7 7 4 5 4 4 5 4 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #7"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_8 : num 4 1 5 3 4 4 4 4 2 4 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #8"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_9 : num 3 5 4 4 3 1 2 2 2 2 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #9"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_10 : num 4 4 7 2 3 3 3 3 5 4 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #10"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_11 : num 2 3 5 4 3 2 3 3 4 2 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #11"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_12 : num 4 7 5 4 4 4 3 3 6 1 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #12"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_13 : num 3 3 4 5 4 3 3 3 3 2 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #13"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_14 : num 5 7 5 5 3 5 5 5 4 2 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #14"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_15 : num 3 7 6 3 4 6 3 3 4 4 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #15"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_16 : num 4 7 5 5 5 4 4 3 5 3 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #16"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_17 : num 4 4 5 3 5 4 3 2 4 2 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #17"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_18 : num 3 5 4 6 4 5 4 5 4 2 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #18"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_19 : num 3 4 5 6 4 4 4 3 3 4 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #19"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
+#> $ Face_20 : num 4 6 6 3 6 4 3 3 4 3 ...
+#> ..- attr(*, "label")= chr "Attractiveness rating for face #20"
+#> ..- attr(*, "format.spss")= chr "F1.0"
+#> ..- attr(*, "display_width")= int 12
Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
Participant_Number |
+Unique number assigned to each participant |
+Numeric |
+
Face_1 |
+Attractiveness rating for face #1 |
+Numeric |
+
Face_2 |
+Attractiveness rating for face #2 |
+Numeric |
+
Face_3 |
+Attractiveness rating for face #3 |
+Numeric |
+
Face_4 |
+Attractiveness rating for face #4 |
+Numeric |
+
Face_5 |
+Attractiveness rating for face #5 |
+Numeric |
+
Face_6 |
+Attractiveness rating for face #6 |
+Numeric |
+
Face_7 |
+Attractiveness rating for face #7 |
+Numeric |
+
Face_8 |
+Attractiveness rating for face #8 |
+Numeric |
+
Face_9 |
+Attractiveness rating for face #9 |
+Numeric |
+
Face_10 |
+Attractiveness rating for face #10 |
+Numeric |
+
Face_11 |
+Attractiveness rating for face #11 |
+Numeric |
+
Face_12 |
+Attractiveness rating for face #12 |
+Numeric |
+
Face_13 |
+Attractiveness rating for face #13 |
+Numeric |
+
Face_14 |
+Attractiveness rating for face #14 |
+Numeric |
+
Face_15 |
+Attractiveness rating for face #15 |
+Numeric |
+
Face_16 |
+Attractiveness rating for face #16 |
+Numeric |
+
Face_17 |
+Attractiveness rating for face #17 |
+Numeric |
+
Face_18 |
+Attractiveness rating for face #18 |
+Numeric |
+
Face_19 |
+Attractiveness rating for face #19 |
+Numeric |
+
Face_20 |
+Attractiveness rating for face #20 |
+Numeric |
+
The data should be in long format with each rating on one row of +data.
+
+# Reformat the data
+DF_long <- pivot_longer(DF, cols = -c(Participant_Number)) %>%
+ dplyr::rename(item = name, score = value)
+
+flextable(head(DF_long)) %>% autofit()
Participant_Number |
+item |
+score |
+
---|---|---|
1 |
+Face_1 |
+1 |
+
1 |
+Face_2 |
+1 |
+
1 |
+Face_3 |
+3 |
+
1 |
+Face_4 |
+3 |
+
1 |
+Face_5 |
+5 |
+
1 |
+Face_6 |
+5 |
+
+# Function for simulation
+var1 <- item_power(data = DF_long, # name of data frame
+ dv_col = "score", # name of DV column as a character
+ item_col = "item", # number of items column as a character
+ nsim = 10,
+ sample_start = 20,
+ sample_stop = 300,
+ sample_increase = 5,
+ decile = .4)
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
What the usual standard error for the data that could be considered +for our stopping rule using the 40%% decile?
+
+# individual SEs
+var1$SE
+#> Face_1 Face_10 Face_11 Face_12 Face_13 Face_14 Face_15
+#> 0.09117808 0.09064190 0.10007472 0.09739767 0.08562437 0.08767230 0.09331351
+#> Face_16 Face_17 Face_18 Face_19 Face_2 Face_20 Face_3
+#> 0.10262632 0.09082536 0.09530433 0.09123386 0.08818665 0.09799754 0.08644573
+#> Face_4 Face_5 Face_6 Face_7 Face_8 Face_9
+#> 0.08915009 0.09127172 0.09078109 0.09968796 0.08977638 0.09481468
+
+var1$cutoff
+#> 40%
+#> 0.09080765
Using our 40%% decile as a guide, we find that 0.091 is our target +standard error for an accurately measured item.
+To estimate minimum sample size, we should figure out what number of +participants it would take to achieve 80%, 85%, 90%, and 95% of the SEs +for items below our critical score of 0.091?
+
+cutoff <- calculate_cutoff(population = DF_long,
+ grouping_items = "item",
+ score = "score",
+ minimum = 1,
+ maximum = 7)
+# showing how this is the same as the person calculated version versus semanticprimeR's function
+cutoff$cutoff
+#> 40%
+#> 0.09080765
Please note that you will always need to simulate larger than the +pilot data sample size to get the starting numbers. We will correct them +below. As shown in our manuscript, we need to correct for the +overestimation of sample sizes based on the original pilot data size. +Given that the pilot data is large: 200, this correction is especially +useful. This correction is built into our function.
+
+final_table <- calculate_correction(
+ proportion_summary = var1$final_sample,
+ pilot_sample_size = nrow(DF),
+ proportion_variability = cutoff$prop_var
+ )
+
+flextable(final_table) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
80.0 |
+230 |
+54.60714 |
+
90.5 |
+245 |
+62.29201 |
+
90.5 |
+245 |
+62.29201 |
+
97.0 |
+255 |
+68.01402 |
+
Our minimum suggested sample size does not exist at exactly 80% of +the items, but instead we can use the first available over 80% +(n = 55 as the minimum).
+While there are many considerations for maximum sample size (time, +effort, resources), the simulation suggests that 68 people would ensure +nearly all items achieve cutoff criterions.
+In any estimate for sample size, you should also consider the +potential for missing data and/or unusable data due to any other +exclusion criteria in your study (i.e., attention checks, speeding, +getting the answer right, etc.). In this study, we likely expect all +participants to see all items and therefore, we could expect to use the +minimum sample size as our final sample size, the point at which all +items reach our SE criterion, or the maximum sample size.
+geller_vignette.Rmd
+knitr::opts_chunk$set(echo = TRUE)
+
+# Set a random seed
+set.seed(3898934)
+
+# Libraries necessary for this vignette
+library(rio)
+library(flextable)
+library(dplyr)
+#>
+#> Attaching package: 'dplyr'
+#> The following objects are masked from 'package:stats':
+#>
+#> filter, lag
+#> The following objects are masked from 'package:base':
+#>
+#> intersect, setdiff, setequal, union
+library(tidyr)
+library(psych)
+library(semanticprimeR)
+
+# Function for simulation
+item_power <- function(data, # name of data frame
+ dv_col, # name of DV column as a character
+ item_col, # number of items column as a character
+ nsim = 10, # small for cran
+ sample_start = 20,
+ sample_stop = 200,
+ sample_increase = 5,
+ decile = .5){
+
+ DF <- cbind.data.frame(
+ "dv" = data[ , dv_col],
+ "items" = data[ , item_col]
+ )
+
+ # just in case
+ colnames(DF) <- c("dv", "items")
+
+ # figure out the "sufficiently narrow" ci value
+ SE <- tapply(DF$dv, DF$items, function (x) { sd(x)/sqrt(length(x)) })
+ cutoff <- quantile(SE, probs = decile)
+
+ # sequence of sample sizes to try
+ samplesize_values <- seq(sample_start, sample_stop, sample_increase)
+
+ # create a blank table for us to save the values in
+ sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(DF$items)))
+
+ # make it a data frame
+ sim_table <- as.data.frame(sim_table)
+
+ # add a place for sample size values
+ sim_table$sample_size <- NA
+
+ iterate <- 1
+
+ for (p in 1:nsim){
+ # loop over sample sizes
+ for (i in 1:length(samplesize_values)){
+
+ # temp that samples and summarizes
+ temp <- DF %>%
+ group_by(items) %>%
+ sample_n(samplesize_values[i], replace = T) %>%
+ summarize(se = sd(dv)/sqrt(length(dv)))
+
+ # dv on items
+ colnames(sim_table)[1:length(unique(DF$items))] <- temp$items
+ sim_table[iterate, 1:length(unique(DF$items))] <- temp$se
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ iterate <- iterate + 1
+ }
+ }
+
+ # figure out cut off
+ final_sample <- sim_table %>%
+ pivot_longer(cols = -c(sample_size, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ group_by(sample_size, nsim) %>%
+ summarize(percent_below = sum(se <= cutoff)/length(unique(DF$items))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+
+ return(list(
+ SE = SE,
+ cutoff = cutoff,
+ DF = DF,
+ sim_table = sim_table,
+ final_sample = final_sample
+ ))
+
+}
Overconfidence for picture cues in foreign language learning
+Data provided by: Jason Geller
+Previous research shows that participants are overconfident in their +ability to learn foreign language vocabulary from pictures compared with +English translations. The current study explored whether this tendency +is due to processing fluency or beliefs about learning. Using self-paced +study of Swahili words paired with either picture cues or English +translation cues, 30 participants provided JOLs to each of the 42 +English-Swahili word pairs from Carpenter and Olson’s (2012) Experiment +2.The English words were one-syllable nouns, ranging between three and +six letters, with an average concreteness rating of 4.86 on a 5-point +scale (SD = .16) (Brysbaert, Warriner, & Kuperman, 2014), and an +average frequency of 106.52 per million (SD = 113.40) (Brysbaert & +New, 2009).
+Participants began the experiment with instructions informing them +that they would be learning Swahili words paired with either pictures or +English translations as cues. To illustrate each type of cue, they were +given an example of an item (Train: Reli) that was not included among +the 42 experimental items. They were informed that each pair of items +(English-Swahili pairs or picture-Swahili pairs) would be presented one +at a time, and they would have as much time as they needed to study it. +Participants were encouraged to do their best to learn each pair, and to +encourage full and meaningful processing of each, they were instructed +to press the spacebar once they felt they had fully “digested” it. For +each participant, 21 items were randomly selected to be presented as +English-Swahili pairs, and 21 as picture-Swahili pairs. Participants saw +each stimulus pair one at a time, in a unique random order with +English-Swahili pairs and picture-Swahili pairs intermixed. Each pair +was presented in the center of the computer screen and remained on +screen until participants pressed the spacebar to move on to the next +pair. After each of the 42 pairs was presented for self-paced study in +this way, the same pairs were presented again for JOLs. During a JOL +trial, each cue-target pair was presented on the screen and participants +were asked to estimate—using a scale from 0% (definitely will NOT +recall) to 100% (definitely will recall)—the likelihood of recalling the +Swahili word from its cue (either the picture or English translation) +after about 5 minutes. Participants entered a value between 0 and 100 +and pressed the ENTER key to advance to the next item.
+Data can be found here: https://osf.io/2byt9/.
+
+#read in data
+DF <- import("data/geller_data.xlsx") %>%
+ select(Experiment, Subject, `CueType[1Word,2Pic]`, Stimulus, EncodeJOL)
+#> Warning: Expecting numeric in J2069 / R2069C10: got 'jico'
+
+str(DF)
+#> 'data.frame': 2898 obs. of 5 variables:
+#> $ Experiment : num 1 1 1 1 1 1 1 1 1 1 ...
+#> $ Subject : num 1 1 1 1 1 1 1 1 1 1 ...
+#> $ CueType[1Word,2Pic]: num 1 1 1 1 1 1 1 1 1 1 ...
+#> $ Stimulus : chr "kidoto" "muhindi" "kiti" "jaluba" ...
+#> $ EncodeJOL : num 1 1 1 1 1 1 1 1 1 1 ...
Carpenter, S. K., & Geller, J. (2020). Is a picture really worth +a thousand words? Evaluating contributions of fluency and analytic +processing in metacognitive judgements for pictures in foreign language +vocabulary learning. Quarterly Journal of Experimental Psychology, +73(2), 211–224. https://doi.org/10.1177/1747021819879416
+Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014). +Concreteness ratings for 40 thousand generally known English word +lemmas. Behavior Research Methods, 46(3), 904–911. https://doi.org/10.3758/s13428-013-0403-5
+Brysbaert, M., & New, B. (2009). Moving beyond Kučera and +Francis: A critical evaluation of current word frequency norms and the +introduction of a new and improved word frequency measure for American +English. Behavior Research Methods, 41(4), 977–990. https://doi.org/10.3758/BRM.41.4.977
+Carpenter, S. K., & Geller, J. (2020). Is a picture really worth +a thousand words? Evaluating contributions of fluency and analytic +processing in metacognitive judgements for pictures in foreign language +vocabulary learning. Quarterly Journal of Experimental Psychology, +73(2), 211–224. https://doi.org/10.1177/1747021819879416
+Carpenter, S. K., & Olson, K. M. (2012). Are pictures good for +learning new vocabulary in a foreign language? Only if you think they +are not. Journal of Experimental Psychology: Learning, Memory, and +Cognition, 38(1), 92–101. https://doi.org/10.1037/a0024828
+Overconfidence, metacognition, processing fluency, analytic +processing, foreign language learning
+Open access with reference to original paper +(Attribution-NonCommercial-ShareAlike CC BY-NC-SA)
+
+metadata <- tibble::tribble(
+ ~Variable.Name, ~Variable.Description, ~`Type (numeric,.character,.logical,.etc.)`,
+ "Experiment", "Experiment 1 (1) or 2 (2) ONLY USE 1", NA,
+ "Subject", "Subject ID", "Numeric",
+ "CueType", "Whether participant was presented with word translation (1) or word with picture (2)", "Numeric",
+ "Stimulus", "Swahili words presented on each trail", "Character",
+ "EncodeJOL", "JOL (1-100) 1=not likely to recall 100=very likely to recall", "Numeric"
+ )
+
+flextable(metadata) %>% autofit()
Variable.Name |
+Variable.Description |
+Type (numeric,.character,.logical,.etc.) |
+
---|---|---|
Experiment |
+Experiment 1 (1) or 2 (2) ONLY USE 1 |
+|
Subject |
+Subject ID |
+Numeric |
+
CueType |
+Whether participant was presented with word translation (1) or word with picture (2) |
+Numeric |
+
Stimulus |
+Swahili words presented on each trail |
+Character |
+
EncodeJOL |
+JOL (1-100) 1=not likely to recall 100=very likely to recall |
+Numeric |
+
+DF <- DF %>%
+ filter(Experiment == 1) %>%
+ filter(!is.na(EncodeJOL))
+
+# Function for simulation
+var1 <- item_power(data = DF, # name of data frame
+ dv_col = "EncodeJOL", # name of DV column as a character
+ item_col = "Stimulus", # number of items column as a character
+ nsim = 10,
+ sample_start = 20,
+ sample_stop = 100,
+ sample_increase = 5,
+ decile = .4)
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
What the usual standard error for the data that could be considered +for our stopping rule using the 40% decile?
+
+# individual SEs
+var1$SE
+#> andiko bao bunduki chaka chapeo chimbule daraja dawati
+#> 5.569863 5.421327 6.467708 5.812192 5.232406 5.964306 5.006286 4.295635
+#> dubu duwara farasi fia fupa gari geli jaja
+#> 6.825573 3.857377 6.126705 5.192728 6.954286 6.092600 5.496488 7.424850
+#> jaluba jicho jiti jumba juya kanisa kelb kidoto
+#> 4.803243 5.649133 6.229624 5.405121 3.783108 6.245590 6.311649 4.324860
+#> kipira kitanda kiti maliki mapwa mkono mlango muhindi
+#> 5.895944 5.515605 6.404627 6.614947 6.634286 7.609701 5.745819 5.471669
+#> muundi papatiko pua rinda riza safina samaki simu
+#> 6.108910 6.681585 6.309811 5.378363 6.045881 6.548968 5.384973 5.490716
+#> ufunguo wardi
+#> 6.229707 5.739385
+
+var1$cutoff
+#> 40%
+#> 5.601571
Using our 40% decile as a guide, we find that 5.602 is our target +standard error for an accurately measured item.
+To estimate minimum sample size, we should figure out what number of +participants it would take to achieve 80%, 85%, 90%, and 95% of the SEs +for items below our critical score of 5.602?
+
+cutoff <- calculate_cutoff(population = DF,
+ grouping_items = "Stimulus",
+ score = "EncodeJOL",
+ minimum = as.numeric(min(DF$EncodeJOL)),
+ maximum = as.numeric(max(DF$EncodeJOL)))
+# showing how this is the same as the person calculated version versus semanticprimeR's function
+cutoff$cutoff
+#> 40%
+#> 5.601571
+
+final_table <- calculate_correction(
+ proportion_summary = var1$final_sample,
+ pilot_sample_size = length(unique(DF$Subject)),
+ proportion_variability = cutoff$prop_var
+ )
+
+flextable(final_table) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
87.85714 |
+35 |
+31.48800 |
+
87.85714 |
+35 |
+31.48800 |
+
97.38095 |
+40 |
+39.17344 |
+
97.38095 |
+40 |
+39.17344 |
+
Our minimum sample size is (n = 31 as the minimum at 80%). +We could consider using 90% (n = 39) or 95% (n = +39).
+While there are many considerations for maximum sample size (time, +effort, resources), if we consider a higher value just for estimation +sake, we could use n = at nearly 100%.
+In any estimate for sample size, you should also consider the +potential for missing data and/or unusable data due to any other +exclusion criteria in your study (i.e., attention checks, speeding, +getting the answer right, etc.). In this study, these values may be +influenced by the pictures/word split in the study.
+heyman_vignette.Rmd
Continuous lexical decision task: classification of Dutch words as +either actual words or nonwords
+Data provided by: Tom Heyman
+Data come from a study reported in Heyman, De Deyne, Hutchison, & +Storms (2015, Behavior Research Methods; henceforth HDHS). More +specifically, the study involved a continuous lexical decision task +intended to measure (item-level) semantic priming effects (i.e., +Experiment 3 of HDHS). It is similar to the SPAML set-up (see https://osf.io/q4fjy/), but +with fewer items and participants. The study had several goals, but +principally we wanted to examine how a different/new paradigm called the +speeded word fragment completion task would compare against a more +common, well-established paradigm like lexical decision in terms of +semantic priming (i.e., magnitude of the effect, reliability of +item-level priming, cross-task correlation of item-level priming +effects, etc.). Experiment 3 only involved a continuous lexical decision +task, so the datafile contains no data from the speeded word fragment +completion task.
+Participants were 40 students from the University of Leuven, Belgium +(10 men, 30 women, mean age 20 years). A total of 576 pairs were used in +a continuous lexical decision task (so participants do not perceive them +as pairs): 144 word–word pairs, 144 word–pseudoword pairs, 144 +pseudoword–word pairs, and 144 pseudoword–pseudoword pairs. Of the 144 +word-word pairs, 72 were fillers and 72 were critical pairs, half of +which were related, the other half unrelated (this was counterbalanced +across participants). The dataset only contains data for the critical +pairs. Participants were informed that they would see a letter string on +each trial and that they had to indicate whether the letter string +formed an existing Dutch word or not by pressing the arrow keys. Half of +the participants had to press the left arrow for word and the right +arrow for nonword, and vice versa for the other half.
+The example dataset also includes R scripts at this location +that used Accuracy in Parameter Estimation in a different fashion.
+
+HDHS<- read.csv("data/HDHSAIPE.txt", sep="")
+str(HDHS)
+#> 'data.frame': 2880 obs. of 8 variables:
+#> $ RT : num 0.52 0.453 0.467 0.534 0.573 ...
+#> $ zRT : num -0.303 -0.492 -0.453 -0.265 -0.153 ...
+#> $ Pp : int 1 1 1 1 1 1 1 1 1 1 ...
+#> $ Type : chr "R" "R" "R" "R" ...
+#> $ Prime : chr "hengst" "matrak" "eland" "erwt" ...
+#> $ Target : chr "veulen" "wapen" "gewei" "wortel" ...
+#> $ accTarget: int 1 1 1 1 1 1 1 1 1 0 ...
+#> $ accPrime : int 1 1 1 1 0 1 1 1 1 0 ...
Heyman, T. (2022, February 4). Dataset AIPE. Retrieved from +osf.io/frxpd [based on Heyman, T., De Deyne, S., Hutchison, K. A., & +Storms, G. (2015). Using the speeded word fragment completion task to +examine semantic priming. Behavior Research Methods, 47(2), +580-606.]
+Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
RT |
+Response time to the target in seconds |
+Numeric |
+
zRT |
+Z-transformed target response times per participant |
+Numeric |
+
Pp |
+Participant identifier (1 to 40) |
+Integer |
+
Type |
+Whether target was preceded by a related prime (R) or an unrelated prime (U) |
+Character |
+
Prime |
+Prime stimulus (in Dutch) |
+Character |
+
Target |
+Target stimulus (in Dutch) |
+Character |
+
accTarget |
+Whether response to target was correct (1) or not (0) |
+Integer |
+
accPrime |
+Whether response to the preceding prime was correct (1) or not (0) |
+Integer |
+
+# pick only correct answers
+HDHScorrect <- HDHS[HDHS$accTarget==1,]
+summary_stats <- HDHScorrect %>% #data frame
+ select(RT, Target) %>% #pick the columns
+ group_by(Target) %>% #put together the stimuli
+ summarize(SES = sd(RT)/sqrt(length(RT)), samplesize = length(RT)) #create SE and the sample size for below
+##give descriptives of the SEs
+describe(summary_stats$SES)
+#> vars n mean sd median trimmed mad min max range skew kurtosis se
+#> X1 1 72 0.05 0.05 0.03 0.04 0.02 0.02 0.31 0.29 3.63 13.19 0.01
+
+##figure out the original sample sizes (not really necessary as all Targets were seen by 40 participants)
+original_SS <- HDHS %>% #data frame
+ count(Target) #count up the sample size
+##add the original sample size to the data frame
+summary_stats <- merge(summary_stats, original_SS, by = "Target")
+##original sample size average
+describe(summary_stats$n)
+#> vars n mean sd median trimmed mad min max range skew kurtosis se
+#> X1 1 72 40 0 40 40 0 40 40 0 NaN NaN 0
+
+##reduced sample size
+describe(summary_stats$samplesize)
+#> vars n mean sd median trimmed mad min max range skew kurtosis se
+#> X1 1 72 38.12 3.09 39 38.83 1.48 22 40 18 -3.29 12.08 0.36
+
+##percent retained
+describe(summary_stats$samplesize/summary_stats$n)
+#> vars n mean sd median trimmed mad min max range skew kurtosis se
+#> X1 1 72 0.95 0.08 0.98 0.97 0.04 0.55 1 0.45 -3.29 12.08 0.01
+
+flextable(head(HDHScorrect)) %>% autofit()
RT |
+zRT |
+Pp |
+Type |
+Prime |
+Target |
+accTarget |
+accPrime |
+
---|---|---|---|---|---|---|---|
0.5202093 |
+-0.3026836 |
+1 |
+R |
+hengst |
+veulen |
+1 |
+1 |
+
0.4532606 |
+-0.4918172 |
+1 |
+R |
+matrak |
+wapen |
+1 |
+1 |
+
0.4670391 |
+-0.4528923 |
+1 |
+R |
+eland |
+gewei |
+1 |
+1 |
+
0.5335296 |
+-0.2650529 |
+1 |
+R |
+erwt |
+wortel |
+1 |
+1 |
+
0.5732744 |
+-0.1527717 |
+1 |
+R |
+ijzel |
+glad |
+1 |
+0 |
+
0.3870141 |
+-0.6789673 |
+1 |
+R |
+sauna |
+warm |
+1 |
+1 |
+
What the usual standard error for the data that could be considered +for our stopping rule?
+
+SE <- tapply(HDHScorrect$RT, HDHScorrect$Target, function (x) { sd(x)/sqrt(length(x)) })
+min(SE)
+#> [1] 0.01511263
+max(SE)
+#> [1] 0.3058638
+
+cutoff <- quantile(SE, probs = .4)
+cutoff
+#> 40%
+#> 0.03113963
The items have a range of 0.0151126 to 0.3058638. We could use the +40% decile SE = 0.0311396 as our critical value for our stopping rule, +as suggested by the manuscript analysis. We could also have a set SE to +a specific target if we do not believe we have representative pilot data +in this example. You should also consider the scale when estimating +these values (i.e., millisecond data has more room to vary than other +smaller scales).
+To estimate minimum sample size, we should figure out what number of +participants it would take to achieve 80% of the SEs for items below our +critical score of 0.0311396?
+
+# sequence of sample sizes to try
+nsim <- 10 # small for cran
+samplesize_values <- seq(20, 500, 5)
+
+# create a blank table for us to save the values in
+sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(HDHS$Target)))
+
+# make it a data frame
+sim_table <- as.data.frame(sim_table)
+
+# add a place for sample size values
+sim_table$sample_size <- NA
+
+iterate <- 1
+
+for (p in 1:nsim){
+
+ # loop over sample sizes
+ for (i in 1:length(samplesize_values)){
+
+ # temp dataframe that samples and summarizes
+ temp <- HDHScorrect %>%
+ group_by(Target) %>%
+ sample_n(samplesize_values[i], replace = T) %>%
+ summarize(se = sd(RT)/sqrt(length(RT)))
+
+ colnames(sim_table)[1:length(unique(HDHScorrect$Target))] <- temp$Target
+ sim_table[iterate, 1:length(unique(HDHScorrect$Target))] <- temp$se
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+ iterate <- 1 + iterate
+ }
+
+}
+
+final_sample <-
+ sim_table %>%
+ pivot_longer(cols = -c(sample_size, nsim)) %>%
+ group_by(sample_size, nsim) %>%
+ summarize(percent_below = sum(value <= cutoff)/length(unique(HDHScorrect$Target))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample %>% head()) %>% autofit()
sample_size |
+percent_below |
+
---|---|
20 |
+0.3958333 |
+
25 |
+0.4416667 |
+
30 |
+0.4527778 |
+
35 |
+0.4972222 |
+
40 |
+0.5430556 |
+
45 |
+0.5708333 |
+
+# use semanticprimer cutoff function for prop variance
+cutoff <- calculate_cutoff(population = HDHScorrect,
+ grouping_items = "Target",
+ score = "RT",
+ minimum = as.numeric(min(HDHScorrect$RT)),
+ maximum = as.numeric(max(HDHScorrect$RT)))
+# showing how this is the same as the person calculated version versus semanticprimeR's function
+cutoff$cutoff
+#> 40%
+#> 0.03113963
+
+final_table <- calculate_correction(
+ proportion_summary = final_sample,
+ pilot_sample_size = HDHScorrect %>% group_by(Target) %>%
+ summarize(sample_size = n()) %>%
+ ungroup() %>% summarize(avg_sample = mean(sample_size)) %>%
+ pull(avg_sample),
+ proportion_variability = cutoff$prop_var
+ )
+
+flextable(final_table) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
80.55556 |
+95 |
+84.96635 |
+
85.41667 |
+120 |
+103.14554 |
+
90.13889 |
+180 |
+141.33833 |
+
Based on these simulations, we can decide our minimum sample size is +likely close to 85 and 88.984375 including information about data +loss.
+In this example, we could set our maximum sample size for 90% power +(as defined as 90% of items below our criterion), which would equate to +141 and 147.609375 with the expected data loss. The final table does not +include 95% of items below our criterion, even after estimating 500 +participants. An investigation of the table indicates that it levels off +at 93-94%.
+In any estimate of sample size, you should also consider the +potential for missing data and/or unusable data due to any other +exclusion criteria in your study (i.e., attention checks, speeding, +getting the answer right, etc.). In this study, we likely expect all +participants to see all items, and therefore, we could expect to use the +minimum sample size as our final sample size, the point at which all +items reach our SE criterion, or the maximum sample size. Note that +maximum sample sizes can also be defined by time, money, or other +means.
+mcfall_vignette.Rmd
Emerging Adulthood Measured at Multiple Institutions 2: The Next +Generation (EAMMi2)
+Data provided by: Joe McFall
+Collaborators from 32 academic institutions primarily in the United +States collected data from emerging adults (Nraw = 4220, Nprocessed = +3134). Participants completed self-report measures assessing markers of +adulthood, IDEA inventory of dimensions of emerging adulthood, +subjective well-being, mindfulness, belonging, self-efficacy, disability +identity, somatic health, perceived stress, perceived social support, +social media use, political affiliation, beliefs about the American +dream, interpersonal transgressions, narcissism, interpersonal +exploitativeness, beliefs about marriage, and demographics.
+Project organizers recruited contributors through social media +(Facebook & Twitter) and listserv invitations (Society of +Personality and Social Psychology, Society of Teaching Psychology).
+
+EAMMi2<- import("data/mcfall_data.sav.zip") %>%
+ select(starts_with("moa1#"), starts_with("moa2#"))
+str(EAMMi2)
+#> 'data.frame': 3134 obs. of 40 variables:
+#> $ moa1#1_1 : num 3 4 4 4 4 4 3 4 4 1 ...
+#> ..- attr(*, "label")= chr "imp_financialindependence"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_2 : num 4 4 4 3 2 3 2 3 3 2 ...
+#> ..- attr(*, "label")= chr "imp_nolongerhom"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_3 : num 3 4 4 4 4 4 1 2 4 1 ...
+#> ..- attr(*, "label")= chr "imp_finishededucation"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_4 : num 2 1 4 3 3 NA 1 1 4 1 ...
+#> ..- attr(*, "label")= chr "imp_married"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_5 : num 3 1 4 3 3 NA 1 1 4 1 ...
+#> ..- attr(*, "label")= chr "imp_havechild"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_6 : num 4 3 3 4 3 4 1 2 4 1 ...
+#> ..- attr(*, "label")= chr "imp_settledcareer"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_7 : num 4 2 1 4 4 4 1 1 3 4 ...
+#> ..- attr(*, "label")= chr "imp_avoiddrunk"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_8 : num 4 3 4 4 4 4 4 1 3 4 ...
+#> ..- attr(*, "label")= chr "imp_avoiddrugs"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_9 : num 4 4 4 4 2 4 4 1 2 4 ...
+#> ..- attr(*, "label")= chr "imp_usecontraception"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#1_10: num 2 2 4 3 3 3 1 1 3 1 ...
+#> ..- attr(*, "label")= chr "imp_committedlongterm"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa1#2_1 : num 1 2 2 3 2 1 2 1 2 3 ...
+#> ..- attr(*, "label")= chr "ach_financialindependence"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa1#2_2 : num 3 1 3 1 2 3 2 2 1 3 ...
+#> ..- attr(*, "label")= chr "ach_nolongerhome"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "achieved"
+#> $ moa1#2_3 : num 1 2 2 2 1 1 2 1 1 3 ...
+#> ..- attr(*, "label")= chr "ach_finishededucation"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "achieved"
+#> $ moa1#2_4 : num 2 1 3 1 1 NA 2 1 1 3 ...
+#> ..- attr(*, "label")= chr "ach_married"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "achieved"
+#> $ moa1#2_5 : num 3 1 3 1 1 NA 2 1 1 2 ...
+#> ..- attr(*, "label")= chr "ach_havechild"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "achieved"
+#> $ moa1#2_6 : num 2 2 1 1 1 1 2 1 1 3 ...
+#> ..- attr(*, "label")= chr "ach_settledcareer"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "achieved"
+#> $ moa1#2_7 : num 3 2 1 3 2 3 2 1 1 3 ...
+#> ..- attr(*, "label")= chr "ach_avoiddrunk"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "achieved"
+#> $ moa1#2_8 : num 3 3 3 3 3 3 2 1 1 3 ...
+#> ..- attr(*, "label")= chr "ach_avoiddrugs"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "achieved"
+#> $ moa1#2_9 : num 3 3 3 3 3 3 2 3 3 3 ...
+#> ..- attr(*, "label")= chr "ach_usecontraception"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa1#2_10: num 3 1 3 1 1 1 2 2 1 2 ...
+#> ..- attr(*, "label")= chr "ach_committedlongterm"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "achieved"
+#> $ moa2#1_1 : num 4 4 3 4 4 3 1 4 4 4 ...
+#> ..- attr(*, "label")= chr "imp_indepedentdecisions"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_2 : num 4 4 4 4 4 3 1 3 4 4 ...
+#> ..- attr(*, "label")= chr "imp_supportfamily"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_3 : num 3 3 4 3 4 1 1 2 4 4 ...
+#> ..- attr(*, "label")= chr "imp_carechildren"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_4 : num 4 4 4 4 3 4 1 4 3 4 ...
+#> ..- attr(*, "label")= chr "imp_acceptresponsibility"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_5 : num 4 3 4 3 3 4 1 3 4 2 ...
+#> ..- attr(*, "label")= chr "imp_employfulltime"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_6 : num 4 4 4 4 4 4 1 2 3 4 ...
+#> ..- attr(*, "label")= chr "imp_avoiddrunkdriving"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_7 : num 3 4 4 3 2 2 1 3 3 4 ...
+#> ..- attr(*, "label")= chr "imp_parentasequal"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_8 : num 4 3 4 4 4 4 1 4 4 4 ...
+#> ..- attr(*, "label")= chr "imp_emotionalcontrol"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_9 : num 3 3 4 4 3 4 1 3 4 2 ...
+#> ..- attr(*, "label")= chr "imp_considerothers"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#1_10: num 2 4 2 3 4 1 1 4 4 1 ...
+#> ..- attr(*, "label")= chr "imp_supportparentsfinance"
+#> ..- attr(*, "format.spss")= chr "F12.0"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:2] 1 4
+#> .. ..- attr(*, "names")= chr [1:2] "not important" "important"
+#> $ moa2#2_1 : num 3 3 2 3 2 3 2 2 3 3 ...
+#> ..- attr(*, "label")= chr "achi_independentdecisions"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_2 : num 2 1 2 2 1 1 2 1 1 2 ...
+#> ..- attr(*, "label")= chr "ach_supportfamily"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_3 : num 3 1 3 2 2 1 2 1 3 3 ...
+#> ..- attr(*, "label")= chr "ach_carechildren"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_4 : num 2 3 2 3 3 3 2 2 2 3 ...
+#> ..- attr(*, "label")= chr "ach_acceptresponsibility"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_5 : num 3 1 1 2 2 3 2 1 3 3 ...
+#> ..- attr(*, "label")= chr "ach_employfulltime"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_6 : num 3 3 3 3 2 3 2 3 2 3 ...
+#> ..- attr(*, "label")= chr "ach_avoiddrunkdriving"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_7 : num 3 3 3 2 2 1 2 1 3 2 ...
+#> ..- attr(*, "label")= chr "ach_parentasequal"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_8 : num 2 3 2 3 2 1 2 2 3 2 ...
+#> ..- attr(*, "label")= chr "ach_emotionalcontrol"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_9 : num 2 3 2 3 3 3 2 2 3 2 ...
+#> ..- attr(*, "label")= chr "ach_considerothers"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
+#> $ moa2#2_10: num 1 1 1 2 1 1 2 1 3 3 ...
+#> ..- attr(*, "label")= chr "ach_supportparentsfinances"
+#> ..- attr(*, "format.spss")= chr "F12.1"
+#> ..- attr(*, "display_width")= int 12
+#> ..- attr(*, "labels")= Named num [1:3] 1 2 3
+#> .. ..- attr(*, "names")= chr [1:3] "no" "somewhat" "yes"
Grahe, J. E., Chalk, H. M., Cramblet Alvarez, L. D., Faas, C., +Hermann, A., McFall, J. P., & Molyneux, K. (2018, January 10). +EAMMi2 Public Data. Retrieved from: https://osf.io/x7mp2/.
+Mostly United States but any English speaker could complete.
+Variable |
+Label |
+
---|---|
moa1#1_1 |
+imp_financialindependence |
+
moa1#1_2 |
+imp_nolongerhom |
+
moa1#1_3 |
+imp_finishededucation |
+
moa1#1_4 |
+imp_married |
+
moa1#1_5 |
+imp_havechild |
+
moa1#1_6 |
+imp_settledcareer |
+
moa1#1_7 |
+imp_avoiddrunk |
+
moa1#1_8 |
+imp_avoiddrugs |
+
moa1#1_9 |
+imp_usecontraception |
+
moa1#1_10 |
+imp_committedlongterm |
+
moa1#2_1 |
+ach_financialindependence |
+
moa1#2_2 |
+ach_nolongerhome |
+
moa1#2_3 |
+ach_finishededucation |
+
moa1#2_4 |
+ach_married |
+
moa1#2_5 |
+ach_havechild |
+
moa1#2_6 |
+ach_settledcareer |
+
moa1#2_7 |
+ach_avoiddrunk |
+
moa1#2_8 |
+ach_avoiddrugs |
+
moa1#2_9 |
+ach_usecontraception |
+
moa1#2_10 |
+ach_committedlongterm |
+
moa2#1_1 |
+imp_indepedentdecisions |
+
moa2#1_2 |
+imp_supportfamily |
+
moa2#1_3 |
+imp_carechildren |
+
moa2#1_4 |
+imp_acceptresponsibility |
+
moa2#1_5 |
+imp_employfulltime |
+
moa2#1_6 |
+imp_avoiddrunkdriving |
+
moa2#1_7 |
+imp_parentasequal |
+
moa2#1_8 |
+imp_emotionalcontrol |
+
moa2#1_9 |
+imp_considerothers |
+
moa2#1_10 |
+imp_supportparentsfinance |
+
moa2#2_1 |
+achi_independentdecisions |
+
moa2#2_2 |
+ach_supportfamily |
+
moa2#2_3 |
+ach_carechildren |
+
moa2#2_4 |
+ach_acceptresponsibility |
+
moa2#2_5 |
+ach_employfulltime |
+
moa2#2_6 |
+ach_avoiddrunkdriving |
+
moa2#2_7 |
+ach_parentasequal |
+
moa2#2_8 |
+ach_emotionalcontrol |
+
moa2#2_9 |
+ach_considerothers |
+
moa2#2_10 |
+ach_supportparentsfinances |
+
IDEA_1 |
+IDEA-manypossibility |
+
IDEA_2 |
+IDEA-exploration |
+
IDEA_3 |
+IDEA-stressed |
+
IDEA_4 |
+IDEA-highpressure |
+
IDEA_5 |
+IDEA-definingself |
+
IDEA_6 |
+IDEA-beliefsvalues |
+
IDEA_7 |
+IDEA-someways |
+
IDEA_8 |
+IDEA-graduallyadult |
+
swb_1 |
+SWB-ideal |
+
swb_2 |
+SWB-excellent |
+
swb_3 |
+SWB-satisfied |
+
swb_4 |
+SWB-important |
+
swb_5 |
+SWB-changenothing |
+
swb_6 |
+SWB-highselfesteem |
+
mindful_1 |
+MIND-emotnotconscious |
+
mindful_2 |
+MIND-breakspill |
+
mindful_3 |
+MIND-difficultfocus |
+
mindful_4 |
+MIND-walknoattention |
+
mindful_5 |
+MIND-notnoticetension |
+
mindful_6 |
+MIND-forgetnames |
+
mindful_7 |
+MIND-runautomatic |
+
mindful_8 |
+MIND-rushactivities |
+
mindful_9 |
+MIND-goalfocus |
+
mindful_10 |
+MIND-jobautomatically |
+
mindful_11 |
+MIND-listensametime |
+
mindful_12 |
+MIND-driveautomatic |
+
mindful_13 |
+MIND-preoccupied |
+
mindful_14 |
+MIND-withoutpayattention |
+
mindful_15 |
+MIND-snackunaware |
+
belong_1 |
+Belong_nobother |
+
belong_2 |
+Belong_avoidrejection |
+
belong_3 |
+Belong_seldomworry |
+
belong_4 |
+Belong_needpeople |
+
belong_5 |
+Belong_othersacceptme |
+
belong_6 |
+Belong_notalone |
+
belong_7 |
+Belong_OKalone |
+
belong_8 |
+Belong_strongNEED |
+
belong_9 |
+Belong_bothernotinplans |
+
belong_10 |
+Belong_feelingseasilyhurt |
+
belnow |
+BELONG-feelIbelong |
+
efficacy_1 |
+EFF-solvetryhard |
+
efficacy_2 |
+EFF-getwhatwant |
+
efficacy_3 |
+EFF-stick2goals |
+
efficacy_4 |
+EFF-dealunexpected |
+
efficacy_5 |
+EFF-resourceful |
+
efficacy_6 |
+EFF-solvenecesseffort |
+
efficacy_7 |
+EFF-remaincalm |
+
efficacy_8 |
+EFF-findseveralsolutions |
+
efficacy_9 |
+EFF-thinkofsolution |
+
efficacy_10 |
+EFF-whatevercomesmyway |
+
support_1 |
+SUP-specialforneed |
+
support_2 |
+SUP-specialjoysorry. |
+
support_3 |
+SUP-familyhelp |
+
support_4 |
+SUP-familyemotionalhelp |
+
support_5 |
+SUP-specialcomfort |
+
support_6 |
+SUP-friendshelp |
+
support_7 |
+SUP-countonfriends |
+
support_8 |
+SUP-talkfamily |
+
support_9 |
+SUP-friendsjoysorrow |
+
support_10 |
+SUP-specialcaresfeelings |
+
support_11 |
+SUP-familyhelpdecisions |
+
support_12 |
+SUP-friendstalkproblems |
+
SocMedia_1 |
+SocMed-avoiddrifting |
+
SocMedia_2 |
+SocMed-friendsplanstonight |
+
SocMedia_3 |
+SocMed-friendsintouch |
+
SocMedia_4 |
+SocMed-friendsupto |
+
SocMedia_5 |
+SocMed-Reconnectwithpeople |
+
SocMedia_6 |
+SocMed-Findoutmore |
+
SocMedia_7 |
+SocMed-someonetoknowbetter |
+
SocMedia_8 |
+SocMed-makenewfriends |
+
SocMedia_9 |
+SocMed-getintouch |
+
SocMedia_10 |
+SocMed-getinformation |
+
SocMedia_11 |
+SocMed-shareinformation |
+
usdream_1 |
+AmDreamImport |
+
usdream_2 |
+AmDreamAchieve |
+
usdream_3 |
+attentionchechshould be1 |
+
transgres_1 |
+trangress-lietoyou |
+
transgres_2 |
+transgress-rumors |
+
transgres_3 |
+transgress-goteven |
+
transgres_4 |
+transgress-degraded |
+
NPI1 |
+NPI1 |
+
NPI2 |
+NPI2 |
+
NPI3 |
+NPI3 |
+
NPI4 |
+NPI4 |
+
NPI5 |
+NPI5 |
+
NPI6 |
+NPI6 |
+
NPI7 |
+NPI7 |
+
NPI8 |
+NPI8 |
+
NPI9 |
+NPI9 |
+
NPI10 |
+NPI10 |
+
NPI11 |
+NPI11 |
+
NPI12 |
+NPI12 |
+
NPI13 |
+NPI13 |
+
exploit_1 |
+EXP-benefitfromothers |
+
exploit_2 |
+EXP-profitfromothers |
+
exploit_3 |
+EXP-usingothers |
+
POQ1 |
+DISID-disabilityinterferes |
+
POQ2 |
+DISID-dontthinkdisabled |
+
POQ3 |
+DISID-lackconfidence |
+
POQ4 |
+DISID-proudtobedisabled |
+
POQ5 |
+DISID-ashamed |
+
POQ6 |
+DISID-notreducedenjoy |
+
POQ7 |
+DISID-limitedfriendships |
+
POQ8 |
+DISID-sourcestrength |
+
POQ9 |
+DISID-accomplishmore |
+
POQ10 |
+DISID-notaproblem |
+
POQ11 |
+DISID-normallife |
+
POQ12 |
+DISID-betterperson |
+
POQ13 |
+DISID-importantpart |
+
POQ14 |
+DISID-proudofdisability |
+
POQ15 |
+DISID-disabilityenriches |
+
physSx_1 |
+Phys_stomach |
+
physSx_2 |
+Phys-back |
+
physSx_3 |
+Phys-appendages |
+
physSx_4 |
+Phys-headaches |
+
physSx_5 |
+Phys-chest |
+
physSx_6 |
+Phys-dizziness |
+
physSx_7 |
+Phys-fainting |
+
physSx_8 |
+Phys-heartpound |
+
physSx_9 |
+Phys-shortness |
+
physSx_10 |
+Phys-constipation |
+
physSx_11 |
+Phys-nausea |
+
physSx_12 |
+Phys-tired |
+
physSx_13 |
+Phys-troublesleeping |
+
stress_1 |
+stress-beenupset |
+
stress_2 |
+stress-unablecontrol |
+
stress_3 |
+stress-nervous |
+
stress_4 |
+stress-confident |
+
stress_5 |
+stress-goingmyway |
+
stress_6 |
+stress-couldnotcope |
+
stress_7 |
+stress-controlirritations |
+
stress_8 |
+stress-ontopofthings |
+
stress_9 |
+stress-beenangered |
+
stress_10 |
+stress-feltdifficulties |
+
Variables in the working file |
+
+
+EAMMi2 <- EAMMi2[complete.cases(EAMMi2),]
+EAMMi2long <- EAMMi2 %>% pivot_longer(cols = everything()) %>%
+ dplyr::rename(item = name, score = value) %>%
+ group_by(item) %>%
+ sample_n(size = 50)
+
+flextable(head(EAMMi2long)) %>% autofit()
item |
+score |
+
---|---|
moa1#1_1 |
+4 |
+
moa1#1_1 |
+3 |
+
moa1#1_1 |
+4 |
+
moa1#1_1 |
+4 |
+
moa1#1_1 |
+4 |
+
moa1#1_1 |
+3 |
+
What the usual standard error for the data that could be considered +for our stopping rule?
+
+SE <- tapply(EAMMi2long$score, EAMMi2long$item, function (x) { sd(x)/sqrt(length(x)) })
+min(SE)
+#> [1] 0.04956958
+quantile(SE, probs = .4)
+#> 40%
+#> 0.09805288
+max(SE)
+#> [1] 0.1781767
+
+cutoff <- quantile(SE, probs = .4)
+
+# we can also use semanticprimer's function
+cutoff_score <- calculate_cutoff(population = EAMMi2long,
+ grouping_items = "item",
+ score = "score",
+ minimum = min(EAMMi2long$score),
+ maximum = max(EAMMi2long$score))
+cutoff_score$cutoff
+#> 40%
+#> 0.09805288
The items have a range of 0.0495696 to 0.1781767. We could use the +40% decile SE = 0.0980529 as our critical value for our stopping rule +given the manuscript results. We could also have a set SE to a specific +item if we do not believe we have representative pilot data in this +example. You should also consider the scale when estimating these values +(i.e., 1-7 scales will have smaller estimates than 1-100 scales).
+To estimate minimum sample size, we should figure out what number of +participants it would take to achieve 80% of the SEs for items below our +critical score of 0.0980529?
+
+# sequence of sample sizes to try
+nsim <- 10 # small for cran
+samplesize_values <- seq(20, 200, 5)
+
+# create a blank table for us to save the values in
+
+sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(EAMMi2long$item)))
+
+# make it a data frame
+sim_table <- as.data.frame(sim_table)
+
+# add a place for sample size values
+sim_table$sample_size <- NA
+
+iterate <- 1
+
+for (p in 1:nsim){
+ # loop over sample sizes
+ for (i in 1:length(samplesize_values)){
+
+ # temp dataframe that samples and summarizes
+ temp <- EAMMi2long %>%
+ group_by(item) %>%
+ sample_n(samplesize_values[i], replace = T) %>%
+ summarize(se = sd(score)/sqrt(length(score)))
+
+ colnames(sim_table)[1:length(unique(EAMMi2long$item))] <- temp$item
+ sim_table[iterate, 1:length(unique(EAMMi2long$item))] <- temp$se
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ iterate <- iterate + 1
+ }
+}
+
+final_sample <-
+ sim_table %>%
+ pivot_longer(cols = -c(sample_size, nsim)) %>%
+ group_by(sample_size, nsim) %>%
+ summarize(percent_below = sum(value <= cutoff)/length(unique(EAMMi2long$item))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample %>% head()) %>% autofit()
sample_size |
+percent_below |
+
---|---|
20 |
+0.1125 |
+
25 |
+0.1425 |
+
30 |
+0.1950 |
+
35 |
+0.2225 |
+
40 |
+0.3050 |
+
45 |
+0.3475 |
+
+final_table <- calculate_correction(
+ proportion_summary = final_sample,
+ pilot_sample_size = EAMMi2long %>% group_by(item) %>%
+ summarize(sample_size = n()) %>% ungroup() %>%
+ summarize(avg_sample = mean(sample_size)) %>% pull(avg_sample),
+ proportion_variability = cutoff_score$prop_var
+ )
+
+flextable(final_table) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
81.00 |
+100 |
+79.10863 |
+
85.50 |
+110 |
+86.31167 |
+
90.25 |
+125 |
+96.91896 |
+
97.25 |
+155 |
+116.99286 |
+
Based on these simulations, we can decide our minimum sample size is +likely close to 79.
+In this example, we could set our maximum sample size for 90% power, +which would equate to 97 participants.
+In any estimate for sample size, you should also consider the +potential for missing data and/or unusable data due to any other +exclusion criteria in your study (i.e., attention checks, speeding, +getting the answer right, etc.). Another important note is that these +estimates are driven by the number of items. Fewer items would require +smaller sample sizes to achieve minimum power. Note: Several redundant +(e.g., reverse coded items) and/or not useful variables (various checks) +were omitted.
+moat_vignette.Rmd
+knitr::opts_chunk$set(echo = TRUE)
+
+# Libraries necessary for this vignette
+library(rio)
+library(flextable)
+library(dplyr)
+#>
+#> Attaching package: 'dplyr'
+#> The following objects are masked from 'package:stats':
+#>
+#> filter, lag
+#> The following objects are masked from 'package:base':
+#>
+#> intersect, setdiff, setequal, union
+library(tidyr)
+library(semanticprimeR)
+set.seed(92747)
+
+# Function for simulation
+item_power <- function(data, # name of data frame
+ dv_col, # name of DV column as a character
+ item_col, # number of items column as a character
+ nsim = 10, # small for cran
+ sample_start = 20,
+ sample_stop = 200,
+ sample_increase = 5,
+ decile = .5){
+
+ DF <- cbind.data.frame(
+ dv = data[ , dv_col],
+ items = data[ , item_col]
+ )
+
+ # just in case
+ colnames(DF) <- c("dv", "items")
+
+ # figure out the "sufficiently narrow" ci value
+ SE <- tapply(DF$dv, DF$items, function (x) { sd(x)/sqrt(length(x)) })
+ cutoff <- quantile(SE, probs = decile)
+
+ # sequence of sample sizes to try
+ samplesize_values <- seq(sample_start, sample_stop, sample_increase)
+
+ # create a blank table for us to save the values in
+ sim_table <- matrix(NA,
+ nrow = length(samplesize_values),
+ ncol = length(unique(DF$items)))
+
+ # make it a data frame
+ sim_table <- as.data.frame(sim_table)
+
+ # add a place for sample size values
+ sim_table$sample_size <- NA
+
+ iterate <- 1
+ for (p in 1:nsim){
+ # loop over sample sizes
+ for (i in 1:length(samplesize_values)){
+
+ # temp that samples and summarizes
+ temp <- DF %>%
+ group_by(items) %>%
+ sample_n(samplesize_values[i], replace = T) %>%
+ summarize(se = sd(dv)/sqrt(length(dv)))
+
+ # dv on items
+ colnames(sim_table)[1:length(unique(DF$items))] <- temp$items
+ sim_table[iterate, 1:length(unique(DF$items))] <- temp$se
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ }
+ }
+
+ # figure out cut off
+ final_sample <- sim_table %>%
+ pivot_longer(cols = -c(sample_size, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ group_by(sample_size, nsim) %>%
+ summarize(percent_below = sum(se <= cutoff)/length(unique(DF$items))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+
+ return(list(
+ SE = SE,
+ cutoff = cutoff,
+ DF = DF,
+ sim_table = sim_table,
+ final_sample = final_sample
+ ))
+
+}
Seeing Is Believing: How Media Type Effects Truth Judgements
+Data provided by: Gianni Ribeiro
+People have been duly concerned about how fake news influences the +minds of the populous since the rise of propaganda in World War One +(Lasswell, 1927). Experts are increasingly worried about the effects of +false information spreading over the medium of video. Members of the +deep trust alliance, a global network of scholars researching deepfakes +and doctored videos, state that ‘a fundamental erosion of trust is +already underway’ (Harrison, 2020). Newman et al. (2015) discovered that +the media type through which information is presented does indeed affect +how true the information feels. Newman speculated that this truthiness +effect could be because images provide participants with more +information than text alone, thus making the source feel more +informationally rich.
+In this experiment, our aim is to test the generalizability of +Newman’s truthiness effect in two ways: first, to see if it extends to +other media types in addition to images, and second, to test if it +applies to other domains. In this study, we will present individuals +with true and false claims presented through three different media +types: (1) text, (2) text alongside a photo, and (3) text alongside a +video. This is a direct replication of Newman’s experiment, just with +the addition of the video condition. Similarly, participants will also +be asked to make truth judgements about trivial claims and claims about +COVID-19, to see if the truthiness effect extends to other domains +besides trivia.
+In this within-subjects design, participants will be presented with +true and false claims about trivia and COVID-19 in counterbalanced +order. These claims will be randomly assigned to appear either as text +alone, text alongside an image, or text alongside a video. Participants +will be asked to rate how true they believe each claim is.
+Participants were largely sourced from the first-year participant +pool at The University of Queensland. Participation was completely +voluntary, and participants can choose to withdraw at any time.
+Thirty matched trivia claims were generated directly from Newman’s +materials. These claims were selected and a true and false version of +each claim was created. Newman’s original claims are available at the +following link: https://data.mendeley.com/datasets/r68dcdjrpc/1
+The second set of materials comprising of matched true and false +claims was generated using information resources from the World Health +Organisation, and various conspiracy websites. These claims were then +fact-checked by Kirsty Short, an epidemiologist and senior lecturer in +the School of Chemistry and Molecular Sciences at The University of +Queensland.
+The claims were also pilot tested to ensure that none of them +performed at floor or ceiling. This pilot test consisted of 56 +participants and subsequently four claims were dropped. The data from +this pilot test was also used to accurately perform a power analysis. +After generating the means of the pilot test, we found that to acquire a +power of 0.8 or greater, there must be a mean difference of 0.4 between +each media type. This mean difference is quite conservative, since we +plan to measure truth ratings on a six-point scale and is easily +achievable with 100 participants.
+The videos were largely sourced from the stock image website Envato +Elements and Screenflow’s Royalty Free Stock Media Library.
+Data can be found here: https://osf.io/zu9pg/
+
+DF <- import("data/moat_data.csv.zip")
+
+str(DF)
+#> 'data.frame': 5040 obs. of 11 variables:
+#> $ id : int 1 1 1 1 1 1 1 1 1 1 ...
+#> $ domain : chr "covid" "covid" "covid" "covid" ...
+#> $ gender : chr "Female" "Female" "Female" "Female" ...
+#> $ age : int 22 22 22 22 22 22 22 22 22 22 ...
+#> $ filename : chr "1_kaitlin_exp_v2b_97832952686.txt" "1_kaitlin_exp_v2b_97832952686.txt" "1_kaitlin_exp_v2b_97832952686.txt" "1_kaitlin_exp_v2b_97832952686.txt" ...
+#> $ medium : chr "claim" "claim" "claim" "claim" ...
+#> $ question_type : chr "drinking" "herd" "hydroxychloroquine" "steam" ...
+#> $ truth : logi TRUE TRUE TRUE TRUE TRUE FALSE ...
+#> $ claim : chr "COVID-19 cannot be contracted through drinking water." "Herd immunity against COVID-19 cannot be achieved by letting the virus spread through the population." "Studies show hydroxychloroquine does not have clinical benefits in treating COVID-19." "Steam inhalation cannot help cure COVID-19." ...
+#> $ filename_other: chr NA NA NA NA ...
+#> $ rating : int 1 5 6 6 6 1 1 1 5 1 ...
Moat, K., Tangen, J., & Newman, E. (2021). Seeing Is Believing: +How Media Type Effects Truth Judgements.
+Open access with reference to original paper +(Attribution-NonCommercial-ShareAlike CC BY-NC-SA)
+
+metadata <- tibble::tribble(
+ ~Variable.Name, ~Variable.Description, ~`Type.(numeric,.character,.logical,.etc.)`,
+ "Id", "Participant ID", "numeric",
+ "Domain", "Whether the trial is a claim about COVID ('covid') or TRIVIA ('trivia)", "character",
+ "Medium", "Whether the trial appears as text alone ('claim'), text alongside an image ('photo'), or text alongside a video ('video')", "character",
+ "Trial_type", "Whether the trial presents a claim that is TRUE ('target') or FALSE ('distractor')", "character",
+ "Rating", "Paritcipant’s truth rating of the claim ranging from 1 (definitely false) to 6 (definitely tue)", "numeric"
+ )
+
+flextable(metadata) %>% autofit()
Variable.Name |
+Variable.Description |
+Type.(numeric,.character,.logical,.etc.) |
+
---|---|---|
Id |
+Participant ID |
+numeric |
+
Domain |
+Whether the trial is a claim about COVID ('covid') or TRIVIA ('trivia) |
+character |
+
Medium |
+Whether the trial appears as text alone ('claim'), text alongside an image ('photo'), or text alongside a video ('video') |
+character |
+
Trial_type |
+Whether the trial presents a claim that is TRUE ('target') or FALSE ('distractor') |
+character |
+
Rating |
+Paritcipant’s truth rating of the claim ranging from 1 (definitely false) to 6 (definitely tue) |
+numeric |
+
+# Function for simulation
+var1 <- item_power(data = DF, # name of data frame
+ dv_col = "rating", # name of DV column as a character
+ item_col = "question_type", # number of items column as a character
+ nsim = 10,
+ sample_start = 20,
+ sample_stop = 300,
+ sample_increase = 5,
+ decile = .4)
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
What is the usual standard error for the data that could be +considered for our stopping rule?
+
+var1$SE
+#> afghan antibiotics bacteria blindness
+#> 0.1733704 0.2192951 0.2550577 0.1787244
+#> breastfeeding chemical children cocaine
+#> 0.1726203 0.2000046 0.2224147 0.2427085
+#> colorvision corn couscous curling
+#> 0.1284245 0.1064393 0.1337200 0.1500109
+#> dartboards denmark dna drinking
+#> 0.1322563 0.1802086 0.2273651 0.1732374
+#> elderly fishing forest foxhunting
+#> 0.2576951 0.1464690 0.1349403 0.1446861
+#> grapes herd hiv houseflies
+#> 0.1573876 0.1877676 0.2135034 0.1907986
+#> hydroxychloroquine infertility lawnbowls lime
+#> 0.1495606 0.2049964 0.1147161 0.1465564
+#> longbow marathon microwave mintonette
+#> 0.1667281 0.1392743 0.1959902 0.1066397
+#> mosquitoes mountains mouthwash nile
+#> 0.1894154 0.0912085 0.2245578 0.1532541
+#> otter oxygen oysters penicillin
+#> 0.1280050 0.2093482 0.1173648 0.1584095
+#> poland prisoners rate rigor
+#> 0.1326624 0.1970849 0.2147593 0.2238266
+#> saline smell snake snowboarding
+#> 0.2132273 0.2432145 0.1899915 0.1359677
+#> steam temperature triathalon twice
+#> 0.2243410 0.2215570 0.1851070 0.2292498
+#> urchin uv vesuvius vitamin-c
+#> 0.1399591 0.2161739 0.1433344 0.2127463
+#> vitamin-d water waterfall zulu
+#> 0.2251918 0.1817230 0.1411920 0.1264954
+var1$cutoff
+#> 40%
+#> 0.1580007
+
+cutoff <- var1$cutoff
+
+# we can also use semanticprimer's function
+cutoff_score <- calculate_cutoff(population = DF,
+ grouping_items = "question_type",
+ score = "rating",
+ minimum = min(DF$rating),
+ maximum = max(DF$rating))
+cutoff_score$cutoff
+#> 40%
+#> 0.1580007
Using our 40% decile as a guide, we find that 0.158 is our target +standard error for an accurately measured item.
+To estimate minimum sample size, we should figure out what number of +participants it would take to achieve 80%, 85%, 90%, and 95% of the SEs +for items below our critical score of 0.158?
+How large does the sample have to be for 80% of the items to be below +our stopping SE rule?
+ +sample_size |
+percent_below |
+
---|---|
300 |
+1 |
+
+
+final_table <- calculate_correction(
+ proportion_summary = var1$final_sample,
+ pilot_sample_size = DF %>% group_by(question_type) %>%
+ summarize(sample_size = n()) %>% ungroup() %>%
+ summarize(avg_sample = mean(sample_size)) %>% pull(avg_sample),
+ proportion_variability = cutoff_score$prop_var
+ )
+
+flextable(final_table) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
100 |
+300 |
+173.0834 |
+
100 |
+300 |
+173.0834 |
+
100 |
+300 |
+173.0834 |
+
100 |
+300 |
+173.0834 |
+
Based on these simulations, we can decide our minimum sample size is +likely close to 173.
+montefinese_vignette.Rmd
Online search trends and word-related emotional response during +COVID-19 lockdown in Italy
+Data provided by: Maria Montefinese
+The strong and long lockdown adopted by the Italian government to +limit the spread of the COVID-19 represents the first threat-related +mass isolation in history that scientists can study in depth to +understand the emotional response of individuals to a pandemic. +Perception of a pandemic threat through invasive media communication, +such as that related to COVID-19, can induce fear-related emotions (Van +Bavel et al., 2020). The dimension theory of emotions (Osgood & +Suci, 1955) assumes that emotive space is defined along three +dimensions: valence (indicating the way an individual judges a stimulus; +from unpleasant to pleasant), arousal (indicating the degree of +activation an individual feels towards a stimulus; from calm to excited) +and dominance (indicating the degree of control an individual feels over +a given stimulus; from out of control to in control). Fear is +characterized as a negatively valenced emotion, accompanied by a high +level of arousal (Witte, 1992; Witte, 1998) and a low dominance +(Stevenson, Mikel & James, 2007). This is generally in line with +previous results showing that participants judged stimuli related to the +most feared medical conditions as the most negative, the most +anxiety-provoking, and the least controllable (Warriner, Kuperman & +Brysbaert, 2013). Fear is also characterized by extreme levels of +emotional avoidance of specific stimuli (Perin et al., 2015) and may be +considered a unidirectional precursor to psychopathological responses +within the current context (Ahorsu et al., 2020). dealing with fear in a +pandemic situation could be easier for some people than others. Indeed, +individual differences have been associated with behavioral responses to +pandemic status (Carvalho Pianowski & Gonçalves, 2020).
+To mitigate the effects of the COVID-19 on the mental health of +individuals, it is imperative to evaluate their emotional response to +this emergency. The internet searches are a direct tool to address this +problem. In fact, COVID-19 has been reported to affect the content that +people explore online (Effenberger et al., 2020), and online media and +platforms offer essential channels where people express their feelings +and emotions and seek health-related information (Kalichman et al., +2003; Reeves, 2001). In particular, Google Trends is an available data +source of real-time internet search patterns, which has been shown to be +a valid indicator of people’s desires and intentions (Payne, +Brown-Iannuzzi & Hannay, 2017; Pelham et al., 2018). Therefore, the +amount of searches related to COVID-19 on the internet revealed by +Google Trends are an indicator of how people feel about concepts related +to the COVID-19 pandemic. A change in online search trends reflects a +change in participants’ interests and attitudes towards a specific +topic. Based on the topic, the context (that is, the reasons for this +change), and this mutated interest per se, it is possible to predict +people’s behavior and affective response to the topic in question. In +this study, our aim was to understand how emotional reaction and online +search behavior have changed in response to the COVID-19 lockdown in the +Italian population.
+Data were collected in the period from 4 May to 17 May 2020, the last +day of complete lockdown in Italy, from 71 native adult Italian speakers +(56 females and 13 males; mean age (SD) = 26.2 (7.9) years; mean +education (SD) = 15.3 (3.2) years). There were no other specific +eligibility criteria. An online survey was conducted using Google Forms +to collect affective ratings during the lockdown caused by the COVID-19 +epidemic in Italy. In particular, we asked participants to complete the +Positive and Negative Affect Schedule (PANAS, Terraciano, McCrae & +Costa, 2003) and Fear of COVID-19 Scale (FCV-19S, Ahorsu et al., 2020) +and judged valence, arousal, and dominance (on a 9-point self-assessment +manikin, Montefinese et al., 2014) of words related or unrelated to +COVID-19, as identified by Google search trends. The word stimuli +consisted of 3 groups of 20 words each. The first group (REL+) consisted +of the words showing the largest positive relation between their search +trends and the search trend for COVID-related terms. On the contrary, +the second group (REL-) consisted of the words showing the largest +negative relation between their search trends and the search trend for +COVID-related terms. In other words, the COVID-19 epidemic in Italy and +the consequent increase in interest in terms related to COVID was +related to a similar increase in interest for the REL+ words and a +decrease in interest for the REL- words. The third group (UNREL) +consisted of the words for which the search trend was unrelated to the +search trend for the COVID-related terms.
+
+DF <- import("data/montefinese_data.csv")
+
+names(DF) <- make.names(names(DF),unique = TRUE)
+
+names(DF)[names(DF) == 'ITEM..ITA.'] <- "item"
+
+DF <- DF %>%
+ filter(StimType != "") %>%
+ filter(Measure == "Valence") %>% # only look at valence score
+ arrange(item) %>% #orders the rows of the data by the target_name column
+ group_by(item) %>% #group by the target name
+ transform(items = as.numeric(factor(item)))%>% #transform target name into a item
+ select(items, item, everything()
+ ) #select all variables from items and target_name
+
+head(DF)
+#> items item ssID Gender Age Education Measure StimType Response
+#> 1 1 affogare 1 F 36 21 Valence UNREL 2
+#> 2 1 affogare 2 M 40 21 Valence UNREL 2
+#> 3 1 affogare 3 F 29 21 Valence UNREL 1
+#> 4 1 affogare 4 M 39 13 Valence UNREL 1
+#> 5 1 affogare 5 F 27 16 Valence UNREL 1
+#> 6 1 affogare 6 M 33 18 Valence UNREL 1
Montefinese M, Ambrosini E, Angrilli A. 2021. Online search trends +and word-related emotional response during COVID-19 lockdown in Italy: a +cross-sectional online study. PeerJ 9:e11858 https://doi.org/10.7717/peerj.11858
+Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
ssID |
+Participant code |
+Numeric |
+
Gender |
+Participants’ gender |
+Character |
+
Age |
+Participants’ age |
+Numeric |
+
Education |
+Participants’ years of education |
+Numeric |
+
Measure |
+Questionnaires and ratings (PANAS, COVID-19 fear, valence, arousal, dominance) |
+Character |
+
ITEM (ITA) |
+Test items and word stimuli |
+Character |
+
Stim Type |
+Word condition (REL+, REL-, UNREL) |
+Character |
+
Response |
+Participants’ scores to the questionnaires and ratings |
+Numeric |
+
In this dataset, there are REL+ and REL- variables. In the REL+ +condition, the words show the largest positive relation between their +search trends and the search trend for the COVID-related terms. In the +REL- condition, the words showed the largest negative relation between +their search trends and the search trends for the COVID-related terms. +The third group (UNREL) consisted in the words for which the search +trend was unrelated to the search trend for the COVID-related terms.
+What the usual standard error for the data that could be considered +for our stopping rule using the 40% decile? Given potential differences +in conditions, we subset the data to each condition to estimate +separately.
+
+### create subset for REL+
+DF_RELpos <- subset(DF, StimType == "REL+")
+
+### create subset for REL-
+DF_RELneg <- subset(DF, StimType == "REL-")
+
+### create subset for UNREL
+DF_UNREL <- subset(DF, StimType == "UNREL")
+
+# individual SEs for REL+ condition
+cutoff_relpos <- calculate_cutoff(population = DF_RELpos,
+ grouping_items = "item",
+ score = "Response",
+ minimum = min(DF_RELpos$Response),
+ maximum = max(DF_RELpos$Response))
+
+SE1 <- tapply(DF_RELpos$Response, DF_RELpos$item, function (x) { sd(x)/sqrt(length(x)) })
+SE1
+#> burro casa cioccolato computer corona famiglia febbre
+#> 0.1557687 0.1766870 0.1537913 0.1998951 0.2176631 0.1481984 0.1481984
+#> lavare libertà mondo muffin notizie peste salute
+#> 0.1860313 0.1916883 0.2017999 0.1846399 0.1636991 0.1621336 0.1569829
+#> salvare sole tempo termometro torta vaiolo
+#> 0.1431403 0.1483131 0.1952478 0.1441268 0.1393278 0.1585099
+cutoff_relpos$cutoff
+#> 40%
+#> 0.1564972
+
+# individual SEs for REL- condition
+cutoff_relneg <- calculate_cutoff(population = DF_RELneg,
+ grouping_items = "item",
+ score = "Response",
+ minimum = min(DF_RELneg$Response),
+ maximum = max(DF_RELneg$Response))
+
+SE2 <- tapply(DF_RELneg$Response, DF_RELneg$item, function (x) { sd(x)/sqrt(length(x)) })
+SE2
+#> autobus costa dormire giacca hotel mangiare matrimonio
+#> 0.17715152 0.18453238 0.16521553 0.15728951 0.17384120 0.14909443 0.20475544
+#> motore palazzo pantalone piazza profumo ristorante sposa
+#> 0.15085177 0.17073238 0.17481656 0.20640961 0.16138025 0.20118108 0.21208427
+#> tram tumore uomo vacanza viaggio villaggio
+#> 0.16872880 0.07001896 0.16715951 0.13096103 0.18557374 0.16663313
+cutoff_relneg$cutoff
+#> 40%
+#> 0.166949
+
+# individual SEs for UNREL condition
+cutoff_unrel <- calculate_cutoff(population = DF_UNREL,
+ grouping_items = "item",
+ score = "Response",
+ minimum = min(DF_UNREL$Response),
+ maximum = max(DF_UNREL$Response))
+
+SE3 <- tapply(DF_UNREL$Response, DF_UNREL$item, function (x) { sd(x)/sqrt(length(x)) })
+SE3
+#> affogare baco cannone cappio corridore dipendente
+#> 0.1312204 0.1622734 0.2131240 0.1750919 0.1781088 0.1796297
+#> disturbare fetore firmamento funerale gusto ladro
+#> 0.1535516 0.1507766 0.1973986 0.1121214 0.1526631 0.1473738
+#> malevolenza mestolo nettare oceano offendersi orgasmo
+#> 0.1379994 0.1218824 0.1755445 0.1511708 0.1652155 0.1458857
+#> perfezione tradire
+#> 0.2210864 0.1158751
+cutoff_unrel$cutoff
+#> 40%
+#> 0.1510131
+# sequence of sample sizes to try
+nsim <- 10 # small for cran
+samplesize_values <- seq(25, 300, 5)
+
+# create a blank table for us to save the values in positive ----
+sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(DF_RELpos$item)))
+# make it a data frame
+sim_table <- as.data.frame(sim_table)
+
+# add a place for sample size values
+sim_table$sample_size <- NA
+sim_table$var <- "Response"
+
+# make a second table for negative -----
+sim_table2 <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(DF_RELneg$item)))
+
+# make it a data frame
+sim_table2 <- as.data.frame(sim_table2)
+
+# add a place for sample size values
+sim_table2$sample_size <- NA
+sim_table2$var <- "Response"
+
+# make a second table for unrelated -----
+sim_table3 <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(DF_UNREL$item)))
+
+# make it a data frame
+sim_table3 <- as.data.frame(sim_table3)
+
+# add a place for sample size values
+sim_table3$sample_size <- NA
+sim_table3$var <- "Response"
+
+iterate <- 1
+
+for (p in 1:nsim){
+
+ # loop over sample size
+ for (i in 1:length(samplesize_values)){
+
+ # related positive temp variables ----
+ temp_RELpos <- DF_RELpos %>%
+ dplyr::group_by(item) %>%
+ dplyr::sample_n(samplesize_values[i], replace = T) %>%
+ dplyr::summarize(se1 = sd(Response)/sqrt(length(Response)))
+
+ # put in table
+ colnames(sim_table)[1:length(unique(DF_RELpos$item))] <- temp_RELpos$item
+ sim_table[iterate, 1:length(unique(DF_RELpos$item))] <- temp_RELpos$se1
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ # related negative temp variables ----
+ temp_RELneg <-DF_RELneg %>%
+ dplyr::group_by(item) %>%
+ dplyr::sample_n(samplesize_values[i], replace = T) %>%
+ dplyr::summarize(se2 = sd(Response)/sqrt(length(Response)))
+
+ # put in table
+ colnames(sim_table2)[1:length(unique(DF_RELneg$item))] <- temp_RELneg$item
+ sim_table2[iterate, 1:length(unique(DF_RELneg$item))] <- temp_RELneg$se2
+ sim_table2[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table2[iterate, "nsim"] <- p
+
+ # unrelated temp variables ----
+ temp_UNREL <-DF_UNREL %>%
+ dplyr::group_by(item) %>%
+ dplyr::sample_n(samplesize_values[i], replace = T) %>%
+ dplyr::summarize(se3 = sd(Response)/sqrt(length(Response)))
+
+ # put in table
+ colnames(sim_table3)[1:length(unique(DF_UNREL$item))] <- temp_UNREL$item
+ sim_table3[iterate, 1:length(unique(DF_UNREL$item))] <- temp_UNREL$se3
+ sim_table3[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table3[iterate, "nsim"] <- p
+
+ iterate <- iterate + 1
+
+ }
+}
Suggestions for REL+ Condition:
+
+# multiply by correction
+cutoff <- quantile(SE1, probs = .4)
+
+final_sample <-
+ sim_table %>%
+ pivot_longer(cols = -c(sample_size, var, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ dplyr::group_by(sample_size, var, nsim) %>%
+ dplyr::summarize(percent_below = sum(se <= cutoff)/length(unique(DF_RELpos$item))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size, var) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size', 'var'. You can override
+#> using the `.groups` argument.
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample %>% head()) %>% autofit()
sample_size |
+var |
+percent_below |
+
---|---|---|
35 |
+Response |
+0.000 |
+
25 |
+Response |
+0.005 |
+
30 |
+Response |
+0.010 |
+
40 |
+Response |
+0.015 |
+
50 |
+Response |
+0.050 |
+
45 |
+Response |
+0.055 |
+
+final_table_pos <- calculate_correction(
+ proportion_summary = final_sample,
+ pilot_sample_size = length(unique(DF_RELpos$ssID)),
+ proportion_variability = cutoff_relpos$prop_var
+ )
+
+flextable(final_table_pos) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
81.0 |
+105 |
+70.33792 |
+
88.5 |
+115 |
+77.34137 |
+
91.0 |
+125 |
+84.39584 |
+
95.5 |
+130 |
+88.08384 |
+
Suggestions for REL- Condition:
+
+cutoff <- quantile(SE2, probs = .4)
+
+final_sample2 <-
+ sim_table2 %>%
+ pivot_longer(cols = -c(sample_size, var, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ dplyr::group_by(sample_size, var, nsim) %>%
+ dplyr::summarize(percent_below = sum(se <= cutoff)/length(unique(DF_RELneg$item))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size, var) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size', 'var'. You can override
+#> using the `.groups` argument.
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample2 %>% head()) %>% autofit()
sample_size |
+var |
+percent_below |
+
---|---|---|
25 |
+Response |
+0.065 |
+
30 |
+Response |
+0.065 |
+
35 |
+Response |
+0.075 |
+
45 |
+Response |
+0.105 |
+
40 |
+Response |
+0.110 |
+
50 |
+Response |
+0.130 |
+
+final_table_neg <- calculate_correction(
+ proportion_summary = final_sample2,
+ pilot_sample_size = length(unique(DF_RELneg$ssID)),
+ proportion_variability = cutoff_relneg$prop_var
+ )
+
+flextable(final_table_neg) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
82.5 |
+95 |
+62.03046 |
+
86.0 |
+105 |
+69.52984 |
+
90.0 |
+110 |
+73.23150 |
+
98.0 |
+120 |
+80.94565 |
+
Suggestions for UNREL Condition:
+
+cutoff <- quantile(SE3, probs = .4)
+
+final_sample3 <-
+ sim_table3 %>%
+ pivot_longer(cols = -c(sample_size, var, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ dplyr::group_by(sample_size, var, nsim) %>%
+ dplyr::summarize(percent_below = sum(se <= cutoff)/length(unique(DF_UNREL$item))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size, var) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size', 'var'. You can override
+#> using the `.groups` argument.
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample3 %>% head()) %>% autofit()
sample_size |
+var |
+percent_below |
+
---|---|---|
25 |
+Response |
+0.030 |
+
35 |
+Response |
+0.055 |
+
30 |
+Response |
+0.060 |
+
40 |
+Response |
+0.120 |
+
45 |
+Response |
+0.125 |
+
50 |
+Response |
+0.200 |
+
+final_table_unrel <- calculate_correction(
+ proportion_summary = final_sample3,
+ pilot_sample_size = length(unique(DF_UNREL$ssID)),
+ proportion_variability = cutoff_unrel$prop_var
+ )
+
+flextable(final_table_unrel) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
81.0 |
+105 |
+69.89133 |
+
86.5 |
+115 |
+76.90026 |
+
90.0 |
+125 |
+83.91580 |
+
96.5 |
+150 |
+100.77698 |
+
Based on these simulations, we can decide our minimum sample size by +examining all three potential scores at 80% of items below the +criterion, \(n_{positive}\) = 70, \(n_{negative}\) = 62, or \(n_{unrelated}\) = 70. These scores are all +very similar, and we should select the largest one.
+roer_vignette.Rmd
The survival processing effect
+Data provided by: Röer, Bell & Buchner (2013)
+The data come from a conceptual replication study on the survival +processing effect. The survival processing effect refers to the finding +that rating words according to their relevance in a survival-related +scenario leads to better retention than processing words in a number of +other fictional scenarios. Participants were randomly assigned to one of +the rating scenarios (survival, afterlife, moving). The to-be-rated +words were presented individually in a random order on the computer +screen. Each word remained on the screen for five seconds. Participants +rated the words by clicking on a 5-point scale that ranged from +completely useless (1) to very useful (5), which was displayed right +below the word.
+Participants were students at Heinrich Heine University Düsseldorf, +Germany that were paid for participating or received course credit. +Their ages ranged from 18 to 55 years. The words to-be-rated consisted +of 30 typical members of 30 categories drawn from the updated Battig and +Montague norms (Van Overschelde, Rawson, & Dunlosky, 2004).
+Data included within this vignette. We drop the scenario column +because the standard deviation and mean of item ratings across the +scenarios were identical. We also add a participant column to keep this +script similar to other ones.
+
+DF <- import("data/roer_data.xlsx")
+drops <- c("Scenario")
+DF <- DF[ , !(names(DF) %in% drops)]
+DF <- cbind(Participant_Number = 1:nrow(DF) , DF)
+
+str(DF)
+#> 'data.frame': 218 obs. of 31 variables:
+#> $ Participant_Number: int 1 2 3 4 5 6 7 8 9 10 ...
+#> $ Item_1 : num 1 3 3 2 3 3 3 1 4 1 ...
+#> $ Item_2 : num 2 5 4 1 4 5 4 3 5 3 ...
+#> $ Item_3 : num 5 3 1 5 2 2 5 2 3 4 ...
+#> $ Item_4 : num 1 4 2 0 3 2 2 1 4 2 ...
+#> $ Item_5 : num 3 5 1 5 1 3 5 2 0 4 ...
+#> $ Item_6 : num 1 3 2 1 4 1 5 2 2 1 ...
+#> $ Item_7 : num 4 2 4 1 4 4 4 1 5 3 ...
+#> $ Item_8 : num 3 5 5 4 4 2 5 5 2 3 ...
+#> $ Item_9 : num 1 5 4 5 4 2 4 1 3 3 ...
+#> $ Item_10 : num 0 5 5 5 0 5 5 4 0 4 ...
+#> $ Item_11 : num 3 5 4 5 3 1 5 5 2 1 ...
+#> $ Item_12 : num 3 5 5 5 5 1 3 5 1 1 ...
+#> $ Item_13 : num 1 1 3 1 2 1 1 4 1 1 ...
+#> $ Item_14 : num 1 2 1 4 1 2 2 1 2 5 ...
+#> $ Item_15 : num 1 4 2 1 5 1 3 2 2 1 ...
+#> $ Item_16 : num 4 5 4 2 3 3 4 3 3 4 ...
+#> $ Item_17 : num 3 5 0 1 3 1 4 3 3 3 ...
+#> $ Item_18 : num 2 5 4 1 5 5 5 2 5 4 ...
+#> $ Item_19 : num 2 1 1 4 1 3 4 2 4 3 ...
+#> $ Item_20 : num 5 3 4 5 2 4 5 3 3 2 ...
+#> $ Item_21 : num 3 0 4 5 3 4 1 1 4 2 ...
+#> $ Item_22 : num 3 5 1 1 2 3 5 4 3 4 ...
+#> $ Item_23 : num 1 1 2 1 1 4 3 1 5 2 ...
+#> $ Item_24 : num 1 4 4 2 3 2 4 5 2 0 ...
+#> $ Item_25 : num 2 5 5 3 5 1 5 4 3 4 ...
+#> $ Item_26 : num 1 4 3 1 3 1 2 1 2 3 ...
+#> $ Item_27 : num 1 5 1 1 5 2 4 4 3 4 ...
+#> $ Item_28 : num 1 5 1 1 4 2 3 5 2 3 ...
+#> $ Item_29 : num 1 5 2 3 2 1 4 3 1 3 ...
+#> $ Item_30 : num 3 5 4 1 1 2 5 4 4 5 ...
Röer, J. P., Bell, R., & Buchner, A. (2013). Is the +survival-processing memory advantage due to richness of encoding? +Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, +1294-1302.
+Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
Items |
+Item ratings for item_1 to item_30 |
+Numeric |
+
Scenario |
+Categorical scenarios- 1,2,3 |
+Numeric |
+
+DF_long <- pivot_longer(DF, cols = -c(Participant_Number)) %>%
+ dplyr:: rename(item = name, score = value)
+
+flextable(head(DF_long)) %>% autofit()
Participant_Number |
+item |
+score |
+
---|---|---|
1 |
+Item_1 |
+1 |
+
1 |
+Item_2 |
+2 |
+
1 |
+Item_3 |
+5 |
+
1 |
+Item_4 |
+1 |
+
1 |
+Item_5 |
+3 |
+
1 |
+Item_6 |
+1 |
+
What is the usual standard error for the data that could be +considered for our stopping rule using the 40% decile?
+
+# individual SEs
+SE <- tapply(DF_long$score, DF_long$item, function (x) { sd(x)/sqrt(length(x)) })
+SE
+#> Item_1 Item_10 Item_11 Item_12 Item_13 Item_14 Item_15
+#> 0.08860091 0.09743922 0.10419876 0.10725788 0.07167329 0.11502337 0.09746261
+#> Item_16 Item_17 Item_18 Item_19 Item_2 Item_20 Item_21
+#> 0.09425595 0.08981453 0.08968434 0.09817041 0.10421504 0.10768420 0.10195369
+#> Item_22 Item_23 Item_24 Item_25 Item_26 Item_27 Item_28
+#> 0.09891678 0.11403914 0.08751293 0.10279479 0.08015419 0.09612067 0.09697587
+#> Item_29 Item_3 Item_30 Item_4 Item_5 Item_6 Item_7
+#> 0.08758437 0.10963101 0.11319421 0.07944272 0.11514133 0.08852481 0.10163361
+#> Item_8 Item_9
+#> 0.09615698 0.09118122
+
+cutoff <- quantile(SE, probs = .40)
+cutoff
+#> 40%
+#> 0.09614246
+
+# we could also use the cutoff score function in semanticprimeR
+cutoff_score <- calculate_cutoff(population = DF_long,
+ grouping_items = "item",
+ score = "score",
+ minimum = min(DF_long$score),
+ maximum = max(DF_long$score))
+
+cutoff_score$cutoff
+#> 40%
+#> 0.09614246
Using our 40% decile as a guide, we find that 0.096 is our target +standard error for an accurately measured item.
+To estimate the minimum sample size, we should figure out what number +of participants it would take to achieve 80%, 85%, 90%, and 95% of the +SEs for items below our critical score of 0.096.
+
+# sequence of sample sizes to try
+nsim <- 10 # small for cran
+samplesize_values <- seq(20, 500, 5)
+
+# create a blank table for us to save the values in
+sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(DF_long$item)))
+
+# make it a data frame
+sim_table <- as.data.frame(sim_table)
+
+# add a place for sample size values
+sim_table$sample_size <- NA
+
+iterate <- 1
+for (p in 1:nsim){
+ # loop over sample sizes
+ for (i in 1:length(samplesize_values)){
+
+ # temp dataframe that samples and summarizes
+ temp <- DF_long %>%
+ group_by(item) %>%
+ sample_n(samplesize_values[i], replace = T) %>%
+ summarize(se = sd(score)/sqrt(length(score)))
+
+ colnames(sim_table)[1:length(unique(DF_long$item))] <- temp$item
+ sim_table[iterate, 1:length(unique(DF_long$item))] <- temp$se
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ iterate <- iterate + 1
+ }
+}
+
+final_sample <-
+ sim_table %>%
+ pivot_longer(cols = -c(sample_size, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ group_by(sample_size, nsim) %>%
+ summarize(percent_below = sum(se <= cutoff)/length(unique(DF_long$item))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample %>% head()) %>% autofit()
sample_size |
+percent_below |
+
---|---|
20 |
+0 |
+
25 |
+0 |
+
30 |
+0 |
+
35 |
+0 |
+
40 |
+0 |
+
45 |
+0 |
+
+final_table <- calculate_correction(
+ proportion_summary = final_sample,
+ pilot_sample_size = DF_long %>% group_by(item) %>%
+ summarize(sample_size = n()) %>% ungroup() %>%
+ summarize(avg_sample = mean(sample_size)) %>% pull(avg_sample),
+ proportion_variability = cutoff_score$prop_var
+ )
+
+flextable(final_table) %>%
+ autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
80.33333 |
+270 |
+60.18641 |
+
85.00000 |
+285 |
+67.54308 |
+
92.33333 |
+300 |
+75.18211 |
+
95.33333 |
+315 |
+82.85322 |
+
Based on these simulations, we can decide our minimum sample size is +likely close to 60.
+suchow_vignette.Rmd
Deep models of superficial trait inferences
+Data provided by: Jordan W. Suchow
+The diversity of human faces and the contexts in which they appear +gives rise to an expansive stimulus space over which people infer +psychological traits (e.g., trustworthiness or alertness) and other +attributes (e.g., age or adiposity). Machine learning methods, in +particular deep neural networks, provide expressive feature +representations of face stimuli, but the correspondence between these +representations and various human attribute inferences is difficult to +determine because the former are high-dimensional vectors produced via +black box optimization algorithms. In this paper, we combined deep +generative image models with over 1 million judgments to model +inferences of more than 30 attributes over a comprehensive latent face +space. The predictive accuracy of the model approached human interrater +reliability, which simulations suggest would not have been possible with +fewer faces, fewer judgments, or lower-dimensional feature +representations. The model can be used to predict and manipulate +inferences with respect to arbitrary face photographs or to generate +synthetic photorealistic face stimuli that evoke impressions tuned along +the modeled attributes.
+In sum, the dataset contains 1.14 million ratings across 1000 items +and 34 traits by 5,000 participants. NOTE: The trait trustworthy in the +dataset was collected twice, so the trait column has 35 traits.
+For the attribute model studies, we used a between-subjects design +where participants evaluated faces with respect to each attribute. +Participants first consented. Then they completed a preinstruction +agreement to answer open-ended questions at the end of the study. In the +instructions, participants were given 25 examples of face images in +order to provide a sense of the diversity they would encounter during +the experiment. Participants were instructed to rate a series of faces +on a continuous slider scale where extremes were bipolar descriptors +such as “trustworthy” to “not trustworthy.” We did not supply +definitions of each attribute to participants and instead relied on +participants’ intuitive notions for each.
+Each participant then completed 120 trials with the single attribute +to which they were assigned. One hundred of these trials displayed +images randomly selected (without replacement) from the full set; the +remaining 20 trials were repeats of earlier trials, selected randomly +from the 100 unique trials, which we used to assess intrarater +reliability. Each stimulus in the full set was judged by at least 30 +unique participants.
+At the end of the experiment, participants were given a survey that +queried what participants believed we were assessing and asked for a +self-assessment of their performance and feedback on any potential +points of confusion, as well as demographic information such as age, +race, and gender. Participants were given 30 min to complete the entire +experiment, but most completed it in under 20 min. Each participant was +paid $1.50.
+https://github.com/jcpeterson/omi
+
+## Please set the work directory to the folder containing the scripts and data
+face_data <- import("data/suchow_data.csv.zip")
+str(face_data)
+#> 'data.frame': 1139300 obs. of 5 variables:
+#> $ participant: int 1256 1256 1256 1256 1256 1256 1256 1256 1256 1256 ...
+#> $ stimulus : int 63 75 73 64 54 46 23 18 74 49 ...
+#> $ trait : int 1 1 1 1 1 1 1 1 1 1 ...
+#> $ response : int 77 99 0 58 69 58 54 47 24 71 ...
+#> $ rt : int 4413 3518 5248 4167 3703 6304 4774 4480 2974 3359 ...
Peterson, J. C., Uddenberg, S., Griffiths, T., Todorov, A., & +Suchow, J. W. (2022). Deep models of superficial face judgments. +Proceedings of the National Academy of Sciences (PNAS).
+For the attribute model studies, we used Amazon Mechanical Turk to +recruit a total of 4,157 participants across 10,974 sessions, of which +10,633 (≈ 97%) met our criteria for inclusion. Participants identified +their gender as female (2,065) or male (2,053), preferred not to say +(21), or did not have their gender listed as an option (18). The mean +age was ∼39 y old. Participants identified their race/ethnicity as +either White (2,935), Black/African American (458), Latinx/a/o or +Hispanic (158), East Asian (174), Southeast Asian (71), South Asian +(70), Native American/American Indian (31), Middle Eastern (12), Native +Hawaiian or Other Pacific Islander (3), or some combination of two or +more races/ethnicities (215). The remaining participants either +preferred not to say (22) or did not have their race/ethnicity listed as +an option (8). Participants were recruited from the United States.
+Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
Participant |
+Unique number assigned to each participant |
+Numeric |
+
Stimulus |
+Face 1 to 1004 |
+Numeric |
+
Trait |
+Trait 1 to 35 |
+Numeric |
+
Response |
+Rating for corresponding rating |
+Numeric |
+
When pilot data is this large, it is important to sample a smaller +subset based on what the participant might actually do in the study. We +will pick 50 faces rated on 10 traits - and then select the highest and +lowest variance to estimate from. This choice is somewhat arbitrary - in +a real study, you could choose to use only the variables you were +interested in and pick the most conservative values or simply average +together estimates from all variables.
+
+# pick random faces
+faces <- unique(face_data$stimulus)[sample(unique(face_data$stimulus), size = 50)]
+# pick random traits
+traits <- unique(face_data$trait)[sample(unique(face_data$trait), size = 10)]
+
+face_data <- face_data %>%
+ filter(trait %in% traits) %>%
+ filter(stimulus %in% faces)
+# all SEs
+SE_full <- tapply(face_data$response, face_data$trait, function (x) { sd(x)/sqrt(length(x)) })
+SE_full
+#> 3 11 15 17 18 20 25 26
+#> 0.5302480 0.6052025 0.6057533 0.7151215 0.5899025 0.5936983 0.8063501 0.8173728
+#> 29 33
+#> 0.7510497 0.6380199
+## smallest variance is trait 4
+face_data_trait4_sub <- subset(face_data, trait == names(which.min(SE_full)))
+
+## largest is trait 30
+face_data_trait30_sub <- subset(face_data, trait == names(which.max(SE_full)))
+# individual SEs for 4 trait
+SE1 <- tapply(face_data_trait4_sub$response, face_data_trait4_sub$stimulus, function (x) { sd(x)/sqrt(length(x)) })
+quantile(SE1, probs = .4)
+#> 40%
+#> 3.230473
+
+# individual SEs for 30 trait
+SE2 <- tapply(face_data_trait30_sub$response, face_data_trait30_sub$stimulus, function (x) { sd(x)/sqrt(length(x)) })
+
+quantile(SE2, probs = .4)
+#> 40%
+#> 4.120649
How large does the sample have to be for 80% and 95% of the items to +be below our stopping SE rule?
+
+# sequence of sample sizes to try
+nsim <- 10 # small for cran
+samplesize_values <- seq(25, 100, 5)
+
+# create a blank table for us to save the values in
+sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(face_data_trait4_sub$stimulus)))
+# make it a data frame
+sim_table <- as.data.frame(sim_table)
+
+# add a place for sample size values
+sim_table$sample_size <- NA
+sim_table$var <- "response"
+
+# make a second table for the second variable
+sim_table2 <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(face_data_trait30_sub$stimulus)))
+
+# make it a data frame
+sim_table2 <- as.data.frame(sim_table2)
+
+# add a place for sample size values
+sim_table2$sample_size <- NA
+sim_table2$var <- "response"
+
+iterate <- 1
+for (p in 1:nsim){
+ # loop over sample sizes for age and outdoor trait
+ for (i in 1:length(samplesize_values)){
+
+ # temp dataframe for age trait that samples and summarizes
+ temp7 <- face_data_trait4_sub %>%
+ dplyr::group_by(stimulus) %>%
+ dplyr::sample_n(samplesize_values[i], replace = T) %>%
+ dplyr::summarize(se1 = sd(response)/sqrt(length(response)))
+
+ #
+ colnames(sim_table)[1:length(unique(face_data_trait4_sub$stimulus))] <- temp7$stimulus
+ sim_table[iterate, 1:length(unique(face_data_trait4_sub$stimulus))] <- temp7$se1
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ # temp dataframe for outdoor trait that samples and summarizes
+ temp35 <-face_data_trait30_sub %>%
+ dplyr::group_by(stimulus) %>%
+ dplyr::sample_n(samplesize_values[i], replace = T) %>%
+ dplyr::summarize(se2 = sd(response)/sqrt(length(response)))
+
+ #
+ colnames(sim_table2)[1:length(unique(face_data_trait30_sub$stimulus))] <- temp35$stimulus
+ sim_table2[iterate, 1:length(unique(face_data_trait30_sub$stimulus))] <- temp35$se2
+ sim_table2[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table2[iterate, "nsim"] <- p
+
+ iterate <- 1 + iterate
+
+ }
+
+}
Calculate the cutoff score with information necessary for +correction.
+
+cutoff_trait4 <- calculate_cutoff(population = face_data_trait4_sub,
+ grouping_items = "stimulus",
+ score = "response",
+ minimum = min(face_data_trait4_sub$response),
+ maximum = max(face_data_trait4_sub$response))
+
+# same as above
+cutoff_trait4$cutoff
+#> 40%
+#> 3.230473
+
+cutoff_trait30 <- calculate_cutoff(population = face_data_trait30_sub,
+ grouping_items = "stimulus",
+ score = "response",
+ minimum = min(face_data_trait30_sub$response),
+ maximum = max(face_data_trait30_sub$response))
+
+cutoff_trait30$cutoff
+#> 40%
+#> 4.120649
Trait 4 Results:
+
+cutoff <- quantile(SE1, probs = .4)
+final_sample <-
+ sim_table %>%
+ pivot_longer(cols = -c(sample_size, var, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ dplyr::group_by(sample_size, var, nsim) %>%
+ dplyr::summarize(percent_below = sum(se <= cutoff)/length(unique(face_data_trait4_sub$stimulus))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size, var) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size', 'var'. You can override
+#> using the `.groups` argument.
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample %>% head()) %>% autofit()
sample_size |
+var |
+percent_below |
+
---|---|---|
25 |
+response |
+0.110 |
+
30 |
+response |
+0.176 |
+
35 |
+response |
+0.288 |
+
40 |
+response |
+0.442 |
+
45 |
+response |
+0.604 |
+
50 |
+response |
+0.720 |
+
Calculate the final corrected scores:
+
+final_scores <- calculate_correction(proportion_summary = final_sample,
+ pilot_sample_size = face_data_trait4_sub %>%
+ group_by(stimulus) %>%
+ summarize(sample_size = n()) %>%
+ ungroup() %>%
+ summarize(avg_sample = mean(sample_size)) %>%
+ pull(avg_sample),
+ proportion_variability = cutoff_trait4$prop_var)
+
+flextable(final_scores) %>% autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
82.6 |
+55 |
+45.47975 |
+
88.6 |
+60 |
+50.61284 |
+
93.4 |
+65 |
+55.82046 |
+
97.2 |
+70 |
+60.93349 |
+
Trait 30 Results:
+
+cutoff <- quantile(SE2, probs = .4)
+final_sample2 <-
+ sim_table2 %>%
+ pivot_longer(cols = -c(sample_size, var, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ dplyr::group_by(sample_size, var, nsim) %>%
+ dplyr::summarize(percent_below = sum(se <= cutoff)/length(unique(face_data_trait30_sub$stimulus))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size, var) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size', 'var'. You can override
+#> using the `.groups` argument.
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample2 %>% head()) %>% autofit()
sample_size |
+var |
+percent_below |
+
---|---|---|
25 |
+response |
+0.336 |
+
30 |
+response |
+0.408 |
+
35 |
+response |
+0.506 |
+
40 |
+response |
+0.626 |
+
45 |
+response |
+0.712 |
+
50 |
+response |
+0.822 |
+
Calculate the final corrected scores:
+
+final_scores2 <- calculate_correction(proportion_summary = final_sample2,
+ pilot_sample_size = face_data_trait30_sub %>%
+ group_by(stimulus) %>%
+ summarize(sample_size = n()) %>%
+ ungroup() %>%
+ summarize(avg_sample = mean(sample_size)) %>%
+ pull(avg_sample),
+ proportion_variability = cutoff_trait30$prop_var)
+
+flextable(final_scores2) %>% autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
82.2 |
+50 |
+46.04663 |
+
89.8 |
+55 |
+51.50307 |
+
93.8 |
+60 |
+56.99324 |
+
98.0 |
+65 |
+62.40286 |
+
Based on these simulations, we can decide our minimum sample size for +80% is likely close to 45 for the trait 4 trials or 46 for the trait 30 +trials, depending on rounding. We can consider only the most variant +trait for power analysis since it would satisfy other traits in the +dataset as well.
+ulloa_vignette.Rmd
Liking effect induced by gaze
+Data provided by: José Luis Ulloa, Clara Marchetti, Marine Taffou +& Nathalie George
+This dataset resulted from a study aiming at investigating how gaze +perception can influence preferences. Previous studies suggest that we +like more the objects that are looked-at by others than non-looked-at +objects (a so-called liking effect). We extended previous studies to +investigate both abstract and manipulable objects. Participants +performed a categorization task (for items that were cued or not by +gaze). Next, participants evaluated how much they liked the items. We +tested if the liking effect could be observed for non-manipulable +(alphanumeric characters) as well as for manipulable items (common +tools).
+Participants were students at Heinrich-Heine-Universität Düsseldorf, +Germany that were paid for participating or received course credit. +Their ages ranged from 18 to 55 years. The words to-be-rated consisted +of 30 typical members of 30 categories drawn from the updated Battig and +Montague norms (Van Overschelde, Rawson, & Dunlosky, 2004).
+José Luis Ulloa, Clara Marchetti, Marine Taffou & Nathalie George +(2014): Only your eyes tell me what you like: Exploring the liking +effect induced by other’s gaze, Cognition & Emotion, DOI: +10.1080/02699931.2014.919899
+Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
suj |
+Unique number assigned to each participant |
+Numeric |
+
congr |
+valid vs invalid |
+Character |
+
item |
+G, K, S, L |
+Character |
+
liking |
+rating response |
+Numeric |
+
In this dataset, there are valid and invalid cue-targeting variable. +In valid cue-targeting condition, stimulus is on the same side of the +gaze. In invalid cue-targeting condition, stimulus was on the opposite +side of the gaze. We consider these two different conditions +separately.
+What the usual standard error for the data that could be considered +for our stopping rule using the 40% decile?
+
+### create subset for valid cue-targeting
+DF_valid <- subset(DF, congr == "valid") %>%
+ group_by(suj, item) %>%
+ summarize(liking = mean(liking, na.rm = T)) %>%
+ as.data.frame()
+#> `summarise()` has grouped output by 'suj'. You can override using the `.groups`
+#> argument.
+
+### create subset for invalid cue-targeting
+DF_invalid <- subset(DF, congr == "invalid") %>%
+ group_by(suj, item) %>%
+ summarize(liking = mean(liking, na.rm = T)) %>%
+ as.data.frame()
+#> `summarise()` has grouped output by 'suj'. You can override using the `.groups`
+#> argument.
+# individual SEs for valid cue-targeting condition
+SE1 <- tapply(DF_valid$liking, DF_valid$item, function (x) { sd(x)/sqrt(length(x)) })
+
+SE1
+#> G K L S
+#> 0.2013228 0.1779694 0.1801060 0.2286006
+cutoff1 <- quantile(SE1, probs = .4)
+cutoff1
+#> 40%
+#> 0.1843494
+
+# individual SEs for invalid cue-targeting condition
+SE2 <- tapply(DF_invalid$liking, DF_invalid$item, function (x) { sd(x)/sqrt(length(x)) })
+
+SE2
+#> G K L S
+#> 0.1982333 0.1749820 0.1724613 0.2132725
+cutoff2 <- quantile(SE2, probs = .4)
+cutoff2
+#> 40%
+#> 0.1796323
+# sequence of sample sizes to try
+nsim <- 10 # small for cran
+samplesize_values <- seq(25, 200, 5)
+
+# create a blank table for us to save the values in
+sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(DF_valid$item)))
+# make it a data frame
+sim_table <- as.data.frame(sim_table)
+
+# add a place for sample size values
+sim_table$sample_size <- NA
+sim_table$var <- "liking"
+
+# make a second table for the second variable
+sim_table2 <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(DF_valid$item)))
+
+# make it a data frame
+sim_table2 <- as.data.frame(sim_table2)
+
+# add a place for sample size values
+sim_table2$sample_size <- NA
+sim_table2$var <- "liking"
+
+iterate <- 1
+for (p in 1:nsim){
+
+ # loop over sample sizes for age and outdoor trait
+ for (i in 1:length(samplesize_values)){
+
+ # temp dataframe for age trait that samples and summarizes
+ temp_valid <- DF_valid %>%
+ dplyr::group_by(item) %>%
+ dplyr::sample_n(samplesize_values[i], replace = T) %>%
+ dplyr::summarize(se1 = sd(liking)/sqrt(length(liking)))
+
+ #
+ colnames(sim_table)[1:length(unique(DF_valid$item))] <- temp_valid$item
+ sim_table[iterate, 1:length(unique(DF_valid$item))] <- temp_valid$se1
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ # temp dataframe for outdoor trait that samples and summarizes
+
+ temp_invalid <-DF_invalid %>%
+ dplyr::group_by(item) %>%
+ dplyr::sample_n(samplesize_values[i], replace = T) %>%
+ dplyr::summarize(se2 = sd(liking)/sqrt(length(liking)))
+
+ #
+ colnames(sim_table)[1:length(unique(DF_invalid$item))] <- temp_invalid$item
+ sim_table2[iterate, 1:length(unique(DF_invalid$item))] <- temp_invalid$se2
+ sim_table2[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table2[iterate, "nsim"] <- p
+
+ iterate <- 1 + iterate
+ }
+
+}
Calculate the cutoff score with information necessary for +correction.
+
+cutoff_valid <- calculate_cutoff(population = DF_valid,
+ grouping_items = "item",
+ score = "liking",
+ minimum = min(DF_valid$liking),
+ maximum = max(DF_valid$liking))
+
+# same as above
+cutoff_valid$cutoff
+#> 40%
+#> 0.1843494
+
+cutoff_invalid <- calculate_cutoff(population = DF_invalid,
+ grouping_items = "item",
+ score = "liking",
+ minimum = min(DF_valid$liking),
+ maximum = max(DF_valid$liking))
+
+cutoff_invalid$cutoff
+#> 40%
+#> 0.1796323
+### for valid cue-targeting condition
+final_sample_valid <-
+ sim_table %>%
+ pivot_longer(cols = -c(sample_size, var, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ dplyr::group_by(sample_size, var, nsim) %>%
+ dplyr::summarize(percent_below = sum(se <= cutoff1)/length(unique(DF_valid$item))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size, var) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size', 'var'. You can override
+#> using the `.groups` argument.
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample_valid %>% head()) %>%
+ autofit()
sample_size |
+var |
+percent_below |
+
---|---|---|
25 |
+liking |
+0.150 |
+
30 |
+liking |
+0.250 |
+
35 |
+liking |
+0.300 |
+
40 |
+liking |
+0.425 |
+
45 |
+liking |
+0.625 |
+
50 |
+liking |
+0.825 |
+
Calculate the final corrected scores:
+
+final_scores <- calculate_correction(proportion_summary = final_sample_valid,
+ pilot_sample_size = length(unique(DF$suj)),
+ proportion_variability = cutoff_valid$prop_var)
+
+# only show first four rows since all 100
+flextable(final_scores %>%
+ ungroup() %>%
+ slice_head(n = 4)) %>% autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
82.5 |
+50 |
+44.43666 |
+
92.5 |
+55 |
+50.02481 |
+
92.5 |
+55 |
+50.02481 |
+
95.0 |
+60 |
+55.49695 |
+
+### for valid cue-targeting condition
+final_sample_invalid <-
+ sim_table2 %>%
+ pivot_longer(cols = -c(sample_size, var, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ dplyr::group_by(sample_size, var, nsim) %>%
+ dplyr::summarize(percent_below = sum(se <= cutoff2)/length(unique(DF_invalid$item))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size, var) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size', 'var'. You can override
+#> using the `.groups` argument.
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample_invalid %>% head()) %>%
+ autofit()
sample_size |
+var |
+percent_below |
+
---|---|---|
25 |
+liking |
+0.075 |
+
30 |
+liking |
+0.250 |
+
35 |
+liking |
+0.400 |
+
40 |
+liking |
+0.600 |
+
45 |
+liking |
+0.675 |
+
50 |
+liking |
+0.825 |
+
Calculate the final corrected scores:
+
+final_scores2 <- calculate_correction(proportion_summary = final_sample_invalid,
+ pilot_sample_size = length(unique(DF$suj)),
+ proportion_variability = cutoff_invalid$prop_var)
+
+# only show first four rows since all 100
+flextable(final_scores2 %>%
+ ungroup() %>%
+ slice_head(n = 4)) %>% autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
82.5 |
+50 |
+44.78681 |
+
87.5 |
+55 |
+50.23831 |
+
92.5 |
+60 |
+55.67221 |
+
100.0 |
+65 |
+61.41225 |
+
Based on these simulations, we can decide our minimum sample size for +80% is likely close to 44 for the valid trials or 45 for the invalid +trials, depending on rounding.
+In this example, we could set our maximum sample size for 95% items +below the criterion, which would equate to 55 for the valid trials or 61 +for invalid trials. In this case, values are equal because the percent +below jumps from 75% to 100%.
+In any estimate for sample size for this study, the dataset has a +large variance in ratings. This dataset need to more sample for items in +each conditions. In fact, we experimented combining two conditions +(valid & invalid cue-targeting) which did not result in any +difference.
+vanpaemel_vignette.Rmd
Exemplar by feature applicability matrices and other Dutch normative +data for semantic concepts
+Data provided by: Wolf Vanpaemel
+This data provides extensive exemplar by feature applicability +matrices covering 15 or 16 different categories (birds, fish, insects, +mammals, amphibians/reptiles, clothing, kitchen utensils, musical +instruments, tools, vehicles, weapons, fruit, vegetables, professions, +and sports), as well as two large semantic domains (animals and +artifacts). For all exemplars of the semantic categories, typicality +ratings, goodness ratings, goodness rank order, generation frequency, +exemplar associative strength, category associative strength, estimated +age of acquisition, word frequency, familiarity ratings, imageability +ratings, and pairwise similarity ratings are described as well. The +structure of the dataset is not programming language friendly. Here, we +only consider typicality.
+The typicality data were collected as part of a larger data +collection. Here we describe the typicality data collection only. The +data collection took place in a large classroom where all the +participants were present at the same time. The participants received a +booklet with instructions on the first page, followed by four sheets +with a semantic category label printed in bold on top. Each of the +category labels was followed by a list of 5–33 items belonging to that +category, referring to exemplars. The participants were asked to +indicate, for every item in the list, how typical it was for the +category printed on top of the page. They used a Likert-type rating +scale, ranging from 1 for very atypical items to 20 for very typical +items. If they encountered an exemplar they did not know, they were +asked to circle it. Every participant completed typicality ratings for +four different categories. The assignment of categories to participants +was randomized. For every category, four different random permutations +of the exemplars were used, and each of these permutations was +distributed with an equal frequency among the participants. All the +exemplars of a category were rated by 28 different participants.
+https://static-content.springer.com/esm/art%3A10.3758%2FBRM.40.4.1030/MediaObjects/DeDeyne-BRM-2008b.zip +and included here.
+
+### for typicality data -- cleaning and processing
+typicality_fnames <- list.files(path = "data/vanpaemel_data",
+ full.names = TRUE)
+
+typicality_dfs <- lapply(typicality_fnames, read.csv)
+
+ID <- c(1:16)
+typicality_dfs <- mapply(cbind, typicality_dfs, "SampleID" = ID, SIMPLIFY = F)
+
+typicality_all_df <- bind_rows(typicality_dfs)
+typicality_all_df_v2 <- typicality_all_df %>%
+ unite("comp_group", X:X.1, remove = TRUE) %>%
+ select(-c(30,31,32,33,34)) %>%
+ drop_na(c(2:29)) %>%
+ filter_all(any_vars(!is.na(.))) %>%
+ dplyr::rename(compType = SampleID)
+# typicality_all_df_v2
+typicality_all_df_v3 <- typicality_all_df_v2 %>%
+ select(starts_with("X"), compType, comp_group) %>%
+ pivot_longer(cols = starts_with("X"),
+ names_to = "participant",
+ values_to = "score")
+
+head(typicality_all_df_v3)
+#> # A tibble: 6 × 4
+#> compType comp_group participant score
+#> <int> <chr> <chr> <int>
+#> 1 1 kikker_frog X.2 18
+#> 2 1 kikker_frog X.3 20
+#> 3 1 kikker_frog X.4 19
+#> 4 1 kikker_frog X.5 12
+#> 5 1 kikker_frog X.6 20
+#> 6 1 kikker_frog X.7 15
De Deyne, S., Verheyen, S., Ameel, E. et al. Exemplar by feature +applicability matrices and other Dutch normative data for semantic +concepts. Behavior Research Methods 40, 1030–1048 (2008). https://doi.org/10.3758/BRM.40.4.1030
+Variable Name |
+Variable Description |
+Type (numeric, character, logical, etc.) |
+
---|---|---|
compType |
+Comparison type for typicality rating |
+Character |
+
comp_group |
+Individual items within compType |
+Character |
+
participant |
+Participant number |
+Character |
+
score |
+Typicality: how typical is the item for the category? |
+Numeric |
+
In this example, we will pick one comparison type and use the items +within that to estimate sample size. This choice is arbitrary!
+
+# individual SEs among different comparison group
+SE <- tapply(typicality_all_df_v3$score, typicality_all_df_v3$compType, function (x) { sd(x)/sqrt(length(x)) })
+SE
+#> 1 2 3 4 5 6 7 8
+#> 0.4847915 0.1868793 0.1894860 0.2326625 0.1862387 0.2310363 0.1433243 0.1751163
+#> 9 10 11 12 14 16
+#> 0.1888044 0.1563060 0.2512611 0.1945454 0.2042343 0.2520606
+
+min(SE)
+#> [1] 0.1433243
+max(SE)
+#> [1] 0.4847915
+
+# comparison type 1: amphibians
+typicality_data_gp1_sub <- subset(typicality_all_df_v3, compType == 1)
+
+# individual SEs for comparison type 1
+SE1 <- tapply(typicality_data_gp1_sub$score, typicality_data_gp1_sub$comp_group, function (x) { sd(x)/sqrt(length(x)) })
+
+SE1
+#> kikker_frog krokodil_crocodile pad_toad
+#> 0.4836714 1.1085074 0.7368140
+#> salamander_salamander schildpad_tortoise
+#> 0.7531742 1.6330366
+# sequence of sample sizes to try
+nsim <- 10 # small for cran
+samplesize_values <- seq(5, 200, 5)
+
+# create a blank table for us to save the values in
+sim_table <- matrix(NA,
+ nrow = length(samplesize_values)*nsim,
+ ncol = length(unique(typicality_data_gp1_sub$comp_group)))
+# make it a data frame
+sim_table <- as.data.frame(sim_table)
+
+# add a place for sample size values
+sim_table$sample_size <- NA
+sim_table$var <- "score"
+
+iterate <- 1
+for (p in 1:nsim){
+
+ # loop over sample sizes for comparison type
+ for (i in 1:length(samplesize_values)){
+
+ # temp dataframe for comparison type 1 that samples and summarizes
+ temp1 <- typicality_data_gp1_sub %>%
+ dplyr::group_by(comp_group) %>%
+ dplyr::sample_n(samplesize_values[i], replace = T) %>%
+ dplyr::summarize(se2 = sd(score)/sqrt(length(score)))
+
+ # add to table
+ colnames(sim_table)[1:length(unique(typicality_data_gp1_sub$comp_group))] <- temp1$comp_group
+ sim_table[iterate, 1:length(unique(typicality_data_gp1_sub$comp_group))] <- temp1$se2
+ sim_table[iterate, "sample_size"] <- samplesize_values[i]
+ sim_table[iterate, "nsim"] <- p
+
+ iterate <- 1 + iterate
+ }
+
+}
Calculate the cutoff score with information necessary for +correction.
+
+cutoff <- calculate_cutoff(population = typicality_data_gp1_sub,
+ grouping_items = "comp_group",
+ score = "score",
+ minimum = min(typicality_data_gp1_sub$score),
+ maximum = max(typicality_data_gp1_sub$score))
+
+cutoff$cutoff
+#> 40%
+#> 0.7466301
+### for response outputs
+# figure out cut off
+final_sample <-
+ sim_table %>%
+ pivot_longer(cols = -c(sample_size, var, nsim)) %>%
+ dplyr::rename(item = name, se = value) %>%
+ dplyr::group_by(sample_size, var, nsim) %>%
+ dplyr::summarize(percent_below = sum(se <= cutoff$cutoff)/length(unique(typicality_data_gp1_sub$comp_group))) %>%
+ ungroup() %>%
+ # then summarize all down averaging percents
+ dplyr::group_by(sample_size, var) %>%
+ summarize(percent_below = mean(percent_below)) %>%
+ dplyr::arrange(percent_below) %>%
+ ungroup()
+#> `summarise()` has grouped output by 'sample_size', 'var'. You can override
+#> using the `.groups` argument.
+#> `summarise()` has grouped output by 'sample_size'. You can override using the
+#> `.groups` argument.
+
+flextable(final_sample) %>% autofit()
sample_size |
+var |
+percent_below |
+
---|---|---|
10 |
+score |
+0.12 |
+
5 |
+score |
+0.16 |
+
15 |
+score |
+0.18 |
+
20 |
+score |
+0.30 |
+
25 |
+score |
+0.38 |
+
30 |
+score |
+0.52 |
+
35 |
+score |
+0.56 |
+
40 |
+score |
+0.56 |
+
45 |
+score |
+0.62 |
+
50 |
+score |
+0.62 |
+
55 |
+score |
+0.66 |
+
60 |
+score |
+0.68 |
+
65 |
+score |
+0.78 |
+
70 |
+score |
+0.80 |
+
75 |
+score |
+0.82 |
+
80 |
+score |
+0.82 |
+
85 |
+score |
+0.82 |
+
95 |
+score |
+0.82 |
+
110 |
+score |
+0.84 |
+
115 |
+score |
+0.84 |
+
100 |
+score |
+0.86 |
+
90 |
+score |
+0.88 |
+
105 |
+score |
+0.88 |
+
125 |
+score |
+0.88 |
+
135 |
+score |
+0.88 |
+
130 |
+score |
+0.90 |
+
150 |
+score |
+0.92 |
+
120 |
+score |
+0.94 |
+
140 |
+score |
+0.94 |
+
155 |
+score |
+0.94 |
+
195 |
+score |
+0.96 |
+
160 |
+score |
+0.98 |
+
170 |
+score |
+0.98 |
+
175 |
+score |
+0.98 |
+
145 |
+score |
+1.00 |
+
165 |
+score |
+1.00 |
+
180 |
+score |
+1.00 |
+
185 |
+score |
+1.00 |
+
190 |
+score |
+1.00 |
+
200 |
+score |
+1.00 |
+
Calculate the final corrected scores:
+
+final_scores <- calculate_correction(proportion_summary = final_sample,
+ pilot_sample_size = length(unique(typicality_data_gp1_sub$participant)),
+ proportion_variability = cutoff$prop_var)
+
+flextable(final_scores) %>% autofit()
percent_below |
+sample_size |
+corrected_sample_size |
+
---|---|---|
80 |
+70 |
+69.05319 |
+
88 |
+90 |
+86.00674 |
+
94 |
+120 |
+108.71054 |
+
100 |
+145 |
+126.05262 |
+