From b9ecd8436d551184e82d2f6d7517f14eac662bf6 Mon Sep 17 00:00:00 2001 From: Paul Ralph <33912489+drpaulralph@users.noreply.github.com> Date: Fri, 15 Mar 2024 12:02:24 -0300 Subject: [PATCH] Create QualitativeSimulation Initial Draft of Qualitative Simulation Standard --- docs/drafts/QualitativeSimulation | 90 +++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 docs/drafts/QualitativeSimulation diff --git a/docs/drafts/QualitativeSimulation b/docs/drafts/QualitativeSimulation new file mode 100644 index 0000000..7161caa --- /dev/null +++ b/docs/drafts/QualitativeSimulation @@ -0,0 +1,90 @@ +# Qualitative Simulation + + +*"A study that involves predominately qualitative analysis of direct observations of humans taking part in a lab-based simulation* + + +## Application + +This standard applies to empirical research that meets the following +conditions. + +- Human participants take part in a simulated task, typically in a contrived or lab-based environment, alone or in groups +- Researchers directly observe (and often record) participants taking part +- Observations are analyzed using qualitative coding + +Protocol Analysis (AKA protocol studies) are a common type of Qualitative Simulation + +## Specific Attributes + +### Essential Attributes + + + + + + + +- [ ] describes the environment in which the simulation took place +- [ ] describes the events, activities, tasks, or objects comprising the simulations +- [ ] describes the qualitative coding procedures + + + +- [ ] presents a clear chain of evidence from observations to findings + + + + + + + + +### Desirable Attributes + + +- [ ] provides supplemental materials such as task materials, coding schemes, coding examples, decision rules, or extended chain-of-evidence tables +- [ ] includes debriefing sessions with participants +- [ ] cross-checks observations against statements made my participants during debriefing sessions +- [ ] describes how prior understandings of the phenomena were managed and/or influenced the research +- [ ] EITHER: evaluates an a priori theory (or model, framework, taxonomy, etc.) using deductive coding with an a priori coding scheme based on the prior theory + OR: synthesizes results into a new, mature, fully-developed and clearly articulated theory (or model, etc.) using some form of inductive coding (coding scheme generated from data) +- [ ] includes autoreflection; i.e., researchers reflect on how their own possible biases may have influenced the research + + +### Extraordinary Attributes + + +- [ ] uses a team-based approach; e.g., multiple raters with analysis of inter-rater reliability (see the [IRR/IRA Supplement](https://github.com/acmsigsoft/EmpiricalStandards/blob/master/Supplements/InterRaterReliabilityAndAgreement.md)) +- [ ] published a protocol beforehand and made it publicly accessible (see the [Registered Reports Supplement](https://github.com/acmsigsoft/EmpiricalStandards/tree/master/Supplements)) + + +## General Quality Criteria + +Qualitative simulations should be evaluated using qualitative validity criteria +such as credibility, multivocality, reflexivity, rigor and transferability (see **Glossary**). + +## Antipatterns + +- Overstating the degree to which the simulation justifies inferences about participants' cognitive states or cognitive processes. The whole point of a qualitative simulation is often to get insight into cognitive states and processes, but these insights should be presented with caution. +- Attempting a qualitative simulation from a positivist epistemological stance. There is no coherent positivist basis for qualitative research in an artificial context. Qualitative simulations make more sense from a realist or interpretivist epistemological stance. + +## Invalid Criticisms + +- Does not present quantitative data. That's why it's called a "qualitative" simulation. +- The task or environment are artificial. The whole point of a qualitative simulation is to study an artificial task in a controlled environment. +- Lack of internal validity. Internal validity is a positivist criterion. + +## Suggested Readings + +Mathew Miles, A Michael Huberman and Saldana Johnny. 2014. *Qualitative +data analysis: A methods sourcebook*. Sage. + +Sarah J. Tracy. 2010. Qualitative Quality: Eight "Big-Tent" Criteria for +Excellent Qualitative Research. *Qualitative Inquiry*. 16, 10, +837–851. DOI: +[10.1177/1077800410383121](https://doi.org/10.1177/1077800410383121) + +## Exemplars + +Mohanani, Rahul, Paul Ralph, Burak Turhan, and Vladimir Mandić. 2021. How templated requirements specifications inhibit creativity in software engineering. *IEEE Transactions on Software Engineering* 48, 10, 4074-4086. DOI: [10.1109/TSE.2021.3112503](https://doi.org/10.1109/TSE.2021.3112503)