#ELIXIR – EXCELERATE Train-the Trainer subtask
- Formative and summative assessment;
- Using questionnaires to promote peer instruction and content delivery;
- The role of questioning in learning; Formative Assessment;
- Diagnostic questionnaires
- [Design of MCQs with distractors] (#design)
- [Assessment of training quality, participant and instructor performance;] (#assessment)
- [Systematic Feedback;] (#systematic)
- [(Short and long term) Post-course feedback.] (#short)
Quotes form "Peer Intruction, Getting students to think in class" by Erik Mazur , pdf available here
".. while listening is largely a passive activity, reading more easily engages the mind and it allows more time for the imagination to explore questions."
"the first exposure to new material comes from reading printed material before the lecture reading."
to be continued .... SEE QUESTIONNAIRE here
(there is a copy in the docs folder)
...
...
In a training course, getting feedback at the end of the event is necessary, as the participants may (should) have developed encompassing, integrated views. However, it is vastly insufficient. Questioning participants frequently diring training provision is rich with information and has very interesting effects. But when should this happen? And how can it be induced so that, as a drug, has as many positive effects and as low adverse effects as possible?
When? Ideally at natural breakpoints such as ending an exercise, shifting to a different subject and right after a wrap-up session.
How? It should be very focused and expedite in execution. The instructor should think of a clearly stated question that has a binary (Yes/no) or garaded (0-5) response. Ideally the isntructor should write the question and display it, ensuring that everybody knows what the question is at the same time and is aware of what the answering method is. Then, the isntructor collects the answers and records trhem in a tally.
This is INSTANT FEEDBACK.
Several methods have been tested, some of the using technology (Clickers, Socrative, Learning Catalytics) or not (the fist of five method). The choice is made according to the availability of the means and how engaging the audience finds it.
###Fist or Five Feedback
by Allegra Via, Kristian Rother and Pedro Fernandes From: Academis by Kristian Rother.
How well was your explanation understood? How useful was an exercise? Is your class enthusiastic or frustrated? During a one-week programming course at IGC, Portugal, we asked after each training module:
"How much did you learn during the lesson? Please show one to five fingers. Raise your hands!"
Then we counted how often each number of fingers occurred. This way, the trainees felt more encouraged to provide critical feedback than if you would simply ask:
"did you understand it or not?"
Not necessarily do trainees utilize all five fingers. Our course participant Patricia commented:
"It is a good feedback and it is immediate. Although I feel sometimes a little bit shy to express my opinion."
The method needs seconds to execute and no preparation, which is a plus for the teacher. But trainees benefit as well. Our course participant Rita commented:
"I like it because it makes me think. It forces me to review and figure out whether I understood the subject or not and how much. It also shows you are interested."
This feedback is not an objective control of students' knowledge; it gives rather an indication of how confident they feel at a given point. You can try to suggest examples what a zero or five means, as in the linked article. The fist or five technique has also been recommended as a voting procedure to reach consensus in group discussions. You may test the method after giving a presentation to evaluate yourself.
The numbers we accumulated over more than a dozen sessions using one consistent method helped us to keep the course on track. The counting itself needed a bit of exercise to do it quickly. When we used the Fist or Five technique for the first time in 2012 with a group of 20 people, we asked for each number from zero to five separately This took a bit longer. For us, the main value of the Fist or Five technique is that it is easy to execute, it is quantitative, it is not stressful, it is immediate and can be repeated many times during a course. We hope you will see lots of 'high fives' in your next course!
BENEFITS WORTH NOTICING:
- For the LEARNER. Carefully implemented instant feedback obliges the learner to introspect, to answer himself first (do I really know this? How easy ist it for me to do this by myself?). With this, it becomes clear that he is made aware of his own progress and this is the smartes way to gain self-confidence. When questioned at the end-of the course questionnaire, he is much more able to make encompassing self assessments
- For the INSTRUCTOR. Multiple ways of checking if what has just been done was effective, as a result of the quality of the question. Useful assessment of the quality of the materials and the performance of the instructor. A way of identifying learners that may be dragging behind and may need more attention. A way of identifying learners that are getting ahead o the others in the group, and can become more active, receive harder assignments, help their colleagues, etc. A way to judge wheather the pace of training delivery is correctly chosen for the audience.
Long term assessments are rather difficult. First becaulse learners move and become more diffuicult to cantact with. Secondly because they forget, as all of us do. In this case they forget what worked for them as hidden details. Those may matter because what we are looking for, here, is the aseesment of impacts that endure.
Interviewing former course participants would be a possibility but it requires a lot of time. Sending them short questions by e-mail has worked with a yeald of about 30%, so unless you are tarining at least several hundreds of people it is likely that you end-up with a very small number of answers. Currently we see some home in the usage of social networks to collect valuable data.
Critical appraisals often happen is casual conversations. One should take notes to record them.
Example: Pedro Fernandes, Pooja Jain, Catarina Moita Training Experimental Biologists in Bioinformatics, Adv Bioinformatics. 2012;2012:672749. doi: 10.1155/2012/672749. Epub 2012 Jan 31. (Open Access)
####Training Evaluation
There are several methods that can be used to evaluate training. One of the most referenced methods comes from Donald Kirpatrick (1924-2014).
####The Kirkpatrick Model
-
Level 1: Reaction The degree to which participants find the training favorable, engaging and relevant to their jobs
-
Level 2: Learning The degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training
-
Level 3: Behavior The degree to which participants apply what they learned during training when they are back on the job
-
Level 4: Results The degree to which targeted outcomes occur as a result of the training and the support and accountability package
This model has been revised and expanded several times, see for example:
http://www.kirkpatrickpartners.com/OurPhilosophy/TheNewWorldKirkpatrickModel/tabid/303/Default.aspx
Applying the Kirkpatrick model and its variants is not easy. One needs to be very careful in checking pre-requisites, assumptions and options in the measurement methods.
The evaluation of training efficiency is a difficult subject. There is an obvious need to standardise to allow for the comparison of observations.
You may like to read an article about applying Kirkpatrick's methods. https://www.mindtools.com/pages/article/kirkpatrick.htm