Options for Student Feedback on Questions
After you’ve selected an answer to a multiple choice question, you have a few ways to interact with author of the question. First, if you supply the same answer as the author, you receive a confirmation that your answer agrees with the author’s answer.
- The system does not necessarily provide a Correct or Incorrect answer. What it allows for is agreement with the author's suggested correct answer. You can imagine this might come into play in questions that are more open ended - perhaps coding solutions. This allows students to collectively decide which answer is the most correct. The value is the group. You can see that on this question, since I am the first one to answer, that the result is report but is qualified that it is does not have enough data to be authoritative. This feedback will hopefully help to build trust, as the system is displaying transparency regarding its data points.
- By participating, you start to collect points. Answering and authoring questions leads to earned badges, which introduce an element of Gamification (another trendy topic in Education Technology) to the system.
Report of Number of Times Question has been answered
- To provide even more context regarding this question, the system displays the number of times this question has been answered. If a student knows, in general, how many students are in the course, and they know the expectations of system use (say, the Instructor has assigned that they must author 1 question and answer 10 questions), they will have a sense of the number of answers each question will have. They will then know if they are answering this question early (and so might come back to it once it matures) and can weight the response rate accordingly.
- Additionally, the students can see the ranking for this question. In addition to being correct/incorrect, students can rate the questions for value. This gives each learner an indication of how their fellow learners felt about the quality of the question. In another part of the interface, students can self-select to quiz using only the highest rated questions. Writing highly rated questions can increase feelings of competence for the author.
Report of Alternative Answers
This table presents the learner with information about their own answer, the number of times that answer was selected by fellow learners, and the number of times that answers was changed after reading the full explanation (written by the question author).
- This reminds the learner which answer they selected.
- First Answers: This column displays the selection of fellow learners. If the learner believes they made the correct choice, they confirm their answer.
- Confirmed Answers: This column displays the selection of fellow learners after they've read the full explanation. If the learner believes there's a more correct answer, they can modify their choice.
The built in ability to change your answer helps to reinforce that this is not a testing tool - it is a practice and review tool. The idea is to learn, not to be "right."
Contribute to improving the answer's Explanation
This is an example of an explanation that I wrote for one of my sample questions.
- Request Help: If I am still confused, I can request help from the question's author.
- Improve Explanation: Now that I'm viewing it as a fellow learner would, I have the option to contribute improvements or further thoughts.
One thing to note: When the author chooses to make a change to the question, the current question is archived and all statistics associated with the question are reset. This is by design, to ensure that students are answering and rating the same question versus versions of the question.
Interface after selecting to "Improve Explanation"
- After selecting on "Improve Explanation," a new WYSIWYG text box appears that allows students to contribute further details to the explanation.
- To submit, the student must choose to "Save explanation."
Interface after adding to Explanation
- Plus 1: Learners can now see that one improvement has been made to the explanation.
- Learners can see the suggestions made. In this case, I wrote about technical errors versus content errors.
- Improve Explanation: Other students can still contribute more to this explanation.
Categorize Questions by Topics
The author of the question can assign tags/categories to the questions they create. This can help make it easier to quiz on selected topics, but, categories have to be managed well to be effective. It might be more effective to have the Instructor provide a list of topics, versus allowing the students to create (and perhaps duplicate) their own.
Leave a Comment
- Unlike improvements to the content of the question, comments allow students to leave other feedback. In hindsight, it would have made more sense to write my comment regarding syntax here.
In my opinion, this is the most important part of the system. This is where students rank the difficult and the quality of the question. Ranking these attributes is difficult because they are so subjective - what is difficult to one person is easy to another. It is also an area that is difficult for the Instructor to translate for the students. One of the questions, likely, most heard by Instructors and TAs is, "How hard will the test be?" perhaps followed by, "What kind of questions will you ask?"
This ranking can give an indication to the teaching team about what their students think is difficult and what they think easy. The teaching team can also rate the questions, providing even more transparency to the students about their perception of the level of the question. Imagine a scenario - students take a questions and mark its level as medium. The Instructor looks at the same question and is surprise - they consider that question difficult, or even extremely difficult (not shown on the scale). By stepping in, and marking the question as difficult, the students receive feedback and can alter their perceptions. Even better, if someone from the teaching team comments about why this question was difficult, students can modify and manage their expectations - stress will decrease and effectiveness of preparation will increase.
- Difficulty: Students can mark how difficult they find this question. This attribute has many layers. Not only should there be a variety of difficult levels (it's ok to have 'easy' questions), it provides transparency to the teaching team about student perception of difficulty. The help text, provided in the system by the "?" icon, to increase consistency states: "Please select what you feel is the most appropriate difficulty level for this question (NOTE: questions of all levels of difficulty are equally valid)."
- Quality: Students can mark how they perceived the quality of the question. The help text, provided in the system by the "?" icon, to increase consistency states: "Take into account the language used in the question, the quality of the alternatives, the quality of the explanation and the overall relevance of the question to your course."
- Report this Question: This measure allows students to anonymously report questions. Students know that they have the ability to identify controversial or inappropriate content.
- Follow author: Although activity in the system is anonymous (the designers are debating allowing non-anonymous options, but these do not exist yet), you can choose to follow authors of questions that you think are particularly valuable.