Feedback from Alison on Slack:
I understand your metrics, but what will success be for this experiment versus failure? is the goal not to have repeat questions or is it that if there are repeaters that means it has more strategic importance? Also is the idea that the score differential would decrease over the week? Finally, I’d love to see the lists of questions, will those be kept in the data section?
I feel like our #Pulse call could look at one of these questions this week or in the future.
Yes, the idea is that the score differential would decrease over the week. I’ll update the experiment to clarify this. Jodie Jepoardy will definitely help us understand how to better assess strategic muscles.
Re: Repeaters. I’m not sure yet how to interpret repeat questions yet, so I’ll track, but no implied success metrics yet. One scenario is that repeated questions _and_ answers is a sign that this exercise is working, because the question is clearly occupying (and perhaps confusing or stressing out) @jodie’s mind, but she keeps coming back to the same answer. Another scenario is that the question keeps coming up, but Jodie keeps coming up with different, possibly diverging answers, which might be an indication that she’s working through possibilities or that she’s flip-flopping. Too soon to tell, so just want to observe and see.
We’ll track this exercise in our usual meeting log, so you’ll be able to follow Jodie’s questions and answers there. As I synthesize, I’ll make that available somewhere as well.