Because we are soon launching our new Quality Barometers for online teachers and courses - which I have spent much time and energy to design, specify and implement.
So, what are quality barometers?
Well, I perceive it as tools that continuously monitor and report quality in transparent ways. I strongly believe that transparent quality barometers are efficient and cost effective quality improvement tools.
Quality barometers are especially useful in online education environments with individual or flexible start-up and progression plans. Continuous monitoring is probably hard to implement in semester based educational environments.
NKI has several years of experience with a response barometer that monitors how much time it takes from a student submits an assignment to the teacher provides feedback and a grade. Since all students have individual progress plans and can submit their assignments 365 days a year, the response barometer continuously provides transparent information about our teachers' individual and collective response time. During the last three months (summer vacation time included), our 140 online teachers have a collective response time of two days. I'm proud to tell that their average collective response time throughout 2010 was 1.8 days.
How could such barometers measure quality?
Thermometers measure temperatures in degrees, and ordinary barometers measure pressure in pascal or bar. I suggest that quality barometers could measure in quality in VESS (meaning VEry Satisfied or Satisfied). The idea comes from the fact that NKI uses a 5-point Likert-scale as shown in the figure. The five alternatives are Very satisfied, Satisfied, Neutral, Dissatisfied and Very Dissatisfied. With these five alternatives, we calculate the percentage of Very satisfied and Satisfied out of the total evaluations. If the evaluations are equally distributed among all five alternatives, the calculated quality would be 40%. A lower VESS reading indicates that you may have quality challenges.
How do NKI's new quality barometers for teachers work?
You have to have some patience, since I need a few more days to finish the English version of the nano-course I'm developing to explain how it works.
I'll be back - in a few days.
I would love to hear your comments on this blog entry.
ReplyDeleteInteresting study. So, are you saying that student satisfaction is the best indicator of quality of online education?
ReplyDeleteI like the barometer idea very much, but I'm very wary indeed of the transparency idea, at least in an automated context. These barometers are useful formative tools that can offer useful *evidence* of success or otherwise, but they cannot ever provide *proof* nor a complete picture. The problem is that (as I think LJL implies) you are looking at a limited set of dimensions of quality.
ReplyDeleteFor instance, timeliness of feedback is undoubtedly a really important factor in helping people to learn but, if the choice were between poor feedback in 2 days or rich feedback in 3, it would not always be in the best interests of the learner to get the former. Given that teachers have limited time and resources, it is sometimes necessary to choose between the lesser of two evils. Given that problem, incorporating feedback on student satisfaction into the results makes sense and helps to balance things better. However, ratings of teachers are not always based solely on the value they give in assisting learning - likeable teachers, for example, often get higher ratings than less likeable teachers, as do those that give higher rather than lower marks, regardless of how effective their feedback may actually be in assisting the learner to learn. It's all useful management information for the teacher that can help him or her to make more informed decisions about their teaching but, if it is shown to others without a lot of rich information to counterbalance it, there is a very high risk that it could reify a world view in which only a limited set of factors are taken into account.
There are also high risks of preferential attachment, whereby views of a given teaching intervention are coloured by expectations based on previous perceptions so that success leads to more success and failure to greater failure. Early raters will have far more effect than later ones. There has been some great work on this in analysing results of presidential primaries - see https://landing.athabascau.ca/pg/blog/read/3594/the-importance-of-being-first. There are ways to reduce that problem, including introducing delay as well as only showing recent results, but it is a tricky balancing act.
Finally, replacing intrinsic motivation with extrinsic motivation that gives what is, in effect, a grade is a bad idea. It encourages exactly the kind of strategic approaches that we try to discourage in learners, whereby teachers would naturally be inclined to aim for success according to these rather limited criteria rather than doing what is best for the student. And, to cap it off, it would almost certainly reduce their overall motivation and job satisfaction (see Deci's work on this for some great experimental evidence of how extrinsic motivation fails).
ReplyDeleteWhen an essay is a problem - get our help!
Wawa that brilliant blog this is dude, I really love to read this.