The Timeline Beta Blog

I’m reading information provided by the TLT group about assessment. Specifically, I’m looking at what they call the “flashlight approach.” This fall, we want to design an assessment process that would be administered at both ends of the course, before the students used the timeline and then after. We’re looking for some good advice on productive ways to approach this task.

The basic goals of the flashlight approach, as reinterpreted by yours truly in the context of this project :

1. Make the survey process meaningful to the survey-takers (most people hate surveys, it needs to be clear to the survey-taker why the survey is useful to them, not just to us!)
2. Pursue “formative” rather than “summative” responses. Summative questions attempt to measure absolute success or failure. Formative questions are designed to discover how to improve results on the next iteration.
3. Study what users actually did with the technology rather than just whether or not they did what you wanted or anticipated. We want to discover what students typically do (student practice), rather than what they could do (hypothetical potential) with the technology.
4. Investigate the “why”. First figure out what students actually did, and then try and identify the deciding factors that made them follow that particular path. These factors may or may not have a direct link to the technology you are trying to evaluate.
5. Don’t expect tidy results. Diversifying learning options for faculty and students also diversifies the process and nature of the results. Students do not arrive at a course with the same levels, skills, or needs. There will be variation. Expect it, embrace it, incorporate it into the _formative_ analysis of your results. Diversity in results can be the reflection of successful personalized environments, rather than the mark of failure to achieve a standardized goal.
6. Recognize that change is never 100% positive. Unless you are willing to identify and validate what is lost, as well as what is gained, when you move to a new process, you can’t work to minimize those losses. For example, assigning media projects may mean assigning fewer papers. You’ve diversified the skills practice and assessment for the course – but students will get less practice writing critical papers. That’s a loss, even if the trade-off seems worth it.
7. The external credibility of any assessment exercise depends upon the perception that the results will be presented no matter what they are. This also goes back to formative vs. summative findings. Summative findings are less credible insofar as (it is assumed) no one is eager to publish negative summative results.
8. Close the feedback loop. Normally, this means that the results will be used to improve the lesson structure for the next round of students. They suggest, however, that students taking these surveys are also entitled to know what the outcome of their survey-response was. So, report back down to the subjects, as well as up to the administration.