The Timeline as a pyramid scheme

April 22, 2008 at 7:55 am (Survey, assessment, pedagogy, use cases)

We always do this with assessment; we frequently to do it in our course material. Our tests/assignments/exercises attempt to teach (or enforce) a larger amount of knowledge than fits into the moment. You still have to do all of the reading, even if we don’t cover it in class. Next week’s test will cover X amount of material, but only ask Y questions. If you knew which questions those were, you could save a lot of time… but you don’t.

One of the things that happens with Web 2.0 is that teachers get all starry-eyed thinking that we’ll be able to use the same old pyramid trick, but with carrots instead of sticks. Instead of getting students to do work by holding them accountable in a traditional sense, we are going to motivate them to cover that ground by sheer coolness. Except that, oops, guess what — students are really good at telling the difference between work and… not-work. Sigh.

Today’s survey results fall on both sides of this question. Since I want to work with four questions from the survey for this point, I’m posting them as images rather than listing the results.

I learned more information researching my post than I posted to the timeline.

Nobody disagreed with this; out of a total of 18 respondents in the two courses, only 3 people were even neutral on this point. This is a classic example of a successful pyramid scheme assignment. It’s also the locus of some of the most active critical thinking around the assignment. Pick a topic, explore it broadly, narrow the field, select the pertinent information, synthesize, post. Fantastic.

Now, the story of that extra information doesn’t end there, however, and I think it would be less successful if it did. By having the students present their timeline posts in class, a lot of the extra information that didn’t make it into the little 150-word post DOES get communicated as part of this presentation, in which the students usually describe their research trajectory as well as their conclusions. Hence our next question(s)….

I learned a lot from other students’ posts during class presentations. / Presenting and discussing timeline posts was a productive in-class activity.

I’ve lumped these two together for discussion. Again, the responses are globally quite positive. Less than a quarter of the students disagreed with this statement, about a quarter of the students were neutral, and slightly over 50% agreed or strongly agreed. I think, too, that these questions are impacted by student attitudes towards peers as a reliable source of information. Students who only listen attentively when the instructor is speaking, and tune out the rest, are not likely to glean much benefit from this format. It should be noted, however, that I am very active during these presentations – in between each one I usually speak for a few minutes, expanding or linking the information presented to the course material at large. I use them as a dialogical framework for information that I might otherwise present as straight lecture.

I learned a lot from other students’ posts outside of class.

What is surprising about this question is that anyone responded in the affirmative at all. Before we implemented the list view on February 14 (3 weeks into the semester) there was no way to identify new posts whatsoever. The list view was an imperfect, but at least functional indicator of recent activity. However, it wasn’t until we put in the color coded new entry function on March 20 that we had a true, visual representation of recent activity, one that was visible on the timeline (and so in context.)

Because there was no real way to monitor recent activity, the only accountability I imposed for reading classmates’ posts was that duplicate posts were marked down. In other words, check and make sure someone else didn’t get there first. This was much more of a factor for the cinema class than the literature class, because they were researching narrower topics (New Wave films, for example) and because there were more students.

I’m unsure whether I want to implement greater accountability for peer information in the next round. The question of quality control becomes much more pressing if the students are to be held accountable for the information. I like the in-class presentations, and I think that the visible “what’s new” function is going to increase the level of browsing. I can easily imagine a different subject area or assignment format, however, where an instructor might want to hold students accountable for the timeline information.

Leave a Comment

Survey results and learning styles

April 21, 2008 at 7:29 am (Survey, assessment, pedagogy, use cases)

Two questions on the survey targeted the issue of learning styles. I was trying to get a sense of where students situated themselves in relationship to this style of presentation of information.

I prefer traditional lecture as a source of contextual information.

This question got the most uniformly diverse response of any of the questions. For the larger class, results were exactly symmetrical – 1 person strongly agreed, 1 strongly disagreed, 3 people agreed, 3 people disagreed, and 6 people checked “neutral”. For the literature class, there was one response per category for all but “strongly agree”.

I find it interesting that the same person who indicated that they strongly preferred traditional lecture also provided the most negative responses in other areas. (see my response to their comment here.)

I consider myself a visual or kinesthetic learner.

This was surprising. According to the VARK project, approximately one third of the population has a visual or kinesthetic learning preference. However, in response to this survey, two-thirds (9) of the cinema students either agreed or strongly agreed with this statement. 4 students were neutral, one student disgreed. For the literature class, 3 of the 4 students strongly agreed with this statement (the 4th student checked “neutral”.) So, the number of students in these classes who consider themselves to be visual or kinesthetic learners was twice as high as what the VARK would have predicted.

However, contrary to my normal practice, I did not ask students this semester to take the VARK survey at the beginning. Over a third of the students have worked with this survey with me in the past; others may have encountered it with other professors. But there was no systematic approach to diagnosing or defining student learning styles at the beginning of the semester, other than my own overtly expressed attempts to cater to all learning styles in my teaching. At the same time, the subject matter itself may have biaised the sample towards visual learners – a course on cinema – and all four of the literature students have worked with me before.

From our grant proposal:

Current research into learning styles indicates that students with a preference for visual and/or kinesthetic modes of information presentation make up approximately one third of the population. These students are typically disadvantaged in higher education compared to students with a strong textual preference, or students with strong aural abilities – the two modes of information presentation that dominate in the traditional classroom. Visual and kinesthetic learners, however, will struggle in this environment. The timeline module will improve the learning experience for these students by providing them with the information they need in the format that they understand and retain the most quickly and easily. It will also enable professors who do not normally (because of their own learning styles) present in a kinesthetic or visual format to better support these students.

An important next step for testing in the Fall would be to diagnose learning styles at the beginning of the course, before the assignment itself has influenced their perception.

Leave a Comment

And now for the results…

April 17, 2008 at 1:41 pm (Survey, assessment, pedagogy)

Let’s start with two survey questions that addressed the beta test aspect of this project.

1. The timeline tool was easy to learn and use.
100% of the literature students and 93% of the cinema students agreed or strongly agreed with this statement. So we are on the right track for useability, even if we still feel that there are import improvements yet to be made. In addition, this means that frustration with the technology was not a major factor in student interaction with the assignment.

2. Participating in a pilot project of this kind was an exciting opportunity.
Answers to this question reveal a split between the two courses, while still falling principally on the positive side. 36% of the cinema students agreed with this statement; 57% chose “neutral,” and 7% (one person) disagreed with the statement. In the literature course, 100% of the students agreed (50%) or strongly agreed (50%) with this statement.

I see two factors involved in the second question. Firstly, the literature course was the original pilot case, and we tried very hard to communicate a sense of ownership and influence to them from the beginning. Their input influenced the design and development process in real time. The cinema course, on the other hand, I believe felt more like guinea pigs and less like co-designers. This is in part because we didn’t bring them into the process until 5 weeks into the semester, and in part because with the larger class, individual input was much reduced.

The purpose of this question was to discover whether student biais against participating in an experimental assignment was a factor in its success. The overwhelming majority of neutral or positive responses would suggest that student biais was not a factor.

Leave a Comment

Comparison of two courses

April 17, 2008 at 9:30 am (Survey, assessment, pedagogy, use cases)

I’m going to start posting the survey results to the blog. It seems unmanageable to post it all at once, so I will take it a few questions at a time, to allow for analysis and discussion. In order for this analyis to make sense, however, I need to begin by outlining the differences between the way the timeline was used in the two courses. These differences had a measureable impact on the survey results, and I will be referring to them frequently.

The principle thing I would like to highlight here from the very beginning is that the timeline assignment was more fully integrated into the literature class than into the cinema class. As we will see from the survey, this did not result in a failed assignment in the cinema class. It could, however, explain the less uniform response from the students in the cinema course. At the same time, it is also important to note that because of the amount of time a show-and-tell session can take with 17 students, it was much less practical to assign the timeline every week in the cinema course, whereas with 4 students, this was quite effective in the literature course.

Leave a Comment

The other side of the coin…

April 4, 2008 at 11:42 am (Survey, assessment, pedagogy)

I’m going to close down the survey at the end of the day today. I’ve got 76% response for the cinema course, after 3 nag notes. In the mean time, I’d like to share another comment that came in this week.

“At first I really wasn’t too sure about this exercise and how it would add to my experience in this film class, but I was astounded by how much I enjoyed gaining a better understanding of the historical context of these films/filmmakers. Plus, on just about every assignment, a few things popped up that were really entertaining and gave interesting insight into the social culture of the time these movies came out.”

Obviously, this is a comment I find tremendously exciting. And it corresponds to the preliminary data – 80% of respondents so far either agreed or strongly agreed with the statement “I learned more information researching my post than I posted to the timeline.” And 100% of respondents either agreed or strongly agreed with this statement: “Historical or contextual information is an important component of a literature/film course.”

There is a chicken-and-egg aspect to this last question, however. Did the students come into the course uniformly strongly believing in the importance of contextual information, or have they come to believe this as a result of the assignment? Unfortunately, I didn’t have a question that would allow me to estimate that. However, I feel that the 100%, especially in the context of the comment cited above, strongly suggests that the exercise itself has informed their conviction.

Leave a Comment

Student prejudice against peer review

March 31, 2008 at 9:15 am (Survey, assessment, pedagogy)

Only half of the students have completed the surveymonkey, so I’m not going to post data yet. Today, however, I’d like to talk about a comment that came in with the surveys submitted so far that I feel really highlights some of the underlying issues at stake for this kind of tool in education.

Here’s what the student said:

“Frankly, I don’t think this is a very good tool. Who cares what a bunch of dumb kids think is relevant to the course? I take classes in order to be taught what the relevant information is…please stick to comprehensive lecturing.”

Wow. Well, my first reaction was dismay, obviously. But then I realized that this comment is fantastically useful, is in fact the gold standard comment, because it puts into words something that MORE than one student feels, something that frequently mitigates the success of technology-driven group knowledge assignments. In a school that places a very high premium on student evaluations as part of the tenure and promotion process, this is a non-trivial consideration for faculty who wish to incorporate new technology into their course structure.

The issue at hand is that blogs, wikis, forums, potentially voicethreads and now timelines all attempt to leverage the potential for social networking supposedly inherent in Web 2.0 to achieve pedagogical goals. Their use in this context is predicated on two basic assumptions. Firstly, these exercises try to extend the benefits of discussion-based teaching beyond the classroom; they are thus predicated upon the idea that discussion, as opposed to lecture, is the ideal pedagogical model. Secondly, they assume that the group effect will always be positive, that students exposed to each others’ work are not equally influenced by good and bad student work, but will collectively move towards a group norm that more closely resembles the best work.

It’s pretty clear which side of this discussion I’m on. The student comment on the timeline exercise, however, serves as an important reminder that the students themselves are not necessarily in agreement with these two hypotheses. Indeed, some of our students consider all course time spent listening to peers as filler at best, an active waste at worst. They do not trust their peers as a source of information, and they do not distinguish between what the instructor says, and what the student learns. In other words, the critical process is unproblematic; it is merely a question of receiving quality information in the first place, and not of one’s own ability to find and evaluate that information. Everything else is simply inefficient.

Finally, there is a learning styles issue at stake here as well. Lecture is a traditional format for presenting information, hallowed, suede-patch professors professing away. This has an ideological effect upon those students whose own learning styles correspond to this method of information presentation. By this I mean that since lecture is the traditional structure, and since this structure works for them, it must therefore be the one true right and natural way, the most efficient format. Bolstered by the evidence of centuries of pedagogical tradition, the lecture student can only be skeptical and impatient with claims that other methods are useful or necessary for other students.

3 Comments

Bring in the surveymonkey!

March 27, 2008 at 6:19 pm (Survey, assessment, pedagogy)

I sent out a surveymonkey to all of the students in both classes today. I thought that it would be good to do this at a point where they have sufficient experience using the timeline tool to respond (well, in my estimation) but far enough from the end of the semester to keep it distinct from the standard course evaluations.

Here are the questions for the survey:

1. Please read the following statements regarding the CLEo timeline tool and indicate the degree to which you agree with each one.
(Options: Strongly agree, Agree, Neutral, Disagree, Strongly disagree)

a. The timeline tool was easy to learn and use.
b. Researching timeline posts enriched my experience of the course.
c. I learned more information researching my post than I posted to the timeline.
d. I learned a lot from other students’ posts during class presentations.
e. I learned a lot from other students’ posts outside of class.
f. Presenting and discussing timeline posts was a productive in-class activity.
g. My research skills in this subject area improved as a result of the timeline assignment.
h. Historical or contextual information is an important component of a literature/film course.
i. I prefer traditional lecture as a source of contextual information.
j. The timeline assignment helped me to feel more engaged with the primary course material.
k. The physical wall was a necessary component to the overall exercise.
l. Using onscreen in-class display of the timeline elements was just as effective as taping printouts to the wall.
m. The purpose of the assignment was clear to me from the beginning.
n. The purpose of the assignment is now clear to me.
o. I would like to use the timeline again in other courses.
p. I consider myself a visual or kinesthetic learner.
q. Participating in a pilot project of this kind was an exciting opportunity.

2. How much time on average did you spend researching and posting to the timeline per assignment ?

a. Less than 30 mins
b. 30 mins – 1 hr
c. 1hr -1.5 hr
d. Over 1.5 hrs

3. Approximately how many sources did you consult per post?

a. 1-5
b. 5-10
c. 10-20
d. 20+

4. How often did you consult a non-internet source, in addition to electronic resources ?

a. Never
b. Occasionally
c. Often
d. Almost always

5. Please use this space to make any additional comments or suggestions not already covered by this survey.

I’ll post the results when they come in.

Leave a Comment