I can’t measure learning, only the symbolic artifacts of learning.
That’s not so strange. We measure civic responsibility by how many people vote, but we can’t measure how “good” those votes are, the extent to which they are backed by intelligent thought or research into the issues. We can only measure outcomes.
As a college instructor with over 200 students and no assistants, I’m in an impossible situation to assess learning. I can only assess outcome achievement. I pretend that I can create assignments that will produce symbolic artifacts of learning. Then I grade the artifacts.
But it’s all a ruse. A student comes in with certain skills. Perhaps they already know how to learn, or have already learned the subject. They get As and Bs because they are engaged and eager to learn. When I give them an A for producing excellent outcomes, I have no idea whether I am grading their learning. What if they already knew or had examined the material before my class? What if they did all the work, but it didn’t change their mind or approach in any way? The “A” is a measurement of outcome achievement, regardless of background.
Similarly, the student who turns in no work at all may have learned something, something amazing, something that may or may not have related to what I taught, but was connected to my class. I’ve had military wives who learned, not history, but how important it was for them to have somewhere to be each day. I had a surfer guy who learned that if he synthesized information and then created his own interpretation, his conclusions were valid and could be important to others. I have students who learn that if they are polite to me and treat me with respect, they will in turn be treated with respect, and students who learn that faceless institutions don’t have to be impersonal.
If my measurement for that were individual, it wouldn’t relate to their grade in History. If my measurement were societal, I’d need to look to society. When I look to society, I see an awful lot of people behaving as if they’ve learned nothing from history. So instead I hope that they learned what they needed, whether or not I was able to assess it.
(this post related to the Rhizo15 class)
I spent last week at the Connected Courses workshop, where amazing people are creating an open online class about, basically, how to teach an open online class. The energy was such that it reminded me of my previous life working in the theatre. The design and beginning development of that class in many ways looks like our POT Cert Class looked last year. Or really, two years ago, when we ran it in WordPress, using the FeedWordpress plugin to aggregate the feeds from participants’ blogs.
But there’s a huge difference between POT Cert and the Connected Courses theatrical productions. Connected Courses is supported by a grant structure and has staff, techies, a paid director, and many resources in addition to the design team I got to be part of. A Best Play Tony would send 20 people up to the stage. POT’s certificate class has been run by community theatre style volunteers: myself, the POT leaders who wanted to work on it, and the generous moderators and mentors (faculty, ed tech folks, and others) who paid it forward after getting their own certificate or joined out of altruism, love, appreciation, or insanity.
La Cage Aux Folles original cast, 1983
We have no money to act as either motivator or thanks – this is not professional theatre. We refused money years ago, because it corrupts our artistic freedom. But this isn’t a world where people can really afford to work for pizza (or retweets or good reviews), and no one wants to run the same show year after year. We must economize. Even Les Miserables and La Cage aux Folles have pared down their production designs. I think a lot of the POT Cert cast and crew have tired of doing it.
Another reason for ennui may be because the class never seems to move forward. Even the best, most experienced online instructor could become bored with the same interpretation of the same play.
I teach History to community college students. While my methods and materials may change each term, the students do not – they are beginners in History in the same way the faculty who need the POT Cert Class are beginners in online teaching. In both cases we’re trying to help newbies, not only by teaching them methods and having them explore content. Like any good play, we have a message. For History, my message is that primary sources can be put together into diverse narratives that answer the needs of society at the time. For online teaching, POT’s message is that faculty must begin with their own pedagogy, and then select and control the technologies that support and expand that pedagogy in the online environment. It’s the reason POT exists – to start faculty with pedagogy rather than letting technology control them. We don’t want an audience who’s seen this show before.
My emphasis in the old days was design, and in many ways it still is. Our current POT Cert design was moved from WordPress to Google Sites last year in order to simplify production with a smaller crew. As always, participants had to set up and run their own blogs, but instead of their posts feeding into a central blog via FeedWordpress, they had to post a link to their work in the discussion, and conversation took place at the Site instead of on their blogs. This worked well with the 25 or so participants we had, though I will never forgive Google Sites (or the many discussion forum alternatives) for not nesting replies cleanly, as WordPress does.
The number of participants in POT Cert has gotten slightly smaller each year, likely because there are now so many alternative shows competing with what we do (and I ain’t no Michael Eisner). Unfortunately, many of these Broadway alternatives provide technology training rather than pedagogical preparation, and are developed by educational technologists rather than in-the-trenches teachers. So what we do continues to be important. We rage against the Disney-fied edtech commercial culture machine.
Last year’s class in Google Sites was hard to run with three facilitators, though it was easier than in WordPress (FeedWordpress can have problems that would frustrate anyone who doesn’t code). And even with audience participation, the show runs too long for current tastes. At 24 weeks (a badge for each semester, and a certificate for completion of two semesters), it is a bit too Angels in America.
So this summer Laura and I began to design a self-paced learning pathway, with only six units, as a static WordPress site. It’s like the TV version of our class. The idea was that people could use the pathway themselves or in cohorts at their institutions. Communities using the content could be run elsewhere if desired, like friends sitting around a living room to experience it together. Or people could do the pathway on their own, and somehow automatically get a badge. But then the Connected Courses workshop reminded me that the cohort aspect of an open, online class is extremely important. The audience must feel and hear each other for it to work. I realized that the “self-paced” idea likely wouldn’t fly.
La Cage Aux Folles 2008 revival, London
I think the new production will involve something like this:
1. Separation of the show from the audience
This allows for more flexible use of the content, and a bit more instruction. And as we write it, Laura and I sense the joy of creation. Perhaps someday it will be a book, its own script.
2. Assigned seating
Although anyone may use the content, we do need to “run” the community, and have continual feedback from other community members and ourselves. Without content, it’s just a community. Without community, it’s a disembodied course. With content and community connected, it’s a class. What happens on stage is only half, or less than half, of a successful show.
3. Audience as creators
Our current class has always required participants to blog every week, with the final post of the semester and year consisting of a list of annotated links to all their previous work. It is that post, combined with their self-assessment, that we used to evaluate for the badge or certificate, since it puts everything in one place. Calling the blog posts something like Portfolio Assignments will make that clear from Day 1.
4. Angels in the Outfield instead of Angels in America
If it has enough content, and more options for more experienced people, it should be possible to put what we need into a 12-week format.
So that’s where we’re headed, at least for now…I think we’ve got a show.
Lately it’s been kind of eerie in the world of open online classes, at least those taught by folks whose work I respect the most.
This year, our Program for Online Teaching leadership for the POT Certificate Class was down to three overworked facilitators, plus our wonderful moderators and those who let us use their videos. The class was definitely a Small Open Online Class, and since it had assigned readings and a schedule, and since MOOCs have become mega-commercial horrors, I no longer call it a MOOC of any sort anyway. For such a small group (60 registered originally), the community was fabulous, both supportive and knowledgeable. A little over a dozen learners completed and earned a badge for spring semester, and/or a certificate for the entire 2013-14 year.
The format of the class was different from the previous year (2012-13), where I had struggled (as a non-programmer) with FeedWordpress to bring in everyone’s feeds. Instead we used a Google Site. We asked everyone to post a link to their blog post at the Site, and engage in discussion at the Site instead of in the blog comments. I was able to bring in blog feeds easily using Gadgets.
I just took a peek at Alec Couros’ DCMOOC, and noticed participants in their Google Plus Community posting links to their weekly blog post. Aha!
Now, when it came to our POT Cert Class this year, there were some issues. I wasn’t delighted with the non-nested discussions in Google Sites, and we discovered that three people couldn’t really run the class effectively, even with moderators, when all three facilitators work full-time plus. But the need, at our college and elsewhere, for pedagogically-based learning about how to teach online is still there. So we decided to create a self-guided Learning Pathway instead.
Then I discovered there was already a Google Plus Community, to which I was invited, called Learning Pathways. Aha!
cc Wavy1 via Flickr, flipped
Anyway, I started creating the new Pedagogy First! Learning Pathway (work in progress is here), and my colleague Laura Paciorek has been helping. The idea is that the pathway is essentially comprised of curated content and assignments for a portfolio, and that any individual or group could participate and use the site for a “class” or individual study. Then for community, we plan to use our own POT Google Plus Community (mostly because some folks don’t like Facebook, where we also have a POT group).
So then I find that Jim Groom has created a self-directed class for ds106. Based on the successful Headless ds106, it is called the Open ds106 Course. Aha!
The synchronicity is striking, or at least it strikes me. And the trends for these classes, and many more, defy a number of assumptions I made when all this cMOOCishness and openness stuff started. I mean waaaay back in 2005 or so (which is also when I started the Program for Online Teaching).
(NB: I am deliberately ignoring xMOOCs, those based in commercial or university-commercial collaborations. My focus here is on what I’ve called Task-Based MOOCs.)
I am surprised to see that when it comes to task-based open online classes:
1. We haven’t ditched the “course”.
While we all acknowledge the importance of connections and helping people be nodes in a network, what this looks like in practice isn’t that different from any other sort of dedicated community that uses online space to interact. And we all continue to create some sort of teacher-designed content, even if it’s just a pathway through assignments or a schedule or a set of expectations.
2. We don’t have a wide variety of platforms from which to choose.
I believe that Alec Couros began designing open courses in wikis, but now is using WordPress. Jim Groom’s ds106 is WordPress-based also. So was OCTEL. Hybrid Pedagogy’s MOOCMOOC used Instructure Canvas, but for most of the open classes and cMOOCs, WordPress seems preferred. I’ve moved back to it myself with the Learning Pathway, although the discussion will be in G+. I recall when the choices were more diverse, and even a time when Alec and I were searching for an open discussion program that featured nested posts, as in Moodle and Ning.
3. The personality/persona of the instructor continues to be a factor in the success of an open class.
Jim Groom, David Wiley, Dave Cormier, George Siemens, Stephen Downes, Alec Couros, Jesse Stommel — all have distinct, interesting personalities and teaching styles. Participants, even while creating communities and connections, are guided not only by the design of the class but by the instructor’s presence. Without a teacher who inspires, an open online class is just a website.
Given these similarities, do we now have models for independent open online classes? And when it comes to designing an open online class, have we hit our stride or are we in a rut?
The whole thing is going the wrong way.
Educational research clearly indicates that effective online teaching includes elements such as professorial enthusiasm, use of multiple tools appropriate to the pedagogy, personalized attention to the students, guided pursuit of student interests, and collaboration, even to the point of creating online communities for learning.
Market forces clearly encourage the use or purchase of set systems to house online courses (Coursera, iVersity, Udacity, Instructure), taught through easily standardized modules that don’t need much monitoring, with flexibility and convenience the prized elements for consumers (oh, sorry…I meant students). They offer products, processes and support that will save universities money while they make money through their product. It is in their interest to have only their product in use, if possible tied in with their own support structures to provide a seamless experience for their…um, students.
Market forces do bow to popularity, of course. Online tools that are used by large numbers of people are integrated or plugged in to the systems. Google Docs, Facebook, and LinkedIn can be part of your class, even in Canvas! (Gosh, that’s exciting.)
So, back to that research (and you thought I’d forgotten). The approach promoted by the POT Certificate Class emphasizes the pedagogy of the individual instructor, supported by the use of appropriate tools. Everything I’ve worked on for the past decade has been in the direction of empowering instructors to empower their students to learn, by emphasizing the instructors’ knowledge and approach, realized through cool web tools that the fit the task.
But the tools will not be there if the market forces prevail in education. They will become expensive (think Ning) or unavailable. Faculty who design their own courses, and teach them using tools that fit their imagination, will become fewer and fewer. It won’t be worth the time to create a course in such an old-fashioned way.
Governments and universities are clearly aligning themselves with market forces, in desperation. That desperation is not just financial. It is, ironically, based on lack of knowledge. Little consideration is taken of the research. Market forces, in the forma of educational product companies, couch their products in the illusion of innovation, but what they offer is packaging. They make the process of learning so much less messy.
Trouble is, learning is messy. It can’t be broken down into outcomes and modules. Teachers know this, and the research shows it. But all that can be ignored, and so much money (and hassle!) saved, by having assistants facilitate carefully packaged courses instead of faculty teaching them. No need for faculty – they can be replaced with “content experts” on teams of course designers (yes, I know, it’s already happening, in lots of places).
So ultimately, programs like our Program for Online Teaching Certificate Class (now between semesters) will become anachronisms, teaching skills that are no longer used, like penmanship and typesetting. And by then, there won’t be many pens or typesets to choose from anyway. It will all fade away.
Last year, the POT Cert Class was set up in a WordPress blog, and I used the FeedWordPress plugin to pull in everyone’s blog posts. As an open class, some people participating in the class were doing cool things other than following the syllabus, so there was a separate Deep End page for their posts, where I used HungryFeed. The whole setup is explained here and here.
It became too complex for one person to handle, particularly when that person is me and problems occurred, like feeds not being pulled in and the fix being code-level.
So this year I had different goals. The plan was that everyone still have their own blog, but share the link to their weekly post in the Pedagogy First Google Plus Community.
But as I set up the WordPress blog (which was there for the syllabus, widgets, static material), I realized that there was no need for two systems. After consulting with my colleagues Jim (our blog meister), Laura (our commenter and organizer), and Todd (our captain of synchronicity), I shifted the whole thing to a Google Site. There was no need for the Community, since I had some old gadget that could do discussion.
The only miscalculation was that my old gadget nested discussions, but when I moved the discussions to the new site, they didn’t nest. This made me angry, but I got over it. In the first place, each person would be posting a link each week, and everyone would reply to them anyway. And in the second place, it wasn’t worth going back to WordPress just for that. I will miss nesting, but it had just become necessary to simplify.
It was also, as Todd pointed out to me (after his initial concerns about Site, which he works with a lot), an opportunity to learn while doing while learning while doing. That’s what POT is all about anyway. So the fact that I’d never run a Google Site shouldn’t matter, and will be a challenge, and what the hell.
Yes, we’re missing some things doing it this way. Some things are bad:
1. I created a Google Form to have people register, then I manually Share the Site with them, getting their info off the registration form. So they must have a Google account (and I thought a gmail address, but it seems OK so long as the whatever address is accessed via Google). Google does not allow people to just have comment permission – everyone who comments must have full edit permission. But I trust everyone in the class – I know they won’t mess up the site. So as names come in, I must Share with full permissions.
2. Sites will only allow use of a limited number of “gadgets”, which are kind of like WordPress’ widgets but far less flexible.
3. I could not figure out at first why people who were logged in to Google already, and were given full permissions, could not comment. Turned out there is a little, tiny, teeny-weeny “sign in” link at the very bottom of the page. I turned it into a big button. I also found out that when other people log in, they see a link to a Google survey. So I did this next to the button:
4. The list of those allowed to Share cannot be reorganized, so it’s hard to see if I’ve added someone already.
5. Gadgets are blind. You cannot see them until you save them, so you can only see them in non-edit mode, so if you use a lot of gadgets it’s hard to see what the hell you’re doing.
6. There is a Google Discussion gadget that I could have used instead of my old gadget (I still have no idea where it came from) but that would require putting everyone into a Google Group, yet another little Google box that would have to be set up. Then the forums would be gadgets. Yick.
But I’ve been able to do some cool stuff:
1. Weekly discussion links in the Navigation menu.
This took a little work, but Site’s More -> Edit site menu settings let me add pages, call them what I wanted, and put them in the order I wanted into the menu.
2. A good substitute for those feeds.
Although it’s not the central way to read everyone’s work, I was able to create a page of feeds using the same gadget over and over.
3. The site can be open yet protected.
Everyone can see everything, but only those with Edit permission can comment.
4. More cool things I don’t know about yet.
But I’ll find them!
Do I have reservations? Only about every five minutes. How could we do this? How could we put our discussions into such a space, dominated and controlled by Evil Empire #2 (or is it #3? I’m losing track)? What if Google cancels Sites next month?
Be flexible, Todd tells me. Be nimble. OK. Let’s do it.
Let’s say that David Wiley is right (and why shouldn’t he be, as king of the open course?). He writes:
Our traditional pedagogies scale poorly beyond 30 or so people because they were developed in the context of teaching 30 or so people. I think it’s safe to assume that, in the same way that our pedagogies-for-30-people degrade as the number of students goes up, pedagogies-for-1000s-of-people degrade as the number of students goes down. Pedagogies for 1000s of people probably function so poorly in the context of 30 people that we’ve never even really tried them before. In other words, we’ve never taught 100,000 people at a time before, and consequently we’ve never developed pedagogies for teaching this many people at once – the last few years just show us trying to shoe-horn pedagogies-for-30 into MOOCs and then publishing articles about the astonishing drop rates.
And I commented there:
Well, some would say that connectivist learning theory is the approach indigenous to the online environment, and it often tends to be attacked in the same breath with MOOCs. But I like the idea that something very new is needed. People keep talking about “scaling up” old pedagogies. Maybe it isn’t about scaling anything up after all, but rather creating something entirely new (maybe not even based on connectivism). Maybe the new model could be something between the one-teacher model and the peer-grading model.
So let’s give it a try. Hmmmm…in between the one-teacher model and the peer-grading model.
I’ve got it!
Start with a team of teachers or professors. They approach the MOOC like writing a textbook – each controls a section that is in their area of expertise. They write the curriculum, assignments, select all materials for that section, record a video if that’s their preferred mode (and only if that’s their preferred mode). And then they moderate the whole class with all the other profs, assessing and providing feedback to students, dividing the workload. We could “scale” based on the number of students – at 30 students per prof, that’s about 33 instructors for a class of 1,000 students.
It’s kind of what we do in our open online class-formerly-known-as-a-SMOOC (or Shhhhmooc, since we like to keep it quiet), the POT Certificate Class, where a different expert moderates discussion each week, based on readings and on their own video introduction to the material. Only this would be bigger.
Think of the employment possibilities, which take care of Jonathan Rees‘ concerns (and mine) about doing away with qualified professors when our society needs them the most. More professors employed!
Think of the quality – no work assessed by uneducated peers, but rather by real professors. No “teams” where the professors are relegated to the role of “content experts” while IDs and ed techs take the lead – they would operate in a clearly supportive role.
Think of the academic freedom – each professor controlling their own content and approach for their section of the class. There would be variety, too, of method, readings, focus.
Think of the connectivism – possible in this environment, but within a more traditionally-organized “course” that can be transferrable and assessable, and thus count for credit at real universities. Instructivist, constructivist and connectivist approaches could all be used in the same class.
It’s certainly one possibility.