I can’t measure learning, only the symbolic artifacts of learning.
That’s not so strange. We measure civic responsibility by how many people vote, but we can’t measure how “good” those votes are, the extent to which they are backed by intelligent thought or research into the issues. We can only measure outcomes.
As a college instructor with over 200 students and no assistants, I’m in an impossible situation to assess learning. I can only assess outcome achievement. I pretend that I can create assignments that will produce symbolic artifacts of learning. Then I grade the artifacts.
But it’s all a ruse. A student comes in with certain skills. Perhaps they already know how to learn, or have already learned the subject. They get As and Bs because they are engaged and eager to learn. When I give them an A for producing excellent outcomes, I have no idea whether I am grading their learning. What if they already knew or had examined the material before my class? What if they did all the work, but it didn’t change their mind or approach in any way? The “A” is a measurement of outcome achievement, regardless of background.
Similarly, the student who turns in no work at all may have learned something, something amazing, something that may or may not have related to what I taught, but was connected to my class. I’ve had military wives who learned, not history, but how important it was for them to have somewhere to be each day. I had a surfer guy who learned that if he synthesized information and then created his own interpretation, his conclusions were valid and could be important to others. I have students who learn that if they are polite to me and treat me with respect, they will in turn be treated with respect, and students who learn that faceless institutions don’t have to be impersonal.
If my measurement for that were individual, it wouldn’t relate to their grade in History. If my measurement were societal, I’d need to look to society. When I look to society, I see an awful lot of people behaving as if they’ve learned nothing from history. So instead I hope that they learned what they needed, whether or not I was able to assess it.
(this post related to the Rhizo15 class)
I confess to disappointment in my recent reading of research on online teaching.
There are many articles now expounding the benefits of faculty being assisted (whether they like it or not) by instructional designers. Many of these are written by people getting degrees in instructional design or PhDs in vague areas of Education. In addition, theses like this one (1) claim that faculty who design their own courses cannot articulate design concepts (or “instructional development processes”), and therefore need serious help. Another study claims that competencies should be based on the many roles online instructors must undertake: developer/designer, educational expert, evaluator, facilitator, scientist of their discipline, lifelong learner, organiser/manager, social director, technologist. (2) It’s a wonder anyone wants to each online at all with a job description like that.
Of course, the fact that such “competencies” have been invented along with the whole field of online instructional design, may be part of the issue. And few acknowledge that the field itself contains serious conflicts of ideas and may be based on techno-utopianism. (3)
One paper (4) does indicate that some of the problems we see in asynchronous online education may be the result of students who do not have the skills to succeed in classes coming to online classes in increasing numbers. But in general, the blame for any problems in online classes falls on the instructors, through the argument that they lack proper training.
Faculty teach. Some cannot articulate educational principles as taught in schools of Education, because they are experts in their field rather than Education. Can quality online classes be developed by people who do not have degrees in education, instructional design, or educational technology? Yes, indeed – I have seen many. The major requirements seem to be passion for ones discipline, passion for the online learning space, and a willingness to learn new skills to create the desired learning experiences for students.
But the pattern in the literature shows a desire to “professionalize” online teaching, via
1. the development of a body of “research” (in this case primarily through the proliferation of doctoral theses based on small sample sizes),
2. the advancement of fields originally focused on supporting educators, but now claiming to be leading educational reform,
3. the creation of “best practices” and “competencies” (designed to ferret out “worse practices” and create standardization), and
4. the promotion of the idea that current online faculty are desperately in need of assistance.
Cui bono? Well, those earning degrees in instructional design and educational technology, particularly those who want jobs in educational administration. For-profit online universities also benefit because they can more easily justify standardized courses taught by poorly-paid staff. So do public and private universities expanding their Ed D programs, online and on-site, to bring in those grad school dollars. The proliferation of journals and associations benefits publishers (many of them closed presses). And those who already have jobs in educational administration have more ammunition to reduce the influence of independent faculty, and limit creativity in the name of accountability.
(1) Raul Mendez, Instructional Development Skills and Competencies for Post-Secondary Faculty-Designers Developing Online Courses (Capella PhD, October 2014)
(2) Diogo Casanova, Antonio Moreira & Nilza Costa, Key competencies to become an e-Learning successful instructor (Santiago Univ, Portugal, n.d.)
(3) Julian Thornton, “We will fix the deficit”: deficit theories in the literature of educational technology adoption (asccilite2014, New Zealand)
(4)Jason Stulo, Asynchronous Distance Education: The Challenge of Teaching Across Time and Space (M Sc, University of Wisconsin-Stout, March 2012)
I spent last week at the Connected Courses workshop, where amazing people are creating an open online class about, basically, how to teach an open online class. The energy was such that it reminded me of my previous life working in the theatre. The design and beginning development of that class in many ways looks like our POT Cert Class looked last year. Or really, two years ago, when we ran it in WordPress, using the FeedWordpress plugin to aggregate the feeds from participants’ blogs.
But there’s a huge difference between POT Cert and the Connected Courses theatrical productions. Connected Courses is supported by a grant structure and has staff, techies, a paid director, and many resources in addition to the design team I got to be part of. A Best Play Tony would send 20 people up to the stage. POT’s certificate class has been run by community theatre style volunteers: myself, the POT leaders who wanted to work on it, and the generous moderators and mentors (faculty, ed tech folks, and others) who paid it forward after getting their own certificate or joined out of altruism, love, appreciation, or insanity.
La Cage Aux Folles original cast, 1983
We have no money to act as either motivator or thanks – this is not professional theatre. We refused money years ago, because it corrupts our artistic freedom. But this isn’t a world where people can really afford to work for pizza (or retweets or good reviews), and no one wants to run the same show year after year. We must economize. Even Les Miserables and La Cage aux Folles have pared down their production designs. I think a lot of the POT Cert cast and crew have tired of doing it.
Another reason for ennui may be because the class never seems to move forward. Even the best, most experienced online instructor could become bored with the same interpretation of the same play.
I teach History to community college students. While my methods and materials may change each term, the students do not – they are beginners in History in the same way the faculty who need the POT Cert Class are beginners in online teaching. In both cases we’re trying to help newbies, not only by teaching them methods and having them explore content. Like any good play, we have a message. For History, my message is that primary sources can be put together into diverse narratives that answer the needs of society at the time. For online teaching, POT’s message is that faculty must begin with their own pedagogy, and then select and control the technologies that support and expand that pedagogy in the online environment. It’s the reason POT exists – to start faculty with pedagogy rather than letting technology control them. We don’t want an audience who’s seen this show before.
My emphasis in the old days was design, and in many ways it still is. Our current POT Cert design was moved from WordPress to Google Sites last year in order to simplify production with a smaller crew. As always, participants had to set up and run their own blogs, but instead of their posts feeding into a central blog via FeedWordpress, they had to post a link to their work in the discussion, and conversation took place at the Site instead of on their blogs. This worked well with the 25 or so participants we had, though I will never forgive Google Sites (or the many discussion forum alternatives) for not nesting replies cleanly, as WordPress does.
The number of participants in POT Cert has gotten slightly smaller each year, likely because there are now so many alternative shows competing with what we do (and I ain’t no Michael Eisner). Unfortunately, many of these Broadway alternatives provide technology training rather than pedagogical preparation, and are developed by educational technologists rather than in-the-trenches teachers. So what we do continues to be important. We rage against the Disney-fied edtech commercial culture machine.
Last year’s class in Google Sites was hard to run with three facilitators, though it was easier than in WordPress (FeedWordpress can have problems that would frustrate anyone who doesn’t code). And even with audience participation, the show runs too long for current tastes. At 24 weeks (a badge for each semester, and a certificate for completion of two semesters), it is a bit too Angels in America.
So this summer Laura and I began to design a self-paced learning pathway, with only six units, as a static WordPress site. It’s like the TV version of our class. The idea was that people could use the pathway themselves or in cohorts at their institutions. Communities using the content could be run elsewhere if desired, like friends sitting around a living room to experience it together. Or people could do the pathway on their own, and somehow automatically get a badge. But then the Connected Courses workshop reminded me that the cohort aspect of an open, online class is extremely important. The audience must feel and hear each other for it to work. I realized that the “self-paced” idea likely wouldn’t fly.
La Cage Aux Folles 2008 revival, London
I think the new production will involve something like this:
1. Separation of the show from the audience
This allows for more flexible use of the content, and a bit more instruction. And as we write it, Laura and I sense the joy of creation. Perhaps someday it will be a book, its own script.
2. Assigned seating
Although anyone may use the content, we do need to “run” the community, and have continual feedback from other community members and ourselves. Without content, it’s just a community. Without community, it’s a disembodied course. With content and community connected, it’s a class. What happens on stage is only half, or less than half, of a successful show.
3. Audience as creators
Our current class has always required participants to blog every week, with the final post of the semester and year consisting of a list of annotated links to all their previous work. It is that post, combined with their self-assessment, that we used to evaluate for the badge or certificate, since it puts everything in one place. Calling the blog posts something like Portfolio Assignments will make that clear from Day 1.
4. Angels in the Outfield instead of Angels in America
If it has enough content, and more options for more experienced people, it should be possible to put what we need into a 12-week format.
So that’s where we’re headed, at least for now…I think we’ve got a show.
Lately it’s been kind of eerie in the world of open online classes, at least those taught by folks whose work I respect the most.
This year, our Program for Online Teaching leadership for the POT Certificate Class was down to three overworked facilitators, plus our wonderful moderators and those who let us use their videos. The class was definitely a Small Open Online Class, and since it had assigned readings and a schedule, and since MOOCs have become mega-commercial horrors, I no longer call it a MOOC of any sort anyway. For such a small group (60 registered originally), the community was fabulous, both supportive and knowledgeable. A little over a dozen learners completed and earned a badge for spring semester, and/or a certificate for the entire 2013-14 year.
The format of the class was different from the previous year (2012-13), where I had struggled (as a non-programmer) with FeedWordpress to bring in everyone’s feeds. Instead we used a Google Site. We asked everyone to post a link to their blog post at the Site, and engage in discussion at the Site instead of in the blog comments. I was able to bring in blog feeds easily using Gadgets.
I just took a peek at Alec Couros’ DCMOOC, and noticed participants in their Google Plus Community posting links to their weekly blog post. Aha!
Now, when it came to our POT Cert Class this year, there were some issues. I wasn’t delighted with the non-nested discussions in Google Sites, and we discovered that three people couldn’t really run the class effectively, even with moderators, when all three facilitators work full-time plus. But the need, at our college and elsewhere, for pedagogically-based learning about how to teach online is still there. So we decided to create a self-guided Learning Pathway instead.
Then I discovered there was already a Google Plus Community, to which I was invited, called Learning Pathways. Aha!
cc Wavy1 via Flickr, flipped
Anyway, I started creating the new Pedagogy First! Learning Pathway (work in progress is here), and my colleague Laura Paciorek has been helping. The idea is that the pathway is essentially comprised of curated content and assignments for a portfolio, and that any individual or group could participate and use the site for a “class” or individual study. Then for community, we plan to use our own POT Google Plus Community (mostly because some folks don’t like Facebook, where we also have a POT group).
So then I find that Jim Groom has created a self-directed class for ds106. Based on the successful Headless ds106, it is called the Open ds106 Course. Aha!
The synchronicity is striking, or at least it strikes me. And the trends for these classes, and many more, defy a number of assumptions I made when all this cMOOCishness and openness stuff started. I mean waaaay back in 2005 or so (which is also when I started the Program for Online Teaching).
(NB: I am deliberately ignoring xMOOCs, those based in commercial or university-commercial collaborations. My focus here is on what I’ve called Task-Based MOOCs.)
I am surprised to see that when it comes to task-based open online classes:
1. We haven’t ditched the “course”.
While we all acknowledge the importance of connections and helping people be nodes in a network, what this looks like in practice isn’t that different from any other sort of dedicated community that uses online space to interact. And we all continue to create some sort of teacher-designed content, even if it’s just a pathway through assignments or a schedule or a set of expectations.
2. We don’t have a wide variety of platforms from which to choose.
I believe that Alec Couros began designing open courses in wikis, but now is using WordPress. Jim Groom’s ds106 is WordPress-based also. So was OCTEL. Hybrid Pedagogy’s MOOCMOOC used Instructure Canvas, but for most of the open classes and cMOOCs, WordPress seems preferred. I’ve moved back to it myself with the Learning Pathway, although the discussion will be in G+. I recall when the choices were more diverse, and even a time when Alec and I were searching for an open discussion program that featured nested posts, as in Moodle and Ning.
3. The personality/persona of the instructor continues to be a factor in the success of an open class.
Jim Groom, David Wiley, Dave Cormier, George Siemens, Stephen Downes, Alec Couros, Jesse Stommel — all have distinct, interesting personalities and teaching styles. Participants, even while creating communities and connections, are guided not only by the design of the class but by the instructor’s presence. Without a teacher who inspires, an open online class is just a website.
Given these similarities, do we now have models for independent open online classes? And when it comes to designing an open online class, have we hit our stride or are we in a rut?
I read carefully a recent article in the San Diego Union-Tribune called Online Class Takers Less Likely to Pass. I am interested in online successful retention rates, the percentage of students passing the class. For online classes at community colleges, successful retention has always hovered around 10% lower for online than traditional classes.
The 10% holds. According to the article, online class successful retention rates are about 60% at California community colleges vs 70% in traditional classes. The article wants to examine why.
But it also presents an even bigger gap. The article says:
Researchers also found that achievement gaps are exacerbated in the online world. For example, the gap between white and African-American students in traditional classes was 12.9 percentage points; that widens to 17.5 points in online courses. They said that might be partly a reflection of the digital divide, where some students don’t have access to computers and broadband.
I’ve heard this argument many times, that the digital divide makes it hard for students to access the technology they need to be successful in online classes. I think someone needs to acknowledge that this is far less true now, in this state, in this country, than anywhere, ever.
Historically, those with lower incomes tend to purchase items for social reasons, even at the risk of sacrificing quality in housing and food. This has been true since at least the Victorian era, and Thorstein Veblen even wrote a book about it in 1899. I have a number of socio-economically disadvantaged students, and they all have smart phones.
As for race, according to Pew research, more AfricanAmerican and Hispanic people in the US have smart phones than “white” people. And, if we want to get away from race and look only at income, 47% of people earning less than $30,000/year have a smart phone.
Although I don’t recommend taking an online class on a smart phone, many of my students do.
In addition, all the college’s students have access via our computer labs on all our campuses, and local public libraries.
This is not that kind of digital divide.
Someone needs to talk about three very real reasons that there is an offset online achievement gap for certain groups:
1. The primary pedagogy of online class continues to be text-based.
For most online classes, the assignments include reading text. Lots of text. And students must write for most of the heavy General Education classes, and they must do it in standard written English, at the college level, on their smart phone or a crowded library. It seems to me likely that those who do poorly with text would have a better chance of success with alternative assignments. That said, such assignments may not be considered appropriate by the instructor as a way of determining whether material was learned. In my history classes, you must write.
2. Students expect flexibility of time to mean less time.
This is mentioned briefly in the article, that students may enroll thinking an online class will be less work. I think it’s just a confusion between flexibility and total time. Self-directed students budget time appropriately, but others assume that because they can work any time, the total time for the class will somehow be different. If you ask students how many classes they can handle during a semester, they often assume they could handle more if the classes were online.
3. The lives of those struggling to meet basic needs is not conducive to the concentration required of online classes.
Here we have to make the jump from race, which is the focus of the article’s statement, to class, which is more to the point. When you are raising your siblings and carrying two low-paying jobs to feed everyone, even if you have a smart phone there may be no good space in which to work. There’s no time to go to the library. I have a number of students from military families and others with very complex arrangements at home. I have students who have been thrown out of the house and are living in their car. For these people, the time spent in a classroom may be the only time they have to focus on the material. Online classes are a poor option.
This last one is the divide no one wants to talk about, because it involves getting into a deeper discussion of poverty, and the lack of social services and public money available to help. Start talking about this, and you must talk about food stamps and living in a country that provides help for tuition at community colleges but can’t help people with hunger and living circumstances. It’s easier to blame poverty itself for failing to provide the means to buy technology.
If the reason for achievement differences were access to technology, the gap would have narrowed in this age of the rapidly increasing adoption of mobile technologies. But it hasn’t – the 10% hasn’t budged. It won’t surprise me if the larger gap holds too.
After over a decade of this, I’m looking forward to the discussion of the digital divide getting broader.
This morning I attended the session Footprints of Emergence, led in the SCoPE community out of British Columbia by Jenny Mackness, Roy Williams and Simone Gumtau based on their recent work published in IRRODL.
I have followed, and even worked a time or two, with Jenny, and am always interested in watching whatever she is working on. Since I missed the first session on November 19, I viewed the recording to catch up on the ideas. Then during the session, I had printed out a footprint map and tried filling it in for the POT Cert Course.
To oversimplify enormously, the idea of the footprint is based on a kind of map for a particular course or “complex learning environment”, and the emphasis is pedagogy and course design. The base map is a circle, with more structured, prescribed learning experiences toward the center, and more “emergent” (self-directed, expansive, connectivist) elements toward the outside, with “chaos” being the ultimate outside edge. The circle is divided into four areas: Open/Structure (the space or environment and how it’s set up), Interactive Environment (the extent of contextualization and interactivity), Presence/Writing (the learning process and product, or the way the learning is realized), and Agency (self-direction and autonomy of learning). A blank map, available in Word (I just printed out the image) looks like this:
Each quarter of the circle contains many factors that can be scaled across from more prescribed to more open (here’s one of the charts to explain each). Each can be marked on the map with a dot, and then the dots connected to make a shape. The more the shape is inwards, the more prescribed and directed the experience. The more near the edges the shape is, the more it emphasizes emergent learning. You can see other people’s examples of their courses here.
My interest at first was mapping out the design of the POT Certificate Class, because I knew that much of it is prescribed and I would like it to be more open, although that’s difficult with beginners. I would be mapping the class from the point of view of the designer. As I began, Scott Johnson, who was also in the session and has been with us at POT Cert, offered to map from the point of view of the student. Here’s mine – a footprint of POT Cert as it actually is, rather than my ideal:
Then Scott emailed me and said something about evaluations, and suddenly many possibilities occurred to me:
- POT workshops could have faculty map their courses. We could guide them through as we were being guided in this workshop.
- My students in history classes could do it, and I could see how their view compared with mine (another form of student evaluation).
- K-12 teachers could use this across the curriculum, sharing their maps with each other.
- Department members who don’t get along could map their own course to discuss differences in pedagogy.
Because what this system does, in addition to providing a way to think through ones own pedagogy, is create a presentation of ones course that can be seen at a glance and compared to others. It’s much easier than visiting a dozen classrooms or clicking through a bunch of online classes. It could spark conversations about pedagogical goals.
What it doesn’t do is dismiss the more prescribed modes of teaching and learning. Although they are closer to the centre and therefore literally less “edgy”, more controlled environments, materials and assessments are by no means considered as irrelevant. This is refreshing, as in my own experience I have found it very difficult to apply the utopian connectivist principles I love as a learner to my role as a teacher of underprepared community college students.
In the chat, Jenny commented that the idea here was balance, but perhaps it is more than that. These map lines can become fluid, changing at various times in the semester, or even for the individuals in the class. Perhaps a class begins with, for example, very limited agency, but as the course continues, that agency becomes more emergent. That’s what happens in my classes – as the semester goes on students have more and more freeedom to bring in resources of interest to them, while at the beginning things are much more instructor-directed.
Although I will undoubtedly make some adaptations, I will be using this somehow, to generate conversation by having participants actually do something (instead of just telling them to “reflect”). A light bulb went on with this – there are many places it could go.