Finding your way into an open online learning community

John van der Baaren ¹, Robert Schuwer ¹, Paul Kirschner ^1,2, and Maaike Hendriks ¹

Abstract: Making educational materials freely available on the web can be seen as a noble enterprise in itself but as a way to address the call for people to become lifelong learners. The world is rapidly changing, requiring us to continuously update our knowledge and skills. A problem with this approach to lifelong learning is that the free materials that are made available are often both incomplete and unsuitable for independent learning in an online setting. The OpenER (Open Educational Resources) project at the Open Universiteit Nederland makes more than 20 short courses, originally developed for independent-study, freely available from the website www.opener.ou.nl. For our research we start from an envisioned online learning environment now under development. We then apply backcasting to select research topics and experiments that form steps from the current to the ultimate situation. This paper reports on two experiments that are an extension to standard forum software and the use of student notes to annotate learning materials: two small steps towards our ultimate open learning environment.

Keywords: Open Educational Resources, peer feedback, annotations, learning community.

1 Current situation

Traditional educational resources that were originally used in face-to-face, traditional educational settings (e.g., lectures, lecture notes, syllabi presentations, computer simulations or animations, et cetera) are increasingly being recycled into open educational resources (OERs) freely available on the web. Unfortunately, open educational resources can be nothing more than the name implies: freely available resources then can be used for education. More is needed to make them truly ÒeducationalÓ. Education involves different forms of student-teacher or student-student interaction and not just student-resource interaction. Most current OERs have been supplemented with tools such as online forums to allow for student-student interactions, but it is unclear if that has really worked. The following scenarios sketch the typical hurdles for an online learner making use of OERs.

1.1 Scenario: Carol in 2008

To improve her IT-skills for her work, Carol has chosen to make use of Initial programming in Java, an OER that she found while surfing the web. This OER contains not only the course materials, but also a discussion board / forum for those making use of the materials. While working on assignment 2.2, she cannot get the Java program running. Neither the text accompanying the assignment nor the solution at the chapterÕs end helps. Carol decides to pose the problem on the courseÕs online forum. After several clicks (open a new browser, click a URL for the domain of study (informatics/computer science), click on the course she is following, click on the year that the course is being given since it has had a number of iterations at the university in which it was given, et cetera) sheÕs finally there. Unfortunately, the two questions she posted previously were still not answered and only 5 visitors had even taken the time to look at her questions. Carol hesitates: ÔWhatÕs the use?Õ Since she has no alternative, she starts: ÒQuestion 2.6 in chapter 2.2 in Initial Programming in Java gives a solution different from mine and I canÕt find my mistake. My solution isÉÓ. After posting this, she clicks around to the other posts on other courses. She finds a nice applet on another site illustrating some programming concepts in the course and wants to share this, but because of the low number of visitors she sees no use in sharing this knowledge. Carol is disappointed. Learning should be more than reading. She decides to ask a colleague at work the next day for help, but for now sheÕs stuck and cannot proceed.

This scenario illustrates the following learner hurdles:

á Several clicks are needed to reach the forum. Carol has to leave her learning context and go to a series of new environments (domain catalogue, course forum). She also needs to toggle between screens / frames in order to cut and paste her solution in her posting on the forum.

á There are few visitors to the forum and therefore, not a very good chance that her questions will be answered. This discourages both knowledge seeking and knowledge sharing.

á Because the question context is not present, it has to be formulated in the question, leading to cumbersome formulations and browsing between forum and learning material and back. Probably, the forum is unformatted text, so this makes posting a computer program even harder!

These learner hurdles lie partly in the world of usability and interface design, but also in the greatly increased transaction costs involved in studying in this way (Ciborra & Olson, 1988). Though originally coined in economics as those - often not very visible - costs incurred in making an economic exchange. In education, and especially in collaborative learning situations and ICT-use situations, transaction costs may either hamper working together (i.e., it can be more inefficient to work together than to work alone; Kirschner, Paas, & Kirschner, In press) or impede the use of ICT (i.e., due to the time that is wasted on inefficient actions and moves).

1.2 Scenario: Bob in 2008

Bob is a judicial advisor in local government. The city is considering new environmental ordinances, but Bob - though a law expert - feels he needs to become aware of the environmental and ethical implications of those ordinances. After a long and tedious search on the internet he finds an OER from a world renowned university, materials from a course on civil society and the environment. There are many interesting literature references, but unfortunately only a few are directly available online. Then he finds an assignment that uses a case very similar to his working situation, but unfortunately it is designed as a group assignment. How is he to find some learners to work on the case? Bob returns to the starting page of the course and clicks on the forum link provided. He finds only a small number of messages in the forum and only 50% have at least one response. He starts writing anyway. ÒI am BobÉ is anybody interested in working on the assignment aboutÉ..Ó. After submitting the message he closes the browser and never returns to the forum again.

This scenario illustrates the following additional learner hurdles:

á not all learning materials referred to in the OER are available online;

á open educational resources are hard to find. There exist local repositories, but there is not yet a global search engine available to find appropriate materials.

á not only the teachers are absent but also fellow learners are difficult to contact;

á no easy to use tools are made available for working together on problems;

2 Ultimate situation

To sketch an ultimate situation where the hurdles of the current situation are not present, letÕs see where Carol is in 10 years.

2.1 Scenario: Carol in 2018

To improve her IT-skills for her work, Carol has chosen to make use of Initial programming in Java, an OER that she found while surfing the web. While working on assignment 2.2, she cannot get the Java program running. Looking at the solution (one click away as a popup next to her solution) she doesnÕt see her mistake. Several visitors before her who also had the same or a similar problem left notes on their problem and assumptions about what the source of the problem might be. Reading the notes, she gets an idea about what went wrong, but wants to check this. An awareness widget on the screen shows that six peers who have indicated they have studied the course and are willing to assist others are online at that moment. Carol connects to one of them (the one who has been rated as an expert by others) and shares her idea with her. During the discussion, they come up with an alternative solution which Carol tries out and which works well. They immediately post this at the relevant place for other learners as well as for the authors so that the OER can be improved.

Here, the hurdles are gone and we now see that:

á There is an environment - an open online learning-community - where learning takes place through interaction with others;

á The environment provides one-click-access to all knowledge resources needed, both material and human;

á There is no distinction between different types of resources;

á All combinations of same/different place/time communication are supported;

á The additional effort needed to interact with the environment is minimal and no longer interferes with learning;

á Participation is obvious, unimpeded, simple, and does not need stimulation.

2.2 Scenario: Bob in 2018

Bob has just updated his blog. Via a RSS-feed he is alerted to the fact that the city has just launched a new project having both judicial and environmental issues and Bob muses about the problem of lack of knowledge about environmental issues involved on his blog. Within an hour a visitor of his blog, alerted by his own RSS-feed, provides him with a link to a freely available course on just these kinds of problems. For some weeks he works on the course every now and then. At a certain moment he starts an assignment which resembles the current project he works on. He has to work on the assignment with another student. He sees that five other students also want to start the assignment. Bob browses through their blogs and sees one of them is a biologist from India with a job very similar to his own. He invites Tilak to work with him and receives a reply within minutesÉ

In addition to the previous advantages, this scenario shows that:

á Relevant open educational resources can be found easily.

á Through blogs and feeds, seamless and effortless contact with others is achieved.

á Sharing your knowledge with colleagues worldwide has become standard practice.

3 Route to the ultimate situation

To overcome the limits of present technology and theories as well as the fact that upcoming - possibly dominant - trends are not predictable, backcasting from the ultimate situation to develop what is currently possible is a good way to begin. Backcasting(Holmberg & Robrt, 2000) is Òa method in which the future desired conditions are envisioned and steps are then defined to attain those conditions, rather than taking steps that are merely a continuation of present methods extrapolated into the futureÓ (p. 294). According to Dreborg (1996) backcasting is particularly useful when the problem that is to be studied is complex, there is a need for major change, the dominant trends are not the solution to the problem, but rather are part of the problem itself, the problem itself is influenced by many factors external to the problem, and the scope of the problem is so broad and the timeframe is sufficiently long that there is room for choice. The problem discussed here conforms to all five of the conditions. In time, better technology will enable more and research will show us what works, what doesnÕt, and why, but what this technology will entail cannot be foreseen. It is also the case here that simply relying on technology is not enough. Acceptance, cooperation and participation of learners to achieve the second scenario are also necessary and achieving this hinges on at least two factors, namely achieving a certain degree of motivation in the learners / participants in the OER-community to work with and for each other and avoiding the situation where the transaction costs of using the system (Ciborra & Olson, 1988) become so high that these learners lose motivation once they have begun.

Backcasting starts from a set of non-overlapping principles of, in our case, a sustainable open online learning community. The two principles mentioned above (motivation and transaction costs) are used in the research described here. There are other important principles which are not used for various reasons. The quality and correctness of the learning content provided is not used as a target principle because we assume that the materials provided (parts of existing distance learning courses) are of sufficient quality for unproblematic use in our current research. Another important principle is the quality and fit of the educational support provided in the materials. Although it might be argued (as we do in our conclusions) that the lack of teacher support in an open learning community will often require adaptation of the learning materials for the OpenER project it was decided to provide the materials as is. A third important principle we did not use if the need for trust a learner should have in the learning community to start using it. In our case we assumed that because the OpenER courses are provided officially by the Open Universiteit Nederland which is known for good quality learning materials, lack of trust should not be an important issue. This might very well change when we move from small 25 hours try-out courses to longer study times. Additional measures will be required to make the learner join our learning community.

Lack of external motivation to help others is often a major problem in forming such communities (online self organising social systems; Wiley & Edwards, 2002). Helping others takes time and effort, and the rewards, other than internal and intrinsic feelings of satisfaction, are often not readily available. Community members must be willing to help others, and relying on intrinsic motivation and/or altruism is not enough. To make an open online learning-community successful, membership must be both attractive and rewarding.

Attractiveness and rewards can be achieved in different ways. One way is by simply making the answer available in the sense that learners experience that asking questions is followed by the receipt of an answer. In the first scenario, Carol wanted to post something, but this was not attractive because she felt that no-one else would profit by it since she didnÕt profit by asking a question (i.e., she had received no answers). This is related to achieving a sense of relatedness (i.e., belongingness or connectedness with others). This construct has quite often been demonstrated (e.g., Ryan & Deci, 2000; see also Furrer & Skinner, 2003 for an overview) to have a positive impact on intrinsic motivation, and thus on engagement and persistence. Relatedness is characterized by fulfilment and involvement with the social world. This social aspect affects relatedness by creating a climate or culture of trust, respect, caring, concern, and a sense of community with others. In a related area Kreijns and Kirschner (2004) have studied the role of this social interaction in collaborative learning. They show that the existence of a sound social space - the network of social relationships amongst the group members embedded in group structures of norms and values, rules and roles, beliefs and ideals - is essential for reinforcing social interaction. A social space is «sound« if it is characterized by affective work relationships, strong group cohesiveness, trust, respect and belonging, satisfaction, and a strong sense of community (cf. Rourke, 2000; Rovai, 2001). The second is allowing for contributors, and especially good contributors, to receive the recognition that they deserve. Recognition-seekers are motivated when answers that they have given are rated so they can earn what could be called Õexpert pointsÕ. Other recognition-seekers could be rewarded through assistance in developing a personal/professional network by becoming a member of an Ôinner circleÕ with the more advanced students, alumnae, or experts. To kick-start a learning community, prior learners or even traditional students can participate for rewards. By setting a good example, participation by others is stimulated. This is related to the construct of perceived competence, the whole complex of beliefs about oneÕs own competences and as such is highly related to self-esteem, the evaluation of oneÕs self-concept. According to Harter (1990), perceived competence is an important psychological mediator of achievement behaviour and motivation among children and adolescents in the academic domain and has often been demonstrated to affect intrinsic motivation.

In a correlational study, children's self-reported perceptions of academic competence and personal control were found to be positively related to their intrinsic interest in schoolwork and preference for challenging school activities (Boggiano, Main, & Katz, 1988). Competence can be perceived through praise, through comparisons with other students or other indications of good performance or through meaningful effort (e.g., Henderlong & Lepper, 2002). Objective mastery praise has been shown to be better than social comparisons in affecting motivation (Henderlong, Tomlinson, & Stanton, 2004).

The concept of transaction costs is more and more used in the field of learning and instruction (Ciborra & Olson, 1988; Kirschner, Paas, & Kirschner, In press; Yamane, 1996). It originates from the field of economics, and concerns those costs, other than the direct financial costs, that are incurred in trading goods or services. Within a collaborative or cooperative learning environment, for example, these transaction costs are Òthe costs of setting up, enforcing, and maintaining the reciprocal obligations, or contracts, that keep the members of a team together [and]Érepresent the ÒoverheadÓ of the teamÉlinked to the resources (time, skills, etc.) employed to allow a work team to produce more than the sum of its partsÓ (Ciborra & Olson, p. 95). In our situation, they refer to the specific extra acts and costs that have to be taken into account when that a learner must carry out when studying, communicating with other learners and coordinating both their own learning and the communication between each other.

4 The OpenER project

In December 2006 the Open Universiteit Nederland launched OpenER (Open Educational Resources), aimed at increasing and broadening participation in Higher Education by lifelong learners. The high-quality OpenER learning materials which are freely accessible on the web is based upon the fact that they were - for the most part - originally designed for accredited university level independent study in informal learning settings, with no references to or need for a teacher, classroom or educational institution. Traditional barriers that may inhibit potential learners from taking part in ÔnormalÕ courses at the Open Universiteit Nederland such as the costs that may be incurred have been removed, while certain aspects of studying at the Open Universiteit Nederland that facilitate learning such as the availability of online and/or face-to-face tutoring and study centres have also been removed. The compact OpenER courses (i.e., 25 hours of individual study as opposed to 100 or 200 hour traditional courses) can be studied by any individual and can be (re)used by any educational institution for non-commercial purposes, without any charge. Almost 500 000 unique users visited the site since the start in December 2006 with 12% of the visitors returning to the site. A survey under 800 visitors showed that most visitors were interested in taking a free course (79%), testing their abilities to take up a university study (32%) or trying a study at the Open Universiteit Nederland (44%). Only 5% of the visitors were interested in reusing the learning materials for their own courses.

One of the ideas behind OpenER is to offer an easy and attractive entry portal to higher education. The OpenER learning materials may be just glanced at, or read intensively, or studied systematically according to oneÕs needs, anywhere, anytime, anyhow, any long. This learning experience may lead to (re)establishment of self-esteem, (re)generation of motivation, (re)discovery of pleasure with learning, and (re)assessment of oneÕs learning capacity. It may therefore tempt individuals in the informal learning context to make subsequent steps to signing up for a study in the formal educational system. One of the targets of OpenER is this conversion rate to amount to approximately five percent.

A unique characteristic is the option of a formal test for some of the OpenER courses, resulting in credits for the Open Universiteit NederlandÕs Bachelor programmes. Informal learning can, thus, be smoothly transitioned into formal learning. At the moment a formal test is possible for 5 of the 22 courses offered. The costs of a test are Û 50 for which the student can try the test twice. When passing the test the student gets a certificate worth 1 EC. Over the past six months we received 71 requests for such a test. Only 21 individuals took the test of which 17 passed and 4 failed. We have the impression that the main reason for taking the formal test is to check whether one would be capable to start a study at university level.

The materials vary from static PDF files through combinations with web pages to fully interactive web-based courses. As mentioned before, there is no support given by the institution to the users of these materials. We will however create opportunities for other users to give peer support.

5 Forum XL

Online forums are often used for asynchronous communication in educational and/or learning situations. The current uses of and possibilities for forums have the problems that were sketched in the scenario. In general, online forums can be useful in situations where learners need information, advice, feedback or even practical information that peers who are working on with the same materials could provide. But when the nature of the support needed requires tutoring or assessment acts or behaviours by a more experienced or knowledgeable learner, another mechanism is required. There is no reason for the experienced learners to actively participate in forums where novices ask their questions. The transaction costs of accessing forum software that is external to the learning materials is already high and if you have to do that several times in order to get the information wanted it will very soon become too high. Many fora associated with OER remain empty or lack recent posts which make visiting them spooky and certainly not motivating.

Forum XL alleviates this by:

1. extending the forum software to have it immediately send the questions that have been posed to expert volunteers whose replies are then sent to the questioner and added to the forum. If no expert reacts quickly enough, if the expert makes known that she/he does not have the required expertise, or if the questioner indicates that the answer to the question was insufficient, then it is resent to other experts. This immediacysolves many of the sketched problems

2. making it rewarding to provide answers which is achieved through the provision of ÔstatusÕ (e.g., kudos) for the answerer or by allowing for the giving of quality ratings to the answers.

Two procedures need to be made operational to implement Forum XL. The first takes care of acquiring a reservoir of experts; the second the question-answering.

The first step in trying to find the experts is to define what kind of expert and what range of expertise is needed. Sometimes it is best to have learners working actively on the same course, but just one or two steps ahead of others while at other times it is better to have learners who have recently completed the course. The second step is to approach the potential experts with a proposition. It is important to be clear about all relevant aspects: ÒHow much work does participation entail?, ÒFor how long?Ó, ÒWhat is the reward?Ó, ÒWho are the learners?Ó The experts who agree to become a part of Forum XL fill in a form about their expertise and the way in which they want to work (e.g., how many questions per week they are willing to answer, whether e-mail follow-up requests on the answers given are allowed or not, etc.). This information is stored in the expert database.

Now the real work begins, namely answering the questions. The following figure depicts this process.

Fig. 1 Flow chart of basic question-handling using the email extension to Forum XL

Step 1: Question is added to forum

A visitor to Forum XL poses a question in the forum. Along with the ÒusualÓ information (i.e., title and content of the posting), information is added on the domain of the question. Sometimes this occurs automatically (e.g., there is a (sub)forum for each course and the course name is used to indicate a specific area of expertise) and sometimes the person asking the question has to manually indicate the domain of expertise required. In the rest of this article, the learner who poses a question is called the question-owner.

Step 2: The question is e-mailed to a random selection of x candidates

After placing the question in the forum, an automated process randomly selects a number of candidates to answer the question from the set available for the domain. Each candidate receives an e-mail. In an HTML-formatted e-mail, the candidate can respond to the question. This response is placed in the forum as well as sent directly to the question owner. When no HTML-formatted e-mail can be used, the e-mail contains a web link to a web form with the same functionality as the HTML-formatted e-mail. The e-mail to the candidate also contains the urgency of the question, an option to reject answering the question (i.e., ÒnoÓ, Òno time availableÓ, Ònot enough experienceÓ) and a score card of the candidate (i.e., Òyou have answered n of m questions with a mean satisfaction score of zÓ).

Step 3a No answer within y days?

When no satisfactory answer has been returned within a set number of days, a new random selection of experts is made to which the question is then sent.

Step 3b Judging the answer

When an answer is given, the question-owner judges whether the answer given has helped her or him and can also rate the answer given. The rating can be made visible in the forum and can be added to the Òscore cardÓ of the person who has given the answer. When a person has satisfactorily answered a predetermined number of questions with a predetermined mean in ratings she or he can be made visible in the forum as an ÒexpertÓ, a Òpreferred userÓ or some such similar qualification.[1]

The answer is also sent to the other candidates who had received the original e-mail containing the question. It is possible that an e-mail exchange will then start between the question-owner and the person who has provided the answer, though this will occur only when the answer-owner has indicated in his or her profile that she or he is open for this possibility. The other candidates can then decide to delete the question since it has been answered, or they can choose to provide additional information or directly contact the answer-owner or question-owner or both by e-mail.

At the moment (April 2008) Forum XL is not yet linked to the real OpenER courses. This will take place mid 2008 when a large scale experiment using several courses and expert groups will begin. A small scale experiment as part of a workshop has shown that the ideas behind Forum XL were appealing to both learners and experts.

In order for backcasting to work well development should be directed at a combination of creating as flexible as possible platforms for further work and picking the low hanging fruits first (Holmberg & Robrt, 2000). We think Forum XL will improve our learning environment by preventing demotivation and often provide additional motivation while keeping the overhead of using the system minimal. Any future learning environment can easily include the functionality provided by Forum XL.

Although Forum XL was designed to help solve problems encountered by learners in an OER, the tool itself is fairly generic. The same tool can be used for different purposes, such as answering questions, giving feedback on performance, giving an assessment of a product or even of a peer tutoring interaction.

6 Looking over the shoulder

A typical alternative for asking questions is looking over a fellow-learnerÕs shoulder. This, however, is not possible online but we can make the annotations a student makes available to fellow learners. The first step in this direction we made is to record information from a limited study-group on answers to exercises/questions, their reflections on the material, the notes that they might leave and the summaries that they might make and then to make this available to the whole group. The effect we hope to achieve is to prevent demotivation in cases where a learner gets stuck. Just by pressing a button the learner gets access to the annotations other learners made in the same place in the learning material (low transaction costs).

This support has already been provided in two of the OpenER courses in PDF format. The course documents contain the notes from between five and eight fellow students. Notes can be made visible or invisible or only a subset of the notes can be made visible. A learner can also add his or her own notes or he/she can comment on the notes that are already there. At the moment, these self-made additions are only available to the individual learner and not to other learners. Figures 2 and 3 illustrate how this looks in the course.

Fig. 2. Snapshot of part of the PDF document of the course on ÒReading literatureÓ. Student annotations are combined in one document. Each student has a separate colour. The content of an annotation becomes visible when mousing over or clicking on an annotation symbol.

Fig. 3 One annotation is used by the student to introduce herself.

Comments and notes are created using the email based review mechanism available in Adobe Acrobat 8 Professional¨ (http://www.adobe.com/products/acrobatpro/ ). Students who volunteer to provide comments are sent the PDFs for annotation. The comments sent by several contributors are combined in one PDF that others can open and see.

Experimental data are being gathered at the moment. Students who studied the OpenER courses on ÒReading LiteratureÓ and ÒInvestingÓ are provided with a PDF based course which included the annotation of fellow students. After they studied the documents, they were asked to fill in a short evaluation questionnaire. At the moment we have only some qualitative feedback from a hands-on workshop suggesting annotations might be helpful during some phases of study, but may be distracting at other times. This suggests that it is important to be able to easily show and hide the comments.

The annotation sharing mechanism we used is expected to become available for all types of digital content. We think the annotation mechanism can be called a low hanging fruit because the overhead of providing access to notes made during study is very low while the potential benefit of immediate access to notes made by peers when a student gets stuck is large.

If our research shows that making annotations available is useful, a broader server-based mechanism can be set up covering all PDF-based courses. All students will be able to share their annotations, but also to comment on the annotations made by others. Technically all of this is already possible for html-based courses (see e.g. Fleck or Diigo) and with tools such as D³E, Ubiquitous D³E, and the PDF Annotation Engine. When these tools become more stable annotation support can be extended to html-based courses.

7 Conclusions

Taking a backcasting perspective has brought some interesting practical research problems to tackle. The project is evaluating an email-based extension to standard online forums that could solve the problem of lack of participation to forums supplied with Open Educational Resources. The software used in the experiments here will be obsolete in ten years. But if we find mechanisms that improve peer-support in an open online learning community, then these can easily be used in future learning systems.

The current OpenER setup tries to improve peer support using extensions of online forums but still relies on volunteers offering support to fellow learners with no other reward than a thank-you, a place in the gallery of fame or a kudo. In many situations this might fall short. An alternative might be to make providing peer support part of the regular learning process. In most professions being competent in assessing, supporting, collaborating with and giving feedback to peers is regarded as important. It seems obvious that a win-win situation arises when education is designed taking into account that some students need support, feedback or to be assessed while others need practice in providing support, feedback or assessment. Unfortunately educational research in this direction is rare (Sluijsmans (2006) being one of the exceptions). Software tools supporting these kinds of learning processes with peer support exist (e.g. Espace, Volder (2007)) but require teacher involvement.

The same can be said about the experiments with sharing annotations. In our current experiment we only added a tool for sharing annotations but a better solution might require both an annotation tool and (hopefully minor) adaptations of the learning materials themselves.

References

Boggiano, A., Main, D., & Katz, P. (1988). Children's preference for challenge: The role of perceived competence and control. Journal of Personality and Social Psychology, 54, 134-141.

Ciborra, C., & Olson, M. H. (1988). Encountering electronic work groups: A transaction costs perspective. In Proceedings of the 1988 ACM Conference on Computer-Supported Cooperative Work, Portland, Oregon, US. Available at http://doi.acm.org/10.1145/62266.62274

Dreborg, K. H. (1996). Essence of backcasting. Futures, 28, 813-828.

Furrer, C., & Skinner, E. (2003). Sense of relatedness as a factor in ChildrenÕs academic engagement and performance. Journal of Educational Psychology, 95, 148-162.

Harter, S. (1990). Causes, correlates and functional role of global self-worth: A life span perspective. In J. Kolligan & R. Stenberg (Eds.), Perceptions of competence and incompetence across the life span. New Haven, CT: Yale University Press.

Henderlong, J., Tomlinson, T., & Stanton, P. (2004). Does social comparison praise undermine childrenÕs intrinsic motivation? Paper presented at the AERA 2004 conference, San Diego, CA.

Henderlong, J., & Lepper, M. (2002). The effects of praise on children's intrinsic motivation: A review and synthesis. Psychological Bulletin, 128, 774-795.

Holmberg, J. & Robrt, K. H. (2000). Backcasting from non-overlapping sustainability principles-a framework for strategic planning. International Journal of Sustainable Development and World Ecology 74, 291-308.

Kirschner, F., Paas, F., & Kirschner, P. A. (In press). Individual and group-based learning from complex cognitive tasks: Effects on retention and transfer efficiency. Computers in Human Behavior.

Kreijns, K. & Kirschner, P. A. (2004). Determining sociability, social space and social presence in (a)synchronous collaborating teams. Cyberpsychology and Behavior, 7, 155-172.

Lambeir, B. (2005). Education as Liberation: The politics and techniques of lifelong learning. Educational Philisophy and Theory, Vol 37, No. 3, 349-355.

Rourke, L. (2000). Operationalizing social interaction in computer conferencing. In Proceedings of the 16th annual conference of the Canadian Association for Distance Education. Quebec City, Canada. Retrieved February 2, 2003 from the World Wide Web: http://www.ulaval.ca/aced2000cade/english/proceedings.html.

Rovai, A. P. (2001). Classroom community at a distance: A comparative analysis of two ALN-based university programs. Internet and Higher Education, 4, 105-118.

Ryan, R.M., & Deci, E.L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well being. American Psychologist, 55, 68-78.

Sluijsmans, D. and Prins, F. (2006). A Conceptual framework for integrating peer assessment in teacher education. Studies in Educational Evaluation 32Õ 6-22.

Volder, M. de, Slootmaker, A., Kurvers, H., Rutjens, M., Baaren, J. van der, Bitter, M., Kappe, R., Roossink, H., Goeijen, J. de, and Reitzema, H. (2007). Espace - A student peer-feedback tool enhances educational quality of assignments without overburdening faculty or adding administrative costs. Educause Quarterly, 30(3), 63-66.

Web, N. M., & Mastergeorge, A. (2003) Promoting effective helping behavior in peer-directed groups. International Journal of Educational Research, 39, 73-97.

Wiley, D. A. & Edwards, E. K. (2002). Online self-organizing social systems: The decentralized future of online learning. Quarterly Review of Distance Education, 3(1), 33-46.

Yamane, D. (1996). Collaboration and its discontents: Steps toward overcoming barriers to successful group projects. Teaching Sociology, 24, 378-383.