Calibrated Peer Review(tm)

Orville L. Chapman,

Arlene A. Russell, and Michael A. Fiore

University of California, Los Angeles

I designed Calibrated Peer Review(tm) (CPR), Michael Fiore wrote the code, and Arlene Russell has made it work in the classroom and has taken charge of the workshops that teach people to use CPR and to write CPR assignments.

The design of CPR is based on peer review of scientific manuscripts and proposals. The design permits only calibrated, competent peers to review student documents. Calibrated Peer Review(tm) ensures careful reading for content, and it makes students conscious of style issues in writing. Finally, the design of CPR does not center on teaching students to write; it focuses on writing to learn.

The motivation for creating Calibrated Peer Review(tm) stemmed from the desire to foster three key elements in science learning:

* Science Literacy

* Constructivist Learning

* Critical Thinking

When we scientists think of science literacy, we naturally focus on science, but we make a major mistake in not focusing on the literacy aspect of science literacy. Literacy is discourse based on comprehension, understanding, organization, and logic. One cannot demonstrate comprehension and understanding without literacy. Multiple choice examinations will never suffice. Writing must comprise a major part of science learning. In no other way do students learn at the same level.

Writing forces students to organize their knowledge, to put together the facts and concepts that they have learned, and to logically argue a case. These factors lie at the heart of constructivist learning. Writing forces the student to construct meaning.

Constructing meaning requires critical thinking. Critical thinking in return requires rigorous thinking, and it demands the use of intellectual models. Critical thinking involves testing ideas and data against existing models, rejecting models, and creating new models.

Science literacy, constructivist learning, and critical thinking are crucial to an understanding of science. If a student wishes to become a scientist, these factors are essential.

Consideration of these factors leads us to Bloom's Taxonomy of the Cognitive Domain:

CPR

* Evaluation-highest level

* Synthesis

* Analysis

Application

* Comprehension

* Knowledge-lowest level

Knowledge is the lowest level of learning, and it is the focus of most courses in lower division chemistry (the first two years). Most courses seek some comprehension, and the best courses try for application. The three higher levels of cognition receive little or no attention. Calibrated Peer Review(tm) impacts five of the six levels of cognition. It requires evaluation in reviewing peer documents and in self-review. CPR requires synthesis of ideas and data, and it demands analysis-critical thinking. It demonstrates comprehension and understanding through organization and logical expression. In writing the student uses whatever knowledge he or she possesses. Writing and reviewing put everything together. These factors are what make CPR great.

Calibrated Peer Review(tm) is a network-based learning tool. You access it through a browser, and consequently it is platform independent. CPR enables writing assignments in classes of any size. It has been used in classes as small as eight students and in classes as large as 400 students. It works in even the largest classes. CPR also enables shared writing assignments among institutions over the Internet. In the United States, several community colleges are sharing CPR assignments over the net.

Two features make CPR special. First CPR is discipline independent. Art, English, social science, history economics, business, law, chemistry, physics, life science, and mathematics are using CPR. Second, CPR is level independent. It serves grades 6-12, colleges, universities, and graduate and professional schools.

Six components form Calibrated Peer Review(tm):

* The Student Interface

* The Instructor Interface

* The Assignment Authoring Tool

* The Reporting Tool

* The Assignment Management System

* The Assignment Libraries

The student interface enables the student to work with the program, but it does not permit access to other features of the program. The Instructor interface enables the instructor to set parameters for student assignments, to view student and class records, and to use the authoring tool. The authoring tool leads assignment authors through the process of creating a new assignment. It ensures that all crucial elements of an assignment are present. The reporting tool provides detailed student reports after the calibration exercise and after the peer review and self-review. The reporting tool also provides instructor reports that enable inspection of individual student work in detail for every element of the student work and reports that summarize class performance on every element of the writing assignment. Assignment libraries are on-line compilation of assignments that have been peer reviewed by experts in the field and edited by people who know how to write and edit.

We turn now to how CPR works. In the beginning, the instructor gives the student a writing assignment. The writing assignment may be as straight forward as giving a journal article to abstract or data to analyze and interpret. The student writes a document and then submits it electronically to CPR. The electronic document submission may come from anywhere on earth, but it will come to one of our servers. The instructor establishes deadlines for submission, completion of the calibration documents, and completion of the peer reviews and the self-review. When the student submits his or her electronic document, CPR delivers three calibration documents for review. The student can go back and forth among these documents. The student answers content and style questions for each document. The student then assigns a score to each document on the basis of a 1-10 scale with 10 being the highest score. The instructor sets the number of correct answers for the content and style questions that he or she considers acceptable. The instructor also assigns a score to each calibration document and sets the allowed deviation from the instructor's score that is permissible. Alternatively, the instructor can opt to simply use the default settings that CPR provides. When the student has completed review of the calibration documents, CPR provides a detailed report of the student's performance on every aspect of the calibration review. Student and instructor answers to all questions are shown as well as student and instructor scores for each document. After the student has successfully completed the calibration review, CPR provides three peer documents for review. The student reviewer again answers content and style questions for each document and assigns a score to each. CPR then gives the student his or her document for self-review. The process for self-review is exactly the same. Finally, CPR delivers a detailed report of the peer reviews and the self-review. This report compares the student reviews with the reviews by the other two reviewers. CPR is a smart program. If two reviewers agree closely and the third diverges to a significant degree, CPR will put a red flag on each of the three reviewers in the instructor report. CPR also retrieves the scores on the calibration reviews for each the three reviewers and presents this in the instructor report. The instructor can thus ascertain the competence level for each reviewer.

The instructor reports that CPR provides deliver a detailed summary report for each student and a detailed summary report for the class. Both reports are broken down among calibration, peer reviews, and self-review. Students are scored on their reviews as well as their writing. Reviewing challenges the students to meet Bloom's highest level of learning, evaluation. Assignment libraries enable instructors to assign writing assignments with the investment of just a few minutes of the instructor's time.

Two substantial assignment libraries exist now: the Molecular Science Library and the California State University, Northridge, Business School Library. Both libraries are growing. Calibrated Peer Review(tm) enables joint assignment development over the Internet. Community college instructors are using this feature to develop and share assignments over the net. Two community college consortia are working in this manner. One consortium involves seven community colleges in New Mexico, and the other involves seven community colleges scattered over the western United States. Joint development across institutions via the Internet enables individual faculty members who wish to use CPR to join similar colleagues anywhere in the world.

Creating CPR assignments follows a logical sequence. The instructor must identify an important course topic for each CPR assignment. The student will invest substantial work in completing the assignment, and he or she must understand that the work is focused on an important topic. Next the instructor must locate suitable source material for the students. The source material might be a journal article to abstract, a newspaper article to criticize, an animation to explain, or data to analyze and interpret. When the source material has been identified, the instructor creates the guiding questions. This task is often difficult, and it requires careful thought and planning. You want to lead the horse to the watering trough, but you should not drink for him. After the guiding questions are in place, the instructor creates three (or more) calibration documents. One calibration document should be an exemplar; the student should always see an excellent response to the writing assignment. In my courses, I write the exemplar and use actual student documents-with the students' written permission-for the other calibration documents. The instructor then creates carefully crafted content and style questions. Finally, the instructor should have experts review the assignment, and someone who writes well should edit the assignment.

Resources to assist instructors in creating CPR assignments are available. The authoring tool leads you through the creation of an assignment, and you can contact Arlene Russell at russell@chem.ucla.edu with questions. Arlene has devoted more time to learning what makes a truly effective CPR assignment than anyone else has. She has also written a manual for creating CPR assignments: Arlene A. Russell, "Workshop Manual and Writing CPR assignments." Another extremely useful guide to using writing assignments in your course is a book by John C. Bean, "Engaging Ideas: The Professor's Guide to Integrating Writing, Critical Thinking, and Active Learning in the Classroom," Jossey-Bass, San Francisco, 1996.

If you create CPR assignments, you will change the way you teach. You will sharpen your focus on what you want your students to learn. You will become more effective as an instructor, and you will be forced to ponder issues that you are currently ignoring. You will be forced to define your assessment strategy first. This latter factor has signal significance. All lecture courses could be improved if the instructor wrote the mid-term examinations and the final examination before the course began.

At UCLA, we begin CPR assignments by creating a database. Often the database is a compilation of experimental data, but it can take many other forms. We have, for example, wrapped CPR around a series of animations of reaction mechanisms. The animations did nothing for students, but the addition of writing assignments to the animations made them amazingly effective. Writing about what they see forces students to observe carefully. We then design an exploration of the database; the guiding questions form an integral part of the exploration. We want the students to learn to ask the questions that really matter. After the exploration is in place, we create the writing assignment. We work hard on the content and style questions, and finally we have colleagues criticize and edit the assignment. We often have difficulty getting colleagues to do something in the educational area, but we never have trouble getting colleagues to criticize assignments. Their criticism is extremely valuable.

Calibrated Peer Review(tm) achieves several critical learning goals. Students learn by writing; they construct meaning. CPR is not about learning to write; it is about writing to learn. Students do, however, improve their skills in writing, organization, argument, and logic. Students learn to evaluate peer documents. In learning this skill, they operate at the highest level of learning, and they also learn exactly how good or bad their own documents are. Students gain skill in abstracting, reading for content, and self-evaluation. Students also think more deeply about important topics that they are studying. But CPR also opens new doors, for example, CPR enables the introduction of topics such as science ethics, which traditional courses usually ignore.

Using CPR, students learn observation through description and comprehension and understanding through analysis and clear, precise expression. Students learn evaluation through critical assessment of peer work, and they learn synthesis of ideas and data as well as argument through organization and logical expression.

Calibrated Peer Review(tm) enables development of an instructor's own writing assignments at a cost of about 6-8 hours per assignment. CPR also fosters collaborative assignment development over the Internet. Using assignment libraries, CPR enables frequent writing assignments with little instructor effort-about 5 minutes per assignment. CPR also enables cooperative distance-learning courses, with or without an instructor. Arlene Russell has created such a course. This course is an advanced placement high school chemistry course offered to rural high school students who would otherwise have no access to advanced placement courses. The Office of the President of the University of California funds this project, which is designed around the use of learning objects from the Molecular Science Curriculum project and Calibrated Peer Review(tm).

Three things make Calibrated Peer Review special. First, CPR is discipline independent; it serves all disciplines. Second, CPR is level independent; it serves middle schools, high schools, colleges, universities, and graduate and professional schools. Third, although CPR is only two and one half years old--the first line of code was written in September, 1997-more than 100,000 students have used Calibrated Peer Review(tm).

Acknowledgements. The National Science Foundation funded the Molecular Science Curriculum project, and the National Science Foundation and the Howard Hughes Medical Institute funded the development of Calibrated Peer Review(tm).

If you have questions about Calibrated Peer Review(tm), contact me at chapman@chem.ucla.edu or Arlene Russell at russell@chem.ucla.edu.