When Selecting Visual Materials To Use For A Project, What Are The Best Questions To Ask Yourself?

Suggested Citation:"4 Guide to Selecting Instructional Materials." National Research Council. 1999. Selecting Instructional Materials: A Guide for K-12 Science. Washington, DC: The National Academies Press. doi: 10.17226/9607.

Section II
Recommended Processes And Tools

This page in the original is blank.

4 Guide to Selecting Instructional Materials

INTRODUCTION TO THE GUIDE

The instructional materials used in K-12 science classes provide the basis for what students can learn and what teachers should teach. The process used to select those materials is critical to providing students and teachers with a solid foundation for achievement and successful teaching. This guide is designed to help school personnel review and select science instructional materials. Specifically, this guide will be most useful to anyone appointed to facilitate the process — for example, a district or state science program administrator, a science department head, or a school principal. The facilitator will work with both the review and selection teams and eventually will seek approval from a school board, advisory board, or principal.

In some cases, individual schools or teachers may work alone to review and select materials; in other cases, communities and states may review and recommend materials for adoption lists. Since the applicable policies and logistical arrangements are highly variable, this guide cannot address all situations. Rather, the guide is based on principles and processes that individuals, committees, and communities may adapt for their unique circumstances and needs.

The review process is designed to be more open-ended than most and to rely heavily on the professional judgments of the reviewers rather than scales, formulas, and averages. As such, it is similar to the type of review used by scientists to evaluate each other's scientific work. This may be perceived to be a drawback because this type of review will be new to most reviewers of instructional materials. In addition, in order to produce a reliable review, reviewers will need to be versed in the standards, to have experience teaching the grade levels for which materials are being considered, and to have the knowledge and understanding of science as described in national standards. In the

end, the experience of carrying out the kind of rigorous review that is common in the scientific world, requiring so much background, will be valuable on many levels. It will provide a significant professional growth experience for many reviewers, help develop a local capacity to select and implement a strong science program successfully, and contribute to developing leadership among local science educators.

Assumptions

This guide and the process it advocates are based on four key assumptions:

The selection of instructional materials can be carried out either for a comprehensive science program or for a small part of such a program. The process in this guide can be used equally well for a variety of selection needs: selecting materials for a multiyear program (for example, K-5, 6-8, or 9-12); meeting a specific goal (such as identifying instructional materials for a new ninth grade physics course); or selecting a single unit of study for part of a year.
The review of instructional materials, which precedes selection, will be based on standards; that is, specific student learning goals. Applying standards to the process makes student learning of important concepts and skills a key factor in making selection decisions. It is also assumed that local policies will determine the source of the standards to be used — national, state, or local.
A curriculum framework (see box) is in place that is based on standards and describes a scope and sequence for student learning. It also is assumed that the selection process involves decisions about which instructional materials are most likely to help students achieve the learning goals given in the framework.
At least two people will review each instructional material, and a group including both experienced teachers and scientists will collaborate in the review process. Experienced teachers contribute their knowledge of how children learn, how to manage a classroom learning environment, and the particular challenges of the local

student population. Scientists can contribute their broad knowledge of science content and scientific inquiry and can be particularly helpful in reviewing the importance of the content and its accuracy.

Review and Selection Process Overview

The review and selection process in this guide differs from some other processes in that it has been designed to rely on the individual and collective judgments of the reviewers, not on checklists, scales, or rubrics. The judgments are based on standards, and incorporate evidence about how likely it is that students will learn through use of the materials. The final products include a review team summary report and recommendations to the decision-making body. Provision is made for consideration of the costs of the materials and reviewer opinions about the need for teacher professional development. These processes are designed to be flexible to suit various purposes, timelines, and available resources.

The review process generates information about the quality of instruction units — the building blocks of a complete science curriculum. The selection of a collection of materials should not be viewed as the equivalent of constructing a multiyear curriculum program. For more information about constructing a complete science program, see Designs for Science Literacy (AAAS, forthcoming a) and Designing Mathematics or Science Curriculum Programs: A Guide for Using Mathematics and Science Education Standards (NRC, 1999a).

The review and selection process presented here is written as a guide for the person responsible for organizing and carrying out the task, the facilitator. The complete process is made up of five steps:

A Facilitator Plans the Review.
Training Reviewers.
Carrying Out the Review of Materials.
Selecting Materials.
Evaluating the Process and Results.

There are consequences for omitting any of the parts, some of which are discussed in sections entitled "Constraints and Cautions." If, over time, the entire process is implemented, and increasing numbers of teachers and community members have an opportunity to participate, the local capacity to select effective instructional materials will be greatly enhanced.

STEP 1: A FACILITATOR PLANS THE REVIEW

As the facilitator, you should begin planning at least a year before final instructional material selections are scheduled to be made. During this planning time, you will be gathering data about the effectiveness of existing science education programs, becoming familiar or reacquainted with state and local policies concerning instructional materials selection, and constructing an action plan and budget. In the process, you will be contacting school personnel and community members for information and opinions, as well as building awareness of the existing program and the possible need for changes.

Recommended Process

Policy information. Compliance with policy is necessary to gain final administrative approval and access to funds for new instructional materials. For example, you will need to know whether your state produces lists of materials from which you must select materials and when state and local funds will become available. Information about deadlines can be especially important in budget planning and for avoiding unnecessary delays. Find out how flexible the policies and regulations are and the consequences of not conforming to policy. Take advantage of the Internet, conferences, and publications to stay current.

If your local plans and needs conflict with state policies or regulations, you have time to build administrative and community support for solutions. Find out about policy waivers and the recent history of how many have been granted. Talk to local administrators about the options available and your concerns in order to gauge their support. Make sure you know the history of local selection practices.

Budget planning. Each review situation will have unique policies and resources for completing the review. At a minimum, develop a budget for two days of training prior to the actual review — one to understand the process and define the criteria and one to do a mock review. In order to make a rough estimate of the time that will be required to do the review, use the following guideline taken from field-test experience: three hours per reviewer (use a minimum of two reviewers) to carry out a review using three standards on one piece of instructional material that is designed to support about eight weeks of the school curriculum.

These minimal time recommendations assume that:

some community scientists are already informed of and involved in

science curriculum planning, implementation, and evaluation, and therefore are comfortable working with school personnel;

your potential reviewers have a reasonably deep understanding of standards; and
professional development for science teachers and ongoing community outreach has developed a broad common understanding of effective science education programs and practices.

If the preceding capabilities are not available, you will need more resources for as many capacity-building activities as possible.

Be sure that you have provided funds for the staff required for the extensive preparation and facilitation of the review and selection processes. Also plan for the time and associated costs required for community outreach activities. Obtaining and organizing the materials to be reviewed can be very time consuming. Your budget should adequately allow for this task and any shipping or storage fees that may be necessary.

Coordination with other science education initiatives. Contact those persons responsible for curriculum and instruction inside and outside your immediate program. Use their advice to compile a broad account of local science education efforts, including a history of recent professional development in science, sources of current funding, and projects and programs in science teaching and learning that are under way or planned. Research new science education initiatives being discussed or to be launched soon in the region or state. Coordination with the plans and proposals of others involved in science education in your area may enable you to share resources for recruiting and training reviewers, developing community support for the science program, and planning for the successful implementation of the new program. Become familiar with the processes used and lessons learned by colleagues in other disciplines who have recently completed instructional materials selection. Make a written summary of these findings. These will be useful later in training reviewers and making presentations to administrators and community groups.

Data collection. Compile and analyze evidence on current student achievement in science, teacher opinions on what is working, elements of the science program in need of revision, and community perceptions of the science program. An anonymous survey of the materials that teachers are actually using may be necessary, since the curriculum prescribed by current policies may not be the one that has been implemented in the classroom. A survey of parents and students will not

only collect useful data, but also increase interest in the review, selection, and approval processes.

In addition to the basic reporting of standardized test scores, a study of the item analyses can provide useful data on student achievement. This information is usually provided along with the overall scores to school administrators. Professional development in how to interpret and apply the test-item analysis information is useful for principals and teachers, who are then better prepared to provide information on student achievement. In regions that disaggregate the test scores in a number of ways — by gender, race, courses, or classrooms — it is possible to further pinpoint needs that should be taken into consideration in selecting instructional materials.

Another source of data related to student science achievement is enrollment data in upper-level science courses, in which students enroll by choice or by meeting prerequisites. Improvements in the science courses should show a trend to increased demand and enrollment for advanced courses, as well as an increased participation of currently underrepresented minorities.

The information collected before the review will help influence final selection decisions and provide compelling background information in support of your recommendations during the approval process.

Identification and involvement of community stakeholders. Support from influential members of the community will be critical when recommendations for the ultimate selections need to be approved and when the new materials are introduced into schools. Selected local scientists and engineers from industry, faculty of local colleges and universities from both the education and science departments, and leaders of science education programs can be made members of an advisory board, along with teachers, students, and parents. Some members of this board may become reviewers and trainers. Participation in the advisory board and in the review and selection process will help educate community members about the curriculum, standards, classroom needs, and available instructional materials.

Involve the community in learning about the science program through district, school-level, and community activities such as open house events, community meetings, and newsletters. Educate participants about program goals and the science standards and gather opinions and suggestions. Keep community members informed through periodic updates using all of the news media available in your community.

Recruit reviewers. Choose highly qualified people whose judgments can be trusted to help increase student achievement in science. Selection criteria should include science content knowledge, demonstrated knowledge of effective teaching practices, and depth of knowledge of science standards. Individuals who have participated in professional development in science will have a common base of experience. Recruitment will be enhanced by including a description of the training to be provided and the professional growth benefits of participation.

Science subject matter knowledge is the most fundamental requirement for reviewers. Teachers often will have acquired this knowledge through classes and experience outside the district's professional development program. Therefore, be sure to collect background information on all potential reviewers, including their college majors, previous experience, and summer internships, through an application process.

To identify a pool of potential teacher reviewers, obtain information on participants in past professional development for science teachers. This may also be a useful exercise for identifying scientists and university faculty who could serve as reviewers. Community advisory groups and partnership activities may also yield potential reviewers, such as practicing scientists and engineers. By all means, try to identify those who have had experience working with school personnel. Consider requesting information from each potential reviewer on possible conflicts of interest and sources of bias, such as participation in professional development sponsored by publishers, past and present consultant agreements, or experience in publisher field tests. Reviewers need not necessarily be excluded because of these activities: when the team convenes, possible biases and conflicts of interest should be declared by each individual and that information then used to avoid potential problems.

Build the capacity of the reviewers. The success of your review and selection process depends on the depth of knowledge of the reviewers — of science subject matter, standards, and effective science teaching. Invest as much as possible in building this knowledge and experience. These professional growth opportunities need not be limited to the reviewers. Wider participation will not only build capacity to review new materials but, more broadly, to accept and implement them.

Resources outside your immediate locale can help you build the necessary capacity. Various organizations provide leadership development opportunities,

many focused on improving science and mathematics education. For example, Project 2061 offers extensive training in the review of instructional materials, which makes for excellent facilitator preparation even when not possible for all reviewers. National or state organizations may offer professional development on the Standards, Benchmarks, and state standards. Universities may offer seminars on how children learn and the efficacy of various assessment strategies. Partnership programs with local science and technology organizations can provide important information on current scientific knowledge and practices.

Pilot-test materials. If there is sufficient lead time (at least six months), plan to have reviewers and others actually use materials in their classrooms. This is particularly valuable when innovative instructional strategies are represented in the materials or when the materials use new technology. Provide training and support for the use of the materials to help ensure that the pilot is a fair test of the quality of the instructional materials. Initially, pilot teachers will be strongly biased by their experiences — good or bad — with the new instructional materials. Sufficient time and frequent opportunities to discuss their experiences with others can moderate the effects of this bias on the review and selection processes.

Constraints and Cautions

If you are short on time, use the policy information and science program effectiveness data that you have on hand. Depend on existing and experienced advisory bodies and educators who are interested in science. Because short timelines are unlikely to produce much of a change from the status quo, consider seeking approval for a postponement of the deadline, if necessary.

If you are short on money, give existing advisory boards preparation tasks or at least seek their help in finding resources. If policy will allow, consider confining the scope of the instructional material review to those areas identified as most in need of improvement.

If you cannot recruit reviewers according to the criteria suggested here, plan to spend more time in training the reviewers. Sometimes members of the review and selection team are political appointees, a situation helpful in gaining eventual approval of the instructional materials recommended. Adequate training will be even more important in developing a common understanding of the task and a common background knowledge about science program goals, if the members of your team have an uneven knowledge about science education standards, effective instruction, and local policies.

If the community lacks knowledge about your science program, consider

who is most likely to affect the selection process and then target your outreach efforts to them. If your community has contradictory ideas about the need for science program improvement, do not skimp on this initial step of preparation. A well-planned and well-executed review process ultimately can be annulled by lack of community support. Schedule frequent progress discussions with other administrators to obtain their advice and commitment as well.

If you arrange for publisher representatives to make presentations to reviewers, try to provide a level playing field for large and small publishers. Give all presenters a common format to follow and forbid the offering or accepting of gifts (which is usually prohibited by local policy anyway). Remember that reviewers can be inappropriately influenced by these presentations, even if they involve only an overview of the program and its components. Caution reviewers to look for evidence to support the claims made by the publishers.

To save time and money, a common impulse is to narrow the field of materials to be reviewed by some kind of prescreening. Various scenarios were examined during the development of this guide, and each carries some risk of undermining a valid review process. The most promising current resources for prescreening are those reviews of science materials published by organizations that have made a large investment in developing both detailed review criteria and the reviewer expertise. Most notably, Project 2061 is producing in-depth reviews based on its Bench-marks for Science Literacy (AAAS, forthcoming c). These reviews compare materials according to various criteria and are available on the Internet (See Chapter 5 ''Resources for Training"). The National Science Resources Center has produced two books of recommended instructional resources, one for elementary school science, and one for middle school science (NSRC, 1996, 1998). The criteria used are provided as appendixes in both books, with the full text available on the Internet (See Chapter 5 "Resources for Training"). Another source of middle school science review information is the Ohio Systemic Initiative (Ohio Systemic Initiative, 1998).

If there is community-wide agreement on the success of some elements of the current science program (e.g., high student achievement and teacher satisfaction), it may be possible to keep those elements in place and focus the review on revising only those parts of the curriculum to be changed.

A publisher's claims of standards addressed or recommended grade levels should be viewed with suspicion. Only a careful review will reveal the degree to which the content of

standards is actually addressed in instructional materials, or how flexible the recommended grade level may be.

Do not reject too quickly instructional materials packages without certain accessories (e.g., bilingual resources and kits of hands-on materials). Instructional materials with a great deal to offer can be too easily discarded in this way. The most appropriate time to compare such support materials is during the subsequent selection process.

STEP 2: TRAINING REVIEWERS

The training of reviewers is an essential step. The goals include developing an understanding of the purpose of the reviews, establishing a common understanding of the role of standards in the review, and fully defining the key terms and criteria to be used in the review. Mock reviews provide the necessary practice, allowing the process outlined in this guide to be adjusted to reflect local needs and values.

The training of the reviewers can also serve to broaden the experiences and background knowledge of the participants, enabling them to envision science education as it could be in local schools, not only as it is. Reviewers should be exposed to recent research in science education content and pedagogy, as well as to outstanding science education programs elsewhere in the nation.

The training process recommended here has been developed through iterative field-test processes. Although the elements have been carefully selected to be those critical for producing a successful review, the facilitator may need to adapt them to meet local needs. Sample agendas, examples, and resources are provided in Chapter 5 "Resources for Training."

Recommended Process

Develop a common understanding of review purpose. The purpose of the review process is improved student science achievement in the near future. The more detail reviewers can bring to their reviews, the more they will be able to make the best choice of instructional materials to meet local needs. In order to provide relevant detail, reviewers will need to develop a common understanding of their work.

First, members of the review team should analyze all the data gathered in Step 1. They should not only discuss the data collected about the effectiveness of the current science education program but also decide on strategies to remedy shortcomings and reinforce strengths. Reviewers should also become familiar with local policies governing the curriculum selection process and reach consensus on any choices that may be open to them about how to proceed.

Develop a depth of knowledge about standards. Each reviewer needs to become familiar with the relevant science content standards (Roseman, 1997a). If the standards document has an overview, that may be a good place to begin. Most standards documents have informative text that precedes the standards, which can provide background information, references to research, and examples of the standards in action.

Be sure that each reviewer under-

stands which part of the main text of the document contains the standards to be used in the review. If your standards are not numbered, a system for referring to an individual standard should be developed (See "Numbering Standards" in Chapter 5).

This may be the first local review to be based on standards or the first experience that individual reviewers have had in applying standards. Reviewers should understand that standards are student learning goals and that the review must determine how likely student are to meet those goals using the instructional materials that will be under review.

Materials do not teach by themselves; so reviewers will need to judge how successfully local teachers will be in using the materials to help students meet learning goals. Reviewers should base their judgments on the explicit guidance and support the materials provide for the teacher — the teacher's guide, lab manual, directions for each lesson, overall format and organization — as well as on the availability of professional development. The reviewers are encouraged to make comments about the knowledge and experience of local teachers in the "Summary Judgment" and "Additional Information" sections of their reviews. The comments recorded for the materials eventually selected will be very helpful to those who plan the professional development that will be required to help the teachers use new materials effectively.

Most instructional material units address more than one content standard, and some will address a standard only partially. The purpose of the review is to evaluate materials against the two to five standards of highest priority. Later, during the selection process, decisions will be made on how best to put together a sequence of instructional materials that meet all student learning goals.

Incorporate the use of selected reference materials. During training activities, introduce reference materials and model their use. The Standards (NRC, 1996) are especially helpful in describing inquiry, providing a broad description of each subject area at K-4, 5-8, and 9-12 grades and outlining good science teaching practices. In Benchmarks (AAAS, 1993) the chapter format enables a reviewer easily to survey a content area from kindergarten through grade 12. Either Standards or Benchmarks should be used to supplement local content standards, particularly when the local standards are lists of topics rather than descriptive of what students should understand. In addition, for some content areas, the Benchmarks chapter on the research base is a convenient reference on children's ideas and recommended teaching strategies.

Information and reference materials obtained from recent professional development in pedagogical or management areas may be helpful. For example, it may be desirable to consider the compatibility of the reviewed science instructional materials with recommended strategies for planning lessons, using cooperative groups, planning an integrated curriculum, developing language skills in content areas, or addressing the needs of bilingual and special education students.

Information about the mathematics curriculum may be necessary to help reviewers determine whether students' knowledge of mathematics is likely to be consistent with demands of particular science instructional materials.

Introduce the review forms. "Directions for Reviewers" is part of Step 3, and all review forms can be found at the end of this chapter. Form 1 is used to record the standards used for reviewing a specific instructional material, as well as the final rating for how the materials address each standard. Forms 2 and 3 include the six criteria that comprise the review — two on science subject matter, and four on student learning or pedagogy, and are to be filled out for each standard. Form 4 asks the reviewer to provide a summary rating. Form 5 records information from the reviewers on which components offered by the publisher they believe are necessary and the professional development needed to support the use of the materials. (See Figure 1 for an overview of the review process.)

All forms can be found at the end of this chapter. If you wish to customize the review forms just described, an electronic version can be copied from the National Academy Press website (<http://www.nap.edu>). It may be especially important to add to or otherwise revise the directions to include specifics about local meetings, process details, or resources.

Define criteria for student learning. The four student learning criteria in Form 3 of the review tool — active engagement (3.1), depth of understanding (3.2), scientific inquiry (3.3), and assessments (3.4) — must be discussed and described by all reviewers. During field tests of the process provided in this guide, group-generated definitions of these terms produced better and faster reviews than the use of prescribed definitions. A recommended process for generating these definitions is provided in Chapter 5 "Resources for Training."

Record, reproduce, and distribute reviewer-generated definitions of criteria produced during this part of the training. Plan to attach these to each review tool packet for reviewer use during the mock review (see below).

Provide examples of how to cite

FIGURE 1 Review Process

evidence. The review forms ask reviewers to provide evidence, including examples, explanations, and references, to back up their judgments. Clear and complete citations will be vital to the selection process. Examples of good and poor evidence help reviewers understand what is meant (See "Citing Evidence" in Chapter 5).

Practice by doing a mock review. Practice is essential before beginning the actual review. Without it, the first review will become the training experience. For the mock review, choose a sample of high-quality instructional materials of sufficient complexity to simulate actual review challenges. For example, this could be a four-to eight-week module that contains a teacher's guide, student materials, assessments, and optional supplements.

Provide copies of Forms 1-5 of the review forms and "Directions to Reviewers." Depending on your schedule, you can answer any questions the reviewers have on the entire process before they begin their mock review, or you can proceed to answer questions one step at a time.

Have reviewers discuss their results after each section. To resolve concerns, refer back to the worksheets, such as "Definitions of Criteria" and "Directions to Reviewers," and to reference materials. Some calibration of the review process is desirable, and the discussion will be quite helpful to the reviewers. However, complete agreement on how to apply the criteria is not desirable. Diverse individual reviews, guided by the standards and backed up with evidence, will produce the most comprehensive, useful results.

Plan for reflection and evaluation of the process. Convene the reviewers at the end of the process to discuss its benefits and drawbacks and how to improve the process. Sometimes the review participants do not participate in the selection process that follows. If this is so, provide them with information about the future uses of their reviews.

Constraints and Cautions

If training for this process is minimal (for reasons of time or budget, or both, for example), reviewers will be likely to produce widely varying reviews and recommendations, and the result will be a prolonged and possibly confusing selection process.

If your standards document is sketchy, the reviewers may not have sufficient information to understand exactly what students are supposed to know or do to meet the standards. This will make review results highly variable. In some cases, the official assessments, used in the region provide more information about what students should know or be able to do. But using assessments as a substitute for stan-

dards is risky. The content addressed in assessments is usually quite narrow, and the content or form of the tests may change with little or no notice. If needed and feasible, make a group decision to supplement your standards with the appropriate sections of the Standards or Benchmarks.

Be alert to the possibility that, while reviewing, reviewers may encounter unfamiliar formats and pedagogical styles. This may cause reviewers to dismiss innovative materials that could be effective (Bush et al., forthcoming).

STEP 3: CARRYING OUT THE REVIEW OF MATERIALS

In Step 3, reviewers identify applicable standards and analyze each potential unit of instructional materials to determine whether the learning goals will be met. The judgments of experienced teachers, informed and focused by training for the process, will be used. The judgments of scientists concerning the accuracy and significance of content and approaches to scientific inquiry — likewise informed and focused by training — will also be harnessed.

You may want to invite others, including school board members and district administrators, to observe the first review session. Doing so will help these key stakeholders become aware of the magnitude of the review task, the qualifications of the reviewers, and the focus on student learning goals. This knowledge will help later on, when these same individuals will be involved in making final decisions, and will help them educate the community about the integrity of the process.

Recommended Process

Make decisions about materials needed, reviewer assignments, and time needed. In assigning materials to reviewer teams, take into account the time required. For planning purposes, estimate that a review of a set of instructional materials covering about eight weeks, against two or three standards, will take at least six hours (when there are two independent reviewers). See also "Budget Planning" in Step 1.

Comprehensive instructional materials packages, such as a yearlong seventh grade science program, will require multiple reviewers. First, decide which standards must be met by the instructional materials and, if feasible, prioritize the standards. Assign each team of two independent reviewers one or two of these standards, which they will then apply in a review of the entire program. This approach ensures that the content coverage and accuracy are given priority. Their review of the student learning criteria should then be carried out for the sections in which their assigned content is found. A conference among all the reviewers is likely to be needed to address overall concerns, such as identification of any gaps, recommended components, and likely needs for professional development. The Review Team Summary (Form 4) will need to be extended to include more standards. In addition, if the reviewers engage in a wide-ranging discussion about the program, it would be helpful to attach a written summary to Form 4.

Each reviewer should be provided with at least the teacher's manual and the assessment materials. For other

components, such as materials kits, videotapes, supplementary materials, and student books (if student books are reproduced in the teacher's manual), one set per team may be adequate. When videotapes or other media materials are an integral part of instruction, make the appropriate playback equipment readily available. If materials include software, CD-ROMs, or probeware, it is advisable to have a technical troubleshooter available.

Be sure to communicate to reviewers your arrangements with the publishers, which usually require that the samples be returned promptly in resalable condition.

If you have provided for adequate release time and space, the reviewers will be able to come to a central site to do the reviews. This way, the complete set of materials can be made readily available, but, more importantly, the reviewers will be available to one another. If the reviewers will be working independently off site, plan to facilitate communications with you and with each other. When the reviewers are finished with their individual reviews, you may want to schedule time for a conference among the reviewers of one set of materials. Each reviewer has made independent decisions, but defending those decisions to others and listening to other opinions may strengthen the review process. If there is a broad range of ratings, reviewers should not be pressured to change their original rating unless they find they truly overlooked or misunderstood something. Alternatively, you can convene a conference only when the reviews indicate a need. Looking back on the quality and sources of evidence cited to support an overall judgment should reveal why the reviews differ and will provide discussion points for a conference. In some cases, it may be necessary to carry out another independent review because of disagreements.

Decide who will identify the standards to be used for each unit of instructional materials. This can be done either by you or by the reviewers. If done by you, the review will get off to a faster start. However, this standards selection is a very time-consuming task requiring reading of many materials. Each pair of reviewers assigned a small number of instructional materials can also accomplish the identification and prioritization of standards themselves. Even though it can be a disorienting experience, this approach actually produces more autonomous, flexible reviewers who appear to understand the framework, standards, and their task better. Allow plenty of time and a flexible schedule for this task. (Also see "Directions for Reviewers" below.)

Revisit the purpose and steps of

the review. During training, you discussed purpose and walked through a mock review. Now you may want to create a flow chart of all the steps in a review to help keep reviewers oriented. (Also see Figures 1 and 2.) It is not uncommon for reviewers to need guidance about what to do next after having been immersed in the detailed examination of materials. Include directions on how you would like the reports filed and when the reviewers of the same materials may confer.

Conduct the review. Provide each reviewer with necessary tools, including the Directions to Reviewers, their criteria definitions, resource materials, and access to all components of the assigned materials. Reiterate how and when to communicate with other reviewers and how to get questions answered.

Compile the review data for each instructional material. Prepare packets of review results for use in the selection process. First, clip together a packet of all review forms for one piece of instructional material from one reviewer. Then gather all reviews for one piece of instructional material and record the results on the Review Team Summary (Form 6). Staple all Form 5 to the back of Form 6. Forms 5 and 6 will be used throughout the selection process. Label and file the individual reviews; they will be needed from time to time in the selection process.

Resolve any discrepant results. When considering multiple reviewer opinions, it is likely that a few will be quite different from the others. Look back through the detailed reviews and let the evidence cited there make the case. At this point, it is possible to disregard a few reviewer decisions if the reviewer did not make a convincing case; but as a general rule, it is better to include than to exclude.

Summarize the results for use in the selection process. This summary is a bridge from the review step to the selection step. You or a small group should apply judgments, make tentative recommendations, and provide a draft of the Selection Recommendations — Form 7 to help get the selection process started.

To do the ranking, look at one program element at a time, such as fifth grade life science or K-2 investigations. Gather the reviews for all instructional materials that may fulfill that program element (some materials will be considered for more than one element). Taking into consideration the opinions and evidence presented by all reviewers, rank the materials from most promising to least promising. Form 7 provides a format for recording this ranked list, with comments. Examples of comments might be ''met 4 of the 5 top standards well, professional development is needed to ensure inquiry standards are met," or "7 standards covered — 3 well, 4

incompletely" or "2 of 4 reviewers rated standards achievement 'not at all', content coverage superficial."

Reviewers will not always reach the same conclusions, and may have each made a convincing case for differing conclusions. Carry forward the controversy to the selection process where it can be openly debated and resolved. A carefully designed neutral summary of the reviews will be a helpful addition to the selection recommendations.

This organizing and preliminary ranking will help the launch of "Step 4: Selecting Materials."

Constraints and Cautions

If you have too little time or too few reviewers to handle the number of materials that need to be reviewed, you may need to review the instructional materials by sampling. By confining the review to a sample, you are assuming that the quality of that sample is consistent with the quality of the whole, a risky assumption. If you must sample, choose what to sample with care (i.e., choose the most critical element of study or perhaps the element most in need of improvement in the current program). When sampling, you may want to have reviewers "specialize" in a particular content area or confine their attention to a small set of the standards.

The review process always seems too laborious and lengthy to some reviewers. However, there is simply no substitute for verifying by careful examination that students are likely to achieve the learning goals — the standards — that the teachers and community have agreed are important. At the end of the review process, reviewers state that they understand the science and the science program much better. So, try to keep the reviewers focused on the benefits to the students and to themselves. Provide comfortable accommodations and a lot of positive reinforcement.

If you need to streamline the process, reviewers can review first by one common criterion agreed to be of highest priority. Then a full review can be done only of those materials that passed this initial screening. Applying one screening criterion will work best when there is unanimity among not only reviewers but also the community at large about the attributes of an excellent science curriculum. Rarely is an instructional materials review that clear cut, however. Upon closer examination, some of the discarded materials may have positive attributes that outweigh what were initially perceived to be weaknesses.

DIRECTIONS FOR REVIEWERS

1. Identify the standards for reviewing each unit of instructional materials.

The primary source of information will be the curriculum framework, which identifies the standards to be achieved by students in particular grades or grade ranges. For example, a curriculum framework may indicate that sixth grade students will learn about ecosystems and biological adaptations. The standards that describe in detail what middle grade students should understand and be able to do are then the standards applicable to instructional materials under consideration for use for sixth grade life science or integrated science classes.

When looking at unfamiliar instructional materials, it is not always obvious what standards they address. Consult the overview in the teacher's guide, the table of contents, the index, or sales materials from the publisher for topical references. Typically, this method of surveying will result in a very long list of standards that will not necessarily be achieved by the students, but will simply be mentioned or partially addressed. This problem will be addressed for this review by prioritizing the standards and by the recommended method for review

You may discover that the instructional materials under review meet a student learning goal that was not previously identified, but is represented in the standards. Consult other reviewers of the same materials to determine whether this is a high priority standard, and, if so, reviewers can add it to their lists.

2. Prioritize the standards.

There is probably not enough time to complete a review of all the materials using each identified standard. Therefore, with your facilitator or team, make a prioritized list of the standards you will use. Record all these prioritized standards in the table on Standards Record and Rating Sheet (Form 1). Put the highest priority standards first. Use the identification or numbering system agreed to in the training of reviewers (See "Numbering Standards" in Chapter 5).

3. Get ready to conduct the review.

Gather all the components of the materials you will be reviewing, a copy of the standards, as well as reference books on science literacy, science content for various grade levels, and science teaching. You also should have definitions of the criteria developed during your training.

Now, look through the materials, especially overviews in the teacher's sections and sales materials, the table of

contents, any assessment materials, and a list of all the components of the set or unit. "Sticky notes" are convenient for marking sections to which you may return. (You should not write on instructional materials under review, because publishers usually ask for their return in salable condition.)

4. Review what students should understand or be able to do.

Make sure that Form 1 contains your name and the title of the unit or set of instructional materials you are reviewing. Now, read the first or highest priority standard and transfer source, grade, and text information about it to the top of Form 2. Now you will begin to do a review, one standard at a time. As you begin, be sure you understand the science content and level of sophistication implied — what students should understand or be able to do. Do not proceed until this is clear to you and your fellow reviewers. Use the content expertise of the scientists on the team and consult your reference books to develop your understanding of what exactly would constitute achieving each standard.

5. Examine the materials for content coverage and scientific accuracy and importance.

Next, examine in detail the materials to look for content coverage and scientific accuracy and importance. Record what you find and where in detail on Form 2. If it turns out that the content of the standard cannot be found in the materials, record why you think so, and give a "not at all" rating for that standard on Form 4 and in the Summary Rating column for that standard on Form 1. Under these circumstances, there is no point in completing Forms 3 and 5.

6. Determine the likelihood the students will learn content.

When the content of the standard is found in the materials, continue to look through the materials for how well and how often the students are engaged in learning about that content. Look also for how well an average teacher would be supported in planning and carrying out the learning experiences. You will be filling out Form 3, using your own judgment and the definitions of the criteria developed during review training. Provide evidence for your conclusions and cite lessons, section numbers, or text, as appropriate.

If instructional materials do not match any standards, but nevertheless seem worthwhile and well designed, you may be confused. In such cases, reviewers often wonder whether the standards are in error. Although this could be the case, usually it is not, and the materials should not be considered for selection. Topics not found in the standards have

usually been intentionally deleted. Also, you may want to refer to standards for other grade levels, since sometimes the topic is found elsewhere and is therefore developmentally inappropriate for the use originally being considered.

Now, make a summary judgment on Form 4 about how likely it is that students would achieve the standard using the unit or set of materials under review. Be sure to give your suggestions for modification or additions as requested. (These will be very helpful in the selection process and for planning professional development if the materials are selected.) Record your rating in the table in Form 1, also. Make notes that will help you complete questions about professional development and essential components in Form 5.

Go back to Form 1 and choose the next standard. Transfer information about that standard to a new set of pages for Forms 2 and 3. Then proceed to judge whether the materials under review contain sufficient amounts of the content in this standard, and so on. When all of the standards listed on Form 1 have been reviewed, complete a summary rating (Form 4).

7. Additional Considerations

On Form 5 you summarize your recommendations for the professional development needed to implement effectively each unit or set of materials you have reviewed. You also identify essential components of the materials needed by the district.

Use a separate Form 5 sheet for each unit or set and be sure to identify at the top which materials you are addressing. When you address professional development, be as specific as possible and apply your knowledge of the level of science background and teaching skills of the "average" teacher. Since you have studied the materials very carefully with student learning in mind, your advice will provide important guidance for those who plan professional development.

Next, you will recommend which components the publisher offers should be purchased, if the materials are selected. Components are parts of the classroom resources that are available separately, such as teacher's guides, student workbooks, videotapes, kits of materials, laser disks, assessment packets, laboratory guides, re-teaching or enrichment materials, and software. List and then rate each, considering its importance to helping students achieve the standards.

What Happens Next?

The facilitator, working with you, will compile a "Review Team Summary" (Form 6), showing how well each reviewed unit or set of instructional materials reflect the prioritized stan-

dards. Your judgments recorded on Forms 1-5 will be used in the selection process. Those who ultimately make selection recommendations will use your summary judgments as well as your estimation of the need for professional development and essential components.

STEP 4: SELECTING MATERIALS

To move closer to the goal of making excellent science instructional materials available to students and teachers, the data and judgments collected during review must now be applied to making selections. First, the review data should be compiled and examined. Up to this point, each set of materials was reviewed only against standards. Now, the sets of materials will be compared with one another.

At this time, considerations of cost and professional development likely to be needed for successful implementation are reconciled with resources. Training and supporting the teachers in using the new materials is just as critical to reaching the goal of increased student achievement as choosing good materials.

Finally, another decision-making body, such as a school board, will usually make the final decision about which materials will be purchased. The recommendations developed through the selection process need to make a strong case, citing evidence to support the validity of the process used while focusing on the role of instructional materials in supporting student learning goals.

Recommended Process

Begin the selection process. The review can continue through selection, with the same participants, so that review and selection constitute a seamless process. However, new people will sometimes be involved at this point, due to local decision-making policies or the need to involve other stakeholders, for example. In any case, some review participants should be part of the selection process in order to provide continuity. Any newcomers should be provided with background information about the process so far and engage in a mock review in order to develop understanding of the review data.

Complete the ranking of comparable instructional materials. As described at the end of "Step 3: Reviewing Materials," the selection process begins with examination of a ranked list of instructional materials suitable for a specific grade and content area in the curriculum framework. The recommendations were drafted by the facilitator to help organize the selection process, but should now be examined and revised as necessary. In taking on the selection tasks the participants need to incorporate four elements: (1) review data, (2) information collected in preparing for the review, (3) comparative cost, and (4) professional development requirements.

1. Review data. At the end of the review process, the facilitator will have compiled the ratings of all

reviewers for each set of instructional materials using the Review Team Summary (Form 6). These data should be readily available.

It is sometimes necessary to remind participants that it is not the purpose of the selection process to do another review. Refer them instead to the quality of the evidence given by the reviewers to support their ratings. The selection process should respect reviewer decisions, not undo or redo the reviews.

2. Information collected in preparing for the review. The preparation step yielded important information about the effectiveness of the current program and the opinions of teachers, administrators, and the community about priorities for improving the science program, as well as policies governing the material review and selection process.

Note: To save time, apply elements 3 and 4 (below) only to the top-ranked materials — meaning those under serious consideration.

3. Comparative cost. Using the Comparative Cost Worksheet, calculate a cost per student for each unit or set of promising materials. This exercise should reflect reviewer advice on which parts of the program to buy. These costs are negotiable; so publishers should be consulted before making final recommendations.

4. Professional development requirements. Consider what resources will be needed to train teachers and provide classroom support for effective use of the most promising materials. Again, the reviewer advice should be taken into account, as well as any available information on the plans of those responsible for future professional development. It is highly desirable for the staff responsible for planning and implementing professional development to be part of this discussion.

For an overview of the selection process see Figure 2.

Fit the most promising instructional materials into the curriculum framework. The most desirable instructional materials for each grade and topical area of the curriculum framework have now been tentatively identified. Put those pieces in place in the framework. The final step of the selection process is studying the proposed overall program, and assuring coherence of the materials at each

FIGURE 2 Selection Process

grade level and across grade spans, while still meeting overall student learning goals. Making final recommendations, particularly for elementary and middle school, entails a great deal of back-and-forth in search of a coherent arrangement, and it may require some adjustments in the original framework to take advantage of the best curriculum materials that are available.

The selection process may fail to identify materials that address content for each grade level. Solutions include continuing to search for materials to put through an ongoing review, writing new materials from scratch, or attempting to fix materials with poor ratings. Constraints and cautions about local efforts to write or fix materials are discussed below.

Complete the selections. Now you have the product of the entire process — recommended instructional materials chosen for their carefully assessed ability to meet student learning goals. Of course, professional development and ongoing materials management are essential for successful implementation. Be sure to specify the recommended components for each selection (software, laser disk, student workbooks, assessment packets, etc.).

Prepare to present a case for approval. The results of planning, training, reviewing, and selecting now must be approved and implemented. The responsibility of selection participants continues with the requirement to communicate with and persuade decision makers to approve their recommendations. In developing the recommendations report, integrate information about the effectiveness of the current program with how the recommended selections will address related concerns. Point out that the focus of the review was on student learning goals and note any relationships those goals have to current or future achievement tests or graduation requirements. Describe the contacts made with the community and school district administrators, as well as their involvement in the process and any pertinent results or findings. Note the credentials of your reviewers and the extent of their training.

In most localities, an oral presentation will be required — most likely to the school board. Anticipate and prepare to answer questions from school board members, other administrators, and the community. You may want to have summary charts or figures on hand. Have a member of the local scientific community, preferably a member of the review and selection team, speak on behalf of the process and recommendations.

Constraints and Cautions

The selection process may result in recommendations of some instructional

materials with lower than desirable ratings. Unfortunately, high-quality instructional materials may not be available for some student learning goals or an unusual sequence or combination of requirements. These important questions should be taken into account:

Is it possible that appropriate instructional materials do exist, but were not included in the review? Are potentially suitable materials currently under development? If so, could local schools use the materials in a developer's field test?
Could it be that the topic is inappropriate for the grade level — too sophisticated or too easy for the average developmental level of students? If so, can the curriculum framework be changed?
If the final result is a mix of materials from various publishers, is the pedagogy consistent across the materials? Will implementation be unduly confusing? Are copyright infringements a possibility?
If extensive teacher education will be required, are sufficient resources available: release time, leadership, ongoing support, and evaluation?
If a local team will need to develop an instructional materials unit, are sufficient human resources — both for teaching strategies and scientific subject matter — available? Are there financial resources to support a sufficiently long period of development, field testing, review, revisions, and publication?
Could a gap in the program remain until suitable instructional materials can be found through a continuing search and mini-review? How long would local policies allow this condition to exist? Are students at risk of failing to pass required tests or meet prerequisites?

For more information on planning and implementing a science program, see Designing Mathematics or Science Curriculum Programs: A Guide for Using Mathematics and Science Education Standards (NRC, 1999a) and Designs for Science Literacy (AAAS, forthcoming a).

Be prepared to encounter community opposition, particularly if you have not informed and involved the community adequately. A facilitator with strong knowledge of local and state policy, familiarity with applicable standards, and who can show the strengths of the review and selection process should be sufficiently prepared to handle this situation. Try to avoid polarization by listening carefully and acknowledging opposing concerns. Address opposition by providing evidence from the review process and criteria to back up the recommendations.

STEP 5: EVALUATING THE PROCESS AND RESULTS

Developing your local capacity to identify and select instructional materials for the best possible local science program will require several years of effort and ongoing evaluation. As you go through each round of review and selection, lessons learned should be noted and applied to revising the criteria and the processes used. The assumptions and implied goals of the review criteria should be checked against subsequent student achievement and teacher feedback. Having just completed the selection process, you — and participants in the review and selection process — are in a position to recommend ongoing monitoring of the effects and to prepare for the next round. Monitoring may identify the need for supplementation of the choices just made.

Although in many areas the facilitator's job description changes once the science instructional materials are selected — to professional development and implementation concerns or to another subject matter area — continual attention needs to be focused on the efficacy of the new instructional materials. To implement a process of continual improvement, the new program should be monitored in a number of ways, and community involvement should be sustained.

Recommended Process

Gather student achievement data. Review how your district and state will gather data on student achievement with the new materials. The coordination of the implementation of new instructional materials with any new assessment plans, the content of professional development, and knowledge of the political climate will enhance success of the science program (DoEd, 1997a). Analysis of district or state test data, surveys of teachers, and interviews of students can provide evidence of the effectiveness of the materials in helping students achieve the standards.

Also consider more informal opportunities to gather feedback on the science program, such as teacher professional development, meetings of principals, science material distribution centers, a district-wide web site, focus groups, and classroom observations. Keep the focus on standards-based student achievement information, not only to collect convincing data but also to reinforce the message that student achievement is the goal of the science program.

Take another look at the process. If this was not done at the end of the review, reconvene those who were involved in reviewing and selecting the materials to discuss what worked and what did not. Collect their suggestions for future modifications, and with the other key feedback — such as student

achievement — begin preparing for the next review and selection round.

The review and selection teams should also discuss their experiences and recommendations with those responsible for professional development, for developing or revising the curriculum framework, and for refurbishing classroom science materials. This kind of internal communication will help develop the capacity to improve the science program continuously.

Continue to strengthen the program. During review and selection, materials may have been recommended that were not highly rated or that were considered incomplete in terms of helping students achieve relevant standards. Identify who will follow through with the suggestions gathered during review and selection. Because new instructional materials are constantly under development, you may want to schedule periodic mini-reviews to identify new materials to replace or supplement those in place.

Continue community involvement. Community interest in the review and selection process should be nurtured and sustained. Be sure that the community is kept aware of — and involved in — the new program in action. Disseminate information gathered in periodic progress reviews. Establish or strengthen a community advisory board. If the community was not wholly supportive of the process or outcome, begin now to involve key community stakeholders in discussions aimed at preparing for the next round of instructional materials selection.

Constraints and Cautions

The demands of implementing the new program may leave no staff or no time to deal with ongoing evaluation and long-term planning. Experience with the information in this guide should help science program administrators articulate their need to collect evidence of program effectiveness continually, develop capacity to understand the role of standards in the science program, keep the community informed, and plan for future reviews of new instructional materials. Resources for evaluation and long-term planning should be given high priority.

If a demand for evidence of student improvement is made in the first year or two, be prepared for student achievement data to be disappointing. Changes in education practice are multidimensional and require numerous changes, such as new teaching approaches and new kinds of materials. All pertinent aspects must change to significantly affect outcome (Fullan, 1991), and that takes time. Gather baseline data and describe reasonable expectations. Explain long-term evaluation plans and methods.

This page in the original is blank.

FORMS FOR USE IN REVIEW AND SELECTION

Forms 1 through 8 referred to in the preceding section are provided here. Each reviewer will need Forms 1-5 for each instructional material and mutiple copies of Forms 2 and 3. Forms 6-8 are used in the selection process.

To revise or customize these forms, you may wish to begin by copying the form from the website of the National Academy Press <http://www.nap.edu>.

This page in the original is blank.