By Margaret Williamson and Leslie Shapard

How can local and state officials be sure they buy goods and services at the best possible value for their constituents? After all, there are many different programs that have objectives they have sworn to support, and they will be held accountable to the public for any incurred expenses.

Government officials make decisions on everything from purchasing office supplies to funding research on such emerging health threats as the Zika virus. It is impossible to be an expert on everything. Where can public managers look for help? They can use peer review.

How the process works: Peer reviewers have comparable knowledge to the people who wrote the proposal or requested the funds. Reviewers are there to judge whether a proposal is feasible, meaningful, and worth the investment.

They will know if similar ideas have been tried, the success rate on those endeavors, or if the idea is a new one. They can also tell if the proposed budget is out of line in either direction.

Peer reviews can help make sense of the wide variety of proposals encountered in government procurements. Done well, peer reviews are a respected decision-making process capable of helping organizations meet and maintain high standards.

They are especially helpful for complex decision making where large dollar amounts are involved, the topic or outcome is controversial, or managers expect a large number of competitors. In these situations, it is beneficial to use reviewers who are external to an organization in order to avoid any appearance of bias or favoritism.

A Successful Process

This article discusses eight leading practices in peer review that come from our experience providing more than 25 years of peer review support to state governments, federal agencies, academic institutions, and international research organizations. Together, these practices build on one another to form the steps needed to prepare a successful peer review process.

1. Know why you want a review. List all the reasons why a review might be beneficial to you and all the stakeholders involved. Because reviews influence buying and funding decisions, their core function is to identify which proposals are most likely to succeed.

Reviews can provide more than just a score. They can also be used to provide valuable information to program managers and to vendors submitting proposals. If you plan on using reviews to provide feedback, referencing your list of stakeholders will guide you as you develop the review evaluation criteria.

2. Develop review criteria. Criteria are the questions you need answered to make informed decisions. Criteria also spell out the factors on which vendor proposals will be evaluated. Reviews should use the criteria specified in the official proposal request to evaluate proposals, but encourage reviewers to address non-scored criteria.

Take advantage of non-scored criteria to ask reviewers' questions about the topic or work proposed. Such simple questions as "What other comments do you have for program managers?" can yield beneficial insights.

If the proposal contains multiple projects, ask reviewers which projects are, in their view, most important to fund and why. This can be particularly useful when funding falls short and choices must be made regarding what can realistically be done.

3. Know what motivates reviewers. Monetary reward, when offered, is one reason, but not the only one. Recognize that reviewers learn from the proposals they evaluate. They learn to become better proposal writers themselves because they now know which factors matter most in evaluations.

This experience hones their abilities to identify well-written and poorly written proposals, and they can apply that knowledge when writing their own proposals.

4. Find knowledgeable reviewers. There are professional review consultants who create robust databases of reviewers and who know how to match reviewer expertise to the proposals being reviewed. Subscription databases, professional societies, and Internet searches using expertise keywords can also be used to locate reviewers.

The key here is matching the expertise described in the reviewer's curricula vitae or resume with the proposal's subject matter and the desired review outcomes.

5. Diversify the review panel. Creating a deliberate mix of perspectives helps to generate a robust review. Diversity can take multiple forms: It can be based on employer (academic, industry, government); a mix of early career and seasoned experts; and variety in the professions or disciplines represented.

Consider having reviewers with complementary, but differing, expertise evaluate the same proposal. You could group the technical or scientific experts with practitioners in the field, for example, or include financial operations experts as reviewers for proposals that depend on leveraged or matched funds.

6. Protect your review from conflict of interest. Define all the circumstances that constitute a conflict beforehand and apply them when you begin identifying potential reviewers. The most obvious conflict occurs when there is potential for a reviewer to benefit financially from a positive or negative review.

Conflicts of interest can be mitigated by instituting such simple rules as excluding potential reviewers who worked in your office within a specified period, such as the last five years. Include a nondisclosure agreement to protect proprietary data and define all other conflicts of interest in a statement. Reviewers must agree to the terms in these documents prior to getting access to the proposals. Record and keep a record of those agreements in the format that works best for you.

7. Support your reviewers. Give your reviewers a reasonable number of proposals to review based on proposal length and complexity, along with an adequate amount of time to do the work. If you ask them to comment on the relevance of a proposal to your organization's strategic plan, then provide that plan to them.

If they are using a specific review system, provide illustrated instructions and/or a webinar demonstrating the system and make a contact person available for their questions.

8. Provide feedback to vendors. A lot of hard work goes into submitting proposals. Ask reviewers to offer ways to strengthen proposals or to articulate any weaknesses. By getting experts to provide constructive feedback, program managers have an opportunity to influence the quality of future proposals and their outcomes.

An Objective Evaluation

Government officials depend on a reputation for sound investments and good stewardship of tax dollars. Peer reviews offer public program managers an objective way to evaluate proposals, thus building confidence and accountability when making funding decisions.

The eight steps presented here provide the framework for a reliable peer review plan. By following these steps, you will be able to thoughtfully plan a review, focus on the right reviewer characteristics, and treat both reviewers and vendors with respect for their contributions.

Margaret Williamson is senior project manager, and Leslie Shapard is associate director, Scientific Assessment and Workforce Development, Oak Ridge Associated Universities, Oak Ridge, Tennessee (margaret.williamson@orau.org; leslie.shapard@orau.org).