2006-04-17 00:07:16Andy Liao
Standards for High-Quality Research
There is no set of absolute standards that, if observed, will automatically ensure the quality of a particular research effort or product. However, there are general (and generally measurable) standards for high-quality research that are widely appreciated in the RAND community and beyond. These standards are applied in judging the quality of RAND’s research.
The research problem should be well formulated.
Problem formulation is recognized as a strength of RAND’s research style. For research to be useful, a problem must be formulated in a way that ensures it can be, if not solved, at least addressed with a substantial measure of success. Research addressing a practical problem must formulate it within as full a context as possible, to avoid coming up with answers that are suboptimal, naïve, or infeasible. So long as the broadly formulated problem remains tractable, the research is potentially more useful, more valuable, and, perhaps, enduring. An improved formulation of the problem can become a major research output.
The research approach should be well designed and well executed.
The selection of the research approach is closely tied to problem formulation, and the approach must be appropriate to the problem. For some types of research, clearly defined formal methodologies are appropriate — e.g., regression analyses, validated models, or survey protocols. The selection of a methodology or combination of methodologies should be buttressed by arguments about the soundness of that approach for the problem on which it is being brought to bear.
Where no clear formal methodologies apply, the analytic approach should always be described, and it should be defended if it is at all novel or controversial. The introduction of a new method or the application of an old method to a new problem obligates the researcher to be prepared to argue its advantages over alternative methods. Obviously, well-designed research does not guarantee high quality. The research approach must also be executed well. High-quality research should display rigor, technical prowess, and creativity in its execution.
The data and assumptions should be sound.
Data and assumptions are key inputs to the research. Data-generation methods and database fields should be clearly specified, and the data should be properly screened and manipulated. The research team should indicate with what confidence data can be used and believed. Information presented as factual should be correct and verifiable.
Major assumptions underlying a study must be explicitly identified and defended. Assumptions can mask uncertainties that affect the validity of findings and the expedience of recommendations. High-quality research often enhances the robustness of its results by varying assumptions in order to analyze alternative scenarios.
The findings should be useful and advance knowledge.
Findings represent new knowledge that must be integrated with existing knowledge. They can be useful in a number of ways to two of RAND’s major audiences. For decisionmakers, high-quality findings bear importantly on a particular decision. Or they may contribute to an ongoing debate, helping to frame or inform the discussions in a policy area. For analysts, findings have empirical value and augment previous knowledge. They may also suggest that a theory can be held with more confidence, needs refinement, or must be rejected.
The implications and recommendations should follow logically from the findings and be explained thoroughly.
Based on its findings, a research team derives implications and may develop recommendations. These critical research activities represent much of the ”think”in a ”think tank.”High-quality research thoroughly explores the implications of its findings. It examines where the new knowledge and old knowledge are congruent and where they are not; it examines whether existing theories and conceptual frameworks have been strengthened or must be modified. To recommend a course of action is a highly accountable step. Because of the nature of RAND’s mission, the actions of its clients may affect the resources or well-being of many millions of individuals — even entire nations or regions. For this reason, recommendations must follow logically from a study’s findings and implications and be strongly supported by them.
The documentation should be accurate, understandable, cogent, and temperate in tone.
Quality standards also apply to the way in which research is documented. Documentation includes paper-based documents, oral presentations, and Web-based formats. The standards apply to both textual and graphical elements. Accuracy is particularly salient as a prima facie indicator of the quality and credibility of research. High-quality research documentation must include all necessary caveats. RAND’s audiences need to know the confidence with which they can regard the accuracy of what is presented.
Much research is rich and complex in its findings and implications and may present numerous recommendations. Cogent documentation makes apparent what is most important and brings it to the foreground. High-quality documentation should make research understandable to its intended audiences. Understandable documentation is well expressed and well organized and may integrate text with graphical or pictorial elements to illuminate complex and novel ideas. High-quality research documentation should be temperate in tone. It should sound neither so flat as to appear unengaged nor so emotional as to appear partisan. Almost all RAND research participates in at least two intersecting sets of discussions: one among analysts and the other among decisionmakers. A temperate tone is best suited to both communities, as well as to wider audiences, such as the general public.
The research should demonstrate understanding of previous related studies.
High-quality research should build on and contribute to other research. The relationships between a given study and its predecessors should be rich and explicit. The research team’s understanding of past research should be evident in many aspects of its work, from the way in which the problem is formulated and approached to the discussion of the findings and their implications. Failure to demonstrate an understanding of previous research lowers the perceived quality of the research and its products despite any other good characteristics they may possess.
The research should be relevant to the client and other stakeholders.
RAND’s research has a strong focus on practical issues, and its research style involves frequent interactions with its clients, who are recognized as an important source of expertise. Nevertheless, because RAND conducts its research in the public interest, a high-quality study must also be relevant to other stakeholders in the policy domain. Interaction with a broad set of stakeholders can help to ensure the relevance of the research and the practicality of its recommendations.
The research should be objective, independent, and balanced.
RAND is trusted in part because its research is respected for its objectivity. Objectivity is valued in science as a contributor to analytic success, and it often helps to improve the likelihood that research will be used. Often, it is RAND’s reputation for objectivity, in combination with other aspects of quality, that results in its research becoming influential.
Independence refers to intellectual independence, not financial independence, which RAND usually cannot claim. In part to counter any inference that its research may be biased by its relationship to research clients or sponsors, RAND has instituted strong policies and mechanisms to ensure intellectual independence. RAND has a rigorous quality assurance process, its researchers commit themselves to seeking and using critical assessments of their work in all phases, and it routinely publishes its peer-reviewed research in the public domain. All RAND research faces the challenge of addressing both scientific perspectives and policy perspectives. RAND research seeks balance among competing perspectives by treating them fairly, portraying them accurately, and weighting them according to merit.
The research problem should be well formulated.
Problem formulation is recognized as a strength of RAND’s research style. For research to be useful, a problem must be formulated in a way that ensures it can be, if not solved, at least addressed with a substantial measure of success. Research addressing a practical problem must formulate it within as full a context as possible, to avoid coming up with answers that are suboptimal, naïve, or infeasible. So long as the broadly formulated problem remains tractable, the research is potentially more useful, more valuable, and, perhaps, enduring. An improved formulation of the problem can become a major research output.
The research approach should be well designed and well executed.
The selection of the research approach is closely tied to problem formulation, and the approach must be appropriate to the problem. For some types of research, clearly defined formal methodologies are appropriate — e.g., regression analyses, validated models, or survey protocols. The selection of a methodology or combination of methodologies should be buttressed by arguments about the soundness of that approach for the problem on which it is being brought to bear.
Where no clear formal methodologies apply, the analytic approach should always be described, and it should be defended if it is at all novel or controversial. The introduction of a new method or the application of an old method to a new problem obligates the researcher to be prepared to argue its advantages over alternative methods. Obviously, well-designed research does not guarantee high quality. The research approach must also be executed well. High-quality research should display rigor, technical prowess, and creativity in its execution.
The data and assumptions should be sound.
Data and assumptions are key inputs to the research. Data-generation methods and database fields should be clearly specified, and the data should be properly screened and manipulated. The research team should indicate with what confidence data can be used and believed. Information presented as factual should be correct and verifiable.
Major assumptions underlying a study must be explicitly identified and defended. Assumptions can mask uncertainties that affect the validity of findings and the expedience of recommendations. High-quality research often enhances the robustness of its results by varying assumptions in order to analyze alternative scenarios.
The findings should be useful and advance knowledge.
Findings represent new knowledge that must be integrated with existing knowledge. They can be useful in a number of ways to two of RAND’s major audiences. For decisionmakers, high-quality findings bear importantly on a particular decision. Or they may contribute to an ongoing debate, helping to frame or inform the discussions in a policy area. For analysts, findings have empirical value and augment previous knowledge. They may also suggest that a theory can be held with more confidence, needs refinement, or must be rejected.
The implications and recommendations should follow logically from the findings and be explained thoroughly.
Based on its findings, a research team derives implications and may develop recommendations. These critical research activities represent much of the ”think”in a ”think tank.”High-quality research thoroughly explores the implications of its findings. It examines where the new knowledge and old knowledge are congruent and where they are not; it examines whether existing theories and conceptual frameworks have been strengthened or must be modified. To recommend a course of action is a highly accountable step. Because of the nature of RAND’s mission, the actions of its clients may affect the resources or well-being of many millions of individuals — even entire nations or regions. For this reason, recommendations must follow logically from a study’s findings and implications and be strongly supported by them.
The documentation should be accurate, understandable, cogent, and temperate in tone.
Quality standards also apply to the way in which research is documented. Documentation includes paper-based documents, oral presentations, and Web-based formats. The standards apply to both textual and graphical elements. Accuracy is particularly salient as a prima facie indicator of the quality and credibility of research. High-quality research documentation must include all necessary caveats. RAND’s audiences need to know the confidence with which they can regard the accuracy of what is presented.
Much research is rich and complex in its findings and implications and may present numerous recommendations. Cogent documentation makes apparent what is most important and brings it to the foreground. High-quality documentation should make research understandable to its intended audiences. Understandable documentation is well expressed and well organized and may integrate text with graphical or pictorial elements to illuminate complex and novel ideas. High-quality research documentation should be temperate in tone. It should sound neither so flat as to appear unengaged nor so emotional as to appear partisan. Almost all RAND research participates in at least two intersecting sets of discussions: one among analysts and the other among decisionmakers. A temperate tone is best suited to both communities, as well as to wider audiences, such as the general public.
The research should demonstrate understanding of previous related studies.
High-quality research should build on and contribute to other research. The relationships between a given study and its predecessors should be rich and explicit. The research team’s understanding of past research should be evident in many aspects of its work, from the way in which the problem is formulated and approached to the discussion of the findings and their implications. Failure to demonstrate an understanding of previous research lowers the perceived quality of the research and its products despite any other good characteristics they may possess.
The research should be relevant to the client and other stakeholders.
RAND’s research has a strong focus on practical issues, and its research style involves frequent interactions with its clients, who are recognized as an important source of expertise. Nevertheless, because RAND conducts its research in the public interest, a high-quality study must also be relevant to other stakeholders in the policy domain. Interaction with a broad set of stakeholders can help to ensure the relevance of the research and the practicality of its recommendations.
The research should be objective, independent, and balanced.
RAND is trusted in part because its research is respected for its objectivity. Objectivity is valued in science as a contributor to analytic success, and it often helps to improve the likelihood that research will be used. Often, it is RAND’s reputation for objectivity, in combination with other aspects of quality, that results in its research becoming influential.
Independence refers to intellectual independence, not financial independence, which RAND usually cannot claim. In part to counter any inference that its research may be biased by its relationship to research clients or sponsors, RAND has instituted strong policies and mechanisms to ensure intellectual independence. RAND has a rigorous quality assurance process, its researchers commit themselves to seeking and using critical assessments of their work in all phases, and it routinely publishes its peer-reviewed research in the public domain. All RAND research faces the challenge of addressing both scientific perspectives and policy perspectives. RAND research seeks balance among competing perspectives by treating them fairly, portraying them accurately, and weighting them according to merit.