Any effort to understand, evaluate, and improve the impact of research must begin with clear concepts and definitions. Currently, key terms to describe research results are used ambiguously, and the most common definitions for these terms are fundamentally flawed. This hinders research design, evaluation, learning, and accountability. Specifically, the terms outcome and impact are often defined and distinguished from one another using relative characteristics, such as the degree, directness, scale, or duration of change. It is proposed instead to define these terms by the kind of change rather than by the degree or temporal nature of change. Research contributions to a change process are modeled as a series of causally inter-related steps in a results chain or results web with three main kinds of results: (i) the direct products of research, referred to as outputs; (ii) changes in the agency and actions of system actors when they are informed/influenced by research outputs, referred to as outcomes; and (iii) tangible changes in the social, economic, environmental, or other physical condition, referred to as realized benefits. Complete definitions for these terms are provided, along with examples. This classification aims to help focus research evaluation appropriately and enhance appreciation of the multiple pathways and mechanisms by which scholarship contributes to change.
There are high expectations from the public, research funding agencies, and researchers themselves to contribute to and document impact resulting from their research (Bornmann, 2012; Edler et al., 2012; Wilsdon et al., 2015). Any effort to understand, evaluate, and improve the impact of research must begin with clear concepts and definitions. Currently there is a debilitating lack of clarity and consistency in the use of key terms that describe the results of any intervention, including changes engendered by research. The terms output, outcome, and impact, which are terms used in a typical logic model, are used ambiguously and the most common definitions for these terms are fundamentally flawed (Belcher and Palenberg, 2018). This hinders evaluation, learning, and accountability in academic research as much or more than in any other field. This essay, based on the authors’ experience with conceptualizing and assessing research impact in the social sciences and humanities, applied research, and research-for-development contexts, takes a systems perspective on research impact and offers precise sub-categories of impact to improve clarity.
Established concepts used in research evaluation such as “impact factor” and “high impact research” refer to measures of publication and citations of research, but do not measure actual use or value beyond the academic realm (DORA, 2012; Hicks et al., 2015). There has been increasing attention to the non-academic impacts of research (Bornmann, 2012; Oancea, 2019; Williams, 2020). Alla et al. (2017) conducted a systematic review of definitions of research impact, finding 108 definitions in 83 publications. However, they noted a dominance of what they called bureaucratic definitions and a widespread failure to actually define the term explicitly. The most highly cited definitions were those of the Research Excellence Framework (“an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (REF, 2011, p. 26)), the Research Councils of the UK (“the demonstrable contribution that excellent research makes to society and the economy” (Economic and Social Research Council, 2021, para.1)), and the Australian Engagement and Impact Assessment framework (“the contribution that research makes to the economy, society, environment or culture, beyond the contribution to academic research” (Australian Research Council, 2018, p. 5)). While these broad, all-encompassing concepts give attention to societal benefits beyond academia, they all lack precision and require further classification to be useful analytically. They also fail to recognize that research typically contributes to change within complex social, economic, technical, and environmental systems, in conjunction with many other factors. Based on their review, Alla et al. (2017) re-emphasize the need for conceptual clarity, while offering their own definition specific to the mental health field: “Research impact is a direct or indirect contribution of research processes or outputs that have informed (or resulted in) development of new (mental) health policy/practices, or revisions of existing (mental) health policy/practices, at various levels of governance (international, national, state, local, organizational, health unit)” (p. 9).
Gow and Redwood (2020) also give considerable attention to the lack of clarity in interpretation of impact. They devote a chapter to discuss impact terminology and suggest a four-part impact typology: Instrumental; Conceptual; Capacity Building, and Procedural. They do not provide precise definitions for these sub-components of impact, and the authors themselves note that the categories are not mutually exclusive.
The term outcome is also widely used to refer to a step in a results chain. Like impact, outcome is also used ambiguously to refer to everything from the products of research to intermediate and shorter-duration changes stimulated by research, and it is often used as a synonym for impact. Most results chains conceptualize outcomes as resulting from outputs and as precursors to impact. The terms outcome and impact are typically distinguished from one another relatively, based on the degree, directness, scale, or duration of change. For example, the influential OECD (2010) glossary of evaluation terms defined outcomes as “The likely or achieved short-term and medium-term effects of an intervention’s outputs” (p. 28) and impacts as “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended” (p. 24). As Belcher and Palenberg (2018) discuss in detail, these definitions do not support clear, unambiguous distinctions between the terms or the concepts they are intended to define. Of particular relevance is the fact that the temporal dimension of these definitions is not helpful for analytical purposes such as research design, evaluation, learning, and accountability.
All the above impact definitions refer to a ‘contribution’ made by research, but devote most of their attention to the locus of change (i.e., beyond academia). They offer little to help specify, understand, or analyze the nature of the contribution research makes, or to ascertain definitively what is included and what is excluded in the definition. To help clarify the concept and advance thinking about research impact, we therefore propose two more precise sub-categories of impact that are defined absolutely, by the kind of change, rather than relatively, by the degree or temporal nature of change. We recognize that change processes happen in complex systems. Research contributes to a change process within a system and can be modeled as a series of causally inter-related steps in a results chain or results web. There are three main kinds of results from research: (i) the products and services of research, produced directly by a research program, which we refer to as outputs; (ii) changes in the agency of other actors when they use and/or are influenced by research outputs, which we refer to as outcomes; and (iii) tangible changes in the social, economic, environmental, or other physical condition, which we refer to as realized benefits. Complete definitions for these terms are provided below, along with examples. This is a classification of the types of contributions of research and scholarship within a theory of change, not a hierarchy of value.
Societal demands for impact naturally focus on positive changes in social, economic, environmental, or other physical condition. Research is supported with the expectation that it will contribute in some way to improvements in human well-being and environmental conditions. In the development field, the term impact is often used to mean mission-level impact (i.e., changes in social, economic, environmental, and/or physical condition) (Belcher and Palenberg, 2018). However, the term impact is used commonly and ambiguously in standard English language, and in the academic realm it has both a particular meaning (often measured by citations) and a general meaning that includes what we have called outcomes as well as realized benefits (and costs), as exemplified by the definitions cited above. The term is so imprecise in its common usage, and so loaded with pre-existing definitions, that it would be difficult to re-define. We have therefore elected not to propose a new or restricted definition of the term impact. Rather, we are proposing a classification of sub-categories of impact, which are based on the nature of the change. We use “impact” as an overarching term to denote any change caused in whole or in part by an action or set of actions, including research actions.
Knowledge, including new insights, technical innovations, institutional models, and other direct products and services produced by a research program. Outputs are produced by actions within a program’s (including partners) sphere of control (see Fig. 1).
A change in knowledge, attitudes, skills, and/or relationships (KASR), ideally manifest as a change in behavior (B), that results in whole or in part from the research process and its outputs. Outcomes may be at the individual, group, organizational, or higher scales.
A change in economic, social, or environmental condition resulting in whole or in part from a chain of events to which research has contributed. This can manifest as a change in flow or change in state. Benefits/costs may be realized at individual, group, organizational, or higher scales.
Figure 1 illustrates a research program Footnote 4 theory of change. The three spheres reflect the fact that the relative influence of any intervention declines as interactions with other actors and processes increase (Hearn, 2010; Montague, 2000). The program has a high level of control over program activities and outputs in the sphere of control. Beyond the program boundary, research outputs inform, influence, and support other actors and their actions (outcomes), alongside many other influences and processes, in the sphere of influence. Ideally, the actions of those other actors will then contribute to realized benefits in the sphere of interest.
In practical terms, the sphere of control includes actions and outputs that can be produced directly by the researcher or research team. This includes actions and outputs produced by collaborators as part of their commitments to a program. If an actor must be persuaded through the provision of knowledge, tools, or advocacy, this change occurs in the sphere of influence. The concept of the sphere of influence attempts to capture the idea that change happens when the KASR of other actors (i.e., not part of the research team) change. These kinds of changes are classified as outcomes of the research if they result in whole or, more likely, in part from the research process and/or output(s). If there are co-produced outputs, it implies that the research process has resulted in KASR and behavior in other actors that would not have happened in the absence of the research, and this change is an outcome. Individually or collectively, changes in behavior that result in part or in whole from the research can lead to realized benefits.
The research program itself is represented in Fig. 1 as a stylized sequence of activities, from top to bottom, within the sphere of control. Activities include developing partnerships with other researchers and/or societal actors and (co-)defining the problem the research will address and the specific questions it seeks to answer. The research then may apply established methods and/or develop new methods to collect and analyze data and (co)create new knowledge and innovations. This list is indicative; not all steps may be present and\or they may occur in different sequence, iteratively, and with or without external actors being involved.
The program’s interactions with and influence in society is represented horizontally, from the sphere of control (program implementation), through the sphere of influence (other actors informed and influenced by research outputs), to the sphere of interest (the tangible benefits to which the research may contribute). The figure tries to represent the dynamic interactions in a complex system. The downward arrows in the sphere of control indicate that each step in the research process contributes to other actions in the research process.
In traditional academic research, the primary aim has been to create new knowledge, search for meaning, and improve understanding. However, research can contribute to outcomes and realized benefits in many ways. Moving from the left to right in the diagram (as indicated by the rightward arrows), each of the individual steps in the research process can produce outputs that contribute independently as well as in combination. For example, the process of developing a partnership may build relationships among stakeholders that have value beyond the program; the research question and/or new methods could stimulate attention and additional research on an important topic; open data policies are increasing the likelihood that data sets will be made available for other uses beyond a program. Each of the steps can contribute to changes in KASR and changes in behavior (B) by other actors. The research process may also be informed and influenced by societal engagement, as represented by the leftward arrows moving from partners, stakeholders, and society back to the program.
The rightward arrow to the second step within the sphere of influence illustrates how changes in KASRB (outcomes) among partners, stakeholders, and society more generally can lead to changes in policy and practice (outcomes) and higher-level system transformations (outcomes), that ultimately lead to changes in social, economic, or environmental condition (realized benefits) in the sphere of interest. This highlights the important role of collaborations and partnerships in co-creating and advancing the use the research-based knowledge and reflects an important rationale for increased use of engaged transdisciplinary research approaches. The circular arrow at the bottom of the diagram represents ongoing stakeholder engagement throughout all stages.
Finally, the figure indicates that the focus of monitoring, evaluation, and learning (MEL) is different at each stage in the impact pathway. Within the sphere of control, the focus is on research quality, broadly defined to include considerations of relevance, credibility, legitimacy, and how research is positioned for use (Belcher et al., 2016; Ofir et al., 2016). Is the research focus, design, and implementation appropriate and sound? Within the sphere of influence, research evaluation needs to focus on whether and how research has contributed to outcomes. Is there evidence that the research has stimulated or contributed to changes in KASRB, and is it reasonable to expect further knock-on changes? In the sphere of interest, the focus is on the scale and scope of realized benefits and analysis of the relative contribution of research.
It is important to emphasize that this is a classification, not a hierarchy of value. It is intended to support research evaluation by distinguishing the kinds of changes that research can enable, catalyze, and contribute to. In order to assess what difference research makes, we need to know what kind of change we are looking for. Change happens in complex systems and, as illustrated in Fig. 1, most change happens outside the control of a research program. The kind and degree of change to which any research program contributes and the timeframe over which that change happens will depend on many other factors, including the nature of the issue, the current state of knowledge, and the political climate. In some domains of research (e.g., many Engineering and Applied Sciences), external stakeholders often have close linkages with researchers, such that the pathway through the spheres of influence and interest to realized benefits can be relatively direct and rapid. In Health research, the interface of researchers with individuals with lived experience of a disease provides engagement and learning, and enables more effective translation of research outputs to practice and realized benefits of the affected communities. The outputs of scholarship in Social Sciences and the Humanities may profoundly influence understanding, appreciation, values, and indeed the actions of individuals, organizations, or society more generally (i.e., outcomes). These kinds of changes are often difficult to observe, difficult to measure, and difficult to attribute, and occur over long timeframes, but have value in and of themselves. They may also contribute to realized benefits, but in most cases the attribution challenges are insurmountable because there are so many other causal factors. This classification aims to help focus research evaluation appropriately and enhance appreciation of contributions that scholarship makes to change in more diffuse ways. In any research evaluation, we need to look at outcomes as the primary indicator of research effectiveness.
There has been a great deal of discussion in the literature about research impact, how to define it, and how to measure it, but current definitions and usage remain vague and ambiguous. This essay combines two main ideas to help achieve conceptual clarity. First, we explicitly recognize that research contributes to change within systems as sequential causal processes (with feedback and iteration), in combination with other processes and other actors. We have provided a generic model of a research-to-impact process that: illustrates the declining relative influence of an intervention in a system, shown as spheres of control, influence, and interest; indicates typical actions within a research process; appreciates that individual actions in the research process may make valuable contributions independently as well as in combination, especially in engaged co-produced research; and identifies that the focus of monitoring, evaluation, and learning is different at each stage in the process. Second, we propose that it is practical and useful to classify research results into different kinds. Outputs are the products and services produced directly by research. Outcomes are the changes in KASR experienced by other actors who have been influenced by the outputs of research. Those changes in KASR may also contribute to changes in behavior and, thereby, to subsequent outcomes. Realized benefits are tangible changes in the social, economic, environmental, or other physical conditions. In this framing, research impact includes both outcomes and realized benefits. This classification aims to help focus research evaluation appropriately and enhance appreciation of the multiple pathways and mechanisms by which scholarship contributes to change.
Change in behavior is understood broadly. It is any action that would not otherwise have taken place. It could be something as simple as one person telling another what they have learned, to transformative changes in individual, organizational, institutional, or societal policies or practices. We are asking “Who does what differently as a result of the research?”
Change is assessed against a (hypothetical) counterfactual; i.e., what would have happened in the absence of the intervention. Thus, the change may be a decision to maintain the status quo or to avoid implementing a program.
In some types of research, such as participatory action research, benefits may be realized by participants.
We use the term “program” to refer to a body of research work done by an individual researcher or a team of researchers. The discussion could equally refer to a “project”.
For their sponsorship of a 2019 workshop by the Federation for the Humanities and Social Sciences and CASRAI, as well as their valuable comments and feedback at various stages in the development of this article, the authors would like to acknowledge Suzanne Board, Laura Beaupre, Yolande Chan, Kyle Demes, Heather Frost, Laura Hillier, Sandra Lapointe, Sharon Murphy, Nilgun Onder, Emile Paquin, David Phipps, Sally Rutherford, Lisa Shapiro, Karine Souffez, Louise Michelle Verrier, and David Watt. We are also indebted to Anna Hatch for her insightful comments on an earlier draft of this article. Brian Belcher’s work on this has been supported by the Canada Research Chairs Programme, the Social Sciences and Humanities Research Council (SSHRC), Ashoka Canada, and the Forests, Trees and Agroforestry Consortium Research Program.
