Should Grants Count As Research - Part 2

Warning: This post might not be of interest to anyone outside of academia. 

Last week I posted the following thought experiment:

Suppose you have two faculty members whose research productivity, publication rates, citation rates, etc. are identical in every respect. The only difference between them is that Faculty member A has two federally funded grants whereas Faculty member B has no grants. How would you rate their research influence/impact.

The post led to a lot of insightful comments, so you should read those if you haven't. The thought experiment was intended to eliminate any correlation between grants and research productivity in order to explore what should "count" as impact/influence. Some commenters denied the premise of the thought experiment when making their judgments (in a thought experiment, the premise doesn't have to be plausible...), so I'd like to address the premise directly before I give my take on the thought experiment.

Within a given research subfield, faculty with grants are likely more productive than those without. It is not clear how strong that correlation is, though. For some areas of psychology, grants are essential to have any productivity (e.g., fMRI, behavioral neuroscience). I can imagine other areas in which grants are largely superfluous once a lab is established. In either of those research areas, the correlation between grants and productivity likely is low. The most interesting cases are those subfields in which grants are helpful, but not absolutely necessary (e.g., traditional cognitive psychology). It would be interesting to explore the within-subfield correlation beween research impact and grant funding in such disciplines. The correlation likely would be positive given that getting grants typically requires prior productivity. I'd hazard a guess that it might not be a strong correlation, though. There must be data—does anyone know the actual numbers?

Now let's return to the issue that motivated the thought experiment. My department, and many others like it, evaluate the contributions of their faculty based on their contributions to research, service, and teaching. Often, the raters are left to their own devices to determine what counts in each category and how to measure it. Some raters use objective metrics, tallying up publications, evaluating citation rates as an index of influence, comparing the relative challenge of publishing in different subfields, etc. Others just grok the overall record and make a subjective assessment. In my discussions with another rater who, like me, prefers an objective approach, we disagreed on how to think about grants. Hence the thought experiment.

As promised, here is my take: 

Provided that a rater has access to a professor's actual research output (publications, citation rates, etc.), grants should be treated as irrelevant when evaluating research impact/influence. So, in the thought experiment, A and B should receive the same research rating. 

In the absence of direct access to the actual research output (and assuming grants are correlated with research impact), grants can serve as a useful, but imperfect proxy for impact. But, when we have access to the things that they proxy for, they add no useful information.

I think an analogy to citation rates is apt. If you lack access to the citation count for an individual paper, you could use the average citation rate of articles in that journal as a proxy for it's citation count. It's imperfect, but better than nothing. However, if you have the actual citations for that paper, there's nothing gained by knowing the average citation rate for that journal. Grants are similarly useful as a proxy when you lack other information. For example, for a more junior professor, a grant promises future productivity, making it a good proxy until that professor has enough of a track record to evaluate their research output directly. For senior faculty, we have the track record, so there is no need for a proxy. Grants are just a means to the end of scholarly productivity (i.e., journal articles), so they should not be treated as a research product themselves.

Note that even citation counts are a proxy for impact/influence. We all can identify papers that are cited frequently because of their topic, but are not necessarily influential (or even read by those citing them. Although citations may tell us little about the quality of an article, they do tell us something objective about its influence and impact.


If grants don't count toward research, should we count them at all?

Although I argue that grants should not count toward ratings of research productivity or impact, they should factor into faculty evaluations in a different way:

Grants should be treated as a component of service to the university rather than as an indicator of research productivity or impact.

Universities value grants because they provide a lot of money. Non-academics might not realize only a subset of the money in a federal grant goes toward research. In fact, more than one third of the grant funds at major universities (in the USA, anyway) go directly to the university to support operating expenses. Consequently, by bringing in grant funding, researchers are doing a service to their university; their grants help the school keep running, so they are highly valued. Even if the researcher produces nothing of research value from the grant, the university still benefits from their service. 

A follow-up thought question:

Let's return to the two faculty people in the thought experiment. Posit that they deserve the same research rating and that the grant-funded researcher should receive credit for their service to the university. Now consider that the funded researcher was less efficient than their unfunded colleague; they used federal funding to achieve the same level of productivity as someone who had no funding. In other words, they used a federal resource for something that apparently did not require it. 

Funding is a limited-sum game. Funding given to one researcher means less funding available for other researchers. If the funding was not needed to conduct the research, should we treat that grant-funded researcher as having done a disservice to their field as a whole. That is, should we credit them with service to their university while simultaneously penalizing their ratings of service to the field or the public because they took needed money away from other researchers? (Note: I'm just being provocative here, but I do think it's worth considering whether more funding should go to those labs that use it most efficiently and productively, those where the increment in funding will lead to the greatest increase in quality research output.)

This further discussion highlights the challenge of evaluating contributions objectively. It's often ambiguous what should count and where it should count. While we're at it,  perhaps we should figure out how to credit academics for all of the other stuff they do, including blogging! Should non-journal or general-audience writing count toward research productivity? Should it count as service to the community? Should it count as non-university teaching? Should it not count at all? I'll be curious to hear your thoughts.