James Brander, President, and Michael Veall, Vice-President On behalf of the Executive Council of the Canadian Economics Association
Guest Contributors
We have reviewed the proposed changes described in the “Briefing on SSHRC’s Renewed Program Architecture” of March 2010. We applaud the SSHRC for seeking to assess and improve its procedures and we recognize that many of the proposed changes will be helpful to the research process. However, we would like to express our concern about two proposed changes.
1. Research Record. Section V on “Research Grants” indicates an intention to reduce the weight placed on the “capability” of the applicant to 20%. The other 80% would be for the proposal itself, dividing into 40% for the “challenge” and 40% for “feasibility.” Compared to the current situation, this change represents a dramatic reduction in the weight placed on the applicant’s research record, which is currently 60%. With a weight of only 20% this criterion would be little more than a marginal consideration.
De-emphasizing the track record of applicants would reward the ability to write grants (sometimes called “grantsmanship”) at the expense of the proven ability to do research. Writing attractive proposals and actually doing research are sufficiently different activities that this is a major problem. It is, for example, increasingly the case that universities and even individuals are turning to professional grant writers for help. The proposed changes might, in effect, cause adjudications to be based more on the ability of professional grant writers than on likely research outcomes.
This problem is worsened by the fact that there is little follow-up regarding what is proposed in successful applications. It is easy to make broad claims and then produce relatively little research. Rarely, if ever, is grant money taken back if the research output falls short of what is promised.
Putting a very high weight on the proposal itself encourages making unrealistic claims and gives insufficient weight to actually doing successful research. More generally, proposal adjudication is a one-time exercise, while publications on a research record reflect repeated assessments by editors and reviewers and are likely to be a more reliable predictor of future success. Also, the evaluation of a track record is more objective than the inherently subjective evaluation of research questions and research plans.
Furthermore, it is the nature of research that it is often hard to predict the likely outcome of a research project. Often the most important results are surprises resulting from opportunistic creativity that is hard to anticipate in a grant proposal. The best indicator of future creativity is a track record of creativity – not grandiose claims.
In addition, one of the major objectives of SSHRC grants is to contribute to mentoring graduate students. Much of the financial support for graduate students comes from research grants. The people who are most successful in getting grants end up with the greatest influence on the next generation of researchers. We would argue that it is the strong researchers who should play the most influential role in mentoring graduate students, not the best grant writers. Moreover, a track record of supporting graduate students and involving them in research should be an important consideration when considering further support, at least for scholars teaching in universities with graduate programs.
As an aside, we also worry that outside observers will conclude that a weight of only 20% on “Capability: The Expertise to Succeed” is very small when compared to other situations when individuals or governments expend resources. This will potentially undermine external support for the program.
Of course we recognize that applicants, even those with strong track records, should be clear about the topics they plan to work on and about how they plan to carry out their research. However, we would propose that the track record should get no less than 50% of the weight in evaluating standard research grants and we would prefer to retain the current weighting of 60%.
We also recognize that scholars just beginning their careers will not have the same kind of track records as established scholars. We are therefore very supportive of the special “new scholar” category where the criteria are different. We add more on this point in the next section.
2. New Scholars. Section VI on Research Development Grants (which includes grants for new scholars) proposes only 5 adjudication committees. Economics is put in a category with the management disciplines, law, political science and administrative studies. This program will not use external reviews.
We have concerns about amalgamating areas in this way, as it is very difficult to make cross-field comparisons. For example legal scholars may have little idea of publication norms, journal quality, current research themes and the like in economics. Conversely, most economists have little idea of the research environment in law. If there is only one economist on such a committee (as seems likely) then the evaluation of the economics proposals will ultimately rely very heavily on that one person, who might not even have much familiarity with the particular sub-field (international trade, macroeconomics or whatever) of some applicants. Similar concerns will arise with other disciplines, and there will be little counterbalance or buffer to deal with any idiosyncrasies or lack of specialized knowledge on the part of the lone representative of a particular discipline.
We are particularly concerned that it is our new scholars who will be put into this new environment. First, if it is agreed that for new scholars relatively little weight can be put on track record, the consequence is that more weight must be put on the assessment of the proposal. We think that proposal assessment is much more likely to be accurate if the committee has collective expertise in the discipline, especially given that there will be no external reviews. Second, new scholars may not understand how to present proposals to such a committee and the advice from senior scholars in their discipline may be unhelpful because the experience of senior scholars will be from applying to a different program. There is the potential that award decisions will be very far from the norms understood within each discipline, conceivably discouraging young scholars from continuing to apply.
The committee structure is of special concern to economics because of the mathematical and statistical tools that economists use, even in policy analysis. These approaches are much less commonly used in law and political science than in economics. In turn, those disciplines also have approaches that will be unfamiliar to most economists. In short, we are concerned that the proposed committee structure will introduce randomness and error in the allocation of research funds to new scholars. Our favoured option would be to have the applications of new scholars adjudicated by the Research Grants Program committees. If there is a concern that new scholars will win too few grants, a separate funding envelope can be assigned. If this is administratively infeasible, we would suggest dividing Committee 5 into two, one with economics and some of the management disciplines, with the other including law, political science and administrative studies.